What’s more, GitHub has basically stopped maintaining their own actions, pushing people to sketchy forks to do basic things. Their entire ecosystem is basically held up with duct tape and gets very little investment.
An interesting things is that GitHub is an expensive service and my guess would be that MS makes good money on it. Our small company paid about 200+ USD monthly for GitHub, much larger cumulative cost than Windows licenses. My believe was that Windows is getting worse, because it is considered legacy business by MS in favor of new offerings such as GitHub subscriptions.
I’d appreciate not being called lazy for mentioning a lack of investment on Microsoft’s side to secure their paid and fairly lucrative service that they bought a popular code hosting platform to integrate with.
Can someone explain what this somewhat recent phenomenon is where people feel the need to defend the worlds biggest billion dollar businesses, that are also often subsidized by tax payer money in weird ways?
How did we go in 20 years from holding these companies to account when they'd misbehave to acting as if they are poor damsels in distress whenever someone points out a flaw?
I won't "defend" Microsoft in this case, but I am always annoyed by phrases like "world's biggest billion-dollar businesses... bablah".
Their size or past misbehaviors shouldn't be relevant to this discussion. Bringing those up feels a bit like an ad hominem. Whether criticism is valid should depend entirely on how GitHub Actions actually works and how it compares to similar services.
The original comment said to stop giving money to these companies if they are not giving you a satisfactory service.
The opposite, to be lazy and to continue giving them money whilst being unhappy with what you get in return, would actually be more like defending the companies.
> How did we go in 20 years from holding these companies to account when they'd misbehave to acting as if they are poor damsels in distress whenever someone points out a flaw?
They hired a ton of people on very very good salaries
There is a massive problem in open source where some people equate pointing out a problem with being too lazy to solve it — when in reality this just stifles the conversation. Especially when a prerequisite to any group project accomplishing anything is to first discuss the problem to be solved.
No that's actually a completely different issue. You're talking about volunteers working on side projects that are sometimes foundational to the way the internet works and then people feel entitled to tell them what to do without contributing.
Here we are talking about one of the worlds most valuable companies that gets all sorts of perks, benefits and preferential treatment from various entities and governments on the globe and somehow we have to be grateful when they deliver garbage while milking the business they bought.
> unless they put the money where their mouth is, it's just noise
I used to work for a Japanese company, and one of their core philosophies was “Don’t complain, unless you have a solution.” In my experience, this did not always have optimal outcomes: https://littlegreenviper.com/problems-and-solutions/
I don’t make the purchasing decision for my employer, but I certainly have to deal with their fallout, so I’ll keep complaining if that’s okay with you.
I've used CircleCI quite a bit in the past; it was pretty good. Feels tough for them to compete with GHA though when you're getting GHA credits for free with your code hosting.
I used Travis rather longer ago, it was not great. Circle was a massive step forward. I don't know if they have improved it since but it only felt useful for very simplistic workflows, as soon as you needed anything complex (including any software that didn't come out of the box) you were in a really awkward place.
What that type of section usually means is "there's someone from Microsoft that signed up for our service using his work account", sometimes it means "there's some tiny team within Microsoft that uses our product", but it very rarely (if ever) means "the entire company is completely reliant on our product".
The quality of setup-* actions has definitely gone down and there are a lot of strange decisions being made. I assume the original authors of these actions have long left the company.
Thank you for your interest in this GitHub repo, however, right now we are not taking contributions.
We continue to focus our resources on strategic areas that help our customers be successful while making developers' lives easier. While GitHub Actions remains a key part of this vision, we are allocating resources towards other areas of Actions and are not taking contributions to this repository at this time. The GitHub public roadmap is the best place to follow along for any updates on features we’re working on and what stage they’re in.
The funny thing about the last one is that those actions ultimately boil down to invoking their CLI tool (which is pre-installed on the runners) with "gh release create ...", so you can just do that yourself and ignore the third-party actions and the issues that come with them. Invoking an action isn't really any easier than invoking the CLI tool.
That issue with their own small private forks has actually raised its head while testing out the AI slop generator thing it has, making anything it produces for you not self hoatable unless you rewrite a lot of basic functions. Sweet irony.
With AI you won't need CI anymore, it's all going straight to prod anyway /s
Actions is one thing, but after all these years where the new finegrained access tokens aren't still supported across all the product endpoints (and the wack granularity) is more telling about their lack of investment in maintenance.
Normally I’d say stop kicking the dead horse, but GHA deserves all the complaints it gets and then some. It’s the embodiment of everything that’s bad in ‘less is more’.
My biggest concern with it is that it’s somehow the de facto industry standard. You could do so much better with relatively small investments, but MS went full IE6 with it… and now there’s a whole generation of young engineers who don’t know how short their end of the stick actually is since they never get to compare it to anything.
It's funny that absolutely everything about GHA fucking sucks, and everyone agrees about this. BUT, the fact that it's free compute, and it's "right there"... means it's very very difficult to say no to!
Personally I've just retired a laptop and I'm planning to turn it into a little home server. I think I'm gonna try spinning up Woodpecker on there, I'm curious to see what a CI system people don't hate is like to live with!
I will say that SourceSafe had one advantage: You could create "composite" repos.
You could add one or two files from one repo, and a few from another, etc. The resulting "avatar" repo would act like they were all in the same repo. It was cool.
> The researchers identified four fundamental security properties that CI/CD systems need: admittance control, execution control, code control, and access to secrets.
Why do CI/CD systems need access to secrets? I would argue need access to APIs and they need privileges to perform specific API calls. But there is absolutely nothing about calling an API that fundamentally requires that the caller know a secret.
I would argue that a good CI/CD system should not support secrets as a first-class object at all. Instead steps may have privileges assigned. At most there should be an adapter, secure enclave style, that may hold a secret and give CI/CD steps the ability to do something with that secret, to be used for APIs that don’t support OIDC or some other mechanism to avoid secrets entirely.
With a secure enclave or an HSM, there's a secret, but the users do not have access to the secret. So, if you have a workflow that needs to, say, sign with a given private key, you would get an API that signs for you. If you need to open a TLS connection with a client certificate, you get a proxy that authenticates for you.
I suppose I would make an exception for license keys. Those have minimal blast radii if they leak.
> I would argue that a good CI/CD system should not support secrets as a first-class object at all. Instead steps may have privileges assigned. At most there should be an adapter, secure enclave style, that may hold a secret and give CI/CD steps the ability to do something with that secret, to be used for APIs that don’t support OIDC or some other mechanism to avoid secrets entirely.
CI/CD does not exist in the vacuum. If you had CI/CD entirely integrated with the rest of the infrastructure it might be possible to do say an app deploy without passing creds to user code (say have the platform APIs that it can call to do the deployment instead of typical "install the client, get the creds, run k8s/ssh/whatever else needed for deploy").
But that's a high level of integration that's very environment specific, and without all that many positives (so what you don't need creds, you still have permission to do a lot of mess if it gets hijacked), and a lot, lot more code to write vs "run a container and pass it some env vars" that had become a standard
We're iterating towards GHA for CI, AWS CodeBuild for the CD. At least on AWS projects. Mainly because managing IAM permissions to permit the github runner to do everything the deployment wants is an astonishingly large waste of time. But you need a secret to trigger one from the other.
How do you e.g. validate that a database product works with all the different cloud databases? Every time you change up SQL generation you're going to want to make sure the SQL parses and evaluates as expected on all supported platforms.
Those tests will need creds to access third party database endpoints.
You might want (or _need_) to sign your binary, for example. Or you might want to trigger a deployment.
Github actually is doing something right here. You can set it up as a trusted identity provider in AWS, and then use Github to assume a role in your AWS account. And from there, you can get access to credentials stored in Secret Manager or SSM.
Yes, their oidc setup was probably their last good feature back when they were actually delivering features back in 2020ish. Everyone else copied it within a few months though.
I agree 100% with what I think is the key phrase, viz. "the results can change without any modification to your code".
I maintain an R package that is quite stable and is widely used. But every month or so, the GHA on one of the R testing machines will report an error. The messages being quite opaque, I typically spend a half hour trying to see if my code is doing something wrong. And then I simply make a calendar item to recheck it each day for a while. Sure enough, the problems always go away after a few days.
This is making me feel quietly vindicated in pushing back on migrating our Jenkins/Ansible setup to GHA simply because corporate wanted the new shiny thing. Fortunately the "this will be a lot of work, i.e. cost" argument won.
Mind you, CI does always involve a surprising amount of maintenance. Update churn is real. And Macs still are very much more fiddly to treat as "cattle" machines.
While I hate defending GHA, the docs do include this:
- Using the commit SHA of a released action version is the safest for stability and security.
- If the action publishes major version tags, you should expect to receive critical fixes and security patches while still retaining compatibility. Note that this behavior is at the discretion of the action's author.
So you can basically implement your own lock file, although it doesn't work for transitive deps unless those are specified by SHA as well, which is out of your control. And there is an inherent trade-off in terms of having to keep abreast if critical security fixes and updating your hashes, which might count as a charitable explanation for why using hashes is less prevalent.
- Using the commit SHA of a released action version is the safest for stability and security.
This is not true for stability in practice: the action often depends on a specific Node version (which may not be supported by the runner at some point) and/or a versioned API that becomes unsupported. I've had better luck with @main.
Depends what you mean by stability. The post is complaining about the lack of lockfiles, and the problem you describe would also be an issue with lockfiles.
Pleased this is being discussed somewhere as it’s something that has troubled me for a while.
There are so many third party actions where the docs or example reference the master branch. A quick malicious push and they can presumably exfiltrate data from a ton of repositories
(Even an explicit tag is vulnerable because it can just be moved still, but master branch feels like not even trying)
it's wild I can wiz through a ton of code for hours on end but seeing a yaml file for something like a CI pipeline actually makes my brain eject i dunno why. my brain has some sort of proverbial capacity limit with how many different configuration-file looking things I can tolerate in a day, and the prospect of becoming intimately familiar with what is effectively an auto integration presented to me as some sort of config makes me completely unjustifiably butthurt for no reason. have i not suffered enough needless and often times limiting abstractions already
We are currently using GitHub Actions for all our CI tasks and I hate it. Yes, the marketplace is nice and there are a lot of utility actions which make life easier, but they all come with the issues the post highlights. Additionally, testing Actions locally is a nightmare. I know that act exists but for us it wasn't working most of the time. Also the whole environment management is kinda odd to me and the fact, that when using an environment (which then allows to access secrets set in that environment) it always creates a new deployment is just annoying [1]
I guess the best solution is to just write custom scripts in whatever language one prefers and just call those from the CI runner. Probably missing out on some fancy user interfaces but at least we'd no longer be completely locked into GHA...
> Some teams vendor actions into their own repos. zizmor is excellent at scanning workflows and finding security issues.
Harsh given GitHub makes it very easy to setup attestations for Artifact (like build & sbom) provenances.
That said, with Zizmor (static analyser for GitHub Actions), Step Security's Harden Runner [0] is also great, even if it requires a bit of an involved setup relative to Zizmor.
Yep. I'm switching our workflows to instead use regular utilities running inside a Docker container.
This works well for _most_ things. There are some issues with doing docker-in-docker for volume mapping, but they're mostly trivial. We're using taskfiles to run tasks, so I can just rely on it for that. It also has a built-in support for nice output grouping ( https://taskfile.dev/docs/reference/schema#output ) that Github actions can parse.
Pros:
1. Ability to run things in parallel.
2. Ability to run things _locally_ in a completely identical environment.
3. It's actually faster!
4. No vendor lock-in. Offramp to github runners and eventually local runners?
Cons:
It often takes quite a while to understand how actions work when you want to run them in your own environment. For example, how do you get credentials to access the Github Actions cache and then pass them to Docker? Most of documentation just tells: "Use this Github Action and stop worrying your pretty little head about it".
Do you have a write up about this? Actions are great, but my #2 gripe with actions, after the tenuous security posture, is that the default practice is not to run/validate actions locally.
I checked out the linked GitHub repo https://github.com/ecosyste-ms/package-manager-resolvers and it appears to be just a README.md that collects summaries of different package managers? How do I know these weren't just LLM-generated?
If I write actions/setup-python@v1, I'm expecting the action to run with the v1 tag of that repository. If I rerun it, I expect it to run with the v1 tag of that repository...which I'm aware may not be the same if the tag was updated.
If I instead use actions/setup-python@27b31702a0e7fc50959f5ad993c78deac1bdfc29 then I'm expecting the action to run with that specific commit. And if I run it again it will run with the same commit.
So, whether you choose the tag or the commit depends on whether you trust the repository or not, and if you want automatic updates. The option is there...isn't it?
You specifying the top level hash doesn't do anything to pin transitive dependencies, and as the article points out, transitive dependencies - especially dependencies common to a lot of actions - would be the juciest target for a supply chain attack.
What if GH actions is considered legacy business in favour of LLMs?
These include
- https://circleci.com/
- https://www.travis-ci.com/
- Gitlab
Open source:
- https://concourse-ci.org/ (discussed in the context of Radicle here https://news.ycombinator.com/item?id=44658820 )
- Jenkins
-etc.
Anyone can complain as much as they want, but unless they put the money where their mouth is, it's just noise from lazy people.
How did we go in 20 years from holding these companies to account when they'd misbehave to acting as if they are poor damsels in distress whenever someone points out a flaw?
Their size or past misbehaviors shouldn't be relevant to this discussion. Bringing those up feels a bit like an ad hominem. Whether criticism is valid should depend entirely on how GitHub Actions actually works and how it compares to similar services.
The opposite, to be lazy and to continue giving them money whilst being unhappy with what you get in return, would actually be more like defending the companies.
They hired a ton of people on very very good salaries
Here we are talking about one of the worlds most valuable companies that gets all sorts of perks, benefits and preferential treatment from various entities and governments on the globe and somehow we have to be grateful when they deliver garbage while milking the business they bought.
I used to work for a Japanese company, and one of their core philosophies was “Don’t complain, unless you have a solution.” In my experience, this did not always have optimal outcomes: https://littlegreenviper.com/problems-and-solutions/
I used Travis rather longer ago, it was not great. Circle was a massive step forward. I don't know if they have improved it since but it only felt useful for very simplistic workflows, as soon as you needed anything complex (including any software that didn't come out of the box) you were in a really awkward place.
Don't waste your time
What that type of section usually means is "there's someone from Microsoft that signed up for our service using his work account", sometimes it means "there's some tiny team within Microsoft that uses our product", but it very rarely (if ever) means "the entire company is completely reliant on our product".
i.e. from https://github.com/actions/cache/?tab=readme-ov-file#note
https://github.com/actions/create-release
(we run a private gitlab instance and a merge request can spawn hundreds of jobs, that's a lot of potential Gitlab credits)
Actions is one thing, but after all these years where the new finegrained access tokens aren't still supported across all the product endpoints (and the wack granularity) is more telling about their lack of investment in maintenance.
My biggest concern with it is that it’s somehow the de facto industry standard. You could do so much better with relatively small investments, but MS went full IE6 with it… and now there’s a whole generation of young engineers who don’t know how short their end of the stick actually is since they never get to compare it to anything.
Personally I've just retired a laptop and I'm planning to turn it into a little home server. I think I'm gonna try spinning up Woodpecker on there, I'm curious to see what a CI system people don't hate is like to live with!
I'm from a generation who had to use VSS for a few years. The sticks are pretty long these days, even the ones you get from github.
I just had trauma!
I will say that SourceSafe had one advantage: You could create "composite" repos.
You could add one or two files from one repo, and a few from another, etc. The resulting "avatar" repo would act like they were all in the same repo. It was cool.
However, absolutely everything else sucked.
I don't miss it.
Why do CI/CD systems need access to secrets? I would argue need access to APIs and they need privileges to perform specific API calls. But there is absolutely nothing about calling an API that fundamentally requires that the caller know a secret.
I would argue that a good CI/CD system should not support secrets as a first-class object at all. Instead steps may have privileges assigned. At most there should be an adapter, secure enclave style, that may hold a secret and give CI/CD steps the ability to do something with that secret, to be used for APIs that don’t support OIDC or some other mechanism to avoid secrets entirely.
I don't really understand what you mean by "secure enclave style"? How would that be different?
I suppose I would make an exception for license keys. Those have minimal blast radii if they leak.
CI/CD does not exist in the vacuum. If you had CI/CD entirely integrated with the rest of the infrastructure it might be possible to do say an app deploy without passing creds to user code (say have the platform APIs that it can call to do the deployment instead of typical "install the client, get the creds, run k8s/ssh/whatever else needed for deploy").
But that's a high level of integration that's very environment specific, and without all that many positives (so what you don't need creds, you still have permission to do a lot of mess if it gets hijacked), and a lot, lot more code to write vs "run a container and pass it some env vars" that had become a standard
Of course the general purpose task runner that both run on does need to support secrets
https://docs.github.com/en/actions/how-tos/secure-your-work/...
Only the CI part needs to build; it needs little else and it's the only part of a coherent setup that needs to build.
Those tests will need creds to access third party database endpoints.
There is if you pay for API access, surely?
Github actually is doing something right here. You can set it up as a trusted identity provider in AWS, and then use Github to assume a role in your AWS account. And from there, you can get access to credentials stored in Secret Manager or SSM.
I maintain an R package that is quite stable and is widely used. But every month or so, the GHA on one of the R testing machines will report an error. The messages being quite opaque, I typically spend a half hour trying to see if my code is doing something wrong. And then I simply make a calendar item to recheck it each day for a while. Sure enough, the problems always go away after a few days.
This might be specific to R, though.
Mind you, CI does always involve a surprising amount of maintenance. Update churn is real. And Macs still are very much more fiddly to treat as "cattle" machines.
- Using the commit SHA of a released action version is the safest for stability and security.
- If the action publishes major version tags, you should expect to receive critical fixes and security patches while still retaining compatibility. Note that this behavior is at the discretion of the action's author.
So you can basically implement your own lock file, although it doesn't work for transitive deps unless those are specified by SHA as well, which is out of your control. And there is an inherent trade-off in terms of having to keep abreast if critical security fixes and updating your hashes, which might count as a charitable explanation for why using hashes is less prevalent.
So in other words the strategy in the docs doesn't actually address the issue
This is not true for stability in practice: the action often depends on a specific Node version (which may not be supported by the runner at some point) and/or a versioned API that becomes unsupported. I've had better luck with @main.
There are so many third party actions where the docs or example reference the master branch. A quick malicious push and they can presumably exfiltrate data from a ton of repositories
(Even an explicit tag is vulnerable because it can just be moved still, but master branch feels like not even trying)
Why not just build the workflows themselves as docker images? I guess running other docker images in the workflow would then become a problem.
> actions/checkout@f43a0e5ff2bd294095638e18286ca9a3d1956744
I guess the best solution is to just write custom scripts in whatever language one prefers and just call those from the CI runner. Probably missing out on some fancy user interfaces but at least we'd no longer be completely locked into GHA...
[1] https://github.com/orgs/community/discussions/36919
Harsh given GitHub makes it very easy to setup attestations for Artifact (like build & sbom) provenances.
That said, with Zizmor (static analyser for GitHub Actions), Step Security's Harden Runner [0] is also great, even if it requires a bit of an involved setup relative to Zizmor.
[0] https://github.com/step-security/harden-runner
> But these are workarounds for a system that lacks the basics. The fix is a lockfile.
Hopefully SLSA drafts in Hermetic build process as a requirement: https://slsa.dev/spec/v1.2/future-directions
This works well for _most_ things. There are some issues with doing docker-in-docker for volume mapping, but they're mostly trivial. We're using taskfiles to run tasks, so I can just rely on it for that. It also has a built-in support for nice output grouping ( https://taskfile.dev/docs/reference/schema#output ) that Github actions can parse.
Pros:
1. Ability to run things in parallel.
2. Ability to run things _locally_ in a completely identical environment.
3. It's actually faster!
4. No vendor lock-in. Offramp to github runners and eventually local runners?
Cons:
It often takes quite a while to understand how actions work when you want to run them in your own environment. For example, how do you get credentials to access the Github Actions cache and then pass them to Docker? Most of documentation just tells: "Use this Github Action and stop worrying your pretty little head about it".
If I write actions/setup-python@v1, I'm expecting the action to run with the v1 tag of that repository. If I rerun it, I expect it to run with the v1 tag of that repository...which I'm aware may not be the same if the tag was updated.
If I instead use actions/setup-python@27b31702a0e7fc50959f5ad993c78deac1bdfc29 then I'm expecting the action to run with that specific commit. And if I run it again it will run with the same commit.
So, whether you choose the tag or the commit depends on whether you trust the repository or not, and if you want automatic updates. The option is there...isn't it?