I've had the displeasure of working with Macs on which we wrote code that would end up being a docker container. But what is compiled on a Mac is an aarch64 container. What is compiled on an AMD64 Linux machine is amd64.
This is annoying when the code you write is native or reaches out to native stuff. Then all of the sudden your container that builds perfectly on the Mac doesn't build on Linux anymore. (and vice versa, looking at you gcc on debian).
GitHub just (January) got Linux ARM runners, but they're not available for private customers yet.
And their Mac runners do not come with Docker, and it cannot be installed.
Just this week I tried giving this another chance to debug some weird CI failures for ruby tests.
I’m on M-series macs so there is an inherent platform mismatch. That coupled with us using depot images as action runners. I was never able to get it running past the dry-run stage.
There is just a lot of differences between CI runners and my local docker images.
I even gave the image from catthehacker a try. Which is like >15GB. It still wouldn’t run.
Finally gave up.
Sounds similar to my own experiences trying to debug GH actions locally.
I've tried twice now to get it working, pulling down many GBs of images and installing stuff and then getting stuck in some obscure configuration or environment issue. I was even running Linux locally which I figured would be the happiest path.
I'm not eager to try again, and unless there's a CI that's very slow or GH actions that need to be updated often for some reason, I feel like it's better to just brute-force it - waiting for CI to run remotely.
GH Actions have always worked flawlessly for me, as long as I don't ever do anything unusual.
Last week I moved from a custom server back to GH Pages + GH Actions, and broke a feature that depended on an out dir existing after one build phase and before another. I have no idea how to fix it. It's probably simple, I'm just burnt out.
It’s even worse than that, it’s 20GB compressed and 60GB uncompressed. Regardless, if your VM runs out of disk space there’s no meaningful error (well 12 months ago) except a failure to start. (Possibly colima-specific I dunno I uninstalled docker-docker)
I do find that some things just work locally and fail in real actions and vice versa. For the most part it’s made it easier to move the bar forward though.
I had luck with my actions I tried to debug. But I also had the issue that the runtime difference is simply too big to 100% trust it. Too bad because I think the tool is quite good.
Every mention of Github Actions is an occasion to start looking for the best alternative in <current_month>, let's go!
Is Dagger usable yet? Is there still hope for Earthly? Are general purpose workflow systems winning any time soon, or are we still afraid of writing code? Any new systems out there that cover all the totally basic features like running locally, unlike Github Actions?
Seems fine to me, I don't really understand why people think CI is hard and they need to shop a different platform. All these systems are is a glorified shell script runner. It seems like developers just keep making up new DSLs over and over to do the same things.
I think we could build something on top of nix that is as easy to use and powerful as earthly, but gets all the nice stuff from nix: reproducibility, caching, just use any command from any nix package, etc
Rather than tying CI & deployments to Github Actions, it is usually better to pull as much of it as possible out to shell scripts and call them in containers in GH actions..
There are optimizations you’ll want (caching downloaded dependencies etc); if you wait to make those after your build is CI-agnostic you’ll be less tempted by vendor specific shortcuts.
Usually means more code - but, easier to test locally. Also, swapping providers later will be difficult but not “spin up a team and spend 6 months” level.
This is always the top comment in these kinds of threads, and I see this as an indication that the current state of CI/CD is pathetically propriety.
It’s like the dark times before free and open source compilers.
When are we going to push back and say enough is enough!?
CI/CD desperately needs something akin to Kubernetes to claw back our control and ability to work locally.
Personally, I’m fed up with pipeline development inner loops that involve a Git commit, push, and waiting around for five minutes with no debugger, no variable inspector, and dumping things to console logs like I’m writing C in the 1980s.
You and I shouldn’t be reinventing these wheels while standing inside the tyre shop.
It's important to note that this tool does not use the same container images or runtime environment that GitHub Actions actually runs. It's an approximation.
For simple use cases that won't matter, but if you have complex GitHub Actions you're bound to find varying behavior. That can lead to frustration when you're debugging bizarre CI failures.
In case anyone else is similarly curious, I managed to get the ubuntu images to build .qcow2 images with some lightweight patches to the packer files and the interior .sh provisioning scripts. I have intention of setting up (heh) GHA to build them on the regular but it has not yet risen to be the highest priority weekend project
If you may say "but, why?!" it's because I really wanted to know what versions of everything I could expect in GHA, and I detest "git commit -amdummy && git push" stupidity so I guess the answer is "because I'm wired that way"
In my experience (and as reflected by the comments on this post already), trying to run complex CI workflow locally is a fairly hopeless enterprise. If you have a fully containerized workflow, you might be able to get close, but even then ensuring you have all of your CI specific environment variables is often not a trivial task, and if your workflow orchestrates things across tasks (e.g. one task uploads an artifact and another task uses that artifact) you'll have a hard time reproducing exactly what is happening in CI. My company (RWX) builds a GitHub Actions competitor and we intentionally do not support local execution -- instead we focused on making it easy to kick off remote builds from your local machine without having to do a `git push` and we made it easy to grab an SSH session before, during, or after a running task to inspect things in the exact build environment that your workflow is running.
Dumb question, but why hasn’t GitHub made a solution that lets you run GitHub Actions locally? Or at the very least a solution that validates the action (giving a bit more certainty that it will succeed, a bit like a dry-run)?
(My war story:) I stopped using GHAs after an optimistic attempt to save myself five key strokes ‘r’ ‘s’ ‘p’ ‘e’ ‘c’ led to 40+ commits and seeing the sunrise but still no successful test run via GHA. Headless browsers can be fragile but the cost benefit ratio against using GHA was horrible, at least for an indy dev.
I'm as big a GitLab fanboy as they come, but they recently axed the gitlab-runner binary's ability to execute local .gitlab-ci.yml files <https://gitlab.com/gitlab-org/gitlab/-/issues/385235>. It now only operates in "receive webhook from the mothership" mode, just like GHA
Pour one out, I guess, but it's okay since I previously was super angry at it for languishing in the uncanny valley of "hello world works, practically nothing else does" -- more or less like nektos/act and its bazillions of forks
It’s the allure of the marketplace that gets people. They’re like “oh I could parse my test report files and upload all of them somewhere and render them with a nice output on my pr! I could spend a week writing and testing all of this or I can slap this action here into the YAML and be done with it in 10 minutes.
The trap and tradeoff is that the thirtieth time you’ve done that is when you realize you’ve screwed yourself and the organization by building this Byzantine half baked DAG with a very sketchy security story that you can’t step through, run locally or test
WARNING: act is great if you use docker. Act does not support podman.
Issues or discussions related to providing support/coverage/compatibility/workarounds for podman are closed with a terse message. Unusual for an open source project.
What are some recommended alternatives for act, if any?
I’ve been a long time user, but I’ve run into tons of problems with act over the last year and am wondering if there are better alternatives out there.
Last time I looked at it, Act was a lot like the “serverless offline” options out there that try to mimic AWS services. If your use case is straightforward then I might work but if you do anything “exotic” (often the types of things I’m trying to debug in a failed run) Act doesn’t fully replicate the GHA experience.
Also, understandably, there is no macOS support and I use macOS on GHA for iOS builds (another place I have to debug that this wouldn’t cover).
I’ve been bedeviled by arm/intel/Mac issues with this.
I want to be able to use this project, I really do. But it’s just not yet there, and this isn’t on Nektos. Nektos is, as best I understand it, trying to approximate GHA, but it’s not easy.
As someone with a deep interest in this space, I spent a non-trivial amount of time tire-kicking that. One can view the issues for my experiences, including the one where I wished the dev all the best but stopped playing QA for the project. It's goal is primarily a TUI for GHA, not an emulation for GHA
like a lot of folks i found nektos lacking. instead, i focused on keeping the gha workflows really light and putting any complexity in Task (taskfile.dev)—that way the meat of the workflows can be run locally, and the gha syntax is just the glue.
Yup. Every time there's one of these threads I renew my call for GHA to have hooks where in-band build tools like nix, bazel, dagger, and friends can propagate upward optimal information to be displayed in the web gui, in particular annotations and info about multiple tasks that run in parallel or series.
You don't get vendor lock in if you're able to run GHA on any compute you'd like (yes, I'm aware of self-hosted runners, and even that their Packer code is open source. That doesn't make it easy to use GHA outside of github.com)
I think this is really cool. We're tackling this problem from the other side by building `pixi` (pixi.sh) which bundles project / package management with a task runner so that any CI should be as simple as `pixi run test` and easy to execute locally or in the cloud.
that’s not really what’s new or special about pixi, is it? poetry (poethepoet) and uv can both do variations of this.
From the outside, pixi looks like a way to replace (conda + pip) with (not-conda + uv). It’s like uv-for-conda, but also uses uv internally.
Better task running is cool, but it would be odd to use pixi if you don’t otherwise need conda stuff. And extra super duper weird if you don’t have any python code!
anyone have any tips for testing actions locally, rather than workflows?
Despite the name, act is really only for the latter. You can try to test a local action by putting it in a workflow and calling that, but if you do a checkout in your workflow that will overwrite the mount of your local code into the act container, meaning you’ll get the version from the remote branch instead. Depending on the action, you may not be able to comment out the checkout step while testing.
In GitHub we use an OIDC token to access some AWS resources. Locally I need to populate tokens etc and so I have an `if: ${{ACT}}` and a not condition to populate it.
I've wanted this so much! My main questions are about permissions. I use environment-specific variables and secrets, I would need to reconfigure them locally and didn't see how. The other issue is workload identity federation.
could you just source vars from an .env file? maybe a bit more work to get the secrets pulled down. i guess ideally you'd want to be able to pull those values directly from github as an option (including secrets) if that's your primary source of truth.
Back in the day we called that a shell script. We had this tool called Jenkins that would run the shell script in a new folder every time you pushed a change to gitlab. It was pretty neat.
If CI/CD tools were code editors, then Jenkins/JFrog would be like Vi/Vim—powerful and traditional. GitLab CI would resemble Helix or Zed, offering a modern and efficient approach. Meanwhile, GitHub Actions would be akin to VSCode, but with a touch of Dreamweaver CC, blending user-friendly features with comprehensive capabilities.
Have you ever use any of those in a realistic scenario? Any existing CI/CD solution is broken like hell and if you try to do more advanced stuff like reusing templates or actions, all hell breaks lose. Don't get me started on clones via SSH, doing commits from pipelines, etc.
CI/CD is one of the topics that is barely solved or usable ...
Actually, I have considerable experience setting up and running various CI/CD tools such as Jenkins, GitLab CI, and Travis CI.
In my experience, GitHub Actions work quite well, especially for smaller projects or when you can avoid using public templates and actions. While shortcuts like third-party libraries or actions can be convenient, they often come with trade-offs.
For scenarios where having your own runners and build images is beneficial, I would recommend GitLab CI.
Jenkins is a great choice if you need to run builds on dedicated hardware or within a virtualized environment.
When discussing the challenges of CI/CD, I often notice that many issues stem from complex chains of dependencies within the software or product itself. For example, consider a release pipeline that typically takes six hours to complete but fails after four hours. In such cases, having to restart the entire pipeline instead of just the failed job can be quite inefficient.
Additionally, debugging can become much easier when you have your own build images on runners, as you can run them locally for testing and troubleshooting.
What specific aspects of CI/CD do you find broken or challenging?
This is annoying when the code you write is native or reaches out to native stuff. Then all of the sudden your container that builds perfectly on the Mac doesn't build on Linux anymore. (and vice versa, looking at you gcc on debian).
GitHub just (January) got Linux ARM runners, but they're not available for private customers yet.
And their Mac runners do not come with Docker, and it cannot be installed.
I've tried twice now to get it working, pulling down many GBs of images and installing stuff and then getting stuck in some obscure configuration or environment issue. I was even running Linux locally which I figured would be the happiest path.
I'm not eager to try again, and unless there's a CI that's very slow or GH actions that need to be updated often for some reason, I feel like it's better to just brute-force it - waiting for CI to run remotely.
Last week I moved from a custom server back to GH Pages + GH Actions, and broke a feature that depended on an out dir existing after one build phase and before another. I have no idea how to fix it. It's probably simple, I'm just burnt out.
I do find that some things just work locally and fail in real actions and vice versa. For the most part it’s made it easier to move the bar forward though.
Is Dagger usable yet? Is there still hope for Earthly? Are general purpose workflow systems winning any time soon, or are we still afraid of writing code? Any new systems out there that cover all the totally basic features like running locally, unlike Github Actions?
Jenkins too going away, but, like Windows XP, people often reach back out for it.
dagger is the only code-based solution. It works, but it does have some edges since it has a much bigger surface area and is constantly growing.
There are optimizations you’ll want (caching downloaded dependencies etc); if you wait to make those after your build is CI-agnostic you’ll be less tempted by vendor specific shortcuts.
Usually means more code - but, easier to test locally. Also, swapping providers later will be difficult but not “spin up a team and spend 6 months” level.
https://github.com/Cloudef/zig-aio/blob/master/flake.nix#L25...
https://github.com/Cloudef/zig-budoux/blob/master/flake.nix#...
The actual GA workflow file is pretty simple: https://github.com/Cloudef/zig-aio/blob/master/.github/workf...
It’s like the dark times before free and open source compilers.
When are we going to push back and say enough is enough!?
CI/CD desperately needs something akin to Kubernetes to claw back our control and ability to work locally.
Personally, I’m fed up with pipeline development inner loops that involve a Git commit, push, and waiting around for five minutes with no debugger, no variable inspector, and dumping things to console logs like I’m writing C in the 1980s.
You and I shouldn’t be reinventing these wheels while standing inside the tyre shop.
For simple use cases that won't matter, but if you have complex GitHub Actions you're bound to find varying behavior. That can lead to frustration when you're debugging bizarre CI failures.
If you may say "but, why?!" it's because I really wanted to know what versions of everything I could expect in GHA, and I detest "git commit -amdummy && git push" stupidity so I guess the answer is "because I'm wired that way"
I use dagger to read these .env/mise env vars and inject dummy values into the test container. Production is taken care of with a secrets manager.
Agreed. I’m thankful for tools like act, but there really should be an officially supported way to run gh actions locally.
(My war story:) I stopped using GHAs after an optimistic attempt to save myself five key strokes ‘r’ ‘s’ ‘p’ ‘e’ ‘c’ led to 40+ commits and seeing the sunrise but still no successful test run via GHA. Headless browsers can be fragile but the cost benefit ratio against using GHA was horrible, at least for an indy dev.
Pour one out, I guess, but it's okay since I previously was super angry at it for languishing in the uncanny valley of "hello world works, practically nothing else does" -- more or less like nektos/act and its bazillions of forks
Your action can be empty and actions generate webhook events.
Do whatever you want with the webhook.
The trap and tradeoff is that the thirtieth time you’ve done that is when you realize you’ve screwed yourself and the organization by building this Byzantine half baked DAG with a very sketchy security story that you can’t step through, run locally or test
Issues or discussions related to providing support/coverage/compatibility/workarounds for podman are closed with a terse message. Unusual for an open source project.
[1] https://github.com/nektos/act/issues/303
I’ve been a long time user, but I’ve run into tons of problems with act over the last year and am wondering if there are better alternatives out there.
Also, understandably, there is no macOS support and I use macOS on GHA for iOS builds (another place I have to debug that this wouldn’t cover).
I want to be able to use this project, I really do. But it’s just not yet there, and this isn’t on Nektos. Nektos is, as best I understand it, trying to approximate GHA, but it’s not easy.
https://github.com/bahdotsh/wrkflw
it works out very well.
I wonder if there is a proposal to support code-based actions. Config-based CI needs to die.
nu is my default shell. Note that I am not talking about dagger shell. https://dagger.io/blog/a-shell-for-the-container-age-introdu...
From the outside, pixi looks like a way to replace (conda + pip) with (not-conda + uv). It’s like uv-for-conda, but also uses uv internally.
Better task running is cool, but it would be odd to use pixi if you don’t otherwise need conda stuff. And extra super duper weird if you don’t have any python code!
Despite the name, act is really only for the latter. You can try to test a local action by putting it in a workflow and calling that, but if you do a checkout in your workflow that will overwrite the mount of your local code into the act container, meaning you’ll get the version from the remote branch instead. Depending on the action, you may not be able to comment out the checkout step while testing.
My alternative was to have a dedicated repository where I can spam the Git and workflow run histories as much as I need to experiment with workflows.
https://nektosact.com/usage/index.html?highlight=Secret#secr...
Which GitHub Actions context variables are emulated or customizable in act, like github.event, github.actor, or secrets
CI/CD is one of the topics that is barely solved or usable ...
In my experience, GitHub Actions work quite well, especially for smaller projects or when you can avoid using public templates and actions. While shortcuts like third-party libraries or actions can be convenient, they often come with trade-offs.
For scenarios where having your own runners and build images is beneficial, I would recommend GitLab CI.
Jenkins is a great choice if you need to run builds on dedicated hardware or within a virtualized environment.
When discussing the challenges of CI/CD, I often notice that many issues stem from complex chains of dependencies within the software or product itself. For example, consider a release pipeline that typically takes six hours to complete but fails after four hours. In such cases, having to restart the entire pipeline instead of just the failed job can be quite inefficient.
Additionally, debugging can become much easier when you have your own build images on runners, as you can run them locally for testing and troubleshooting.
What specific aspects of CI/CD do you find broken or challenging?