Just this week I tried giving this another chance to debug some weird CI failures for ruby tests.
I’m on M-series macs so there is an inherent platform mismatch. That coupled with us using depot images as action runners. I was never able to get it running past the dry-run stage.
There is just a lot of differences between CI runners and my local docker images.
I even gave the image from catthehacker a try. Which is like >15GB. It still wouldn’t run.
Finally gave up.
Sounds similar to my own experiences trying to debug GH actions locally.
I've tried twice now to get it working, pulling down many GBs of images and installing stuff and then getting stuck in some obscure configuration or environment issue. I was even running Linux locally which I figured would be the happiest path.
I'm not eager to try again, and unless there's a CI that's very slow or GH actions that need to be updated often for some reason, I feel like it's better to just brute-force it - waiting for CI to run remotely.
There's another alternative - debug in CI itself. There's a few ways that you can pause your CI at a specific step and get a shell to do some debugging, usually via SSH. I've found that to be the most useful.
It’s slow and arduous work to inject at the right point and find out what went wrong. But it’s miles better than act. act is a noble idea, and I hope it can meet its target, but it falls short currently.
ssh-action gets you on the host, and lets you quickly establish what is happening and implement fixes. It’s top notch!
GH Actions have always worked flawlessly for me, as long as I don't ever do anything unusual.
Last week I moved from a custom server back to GH Pages + GH Actions, and broke a feature that depended on an out dir existing after one build phase and before another. I have no idea how to fix it. It's probably simple, I'm just burnt out.
You can pass outputs between jobs by declaring one job as a dependency of another with needs:, then reference that dependency's outputs with needs.<job>.outputs
ChristopherHX publishes Docker images which are created directly from packing up the root filesystem into a tarball from inside a runner. The images are huge though.
It’s even worse than that, it’s 20GB compressed and 60GB uncompressed. Regardless, if your VM runs out of disk space there’s no meaningful error (well 12 months ago) except a failure to start. (Possibly colima-specific I dunno I uninstalled docker-docker)
I do find that some things just work locally and fail in real actions and vice versa. For the most part it’s made it easier to move the bar forward though.
This is really only for debugging the logic of your Actions, isn't it?
On my team, Nix seems to work pretty well for encapsulating most of the interesting things CI jobs do. We don't use virtualization to run local versions of our jobs, and we don't run into architecture mismatch issues.
My experience with Ruby on Apple Silicon as been far from seamless. The only way I could thoroughly get rid of problems was by running the most recent Ruby release and dealing with what THAT causes.
I’m pretty sure GH actions don’t run the latest Ruby version.
My experience with both Ruby and Python Apple Silicon has been hit or miss. Mise has made it a lot simpler and seamless but if you need an older version or the absolute latest when it is released, you might have some issues. Mise has helped a lot, I generally do not expect to get an error.
Are there specific problems? I’ve been primarily Ruby at work the last 6 or so years (before, also doing Ruby, but also lots of Java, Kotlin, Swift, FE stuff, and some Python)… exactly that period… and haven’t really noticed any issues. Granted, I don’t typically use a lot of obscure binary-only dependencies, or things whose C/C++/machine code wouldn’t mostly cross-compile fine.
Granted, all this time I’ve always only used rbenv-managed Ruby versions (and rvm before that). I’ve long disliked/avoided Python because its ecosystem lacked any equivalent, but I know “uv” has gained popularity there, and perhaps Mise is good in Python land as well.
I had luck with my actions I tried to debug. But I also had the issue that the runtime difference is simply too big to 100% trust it. Too bad because I think the tool is quite good.
Every mention of Github Actions is an occasion to start looking for the best alternative in <current_month>, let's go!
Is Dagger usable yet? Is there still hope for Earthly? Are general purpose workflow systems winning any time soon, or are we still afraid of writing code? Any new systems out there that cover all the totally basic features like running locally, unlike Github Actions?
Earthly has pivoted away from their CI/CD products: they’re shutting down Satellite, and they’re stopping active maintenance of the Earthly open source project.
I think we could build something on top of nix that is as easy to use and powerful as earthly, but gets all the nice stuff from nix: reproducibility, caching, just use any command from any nix package, etc
That's mostly what we've built with Flox [1] (though I'm not exactly sure what you mean by run any command from any Nix package). It looks and feels like an amped up package manager, but uses Nix as kind of an infrastructure layer under the hood. Here's a typical workflow for an individual developer:
- cd into repo
- `flox activate` -> puts you into a subshell with your desired tools, environment variables, services, and runs any setup scripts you've defined
- You do your work
- `exit` -> you're back in your previous shell
Setting up and managing an environment is also super easy:
- cd into project directory
- `flox init` -> creates an "environment"
- `flox install python312` -> installing new packages is very simple
- `flox edit` -> add any environment variables, setup scripts, services in an editor
- `flox activate` -> get to work
The reason we call these "environments" instead of "developer environments" is that what we provide is a generalization of developer environments, so they're useful in more than just local development contexts. For example, you can use Flox to replace Homebrew by creating a "default" environment in your home directory [2]. You can also bundle an environment up into a container [3] to fit Flox into your existing deployment tools, or use Flox in CI [4].
All that stuff I described is free, but we have some enterprise features in development that won't be, and I think people are going to find those very appealing.
I built something like this out using an old version of Dagger and found it enormously complicated, and the they rewrote everything and abandoned the version of Dagger I used.
When I they did, I said "fuck it" and just started distributing a Nix flake with wrappers for all the tools I want to run in CI so that at least that part gets handled safely and is easy to run locally.
The worst and most annoying stuff to test is the CI platforms' accursed little pseudo-YAML DSLs. I still would like that piece.
devenv.sh tasks might be a good basis for building on it. I think maybe bob.build also wants to be like this
Yes! I knew that those are comparable to devenv.sh (what my team uses; love it!) in terms of environment management but wasn't yet aware of whether or not they also have task runners like one might use to build CI/CD pipeline logic.
I also want to say that this approach has largely spared my team the pain many users seem to have with Docker on aarch64 Macs. Nix works well on both aarch64 and x86_64, and doesn't require emulation to run. This is really more appropriate for running development tools locally, I think.
Gitlab? It's been around longer than Actions, it's much better and honestly not sure why anyone would consider anything else (but happy to hear if there are reasons).
In the context of this submission, one will want to exercise caution because they and their forgejo friends are using forks of nektos/act, with all the caveats of both "act" and "fork"
Seems fine to me, I don't really understand why people think CI is hard and they need to shop a different platform. All these systems are is a glorified shell script runner. It seems like developers just keep making up new DSLs over and over to do the same things.
We ended up running every job in a docker container, otherwise the Jenkins admin had to keep installing various tools on the runners. Gitlab supports this workflow natively so seemed the obvious choice.
Jenkins also supports this out of the box, and if it didn't have it for the pipeline DSL, again, Jenkins is a general purpose runner, you can just install docker, and run the script to run builds in the docker containers
I wish jenkins had a better config as code option. Not crazy about groovy and you still need to click in the UI to get some set up before you can use. Maybe im stuck with an old version or something
I mostly like Jenkins though, idk why people so desperately want to always move to something new. I guess I've never been on the maintenance side of running Jenkins for thousands of engineers though.
it's hard since you don't have the right environment, you can't debug, can't easily run locally and you have to push a change, and wait for the code to run on a remote machine 100x less powerful than you old laptop.
If these systems thought a moment about the developer experience, they'd be magical, but they do not.
It's still a miserable experience to maintain it, update it, deal with mostly old plugins, dynamically loading the tooling, groovy idiosyncrasies.. and UI/UX that despite recent efforts continues to feel terrible.
Managing the underlying infra is painful, and it remains a security liability even when not making obvious mistakes like exposing it to any open network.
And good luck if you're having that fun at a windows shop and aren't using a managed instance (or 5).
>It's still a miserable experience to maintain it...
How so? I've been maintaining an instance for a decade, and it really doesn't seem that bad. Updating plugins we do about monthly, it's largely clicking a couple buttons in the UI. Updating Jenkins itself is an apt update. Groovy takes a bit to grok, sure, LLMs help a lot here. The UX isn't that bad, IMHO, but I can see why some would say that. We've switched over almost entirely to using a couple runners, docker, and Jenkinsfiles, which works great. We do still run deploys directly on the Jenkins box, largely because we need to run them single-threaded so they don't step on each-other with load balancer rotations.
Your use case is just fine because you're barely actually using Jenkins.
I can give you a few examples of where it falls short:
- security: there are constant CVEs about anything and everything in Jenkins
- upgrade paths: if your company uses lots of plugins, the resulting spaghetti is of Italian proportions
- if said company is in a Windows-only infra, on prem, and they still decided to use Jenkins then good luck doing anything. Try putting an agent on a non-system disk for example, Windows paths aren't handled and you find yourself already passing very specific "pre" commands that your master will send over ssh.
- Said ssh connection can be lost due to a variety of things for which there are quite a few combinations of parameters when invoking Jenkins
- While we're at it, SSO isn't exactly supported in Windows environments. There are two external plugins you can try, one created because the other doesn't work, and even then good luck with that.
- At scale you end up having to be at least minimally interested in GC tuning, as Jenkins runs within the JVM
- UI: normal "Views" are not informative, and a bunch of custom views need to be made but rarely work with all sorts of plugins that people using your CI can consider crucial (say, parametrized cronjobs)
- Using the functionnalities Jenkins offers to "install tooling". Try to get it to use a certain version of node in a pipeline. Any current typical CI solution turns that into a straightforward task that's extendable, but in Jenkins you have to configure a very archaic and barely-working "tooling" area in your system to use that, and this barely works beyond the most basic tools.
- Having to maintain enterprise-level groovy libraries. Good luck. It's Groovy but not exactly Groovy. It's all inside the JVM, but inside Jenkins's abstractions of it inside of the JVM.
- Good luck monitoring lots of agents and doing typical tasks with them. Maintenance Windows are slowly coming in, monitoring sort-of-kinda-works via plugins..
I've maintainted instances in small companies and larger ones. With less custom stuff and with a lot more. Compiling C++, C#, Java/Kotlin, Objective-C/Swift. Building webapps, iOS SDKs, Android SDKs, and native Windows apps.
Jenkins can do everything if you bend it the way you want with some plugins and custom code. That's a strength that nothing else offers, but by and large it is a weakness of CI in the long run. Being opinionated isn't amazing, but sometimes it is required to be less complex, easier to maintain, more secure, easier to use, etc.
Let me throw another one out there: how about TeamCity on premise? Still can be done for free and the last time I used it (years ago) it left a very complete, stable and easy to use impression on me. Jenkins by comparison seems heavy and complicated.
Rather than tying CI & deployments to Github Actions, it is usually better to pull as much of it as possible out to shell scripts and call them in containers in GH actions..
There are optimizations you’ll want (caching downloaded dependencies etc); if you wait to make those after your build is CI-agnostic you’ll be less tempted by vendor specific shortcuts.
Usually means more code - but, easier to test locally. Also, swapping providers later will be difficult but not “spin up a team and spend 6 months” level.
This is always the top comment in these kinds of threads, and I see this as an indication that the current state of CI/CD is pathetically propriety.
It’s like the dark times before free and open source compilers.
When are we going to push back and say enough is enough!?
CI/CD desperately needs something akin to Kubernetes to claw back our control and ability to work locally.
Personally, I’m fed up with pipeline development inner loops that involve a Git commit, push, and waiting around for five minutes with no debugger, no variable inspector, and dumping things to console logs like I’m writing C in the 1980s.
You and I shouldn’t be reinventing these wheels while standing inside the tyre shop.
We've had open source CI. We still have it. I remember ye olde days of Hudson, before it was renamed Jenkins. Lotsa orgs still use Jenkins all over the place, it just doesn't get much press. It predates GHA, Circle, and many of the popular cloud offerings.
Turns out CI/CD is not an easy problem. I built a short-lived CI product before containers were really much of a thing ...you can guess how well that went.
Also, I'll take _any_ CI solution, closed or open, that tries to be the _opposite_ of the complexity borg that is k8s.
Having a container makes debugging possible, but it's still generally going to be an unfriendly experience, compared to a script you can just run and debug immediately.
It's inevitable that things will be more difficult to debug once you're using a third party asynchronous tool as part of the flow.
- You may need something to connect the dots between code changes and containers. It's not always possible to build everything on every change, especially in a multi/mono-repo setup.
- You probably need some way to connect container outcomes back to branch protection rules. Again, if you are building everything, every time, it's pretty simple, but less so otherwise.
- You likely want to have some control over the compute capacity on which the actions run, both for speed and cost control. And since caching matters, some compute solutions are better than others.
I don't think GitHub Actions solves any of these problems well, but neither do containers on their own.
In my experience (and as reflected by the comments on this post already), trying to run complex CI workflow locally is a fairly hopeless enterprise. If you have a fully containerized workflow, you might be able to get close, but even then ensuring you have all of your CI specific environment variables is often not a trivial task, and if your workflow orchestrates things across tasks (e.g. one task uploads an artifact and another task uses that artifact) you'll have a hard time reproducing exactly what is happening in CI. My company (RWX) builds a GitHub Actions competitor and we intentionally do not support local execution -- instead we focused on making it easy to kick off remote builds from your local machine without having to do a `git push` and we made it easy to grab an SSH session before, during, or after a running task to inspect things in the exact build environment that your workflow is running.
It's important to note that this tool does not use the same container images or runtime environment that GitHub Actions actually runs. It's an approximation.
For simple use cases that won't matter, but if you have complex GitHub Actions you're bound to find varying behavior. That can lead to frustration when you're debugging bizarre CI failures.
Yeah, this is lame. With Gitlab you can choose the exact image that runs and provide your own. So you can easily run the exact same code in the exact same environment locally.
I guess it would be nice to have a tool to convert a Gitlab YAML to Docker incantations, but I've never needed to do it that often for it to be a problem.
In case anyone else is similarly curious, I managed to get the ubuntu images to build .qcow2 images with some lightweight patches to the packer files and the interior .sh provisioning scripts. I have intention of setting up (heh) GHA to build them on the regular but it has not yet risen to be the highest priority weekend project
If you may say "but, why?!" it's because I really wanted to know what versions of everything I could expect in GHA, and I detest "git commit -amdummy && git push" stupidity so I guess the answer is "because I'm wired that way"
Feel like the correct way to use any CI these days is to just have it do almost nothing. The CI just triggers a script. And then you can run that script locally.
exactly. I am surprised by the amount of comments in this and similar threads praying for github to add more complexity to an already over-complicated and fragile solution.
Because you may want caching, or to install certain software in the VM or container that you already have locally. Separating scripts could help, but IME the CI environment is always different enough that some branching or mimicking is needed to harmonize them.
Caching is fine, you likely don’t have to replicate that locally as in theory it should do nothing. For installing software, create a docker image that already has everything you need.
I'm convinced that they specifically won't do that so that people won't use them to build competing products. Actions is one of the pieces that make GitHub money.
Almost every company I know of is in love with self hosted, or 3rd party, runners because they can't reason about OIDC auth into their cloud, or it's more evil sibling VPN
I'm open to your suggestion but I struggle to get onboard that Microsoft thinks that CI compute billing is the future
WARNING: act is great if you use docker. Act does not support podman.
Issues or discussions related to providing support/coverage/compatibility/workarounds for podman are closed with a terse message. Unusual for an open source project.
Dumb question, but why hasn’t GitHub made a solution that lets you run GitHub Actions locally? Or at the very least a solution that validates the action (giving a bit more certainty that it will succeed, a bit like a dry-run)?
(My war story:) I stopped using GHAs after an optimistic attempt to save myself five key strokes ‘r’ ‘s’ ‘p’ ‘e’ ‘c’ led to 40+ commits and seeing the sunrise but still no successful test run via GHA. Headless browsers can be fragile but the cost benefit ratio against using GHA was horrible, at least for an indy dev.
It’s the allure of the marketplace that gets people. They’re like “oh I could parse my test report files and upload all of them somewhere and render them with a nice output on my pr! I could spend a week writing and testing all of this or I can slap this action here into the YAML and be done with it in 10 minutes.
The trap and tradeoff is that the thirtieth time you’ve done that is when you realize you’ve screwed yourself and the organization by building this Byzantine half baked DAG with a very sketchy security story that you can’t step through, run locally or test
I'm as big a GitLab fanboy as they come, but they recently axed the gitlab-runner binary's ability to execute local .gitlab-ci.yml files <https://gitlab.com/gitlab-org/gitlab/-/issues/385235>. It now only operates in "receive webhook from the mothership" mode, just like GHA
Pour one out, I guess, but it's okay since I previously was super angry at it for languishing in the uncanny valley of "hello world works, practically nothing else does" -- more or less like nektos/act and its bazillions of forks
Last time I looked at it, Act was a lot like the “serverless offline” options out there that try to mimic AWS services. If your use case is straightforward then I might work but if you do anything “exotic” (often the types of things I’m trying to debug in a failed run) Act doesn’t fully replicate the GHA experience.
Also, understandably, there is no macOS support and I use macOS on GHA for iOS builds (another place I have to debug that this wouldn’t cover).
What are some recommended alternatives for act, if any?
I’ve been a long time user, but I’ve run into tons of problems with act over the last year and am wondering if there are better alternatives out there.
I’ve been bedeviled by arm/intel/Mac issues with this.
I want to be able to use this project, I really do. But it’s just not yet there, and this isn’t on Nektos. Nektos is, as best I understand it, trying to approximate GHA, but it’s not easy.
As someone with a deep interest in this space, I spent a non-trivial amount of time tire-kicking that. One can view the issues for my experiences, including the one where I wished the dev all the best but stopped playing QA for the project. It's goal is primarily a TUI for GHA, not an emulation for GHA
Yup. Every time there's one of these threads I renew my call for GHA to have hooks where in-band build tools like nix, bazel, dagger, and friends can propagate upward optimal information to be displayed in the web gui, in particular annotations and info about multiple tasks that run in parallel or series.
like a lot of folks i found nektos lacking. instead, i focused on keeping the gha workflows really light and putting any complexity in Task (taskfile.dev)—that way the meat of the workflows can be run locally, and the gha syntax is just the glue.
You don't get vendor lock in if you're able to run GHA on any compute you'd like (yes, I'm aware of self-hosted runners, and even that their Packer code is open source. That doesn't make it easy to use GHA outside of github.com)
In GitHub we use an OIDC token to access some AWS resources. Locally I need to populate tokens etc and so I have an `if: ${{ACT}}` and a not condition to populate it.
I've had the displeasure of working with Macs on which we wrote code that would end up being a docker container. But what is compiled on a Mac is an aarch64 container. What is compiled on an AMD64 Linux machine is amd64.
This is annoying when the code you write is native or reaches out to native stuff. Then all of the sudden your container that builds perfectly on the Mac doesn't build on Linux anymore. (and vice versa, looking at you gcc on debian).
GitHub just (January) got Linux ARM runners, but they're not available for private customers yet.
And their Mac runners do not come with Docker, and it cannot be installed.
In fairness, the opposite way usually ends up being much cheaper: make prod run arm64
I would guess that just `alias docker="docker --platform=linux/amd64"` type thing would go a long way toward evicting those platform mismatches[1]. I would also guess there's a way to switch off the binfmt "helper" in colima such that trying to run arm64 in the VM would immediately puke rather than silently helping you
1: or, of course, don't push production containers from your workstation, but the years have taught me that I'm talking to myself
I've been using this recently but there's many issues, and things that can only be tested remotely. In some cases I've had to guard steps if: ${{ !env.ACT }} so they don't run locally.
I use this for work - but there are edge cases all over the place that I keep running into (e.g. Yarn being installed on Github-hosted runners, but not self-hosted ones or act - https://github.com/actions/setup-node/issues/182)
Same experience here. Edge cases everywhere, though most can be worked around.
You can specify different runners to use. The default images are a compromise to keep size down. There is a very large image that tries to include everything you might want. I would suggest trying that if you don’t mind the very large (15GB IIRC) image.
I definitely remember considering the larger images - I think we ended up not using them since my work's usecase for act is running user github workflows on-demand on temporary VMs. The hope was that most usage is covered by the smaller images - and in fairness that has been true so far.
I think this is really cool. We're tackling this problem from the other side by building `pixi` (pixi.sh) which bundles project / package management with a task runner so that any CI should be as simple as `pixi run test` and easy to execute locally or in the cloud.
My team has a setup that sounds essentially the same using Nix via devenv.sh. We deterministically bundle and run everything from OpenTofu and all its providers to our programming languages runtimes to our pre-commit hooks this way, and it also features a task runner that builds a dependency graph and runs things in parallel and so on.
Our commands for CI are all just one liners that go to wrappers than pin all our dependencies.
Lately I've been working with a lot of cross-platform Bash scripts that run natively on macOS, WSL, and Linux servers, with little to no consideration for the differences. It's been good!
that’s not really what’s new or special about pixi, is it? poetry (poethepoet) and uv can both do variations of this.
From the outside, pixi looks like a way to replace (conda + pip) with (not-conda + uv). It’s like uv-for-conda, but also uses uv internally.
Better task running is cool, but it would be odd to use pixi if you don’t otherwise need conda stuff. And extra super duper weird if you don’t have any python code!
I've wanted this so much! My main questions are about permissions. I use environment-specific variables and secrets, I would need to reconfigure them locally and didn't see how. The other issue is workload identity federation.
could you just source vars from an .env file? maybe a bit more work to get the secrets pulled down. i guess ideally you'd want to be able to pull those values directly from github as an option (including secrets) if that's your primary source of truth.
Back in the day we called that a shell script. We had this tool called Jenkins that would run the shell script in a new folder every time you pushed a change to gitlab. It was pretty neat.
anyone have any tips for testing actions locally, rather than workflows?
Despite the name, act is really only for the latter. You can try to test a local action by putting it in a workflow and calling that, but if you do a checkout in your workflow that will overwrite the mount of your local code into the act container, meaning you’ll get the version from the remote branch instead. Depending on the action, you may not be able to comment out the checkout step while testing.
act is a great helper tool for another tool that is an utter shit.
Ive given up on gh actions a while ago and transitioned to bash scripts in docker containers. I can reproduce stuff locally, can test it, can debug it, etc.
I dont get how GH Actions do not yet have a tool that looks and works like jupyter workbook. Then it would have been fantastic.
If CI/CD tools were code editors, then Jenkins/JFrog would be like Vi/Vim—powerful and traditional. GitLab CI would resemble Helix or Zed, offering a modern and efficient approach. Meanwhile, GitHub Actions would be akin to VSCode, but with a touch of Dreamweaver CC, blending user-friendly features with comprehensive capabilities.
Have you ever use any of those in a realistic scenario? Any existing CI/CD solution is broken like hell and if you try to do more advanced stuff like reusing templates or actions, all hell breaks lose. Don't get me started on clones via SSH, doing commits from pipelines, etc.
CI/CD is one of the topics that is barely solved or usable ...
Actually, I have considerable experience setting up and running various CI/CD tools such as Jenkins, GitLab CI, and Travis CI.
In my experience, GitHub Actions work quite well, especially for smaller projects or when you can avoid using public templates and actions. While shortcuts like third-party libraries or actions can be convenient, they often come with trade-offs.
For scenarios where having your own runners and build images is beneficial, I would recommend GitLab CI.
Jenkins is a great choice if you need to run builds on dedicated hardware or within a virtualized environment.
When discussing the challenges of CI/CD, I often notice that many issues stem from complex chains of dependencies within the software or product itself. For example, consider a release pipeline that typically takes six hours to complete but fails after four hours. In such cases, having to restart the entire pipeline instead of just the failed job can be quite inefficient.
Additionally, debugging can become much easier when you have your own build images on runners, as you can run them locally for testing and troubleshooting.
What specific aspects of CI/CD do you find broken or challenging?
Just this week I tried giving this another chance to debug some weird CI failures for ruby tests. I’m on M-series macs so there is an inherent platform mismatch. That coupled with us using depot images as action runners. I was never able to get it running past the dry-run stage. There is just a lot of differences between CI runners and my local docker images. I even gave the image from catthehacker a try. Which is like >15GB. It still wouldn’t run. Finally gave up.
Sounds similar to my own experiences trying to debug GH actions locally.
I've tried twice now to get it working, pulling down many GBs of images and installing stuff and then getting stuck in some obscure configuration or environment issue. I was even running Linux locally which I figured would be the happiest path.
I'm not eager to try again, and unless there's a CI that's very slow or GH actions that need to be updated often for some reason, I feel like it's better to just brute-force it - waiting for CI to run remotely.
There's another alternative - debug in CI itself. There's a few ways that you can pause your CI at a specific step and get a shell to do some debugging, usually via SSH. I've found that to be the most useful.
https://github.com/namespacelabs/breakpoint
Came here to say this, and to recommend: https://github.com/appleboy/ssh-action
It’s slow and arduous work to inject at the right point and find out what went wrong. But it’s miles better than act. act is a noble idea, and I hope it can meet its target, but it falls short currently.
ssh-action gets you on the host, and lets you quickly establish what is happening and implement fixes. It’s top notch!
(I’m unaffiliated, just a big fan of ssh-action).
GH Actions have always worked flawlessly for me, as long as I don't ever do anything unusual.
Last week I moved from a custom server back to GH Pages + GH Actions, and broke a feature that depended on an out dir existing after one build phase and before another. I have no idea how to fix it. It's probably simple, I'm just burnt out.
Each job runs on a separate runner. If you want to pass data across, use artifacts to upload and then retrieve the specified file(s).
You can pass outputs between jobs by declaring one job as a dependency of another with needs:, then reference that dependency's outputs with needs.<job>.outputs
ChristopherHX publishes Docker images which are created directly from packing up the root filesystem into a tarball from inside a runner. The images are huge though.
https://github.com/ChristopherHX/runner-image-blobs/pkgs/con...
Something like this may help by letting you ssh into the running instance so you can debug with the exact context: https://docs.warpbuild.com/tools/action-debugger
It’s even worse than that, it’s 20GB compressed and 60GB uncompressed. Regardless, if your VM runs out of disk space there’s no meaningful error (well 12 months ago) except a failure to start. (Possibly colima-specific I dunno I uninstalled docker-docker)
I do find that some things just work locally and fail in real actions and vice versa. For the most part it’s made it easier to move the bar forward though.
This is really only for debugging the logic of your Actions, isn't it?
On my team, Nix seems to work pretty well for encapsulating most of the interesting things CI jobs do. We don't use virtualization to run local versions of our jobs, and we don't run into architecture mismatch issues.
My experience with Ruby on Apple Silicon as been far from seamless. The only way I could thoroughly get rid of problems was by running the most recent Ruby release and dealing with what THAT causes.
I’m pretty sure GH actions don’t run the latest Ruby version.
My experience with both Ruby and Python Apple Silicon has been hit or miss. Mise has made it a lot simpler and seamless but if you need an older version or the absolute latest when it is released, you might have some issues. Mise has helped a lot, I generally do not expect to get an error.
Are there specific problems? I’ve been primarily Ruby at work the last 6 or so years (before, also doing Ruby, but also lots of Java, Kotlin, Swift, FE stuff, and some Python)… exactly that period… and haven’t really noticed any issues. Granted, I don’t typically use a lot of obscure binary-only dependencies, or things whose C/C++/machine code wouldn’t mostly cross-compile fine.
Granted, all this time I’ve always only used rbenv-managed Ruby versions (and rvm before that). I’ve long disliked/avoided Python because its ecosystem lacked any equivalent, but I know “uv” has gained popularity there, and perhaps Mise is good in Python land as well.
Usually just errors building it. Mostly python but I think I had an error with 3.4.3 the other day.
I had luck with my actions I tried to debug. But I also had the issue that the runtime difference is simply too big to 100% trust it. Too bad because I think the tool is quite good.
Every mention of Github Actions is an occasion to start looking for the best alternative in <current_month>, let's go!
Is Dagger usable yet? Is there still hope for Earthly? Are general purpose workflow systems winning any time soon, or are we still afraid of writing code? Any new systems out there that cover all the totally basic features like running locally, unlike Github Actions?
Earthly has pivoted away from their CI/CD products: they’re shutting down Satellite, and they’re stopping active maintenance of the Earthly open source project.
https://earthly.dev/blog/shutting-down-earthfiles-cloud/
I think we could build something on top of nix that is as easy to use and powerful as earthly, but gets all the nice stuff from nix: reproducibility, caching, just use any command from any nix package, etc
That's mostly what we've built with Flox [1] (though I'm not exactly sure what you mean by run any command from any Nix package). It looks and feels like an amped up package manager, but uses Nix as kind of an infrastructure layer under the hood. Here's a typical workflow for an individual developer:
- cd into repo
- `flox activate` -> puts you into a subshell with your desired tools, environment variables, services, and runs any setup scripts you've defined
- You do your work
- `exit` -> you're back in your previous shell
Setting up and managing an environment is also super easy:
- cd into project directory
- `flox init` -> creates an "environment"
- `flox install python312` -> installing new packages is very simple
- `flox edit` -> add any environment variables, setup scripts, services in an editor
- `flox activate` -> get to work
The reason we call these "environments" instead of "developer environments" is that what we provide is a generalization of developer environments, so they're useful in more than just local development contexts. For example, you can use Flox to replace Homebrew by creating a "default" environment in your home directory [2]. You can also bundle an environment up into a container [3] to fit Flox into your existing deployment tools, or use Flox in CI [4].
All that stuff I described is free, but we have some enterprise features in development that won't be, and I think people are going to find those very appealing.
[1] https://flox.dev
[2] https://flox.dev/docs/tutorials/migrations/homebrew/
[3] https://flox.dev/docs/reference/command-reference/flox-conta...
[4] https://flox.dev/docs/tutorials/ci-cd/
I built something like this out using an old version of Dagger and found it enormously complicated, and the they rewrote everything and abandoned the version of Dagger I used.
When I they did, I said "fuck it" and just started distributing a Nix flake with wrappers for all the tools I want to run in CI so that at least that part gets handled safely and is easy to run locally.
The worst and most annoying stuff to test is the CI platforms' accursed little pseudo-YAML DSLs. I still would like that piece.
devenv.sh tasks might be a good basis for building on it. I think maybe bob.build also wants to be like this
In addition to https://devenv.sh and https://bob.build , here are 2 others that are based on Nix.
- https://flox.dev
- https://www.jetify.com/devbox
Yes! I knew that those are comparable to devenv.sh (what my team uses; love it!) in terms of environment management but wasn't yet aware of whether or not they also have task runners like one might use to build CI/CD pipeline logic.
I also want to say that this approach has largely spared my team the pain many users seem to have with Docker on aarch64 Macs. Nix works well on both aarch64 and x86_64, and doesn't require emulation to run. This is really more appropriate for running development tools locally, I think.
that's actually the only good use for nix.
Gitlab? It's been around longer than Actions, it's much better and honestly not sure why anyone would consider anything else (but happy to hear if there are reasons).
because they didn't keep sha1 and so it's not promoted as heavily as gh.
/me puts tinfoilhat
Gitea can be easily self-hosted and supports Gitea Actions, which are largely compatible with GitHub Actions. I’m about to try it out. https://docs.gitea.com/next/usage/actions/overview
In the context of this submission, one will want to exercise caution because they and their forgejo friends are using forks of nektos/act, with all the caveats of both "act" and "fork"
https://gitea.com/gitea/act -> https://gitea.com/gitea/act_runner
https://code.forgejo.org/forgejo/act -> https://code.forgejo.org/forgejo/runner
How's Jenkins looking these days?
Seems fine to me, I don't really understand why people think CI is hard and they need to shop a different platform. All these systems are is a glorified shell script runner. It seems like developers just keep making up new DSLs over and over to do the same things.
We ended up running every job in a docker container, otherwise the Jenkins admin had to keep installing various tools on the runners. Gitlab supports this workflow natively so seemed the obvious choice.
Jenkins also supports this out of the box, and if it didn't have it for the pipeline DSL, again, Jenkins is a general purpose runner, you can just install docker, and run the script to run builds in the docker containers
More yaml will surely save us
I wish jenkins had a better config as code option. Not crazy about groovy and you still need to click in the UI to get some set up before you can use. Maybe im stuck with an old version or something
I mostly like Jenkins though, idk why people so desperately want to always move to something new. I guess I've never been on the maintenance side of running Jenkins for thousands of engineers though.
I wanted to like the new Jenkinsfile approach, but the developer experience was beyond awful
it's hard since you don't have the right environment, you can't debug, can't easily run locally and you have to push a change, and wait for the code to run on a remote machine 100x less powerful than you old laptop.
If these systems thought a moment about the developer experience, they'd be magical, but they do not.
They use github actions
https://github.com/jenkinsci/jenkins/tree/master/.github/wor...
It's still a miserable experience to maintain it, update it, deal with mostly old plugins, dynamically loading the tooling, groovy idiosyncrasies.. and UI/UX that despite recent efforts continues to feel terrible.
Managing the underlying infra is painful, and it remains a security liability even when not making obvious mistakes like exposing it to any open network.
And good luck if you're having that fun at a windows shop and aren't using a managed instance (or 5).
>It's still a miserable experience to maintain it...
How so? I've been maintaining an instance for a decade, and it really doesn't seem that bad. Updating plugins we do about monthly, it's largely clicking a couple buttons in the UI. Updating Jenkins itself is an apt update. Groovy takes a bit to grok, sure, LLMs help a lot here. The UX isn't that bad, IMHO, but I can see why some would say that. We've switched over almost entirely to using a couple runners, docker, and Jenkinsfiles, which works great. We do still run deploys directly on the Jenkins box, largely because we need to run them single-threaded so they don't step on each-other with load balancer rotations.
Your use case is just fine because you're barely actually using Jenkins.
I can give you a few examples of where it falls short:
- security: there are constant CVEs about anything and everything in Jenkins
- upgrade paths: if your company uses lots of plugins, the resulting spaghetti is of Italian proportions
- if said company is in a Windows-only infra, on prem, and they still decided to use Jenkins then good luck doing anything. Try putting an agent on a non-system disk for example, Windows paths aren't handled and you find yourself already passing very specific "pre" commands that your master will send over ssh.
- Said ssh connection can be lost due to a variety of things for which there are quite a few combinations of parameters when invoking Jenkins
- While we're at it, SSO isn't exactly supported in Windows environments. There are two external plugins you can try, one created because the other doesn't work, and even then good luck with that.
- At scale you end up having to be at least minimally interested in GC tuning, as Jenkins runs within the JVM
- UI: normal "Views" are not informative, and a bunch of custom views need to be made but rarely work with all sorts of plugins that people using your CI can consider crucial (say, parametrized cronjobs)
- Using the functionnalities Jenkins offers to "install tooling". Try to get it to use a certain version of node in a pipeline. Any current typical CI solution turns that into a straightforward task that's extendable, but in Jenkins you have to configure a very archaic and barely-working "tooling" area in your system to use that, and this barely works beyond the most basic tools.
- Having to maintain enterprise-level groovy libraries. Good luck. It's Groovy but not exactly Groovy. It's all inside the JVM, but inside Jenkins's abstractions of it inside of the JVM.
- Good luck monitoring lots of agents and doing typical tasks with them. Maintenance Windows are slowly coming in, monitoring sort-of-kinda-works via plugins..
I've maintainted instances in small companies and larger ones. With less custom stuff and with a lot more. Compiling C++, C#, Java/Kotlin, Objective-C/Swift. Building webapps, iOS SDKs, Android SDKs, and native Windows apps.
Jenkins can do everything if you bend it the way you want with some plugins and custom code. That's a strength that nothing else offers, but by and large it is a weakness of CI in the long run. Being opinionated isn't amazing, but sometimes it is required to be less complex, easier to maintain, more secure, easier to use, etc.
With Git Actions at full speed..... it is dying.
Every org I have spoken with recently are in GH action cost reduction mode -- which always, always, always causes issues.
Jenkins too going away, but, like Windows XP, people often reach back out for it.
Still awful last I looked.
nyuk, nyuk, nyuk
Let me throw another one out there: how about TeamCity on premise? Still can be done for free and the last time I used it (years ago) it left a very complete, stable and easy to use impression on me. Jenkins by comparison seems heavy and complicated.
I think Earthly is dead, but Dagger seems to be healthy still...
Earthly was amazing. The exact same setup in CI and locally. They're reviving it with a community effort, but I'm not sure if it'll live
earthly is config-based, so it's in the same league as GHA in my book. https://docs.earthly.dev/docs/earthly-config
dagger is the only code-based solution. It works, but it does have some edges since it has a much bigger surface area and is constantly growing.
That's not really related... that config is machine specific, not really related to CI.
This is what Earthfiles look like: https://docs.earthly.dev/docs/earthfile
Can’t beat GitLab
Rather than tying CI & deployments to Github Actions, it is usually better to pull as much of it as possible out to shell scripts and call them in containers in GH actions..
There are optimizations you’ll want (caching downloaded dependencies etc); if you wait to make those after your build is CI-agnostic you’ll be less tempted by vendor specific shortcuts.
Usually means more code - but, easier to test locally. Also, swapping providers later will be difficult but not “spin up a team and spend 6 months” level.
This is always the top comment in these kinds of threads, and I see this as an indication that the current state of CI/CD is pathetically propriety.
It’s like the dark times before free and open source compilers.
When are we going to push back and say enough is enough!?
CI/CD desperately needs something akin to Kubernetes to claw back our control and ability to work locally.
Personally, I’m fed up with pipeline development inner loops that involve a Git commit, push, and waiting around for five minutes with no debugger, no variable inspector, and dumping things to console logs like I’m writing C in the 1980s.
You and I shouldn’t be reinventing these wheels while standing inside the tyre shop.
We've had open source CI. We still have it. I remember ye olde days of Hudson, before it was renamed Jenkins. Lotsa orgs still use Jenkins all over the place, it just doesn't get much press. It predates GHA, Circle, and many of the popular cloud offerings.
Turns out CI/CD is not an easy problem. I built a short-lived CI product before containers were really much of a thing ...you can guess how well that went.
Also, I'll take _any_ CI solution, closed or open, that tries to be the _opposite_ of the complexity borg that is k8s.
Having a container makes debugging possible, but it's still generally going to be an unfriendly experience, compared to a script you can just run and debug immediately.
It's inevitable that things will be more difficult to debug once you're using a third party asynchronous tool as part of the flow.
That's mostly true.
- You may need something to connect the dots between code changes and containers. It's not always possible to build everything on every change, especially in a multi/mono-repo setup.
- You probably need some way to connect container outcomes back to branch protection rules. Again, if you are building everything, every time, it's pretty simple, but less so otherwise.
- You likely want to have some control over the compute capacity on which the actions run, both for speed and cost control. And since caching matters, some compute solutions are better than others.
I don't think GitHub Actions solves any of these problems well, but neither do containers on their own.
Nix is in fact perfect for this.
As a non-Nix user, could you give me an example?
While flakes have their own "checks" thing, what I do is just have a app definition in flake with the required dependencies.
https://github.com/Cloudef/zig-aio/blob/master/flake.nix#L25...
https://github.com/Cloudef/zig-budoux/blob/master/flake.nix#...
The actual GA workflow file is pretty simple: https://github.com/Cloudef/zig-aio/blob/master/.github/workf...
In my experience (and as reflected by the comments on this post already), trying to run complex CI workflow locally is a fairly hopeless enterprise. If you have a fully containerized workflow, you might be able to get close, but even then ensuring you have all of your CI specific environment variables is often not a trivial task, and if your workflow orchestrates things across tasks (e.g. one task uploads an artifact and another task uses that artifact) you'll have a hard time reproducing exactly what is happening in CI. My company (RWX) builds a GitHub Actions competitor and we intentionally do not support local execution -- instead we focused on making it easy to kick off remote builds from your local machine without having to do a `git push` and we made it easy to grab an SSH session before, during, or after a running task to inspect things in the exact build environment that your workflow is running.
For local development you can use .env files and mise. https://mise.jdx.dev/environments/#env-file
I use dagger to read these .env/mise env vars and inject dummy values into the test container. Production is taken care of with a secrets manager.
It's important to note that this tool does not use the same container images or runtime environment that GitHub Actions actually runs. It's an approximation.
For simple use cases that won't matter, but if you have complex GitHub Actions you're bound to find varying behavior. That can lead to frustration when you're debugging bizarre CI failures.
Yeah, this is lame. With Gitlab you can choose the exact image that runs and provide your own. So you can easily run the exact same code in the exact same environment locally.
I guess it would be nice to have a tool to convert a Gitlab YAML to Docker incantations, but I've never needed to do it that often for it to be a problem.
AWS Lambda publishes Docker images (e.g. public.ecr.aws/lambda/python:3.12-arm64), does Github Actions have something similar?
https://github.com/actions/runner-images
In case anyone else is similarly curious, I managed to get the ubuntu images to build .qcow2 images with some lightweight patches to the packer files and the interior .sh provisioning scripts. I have intention of setting up (heh) GHA to build them on the regular but it has not yet risen to be the highest priority weekend project
If you may say "but, why?!" it's because I really wanted to know what versions of everything I could expect in GHA, and I detest "git commit -amdummy && git push" stupidity so I guess the answer is "because I'm wired that way"
GitHub really needs to support local development with GitHub actions. Sheesh. What a step backwards.
Feel like the correct way to use any CI these days is to just have it do almost nothing. The CI just triggers a script. And then you can run that script locally.
exactly. I am surprised by the amount of comments in this and similar threads praying for github to add more complexity to an already over-complicated and fragile solution.
CI must be local-first and platform-agnostic.
Because you may want caching, or to install certain software in the VM or container that you already have locally. Separating scripts could help, but IME the CI environment is always different enough that some branching or mimicking is needed to harmonize them.
Caching is fine, you likely don’t have to replicate that locally as in theory it should do nothing. For installing software, create a docker image that already has everything you need.
I'm convinced that they specifically won't do that so that people won't use them to build competing products. Actions is one of the pieces that make GitHub money.
> Actions is one of the pieces that make GitHub money.
How do you figure that? I'd buy "lock people in the platform," but in that way GitHub Issues has been the moat for far longer than Actions has
I figure because they charge for minutes, I'm not saying that it actually turns a profit. My company also pays for certain "xl" machines.
Almost every company I know of is in love with self hosted, or 3rd party, runners because they can't reason about OIDC auth into their cloud, or it's more evil sibling VPN
I'm open to your suggestion but I struggle to get onboard that Microsoft thinks that CI compute billing is the future
> GitHub really needs to support local development with GitHub actions.
Agreed. I’m thankful for tools like act, but there really should be an officially supported way to run gh actions locally.
WARNING: act is great if you use docker. Act does not support podman.
Issues or discussions related to providing support/coverage/compatibility/workarounds for podman are closed with a terse message. Unusual for an open source project.
There's a long-open issue[1] for Podman support. Maybe they're just tired of dups?
[1] https://github.com/nektos/act/issues/303
Seems like that was also closed for about half a year, maybe GP came across [this discussion](https://github.com/nektos/act/issues/553#issuecomment-791562...) during that period and (understandably) came to their conclusion from that.
Dumb question, but why hasn’t GitHub made a solution that lets you run GitHub Actions locally? Or at the very least a solution that validates the action (giving a bit more certainty that it will succeed, a bit like a dry-run)?
(My war story:) I stopped using GHAs after an optimistic attempt to save myself five key strokes ‘r’ ‘s’ ‘p’ ‘e’ ‘c’ led to 40+ commits and seeing the sunrise but still no successful test run via GHA. Headless browsers can be fragile but the cost benefit ratio against using GHA was horrible, at least for an indy dev.
Why would GitHub (a Microsoft service) want you to be less dependent on GitHub? That just makes the extinguishing part harder.
They do.
Your action can be empty and actions generate webhook events.
Do whatever you want with the webhook.
It’s the allure of the marketplace that gets people. They’re like “oh I could parse my test report files and upload all of them somewhere and render them with a nice output on my pr! I could spend a week writing and testing all of this or I can slap this action here into the YAML and be done with it in 10 minutes.
The trap and tradeoff is that the thirtieth time you’ve done that is when you realize you’ve screwed yourself and the organization by building this Byzantine half baked DAG with a very sketchy security story that you can’t step through, run locally or test
Like https://docs.gitlab.com/runner/ ?
I'm as big a GitLab fanboy as they come, but they recently axed the gitlab-runner binary's ability to execute local .gitlab-ci.yml files <https://gitlab.com/gitlab-org/gitlab/-/issues/385235>. It now only operates in "receive webhook from the mothership" mode, just like GHA
Pour one out, I guess, but it's okay since I previously was super angry at it for languishing in the uncanny valley of "hello world works, practically nothing else does" -- more or less like nektos/act and its bazillions of forks
Disappointing. That was very useful for troubleshooting issues.
Last time I looked at it, Act was a lot like the “serverless offline” options out there that try to mimic AWS services. If your use case is straightforward then I might work but if you do anything “exotic” (often the types of things I’m trying to debug in a failed run) Act doesn’t fully replicate the GHA experience.
Also, understandably, there is no macOS support and I use macOS on GHA for iOS builds (another place I have to debug that this wouldn’t cover).
What are some recommended alternatives for act, if any?
I’ve been a long time user, but I’ve run into tons of problems with act over the last year and am wondering if there are better alternatives out there.
I’ve been bedeviled by arm/intel/Mac issues with this.
I want to be able to use this project, I really do. But it’s just not yet there, and this isn’t on Nektos. Nektos is, as best I understand it, trying to approximate GHA, but it’s not easy.
I've seen (but not used) this tool recently, which seems like it does a similar thing. Curious if it is any better experience.
https://github.com/bahdotsh/wrkflw
As someone with a deep interest in this space, I spent a non-trivial amount of time tire-kicking that. One can view the issues for my experiences, including the one where I wished the dev all the best but stopped playing QA for the project. It's goal is primarily a TUI for GHA, not an emulation for GHA
I try to do as much as possible in a (dagger) script and use the GHA to call it. That way I can test the script locally.
I wonder if there is a proposal to support code-based actions. Config-based CI needs to die.
Yup. Every time there's one of these threads I renew my call for GHA to have hooks where in-band build tools like nix, bazel, dagger, and friends can propagate upward optimal information to be displayed in the web gui, in particular annotations and info about multiple tasks that run in parallel or series.
Why dagger and not just... any language? (Nushell for example https://www.nushell.sh/)
Because I'm typically building and running tests in containers in CI, which is what dagger is for.
nu is my default shell. Note that I am not talking about dagger shell. https://dagger.io/blog/a-shell-for-the-container-age-introdu...
like a lot of folks i found nektos lacking. instead, i focused on keeping the gha workflows really light and putting any complexity in Task (taskfile.dev)—that way the meat of the workflows can be run locally, and the gha syntax is just the glue.
it works out very well.
It's amazing that Microsoft built a whole IDE and programming environment for builds but left all the debugging tools up to the community.
You don't get vendor lock in if you're able to run GHA on any compute you'd like (yes, I'm aware of self-hosted runners, and even that their Packer code is open source. That doesn't make it easy to use GHA outside of github.com)
Lots of changes have to be made to your github action workflow to actually make this work properly.
What things have you needed to change? Can't say I've ever needed to do that but of course I've only used it on a few projects.
In GitHub we use an OIDC token to access some AWS resources. Locally I need to populate tokens etc and so I have an `if: ${{ACT}}` and a not condition to populate it.
I've had the displeasure of working with Macs on which we wrote code that would end up being a docker container. But what is compiled on a Mac is an aarch64 container. What is compiled on an AMD64 Linux machine is amd64.
This is annoying when the code you write is native or reaches out to native stuff. Then all of the sudden your container that builds perfectly on the Mac doesn't build on Linux anymore. (and vice versa, looking at you gcc on debian).
GitHub just (January) got Linux ARM runners, but they're not available for private customers yet.
And their Mac runners do not come with Docker, and it cannot be installed.
> And their Mac runners do not come with Docker, and it cannot be installed.
?????
If you have a Docker runner on macOS, you're just running a Linux runner in a VM. So just register a Linux runner in a VM, right?
Sorry for the late reply.
GitHub Actions' macOS runners do not come with Docker installed, because of licensing issues: https://github.com/actions/runner-images/issues/17
Then there is the issue that the macOS runners are virtual instances, and pass-through virtualization has not been enabled:
https://github.com/douglascamata/setup-docker-macos-action#:...
Match dev env to prod env?
No, as their MDM solution only works on Macs.
In fairness, the opposite way usually ends up being much cheaper: make prod run arm64
I would guess that just `alias docker="docker --platform=linux/amd64"` type thing would go a long way toward evicting those platform mismatches[1]. I would also guess there's a way to switch off the binfmt "helper" in colima such that trying to run arm64 in the VM would immediately puke rather than silently helping you
1: or, of course, don't push production containers from your workstation, but the years have taught me that I'm talking to myself
All of this is an effort to validate that building works on all environments.
> alias docker="docker --platform=linux/amd64"
I think that on its own breaks a Rust project which uses openssl with the vendored feature.
I've been using this recently but there's many issues, and things that can only be tested remotely. In some cases I've had to guard steps if: ${{ !env.ACT }} so they don't run locally.
I use this for work - but there are edge cases all over the place that I keep running into (e.g. Yarn being installed on Github-hosted runners, but not self-hosted ones or act - https://github.com/actions/setup-node/issues/182)
Apart from that it's been quite good!
Same experience here. Edge cases everywhere, though most can be worked around.
You can specify different runners to use. The default images are a compromise to keep size down. There is a very large image that tries to include everything you might want. I would suggest trying that if you don’t mind the very large (15GB IIRC) image.
I definitely remember considering the larger images - I think we ended up not using them since my work's usecase for act is running user github workflows on-demand on temporary VMs. The hope was that most usage is covered by the smaller images - and in fairness that has been true so far.
I'm convinced that the only way is to do the reverse. Make the Github actions just be a thin wrapper over docker-compose invocations.
I think this is really cool. We're tackling this problem from the other side by building `pixi` (pixi.sh) which bundles project / package management with a task runner so that any CI should be as simple as `pixi run test` and easy to execute locally or in the cloud.
My team has a setup that sounds essentially the same using Nix via devenv.sh. We deterministically bundle and run everything from OpenTofu and all its providers to our programming languages runtimes to our pre-commit hooks this way, and it also features a task runner that builds a dependency graph and runs things in parallel and so on.
Our commands for CI are all just one liners that go to wrappers than pin all our dependencies.
Lately I've been working with a lot of cross-platform Bash scripts that run natively on macOS, WSL, and Linux servers, with little to no consideration for the differences. It's been good!
I'd be interested to know more about a team that uses Nix and Guix. Is there a website or email an interested party can contact?
that’s not really what’s new or special about pixi, is it? poetry (poethepoet) and uv can both do variations of this.
From the outside, pixi looks like a way to replace (conda + pip) with (not-conda + uv). It’s like uv-for-conda, but also uses uv internally.
Better task running is cool, but it would be odd to use pixi if you don’t otherwise need conda stuff. And extra super duper weird if you don’t have any python code!
I would love for this to actually work, but apart from trivial workflows, it never replaced the dreaded game of CI ping/pong.
My alternative was to have a dedicated repository where I can spam the Git and workflow run histories as much as I need to experiment with workflows.
I've wanted this so much! My main questions are about permissions. I use environment-specific variables and secrets, I would need to reconfigure them locally and didn't see how. The other issue is workload identity federation.
Secrets can be loaded from a file.
https://nektosact.com/usage/index.html?highlight=Secret#secr...
could you just source vars from an .env file? maybe a bit more work to get the secrets pulled down. i guess ideally you'd want to be able to pull those values directly from github as an option (including secrets) if that's your primary source of truth.
Back in the day we called that a shell script. We had this tool called Jenkins that would run the shell script in a new folder every time you pushed a change to gitlab. It was pretty neat.
I'm using this to run on a risc-v board I have at home, it's pretty nice.
My action just calls `make build test` but it is convenient to have it hooked to GH for automated testing on PRs
anyone have any tips for testing actions locally, rather than workflows?
Despite the name, act is really only for the latter. You can try to test a local action by putting it in a workflow and calling that, but if you do a checkout in your workflow that will overwrite the mount of your local code into the act container, meaning you’ll get the version from the remote branch instead. Depending on the action, you may not be able to comment out the checkout step while testing.
I needed to do this recently, the only way I could find was with a proxy workflow.
This didn't work for me. I found just running Ubuntu in VirtualBox was close enough for my use case.
`act`is great, but it's still annoying dealing with secrets!
Anyone use kubernetes argo workflow for CI automation?
wow this is a popular project^_^
Which GitHub Actions context variables are emulated or customizable in act, like github.event, github.actor, or secrets
We have come full circle
act is a great helper tool for another tool that is an utter shit.
Ive given up on gh actions a while ago and transitioned to bash scripts in docker containers. I can reproduce stuff locally, can test it, can debug it, etc.
I dont get how GH Actions do not yet have a tool that looks and works like jupyter workbook. Then it would have been fantastic.
If CI/CD tools were code editors, then Jenkins/JFrog would be like Vi/Vim—powerful and traditional. GitLab CI would resemble Helix or Zed, offering a modern and efficient approach. Meanwhile, GitHub Actions would be akin to VSCode, but with a touch of Dreamweaver CC, blending user-friendly features with comprehensive capabilities.
Have you ever use any of those in a realistic scenario? Any existing CI/CD solution is broken like hell and if you try to do more advanced stuff like reusing templates or actions, all hell breaks lose. Don't get me started on clones via SSH, doing commits from pipelines, etc.
CI/CD is one of the topics that is barely solved or usable ...
Actually, I have considerable experience setting up and running various CI/CD tools such as Jenkins, GitLab CI, and Travis CI.
In my experience, GitHub Actions work quite well, especially for smaller projects or when you can avoid using public templates and actions. While shortcuts like third-party libraries or actions can be convenient, they often come with trade-offs.
For scenarios where having your own runners and build images is beneficial, I would recommend GitLab CI.
Jenkins is a great choice if you need to run builds on dedicated hardware or within a virtualized environment.
When discussing the challenges of CI/CD, I often notice that many issues stem from complex chains of dependencies within the software or product itself. For example, consider a release pipeline that typically takes six hours to complete but fails after four hours. In such cases, having to restart the entire pipeline instead of just the failed job can be quite inefficient.
Additionally, debugging can become much easier when you have your own build images on runners, as you can run them locally for testing and troubleshooting.
What specific aspects of CI/CD do you find broken or challenging?