I haven't used systemd timers enough to disagree, but
> Ambiguous $PATH settings make cron script execution difficult to predict.
What makes you say that? You can set the PATH right in the crontab. Is that harder to "predict" than it being set in /etc/bashrc, ~/.bashrc, ~/.profile, ~/.bash_profile, /etc/systemd/…, or wherever else?
> You might feel cool knowing the scheduling grammar by heart
I've used Linux since 1994 and I don't know it by heart. But luckily it's pre-printed in the crontab as comments:
# For more information see the manual pages of crontab(5) and cron(8)
#
# m h dom mon dow command
You just put numbers aligned with the titles.
The rest of the complaints, sure. Next time I need a cronjob, I'll try it out.
That is not a fair summarization of their point because that is not the grammar. There's commas, slashes, asterisks, combinations, and then if you want randomization you need to put it in the command itself because cron can't do it. (Some crons can, but it's not a general capability of cron.) Writing a non-trivial cron spec is not easy.
I am familiar with the syntax, so I am biased ("*/3" and "12,14,20" makes sense if you are familiar with Unix tools), but it is still more intuitive to me than the systemd unit file syntax and usage. I know that I just have to edit /etc/cron or throw any executable file into /etc/cron.d/monthly and it will work on my system, but I cannot write a systemd timer file from scratch without looking it, and to do that I first have to find the directory where the other examples are located. /etc/systemd doesn't appear to be it.
This is generally my only real complaint about systemd. I don't care if it is too monolitic, written in C or whatever, I just want a straightforward syntax for straightforward operations. I'd like it if systemd could recognize if a .target file is a shell script and just do "the right thing". Perhaps it would make sense for a timer file to recognize cron syntax as well. Or at least allow for a kind of extensibility so that I can have it supported.
If systemd had a little more respect for existing conventions, I am pretty sure it wouldn't be so controversial. After all, system administrators like it because they use it all the time, but a regular, full-timer user like me, who only deals with it when something is broken or have to use it as a means-to-an-end to set something up, then all friction is annoying and bad UX. (And no, using Nix is not the solution)
Yeah, it would be nice to have a folder like /etc/systemd-jobs/ where I could put them and where there are no files unrelated to job scheduling. There is /etc/systemd/user, but it does get a bit of pollution depending on the system.
$ systemctl cat public-inbox-watch@.timer
# /etc/systemd/system/public-inbox-watch@.timer
[Unit]
Description=Periodic fetch of public mailing list
[Timer]
# twice a day
OnCalendar=*-*-* 5,17:35
RandomizedDelaySec=1h
Persistent=true
[Install]
WantedBy=multi-user.target
[1] This question is sarcasm. SystemD is often like this... dead simple things look dead simple, but complex things are -if they're possible at all- at least as complex as they are everywhere else.
That's simple but consider "run something 4x per day but randomize a delay by hour so all of the 200 servers doing that task won't run it all at once"
In cron, you basically have to either use your configuration management to generate those times, or have a random delay script running before the command
In systemd timers, it's just
OnCalendar=0/6:00:00
RandomizedOffsetSec=60m
and the offset generated will be stable for the job on a given machine (i.e. always same on this machine but different on others) so you will get nice uniform distribution of load.
If you add
Persist=true
the job will also be run once if there was one or more scheduled runs when the machine was down
> In cron, you basically have to either use your configuration management to generate those times, or have a random delay script running before the command
Nope. From crontab(5)
The RANDOM_DELAY variable allows delaying job startups by random amount
of minutes with upper limit specified by the variable. The random scal‐
ing factor is determined during the cron daemon startup so it remains
constant for the whole run time of the daemon.
That's from my cronie install, but it looks like this has been a feature of some crons for at least a decade. (Notice that the post date of [0] is in 2016.) Given that cronie is based on vixie-cron, and I think I was was using vixie-cron in 2002, I bet it's been a thing for at least twenty years.
Having had to work on an application supposedly supporting cron expressions: the numbers are just the basic parts of the language.
When someone inputs something ridiculous like "5,3/4 4-8,11 1 4,5,6,9-11 */2" you get to enjoy the fun of reverse engineering what they meant (it's never what they actually wrote).
And that's before you get to all the extensions supported in some cron environments (but not all).
I find systemd timers a lot more manageable. Things like having control over whether or not long-running jobs are allowed to overlap and the ability to run tasks between start-finish rather than a fixed time window are major improvements for me. At some point my VPS went down because the backup job ran into some kind of symlink loop and cron just kept spawning more and more backup tasks even though none of them finished.
Having to re-write commands and scripts because CRON had its own special PATH was also a pain point, but the same can be true for some types of systemd timers. But: you can execute those timers manually if you want instead of updating the crontab to trigger in 30 seconds and simply waiting.
Complex expressions are one of the things I don't like in cron. On Debian/Ubuntu servers, I bite the bullet with systemd timers. On my workstation, I have a personal job scheduler that feels easier and more fun to tinker with. The scheduler uses Starlark functions instead. For example:
# Run if at least a day has passed since the last run
# and it isn't the weekend.
def should_run(finished, timestamp, dow, **_):
return dow not in [0, 6] and timestamp - finished >= one_day
What's so hard about "At 5 minutes past the hour and every 4 minutes, starting at 3 minutes past the hour, at 04:00 AM through 08:59 AM and 11:00 AM, on day 1 of the month, every 2 days of the week, only in April, May, June, and September through November"?
The main nice thing about the environment in systemd is that it is standard and mostly a blank slate, whereas at least for me I was always getting bit by the fact that the environment in Crontab was completely different from say, the environment inherited by supervisord or sysvinit scripts. In systemd the actual unit that gets executed is the same regardless of what triggers it, so there is no gap.
That does require you to still know what the default environment is, but it is a mostly completely clean environment, without any influence from any shell.
I'd have to concur that I agree this is an advantage of systemd.
> That does require you to still know what the default environment is, but it is a mostly completely clean environment, without any influence from any shell.
Odd. This script
#!/bin/bash
set > /tmp/set.txt
when scheduled like so
* * * * * $HOME/bin/testCronScript.sh
Produces this file in /tmp/set.txt which has had a handful of values (HOME, UID, etc) lightly redacted prior to posting here -to remove PII or for length- but its keys are entirely untouched:
Seems pretty clean to me. Even when I run this via /etc/crontab, rather than as a user cron job:
* * * * * root /home/user/bin/testCronScript.sh
I get effectively the same results.
Maybe your distro's default cron environment was bad, and you never bothered to check and unset the badness? I'd be surprised if they were unable to make the default environment for Timer Units to be bad.
We are now considered old and therefore irrelevant. The new generation uses timers and couldn't care less about cron that has served us just fine for decades.
I use cron and my general attitude towards LP and systemd is very similar to the attitude of LP and systemd to us.
> That's true, but most people don't know the numbered manual sections, so they get the docs for the cron table command not the cron table config file.
problem with vars is that they apply to any subsequent entry in the file so you need to take that into consideration; the nice thing about timers is that all settings are self contained and not affected by previous entries. The standard /10 and similar cron expressions also have thundering herd problem when on bunch of servers, tho some variants like in Jenkins use variant H/10 (H standing for hash) where the thing is randomly shifted in time to not hit same minute on same server/job
another benefit is having logs in one place for the job; cron's "send a mail when there is any amount of output text" is just annoying behaviour, but also only place to get the job output unless you redirect it somewhere. Also starting from timer vs just doing systemctl start job.service is the same so easier to debug
other than that the few improvements in how to specify run time have been pretty useful.
For example, setting timer as "persistent" will mean any run "lost" to machine powered off will just be ran next time after boot, so you can have job on your PC that is just "run backup at 2AM" and if you turn it off before that you get the backup done first thing in the morning
There is also both random, and fixed (depending on machine UUID) random delay so avoiding thundering herd problem with backups is also pretty convenient.
There is even option to wake a device for the job if necessary tho the problem of shutdown is left to the user. And picking whether to start counting to next timer from previous one or from the job's end.
What I would like also is to have job summary page ("hey this job was done X times but failed Y times") but that's probably better left to external tooling
> You can set the PATH right in the crontab. Is that harder to "predict" than it being set in /etc/bashrc, ~/.bashrc, ~/.profile, ~/.bash_profile, /etc/systemd/…, or wherever else?
There is* a common trap as the cron PATH is usually just /usr/bin:/bin so anything in /usr/local/bin, or in /sbin won't be there.
I have a Canon printer, I actually can't trust that their print nozzle won't get jammed up after sitting idle for a while. So I had claude setup a systemd script to print a picture of my dog every week, I ensure it has enough CMYK spectrum to stress the printer. Its a nice surprise every monday as I sit on my desk to see a sudden picture pop up from the printer :)
I wish printers could have a mode like this to print random images from an album, or a calendar, rather than wastefully draining ink into a sponge every few days.
If nothing else, maybe it could be some kid's high school science fair project idea.
How about printing a QR code for a randomly generated private key for Satoshi Nakamoto's Bitcoin wallet, then every few days you get a tiny moment of excitement, hope, and then disappointment. It's still wasteful, but it could pay off big time?
Maybe I'm misremembering, but I'm sure there was something on HN a few weeks ago about an electric typewriter that someone had connected to (I'm guessing) a Raspberry Pi? My search-fu is currently failing to find anything particularly recently, at the moment.
It sat unused and powered off for a couple of years after he passed, until I needed a color print.
Didn't do anything but hook it up to power and print. Took about 1/5 of a page until all colors were back in action, after that it printed about 20 pages flawlessly.
This is the part where I get to point out that Brother inkjets do a little dance ~every day that keeps the heads fresh. They do this on their own as for long as they're powered up.
This allows them to work well even if years go by between prints. It's a very thoughtful design element.
(They don't survive sitting for months and months unpowered on a shelf very well, but... you'll have that.)
I was about to recommend a cheap OKI LED color printer (I think C322dn); alas they withdrew from consumer market :/
The colors are super nice and uniform, even if the maximum resolution is only 600 dpi - and the toner won't dry out, which was my brother's crucial purchase criterion; we had HP inkjet clogged more than once.
Moved from cronie to systemd timers because they are resilient to system startup times. My backup strategy is to create a borg archive entry every day at a fixed time. With cronie the system needs to be running at the scheduled time, but systemd timer tolerates this and runs the service as soons as the system is available.
Cronie has a mechanism for this, called "anacron", which is called hourly by cron (on my system, /etc/cron.hourly/0anacron), and performs all the /etc/cron.{daily,weekly,monthly} tasks, no matter if the earliest possible schedule was missed (and with a configurable random delay). You can modify /etc/anacrontab to create custom schedules.
To do this at the user level, you can add something like "@hourly anacron -t /path/to/anacrontab -S /path/to/spooldir" to the user's crontab, though I've never tried this.
Many cron implementations have a similar mechanism.
This isn't the same as with systemd timer because timer lets you specify when you want to run your service exactly and will fallback to running when the system comes online. With @hourly I lose this control and multiple machines could potentially trigger backups at the same time, hogging the physical hard drives and the network.
> fallback to running when the system comes online.
That isn't something I'd want to happen, it sounds like it creates a potential queue of scripts that will flood the system on start, if it works the way you described.
I prefer the deterministic behavior of cron, the script will run when it is specified to run, as you said earlier, as long as the system is running; and as I stated in a separate comment, it will run @reboot if I need it to run then.
> With @hourly I lose this control and multiple machines could potentially trigger backups at the same time
Then don't use @hourly, use staggered times, it's very easy.
> That isn't something I'd want to happen, it sounds like it creates a potential queue of scripts that will flood the system on start, if it works the way you described.
This isn't what happens. If you leave it offline for days it'll only trigger the service only a single time.
I interpreted it more like "I have these 500 different cronjobs all spread out across $unit_of_time. If the system is down for longer than $unit_of_time and then comes back, does all 500 jobs start running instantly (since they missed their previous deadline)?"
Just to be clear, this isn't default systemd timer behaviour, you need to opt in by setting Persistent=true. If you have hundreds of jobs like this you need a proper queue and neither cronie nor systemd is the right tool because at that scale you'd surely need better observability
If you have 100 different jobs that were supposed to run over the past week, but didn't because offline, when you restart, they they all flood the system on start.
100 jobs all running at different times throughout the week is a very different load than them all falling back and running at the same time on system boot.
> That isn't something I'd want to happen, it sounds like it creates a potential queue of scripts that will flood the system on start, if it works the way you described.
There are two options to fix it;
Disable persist so no catching up on missing scripts. Set OnBoot=5m so it gets ran 5 minutes after boot, so your script (say backup) is ran on boot first, then every time on schedule
Enable persist but just add sleep in ExecStartPre - very "cron" way but there is just no in-systemd option to enable "catch up" script to be delayed
Sadly no option to "run catch-up timers with delay" at least yet
> Then don't use @hourly, use staggered times, it's very easy.
Not in cron. In systemd it's just RandomizedOffsetSec=30m and it is "stable" - same host with same job will always have same delay so on multiple hosts it is spread nicely. There is also non-stable version
> With cronie the system needs to be running at the scheduled time, but systemd timer tolerates this and runs the service as soons as the system is available.
I love systemd timers! I've slowly moved all of my ansible-deployed cron jobs to timers (now just an ansible copy!). The integration with journalctl, especially in a newer OS like Debian 13 where syslog is gone, is really nice. It's also really nice to be able to start the service manually for debug. Having a cron job that didn't work was an annoying exercise in copy/pasting or writing an extra shell script. Don't even get me started on the black hole of cron job stdout. I can monitor systemd services like I already do and get a notification on failure.
I've noticed more and more open source projects recommending timers as a deployment method and I think that's great!
Timers can work with arbitrary units (not just a similarly-named service unit) so they can be surprisingly flexible. I have a timer on my servers that starts a backup.target that fires off a full "restic backup","restic prune", "restic forget" backup cycle each morning with randomized start times and notifications. The actual restic-* units are Podman Quadlets so the whole setup runs agnosticaly of what's on the server, just as long as it has Podman and Systemd installed.
I will admit thought, timers are up there in terms of being the clunkiest systemd unit type to use on a regular basis. I get why they're split up into two files and require different start vs enable syntax's, but man sometimes I just want to create a file that runs a script and be done with it.
I feel like systemd units could need a layer of abstraction above them, so instead of editing the files manually, a tool would do it, some kind of declarative CLI or something. Probably not really a concern in the age of LLMs anymore, but it feels just slightly too tedious every time.
Should have been more clear: I use RandomizedOffsetSec= to add a random offset to a set start time (usually 4am), to prevent overloading the backup server, not truly random start times.
I will use what I am comfortable with and so should others. CronD, SystemD, atD, multiple conditional checks in a shell script, whatever tickles your fancy. There is no wrong answer, just document what you did and add a comment. Comments are permitted in cron. If someone keeps putting complex obfuscated time structures into cron make them decipher their incantation and keep nagging them until they keep it simple, comment their cron entries or until they and their manager resign.
For what it's worth there are usually web apps popping up that can decipher goofy cron time/date incantations. [1] This one has a git repo in the top right, not my repo. Maybe clone it just in case their site goes away some day.
I believe one of the major distro lines (redhat or debian, I forget which) uses systemd-cron, where cron is just a thin wrapper around systemd. You get more power from writing the unit files directly, but if all you ever need is a simple cron job, you have the old interface still available.
Yep, I use this for a @reboot job and a few regular jobs on my home server. I use user crontabs, so I can get around the "unknown shell/path/etc." by prefixing every job with
/some/shell -l myjob.sh
or sometimes
. ~/.profile && cd /some/where && ./job >>cron.log 2>&1
NixOS comes with systemd, so I've been using it as a first-class part of managing stuff. It's great, especially coming from macOS' launchd.
Which makes it nice to distribute a tool for NixOS so that it can lean into systemd instead of as some bolted-on afterthought.
Makes me wonder what you'd do if you were distributing a lifecycle-heavy tool for Linux users in general since systemd isn't ubiquitous.
I use a systemd timer to run a monthly scrub for my btrfs pool. Kinda cool how you can do increasingly useful things like skip the next scheduled event if the user initiates a scrub, do or don't accumulate tasks if you have a monthly task but the machine was offline for 6 months -- or fold them into a single task, etc.
Have you been defining them directly in your flake.nix file? I too am on nixos but I keep all my configurations in their native format and symlink them with nix, that way I can take and reuse that config on a non nixos system easily.
The problem I have found is that nixos doesn't seem to pickup and run systemd timers and services placed into the ~/.config/systems/user folder and additionally things like WantedBy=default.target have no effect.
So after I restart all my services manually on reboot I agree, systems timers are cool.
I haven't always been the biggest fan of systemd in some regards, but I will say that I mostly agree with this sentiment. I've almost completely quit using cron, and now favor systemd timers for scheduled jobs - at the "system" level anyway. I might still embed Quartz for scheduling that's scoped to a particular application or something.
Why? It's one of those fuzzy and somewhat hard to explain things. The systemd approach just maps more cleanly to my mental model of "how things should work" I guess. And maybe some of it is that I did indeed experience plenty of " Ambiguous $PATH settings make cron script execution difficult to predict" in the past, although it's not just that.
I won't sit here and claim that systemd timers are necessarily better than cron in any universal / objective sense. But they've won me over, for what it's worth.
Has it actually served you well? Because it hasn't served me well at all.
I am not the biggest fan of systemd, but today I will always reach for a systemd timer over cron simply due to the sheer amount of bad experiences I've had with cron. Hours upon hours wasted trying to troubleshoot crons that weren't working due to some stupid obscure issue, having to use dirty hacks to monitor for success or retry failed jobs.
A few years ago I was trying to run a very simple bash script with cron and the script just died halfway through for no reason. Nothing in logs, worked fine when run directly, but in cron it just stopped halfway through a loop. Never figured out the cause, just gave up and used a timer instead, which worked fine. Never touched cron again after that.
The ease and convenience of monitoring and troubleshooting alone are worth switching over.
I'm sympathetic, but "bad script" is an awful assertion.
We are all guilty of making bad scripts, bash is a disgusting degenerate language (and I love it). The way we learn to write good scripts is by writing bad scripts in enough amounts to get bitten by all the warts.
One thing I really love about cron, is that if you set up mail on the server (which: you should btw), then cron actually sends emails if it sees anything in stdout and stderr.
I am a dyed in the wool systemd non-believer, but I really do like the timers.
I don't agree that these are just limitations. The fundamental problem cron tries to solve is very simple: I want to run a program automatically at specific times. There are probably many features of systemd timers that can be considered niche or extraneous in solving this problem, but the ability to easily know when the program last ran and what its exit code and stderr output were is not one of them. I believe that if an alleged solution to this problem doesn't provide at least this, it's not really solving the problem.
> Unrelated to cron. Bad script
Again, worked fine when run manually, worked fine in a systemd timer. Pretty sure I still have it running today and it continues to work fine without ever failing.
> Like imagine trying to explain systemd timers and services and unit files to a beginner.
I think it's... easier? Like "systemd is the place where your system manages all the processes it needs to run. Part of those processes can be run on a schedule, or on a timer, and you define them using this simple text file".
cron is easier for easy stuff ("just run this every 10 minutes") but harder for hard stuff ("run it every 8 hours but with randomized offset so not all machines at once do it, but also if machine was down when it should run, run it immediately").
It is also easier to debug as every job gets its own log rather than trying to write to system mailer nobody had set up with the job errors
This is a very good intro to systemd timers -- I think you convinced me to finally start using them. Love the "list-timers" thing as well. With cron, it never seemed easy to me to get a picture of all the cron jobs running on a box. I'd need to check crontab for all users, as well as /etc/cron.d/, as well as the daily/hourly/monthly directories.
And in fact I do have a use-case for needing to run something ~5 minutes after the system boots and then every ~12 hours onward from there. It's great that systemd timers has me covered!
I hard disagree. Previously I could use what I knew about Linux commands to read logs and administer the systemm. Systemctl knowledge does not generalize.
I have done scheme all my life, which is why I prefer shepherd. Not only is it in a syntax that i can use elsewhere, I get completion in Emacs.
This is such a modern view. People used to HATE systemd when it first came out, but I always liked it and knew people would eventually come around and its nice to see they finally did!
And you immediately lose the ability to do `crontab -l` on any server to know its scheduled tasks.
Now you get to look around the myriad of places where you can put systemd files, and figure out which ones are base services and which ones are custom, with no general convention to go about it. Nope.
Never thought I'd see hackers saying INI format looked ugly of all things. It's basic, sure, but that's a good thing for something meant to be easily editable by hand from any editor. Otherwise, it's just key value pairs in named sections, how ugly can it be about that?
Is this one of those cases where at one point you had an error in the file and you figured it was down to spaces? You fixed that issue, it still didn't work but from that point you never thought to question the assumption.
I find myself doing this sort of thing all the time..
Somewhere in my head I had that spaces caused a syntax error, and the UI for systemd is not obvious when you first start using it ... so if it's working then leave well alone. I'll be making all of my .service files (not so many) human-readable in the near future!
There's definitely some weirdness to certain parts of systemd service files, but was a huge improvement over Upstart and the old SysV-style init scripts.
Over all I think Systemd get way to much criticism. You don't have to use all the parts, but if you care to go through the documentation you'll find interesting features such as journald log-shipping and systemd-machined which can manage containers and VMs.
Oh yes, because the well documented clean syntax of sys v init shell scripts was so nice.
If I never recall hacking in ulimit calls in the top of buggy shell scripts for crappy old services that done respect pam_limits it won’t be soon enough.
Since systemd is successfully parsing its INI files, and barks at you when you put weird shit into them, a grammar for them does exist as well.
XML is that wonderful format that gave us vulnerabilities like death by million laughs, up to a certain moment, you could MitM DTDs, and a whole slew of everything-XML stuff back when XML was like AI is today, none of which I miss today.
Oh, and remember times when programmers would argue whether argument order in XML files should be significant or not?
But XML books with their idealized XML future description did give me the same warm fuzzies as some intricate clockwork mechanism to a Victorian geek.
Does systemd ship with something to upgrade your cron jobs for you? That would be the friendly way. Write your old school cron jobs, and then a script that converts them to do things the systemd way, documenting its steps, i.e. I created this file and this is why. Friendly "I help you do things better" rather than standoffish "your way is obsolete, you need to do it our way". Oh wait. I get it. LLM agents can do exactly that for you can't they. Another way I'm behind the curve.
I have knocked together a systemd service or three based on google copypasta. But generally, for cron jobs, why make it complicated? One line in /etc/crontab and done. I generally call an encapsulation script that sets the right environment variables, uses absolute paths, captures stdout/stderr if required and so on. I just want the simplest possible way to launch that script on a schedule.
I'm fully ready to drink the "just let systemd do all the things" kool-aid, but I would love to see some sort of introductory/tutorial info into some of the things it can do other than services - i.e. containers and timers. I know man pages exist, but it would be nice if there was more scannable intro out there.
I design all my services expecting to receive sockets this way. It makes sandboxing easy as the service itself doesn't need network access to have a listening socket.
It's a shame docker never supported it. I feel like if they had got on board all those years ago there would be broad support across the software ecosystem for it and we wouldn't need half of these complicated iptables rules and proxies and service mesh. It would be a step towards a capability based system.
This is very interesting. I'm not sure what I'd use it for yet, but I imagine it could be useful for triggering ad hoc jobs over the network. Maybe have Home Assistant make a network call to kick off a daily back up when I leave the office at the end of a work day.
I believe its original motivation was just speeding up boot times by starting fewer services, even if you'd eventually want the service running. This was achieved in the past with xinetd, but systemd made the approach more popular for the masses.
inetd began to fall out of favour in the mid-late 90s as services became more heavyweight and startup times became longer (think of the initial crypto setup needed by sshd vs rsh/telnetd)
CPU speeds have increased & and i/o latency has decreased so much since then that startup times are generally imperceptible, so the pendulum has swung back to favouring socket activation.
The anti-systemd "traditionalists" never seem to acknowledge that history, though!
This is actually something that I like in systemd.
I am dealing with mostly non systemd system: BSD, Alpine, termux
On BSD anacron works well, but I do not why I am always running into problems with the cronie anacron implementation. And it is very hard to debug.
I would really like a simple modern cron/anacron alternative.
Cronicle looked cool but it is node.js, a bit heavy and being replace now by their new product called xyOps anyway.
In decades of trying, I do not believe there was one time that I ever got a cron job to work properly in the first attempt. Systemd timers are a godsend.
You will love SystemD [0] timers until they fuck you over in an entirely inscrutable way and the SystemD maintainers don't care to either fix the problem or update the docs to warn of the shortcoming.
One of our customers called in with a production down incident caused by a full disk. We got a copy of the VM and took a look. Investigation revealed that / was full because /var/log was full and that our 'logrotate' timer unit that was scheduled to run once a day had run either exactly never or exactly once... I can't remember which. Further investigation revealed no difference in software load or configuration between this VM and a VM that had a functional logrotate timer unit. Exactly one VM out of hundreds of identical VMs at this site (and many multiples of that at other customer's sites) were affected by this. Advising the customer to clear out /var/log and reboot did not unstick 'logrotate', and none of the diagnostics or fixes we could find anywhere unstuck it. Once "systemd-crond" decided to never schedule this job ever again, it stuck to that decision.
After a lot of searching, we found an open bug report from a year or three prior where someone reported exactly the same symptoms and was scheduling a unit with pretty much the same set of unit configuration flags that we were using. The conversation from the core devs ran through the pattern that one gets used to seeing when one runs into SystemD bugs that are caused by extremely complex unanticipated interactions between parts of the project: "That's not a bug, only an idiot would want that to work.", "Oh, we don't document that that's not supposed to work?", "Wow, okay, yeah, I can see how that maybe should work. That it doesn't sure does seem weird.", "Having said that, I don't know if it's supposed to work, or if it's unsupported. Someone should really either document that or fix it."... and then the behavior is neither fixed nor documented. [1] Absent any actual explanation for the failure, we ended up swizzling the options in our 'logrotate' unit and praying that satisfied whatever gremlin arose from the depths to trouble our customer.
SystemD contains an enormous -and ever-growing- amount of accidental complexity, and has a set of core maintainers who are generally disinterested in either documenting the places where one or more complex systems bind together to cause stop-the-world problems or fixing the systems involved so that they don't bind up. It's a fine project until it's very, very suddenly not, and then you're absolutely SOL. If you're lucky, you can shuffle around what you're doing [2] and hope that avoids the problem. [3]
[0] Some folks use the spelling "SystemD" to mock the project. I use the spelling "SystemD" to distinguish between "the entire systemd project" and systemd(1). I do this because some folks will make a claim like "systemd is very, very small and self-contained. I don't understand why anyone would say otherwise.", but what they are actually saying is that systemd(1) is a fairly small program that doesn't do all that much when run as PID 1. It sucks minor amounts of ass that the project and the program it runs as PID 1 share the same name, but what can you do?
[1] No, I don't have a link to the open bug report. This was more than a year ago, so the bug ID has been long forgotten.
[2] The term of art for this practice is "wave a dead chicken at it".
[3] Plus, like, even disregarding most of the rest of my report... how in the hell do you design a cron that knows a job is scheduled to be run periodically, can tell you how long it has been since it last ran, but never manages to run it? To me, that's unforgivable. It's a "You had one job!"-tier cockup.
I haven't used systemd timers enough to disagree, but
> Ambiguous $PATH settings make cron script execution difficult to predict.
What makes you say that? You can set the PATH right in the crontab. Is that harder to "predict" than it being set in /etc/bashrc, ~/.bashrc, ~/.profile, ~/.bash_profile, /etc/systemd/…, or wherever else?
> You might feel cool knowing the scheduling grammar by heart
I've used Linux since 1994 and I don't know it by heart. But luckily it's pre-printed in the crontab as comments:
You just put numbers aligned with the titles.The rest of the complaints, sure. Next time I need a cronjob, I'll try it out.
"You just put numbers aligned with the titles."
That is not a fair summarization of their point because that is not the grammar. There's commas, slashes, asterisks, combinations, and then if you want randomization you need to put it in the command itself because cron can't do it. (Some crons can, but it's not a general capability of cron.) Writing a non-trivial cron spec is not easy.
I am familiar with the syntax, so I am biased ("*/3" and "12,14,20" makes sense if you are familiar with Unix tools), but it is still more intuitive to me than the systemd unit file syntax and usage. I know that I just have to edit /etc/cron or throw any executable file into /etc/cron.d/monthly and it will work on my system, but I cannot write a systemd timer file from scratch without looking it, and to do that I first have to find the directory where the other examples are located. /etc/systemd doesn't appear to be it.
This is generally my only real complaint about systemd. I don't care if it is too monolitic, written in C or whatever, I just want a straightforward syntax for straightforward operations. I'd like it if systemd could recognize if a .target file is a shell script and just do "the right thing". Perhaps it would make sense for a timer file to recognize cron syntax as well. Or at least allow for a kind of extensibility so that I can have it supported.
If systemd had a little more respect for existing conventions, I am pretty sure it wouldn't be so controversial. After all, system administrators like it because they use it all the time, but a regular, full-timer user like me, who only deals with it when something is broken or have to use it as a means-to-an-end to set something up, then all friction is annoying and bad UX. (And no, using Nix is not the solution)
Yeah, it would be nice to have a folder like /etc/systemd-jobs/ where I could put them and where there are no files unrelated to job scheduling. There is /etc/systemd/user, but it does get a bit of pollution depending on the system.
How do you express those things in a systemd timer? E.g. run something 4x per day, */6 in cron.
Something like:
You can test it with: The format is `DayOfWeek Year-Month-Day Hour:Minute:Second`https://www.freedesktop.org/software/systemd/man/latest/syst...
I, er, what? It's... the same as cron? I'm confused now. It's not exactly the same, I guess?
Sure, I'll pile on here. To do nontrivial scheduling you'd use the entirely-obvious-and-intuitive syntax described at [0]. For example:
Who'd ever want to go back crontab format for nontrivial scheduling? [1][0] <https://www.freedesktop.org/software/systemd/man/latest/syst...>
[1] This question is sarcasm. SystemD is often like this... dead simple things look dead simple, but complex things are -if they're possible at all- at least as complex as they are everywhere else.
That's simple but consider "run something 4x per day but randomize a delay by hour so all of the 200 servers doing that task won't run it all at once"
In cron, you basically have to either use your configuration management to generate those times, or have a random delay script running before the command
In systemd timers, it's just
and the offset generated will be stable for the job on a given machine (i.e. always same on this machine but different on others) so you will get nice uniform distribution of load.If you add
the job will also be run once if there was one or more scheduled runs when the machine was down> In cron, you basically have to either use your configuration management to generate those times, or have a random delay script running before the command
Nope. From crontab(5)
That's from my cronie install, but it looks like this has been a feature of some crons for at least a decade. (Notice that the post date of [0] is in 2016.) Given that cronie is based on vixie-cron, and I think I was was using vixie-cron in 2002, I bet it's been a thing for at least twenty years.[0] <https://stackoverflow.com/a/34815984>
Having had to work on an application supposedly supporting cron expressions: the numbers are just the basic parts of the language.
When someone inputs something ridiculous like "5,3/4 4-8,11 1 4,5,6,9-11 */2" you get to enjoy the fun of reverse engineering what they meant (it's never what they actually wrote).
And that's before you get to all the extensions supported in some cron environments (but not all).
I find systemd timers a lot more manageable. Things like having control over whether or not long-running jobs are allowed to overlap and the ability to run tasks between start-finish rather than a fixed time window are major improvements for me. At some point my VPS went down because the backup job ran into some kind of symlink loop and cron just kept spawning more and more backup tasks even though none of them finished.
Having to re-write commands and scripts because CRON had its own special PATH was also a pain point, but the same can be true for some types of systemd timers. But: you can execute those timers manually if you want instead of updating the crontab to trigger in 30 seconds and simply waiting.
Complex expressions are one of the things I don't like in cron. On Debian/Ubuntu servers, I bite the bullet with systemd timers. On my workstation, I have a personal job scheduler that feels easier and more fun to tinker with. The scheduler uses Starlark functions instead. For example:
This was inspired by GNU mcron. In mcron, jobs can calculate the next time they should run using Guile (https://www.gnu.org/software/mcron/manual/mcron.html#Guile-S...): I found mcron's scheduling counterintuitive and decided I wanted a function that returned a boolean. I can recommend this approach so far.> 5,3/4 4-8,11 1 4,5,6,9-11 */2
What's so hard about "At 5 minutes past the hour and every 4 minutes, starting at 3 minutes past the hour, at 04:00 AM through 08:59 AM and 11:00 AM, on day 1 of the month, every 2 days of the week, only in April, May, June, and September through November"?
(I used https://crontab.cronhub.io/ to decode it, to be fair)
The main nice thing about the environment in systemd is that it is standard and mostly a blank slate, whereas at least for me I was always getting bit by the fact that the environment in Crontab was completely different from say, the environment inherited by supervisord or sysvinit scripts. In systemd the actual unit that gets executed is the same regardless of what triggers it, so there is no gap.
That does require you to still know what the default environment is, but it is a mostly completely clean environment, without any influence from any shell.
I'd have to concur that I agree this is an advantage of systemd.
> That does require you to still know what the default environment is, but it is a mostly completely clean environment, without any influence from any shell.
Odd. This script
when scheduled like so Produces this file in /tmp/set.txt which has had a handful of values (HOME, UID, etc) lightly redacted prior to posting here -to remove PII or for length- but its keys are entirely untouched: Seems pretty clean to me. Even when I run this via /etc/crontab, rather than as a user cron job: I get effectively the same results.Maybe your distro's default cron environment was bad, and you never bothered to check and unset the badness? I'd be surprised if they were unable to make the default environment for Timer Units to be bad.
I use cron in OpenBSD and it's a deterministic environment and mostly clean[0]. I like that instead of having other subsystems creep in.
[0]: https://man.openbsd.org/crontab.5#ENVIRONMENT
> I've used Linux since 1994
Same here.
We are now considered old and therefore irrelevant. The new generation uses timers and couldn't care less about cron that has served us just fine for decades.
I use cron and my general attitude towards LP and systemd is very similar to the attitude of LP and systemd to us.
> But luckily it's pre-printed in the crontab as comments
That's true, but most people don't know the numbered manual sections, so they get the docs for the cron table command not the cron table config file.
> That's true, but most people don't know the numbered manual sections, so they get the docs for the cron table command not the cron table config file.
No `man man`? ;)
problem with vars is that they apply to any subsequent entry in the file so you need to take that into consideration; the nice thing about timers is that all settings are self contained and not affected by previous entries. The standard /10 and similar cron expressions also have thundering herd problem when on bunch of servers, tho some variants like in Jenkins use variant H/10 (H standing for hash) where the thing is randomly shifted in time to not hit same minute on same server/job
another benefit is having logs in one place for the job; cron's "send a mail when there is any amount of output text" is just annoying behaviour, but also only place to get the job output unless you redirect it somewhere. Also starting from timer vs just doing systemctl start job.service is the same so easier to debug
other than that the few improvements in how to specify run time have been pretty useful.
For example, setting timer as "persistent" will mean any run "lost" to machine powered off will just be ran next time after boot, so you can have job on your PC that is just "run backup at 2AM" and if you turn it off before that you get the backup done first thing in the morning
There is also both random, and fixed (depending on machine UUID) random delay so avoiding thundering herd problem with backups is also pretty convenient.
There is even option to wake a device for the job if necessary tho the problem of shutdown is left to the user. And picking whether to start counting to next timer from previous one or from the job's end.
What I would like also is to have job summary page ("hey this job was done X times but failed Y times") but that's probably better left to external tooling
> You can set the PATH right in the crontab. Is that harder to "predict" than it being set in /etc/bashrc, ~/.bashrc, ~/.profile, ~/.bash_profile, /etc/systemd/…, or wherever else?
There is* a common trap as the cron PATH is usually just /usr/bin:/bin so anything in /usr/local/bin, or in /sbin won't be there.
I have a Canon printer, I actually can't trust that their print nozzle won't get jammed up after sitting idle for a while. So I had claude setup a systemd script to print a picture of my dog every week, I ensure it has enough CMYK spectrum to stress the printer. Its a nice surprise every monday as I sit on my desk to see a sudden picture pop up from the printer :)
I wish printers could have a mode like this to print random images from an album, or a calendar, rather than wastefully draining ink into a sponge every few days.
If nothing else, maybe it could be some kid's high school science fair project idea.
How about printing a QR code for a randomly generated private key for Satoshi Nakamoto's Bitcoin wallet, then every few days you get a tiny moment of excitement, hope, and then disappointment. It's still wasteful, but it could pay off big time?
this is an amazing youtube video idea if you could get a type writer to do it.
Maybe I'm misremembering, but I'm sure there was something on HN a few weeks ago about an electric typewriter that someone had connected to (I'm guessing) a Raspberry Pi? My search-fu is currently failing to find anything particularly recently, at the moment.
Or if you have a printer/scanner combo, you can turn it into a pen pal!
Dad had an Deskjet 720 or something like that.
It sat unused and powered off for a couple of years after he passed, until I needed a color print.
Didn't do anything but hook it up to power and print. Took about 1/5 of a page until all colors were back in action, after that it printed about 20 pages flawlessly.
Laser printers are your friend. The savings on consumables alone will make it pay for itself.
Epson Ecotank. I’ve been using mine for years and I only had to buy new ink once.
And I printed a lot of photos, notes, documents, etc
Those will still get their nozzles clogged if you don't run them.
This is the part where I get to point out that Brother inkjets do a little dance ~every day that keeps the heads fresh. They do this on their own as for long as they're powered up.
This allows them to work well even if years go by between prints. It's a very thoughtful design element.
(They don't survive sitting for months and months unpowered on a shelf very well, but... you'll have that.)
Not to mention more water resistant, when printing things like envelopes.
I was about to recommend a cheap OKI LED color printer (I think C322dn); alas they withdrew from consumer market :/ The colors are super nice and uniform, even if the maximum resolution is only 600 dpi - and the toner won't dry out, which was my brother's crucial purchase criterion; we had HP inkjet clogged more than once.
Moved from cronie to systemd timers because they are resilient to system startup times. My backup strategy is to create a borg archive entry every day at a fixed time. With cronie the system needs to be running at the scheduled time, but systemd timer tolerates this and runs the service as soons as the system is available.
Btw this is my repo for the backup automation: https://github.com/gchamon/borg-automated-backups
Cronie has a mechanism for this, called "anacron", which is called hourly by cron (on my system, /etc/cron.hourly/0anacron), and performs all the /etc/cron.{daily,weekly,monthly} tasks, no matter if the earliest possible schedule was missed (and with a configurable random delay). You can modify /etc/anacrontab to create custom schedules.
To do this at the user level, you can add something like "@hourly anacron -t /path/to/anacrontab -S /path/to/spooldir" to the user's crontab, though I've never tried this.
Many cron implementations have a similar mechanism.
EDITED
This isn't the same as with systemd timer because timer lets you specify when you want to run your service exactly and will fallback to running when the system comes online. With @hourly I lose this control and multiple machines could potentially trigger backups at the same time, hogging the physical hard drives and the network.
Cron also has @reboot. Not exactly the same, but has been sufficient for me so far.
https://news.ycombinator.com/item?id=48371021
> fallback to running when the system comes online.
That isn't something I'd want to happen, it sounds like it creates a potential queue of scripts that will flood the system on start, if it works the way you described.
I prefer the deterministic behavior of cron, the script will run when it is specified to run, as you said earlier, as long as the system is running; and as I stated in a separate comment, it will run @reboot if I need it to run then.
> With @hourly I lose this control and multiple machines could potentially trigger backups at the same time
Then don't use @hourly, use staggered times, it's very easy.
> That isn't something I'd want to happen, it sounds like it creates a potential queue of scripts that will flood the system on start, if it works the way you described.
This isn't what happens. If you leave it offline for days it'll only trigger the service only a single time.
I interpreted it more like "I have these 500 different cronjobs all spread out across $unit_of_time. If the system is down for longer than $unit_of_time and then comes back, does all 500 jobs start running instantly (since they missed their previous deadline)?"
Just to be clear, this isn't default systemd timer behaviour, you need to opt in by setting Persistent=true. If you have hundreds of jobs like this you need a proper queue and neither cronie nor systemd is the right tool because at that scale you'd surely need better observability
If you have 100 different jobs that were supposed to run over the past week, but didn't because offline, when you restart, they they all flood the system on start.
100 jobs all running at different times throughout the week is a very different load than them all falling back and running at the same time on system boot.
I don't, it's a single backup service.
> That isn't something I'd want to happen, it sounds like it creates a potential queue of scripts that will flood the system on start, if it works the way you described.
There are two options to fix it;
Disable persist so no catching up on missing scripts. Set OnBoot=5m so it gets ran 5 minutes after boot, so your script (say backup) is ran on boot first, then every time on schedule
Enable persist but just add sleep in ExecStartPre - very "cron" way but there is just no in-systemd option to enable "catch up" script to be delayed
Sadly no option to "run catch-up timers with delay" at least yet
> Then don't use @hourly, use staggered times, it's very easy.
Not in cron. In systemd it's just RandomizedOffsetSec=30m and it is "stable" - same host with same job will always have same delay so on multiple hosts it is spread nicely. There is also non-stable version
> With cronie the system needs to be running at the scheduled time, but systemd timer tolerates this and runs the service as soons as the system is available.
Cronie doesn't have a `@reboot` meta-trigger?
https://news.ycombinator.com/item?id=48371021
I'm sorry, I tried Googling the word "tolates" but I can't find any definition that makes sense?
> runs the service as soons as the system is available.
cron has the @reboot option which I use for a few scripts and works great.
Typo, I meant tolerates. Fixed it.
Not an option either, because if I reboot two machines and the backup starts in both of them it'll cripple my NAS
"tolerates".
I love systemd timers! I've slowly moved all of my ansible-deployed cron jobs to timers (now just an ansible copy!). The integration with journalctl, especially in a newer OS like Debian 13 where syslog is gone, is really nice. It's also really nice to be able to start the service manually for debug. Having a cron job that didn't work was an annoying exercise in copy/pasting or writing an extra shell script. Don't even get me started on the black hole of cron job stdout. I can monitor systemd services like I already do and get a notification on failure.
I've noticed more and more open source projects recommending timers as a deployment method and I think that's great!
> more and more open source projects recommending timers
I am perfectly happy with projects recommending timers as long as I can ignore them and use cron.
Timers can work with arbitrary units (not just a similarly-named service unit) so they can be surprisingly flexible. I have a timer on my servers that starts a backup.target that fires off a full "restic backup","restic prune", "restic forget" backup cycle each morning with randomized start times and notifications. The actual restic-* units are Podman Quadlets so the whole setup runs agnosticaly of what's on the server, just as long as it has Podman and Systemd installed.
I will admit thought, timers are up there in terms of being the clunkiest systemd unit type to use on a regular basis. I get why they're split up into two files and require different start vs enable syntax's, but man sometimes I just want to create a file that runs a script and be done with it.
I feel like systemd units could need a layer of abstraction above them, so instead of editing the files manually, a tool would do it, some kind of declarative CLI or something. Probably not really a concern in the age of LLMs anymore, but it feels just slightly too tedious every time.
Why do you randomize your backup times?
Should have been more clear: I use RandomizedOffsetSec= to add a random offset to a set start time (usually 4am), to prevent overloading the backup server, not truly random start times.
I will use what I am comfortable with and so should others. CronD, SystemD, atD, multiple conditional checks in a shell script, whatever tickles your fancy. There is no wrong answer, just document what you did and add a comment. Comments are permitted in cron. If someone keeps putting complex obfuscated time structures into cron make them decipher their incantation and keep nagging them until they keep it simple, comment their cron entries or until they and their manager resign.
For what it's worth there are usually web apps popping up that can decipher goofy cron time/date incantations. [1] This one has a git repo in the top right, not my repo. Maybe clone it just in case their site goes away some day.
[1] - https://crontab.cronhub.io/
I believe one of the major distro lines (redhat or debian, I forget which) uses systemd-cron, where cron is just a thin wrapper around systemd. You get more power from writing the unit files directly, but if all you ever need is a simple cron job, you have the old interface still available.
Yep, I use this for a @reboot job and a few regular jobs on my home server. I use user crontabs, so I can get around the "unknown shell/path/etc." by prefixing every job with
or sometimesNixOS comes with systemd, so I've been using it as a first-class part of managing stuff. It's great, especially coming from macOS' launchd.
Which makes it nice to distribute a tool for NixOS so that it can lean into systemd instead of as some bolted-on afterthought.
Makes me wonder what you'd do if you were distributing a lifecycle-heavy tool for Linux users in general since systemd isn't ubiquitous.
I use a systemd timer to run a monthly scrub for my btrfs pool. Kinda cool how you can do increasingly useful things like skip the next scheduled event if the user initiates a scrub, do or don't accumulate tasks if you have a monthly task but the machine was offline for 6 months -- or fold them into a single task, etc.
+1, NixOS makes working with systemd a breeze. Defining units in Nix beats wrangling INI files.
Have you been defining them directly in your flake.nix file? I too am on nixos but I keep all my configurations in their native format and symlink them with nix, that way I can take and reuse that config on a non nixos system easily.
The problem I have found is that nixos doesn't seem to pickup and run systemd timers and services placed into the ~/.config/systems/user folder and additionally things like WantedBy=default.target have no effect.
So after I restart all my services manually on reboot I agree, systems timers are cool.
I define all units in Nix because:
a) It is way nicer and you get decent validation at build time
b) A LLM can port units over if the need arises; it’s a very light abstraction around systemd syntax
c) I personally don’t see how I would ever move to another distro :)
I've been using Linux for over 20 years, systemd for over 10.
Yet there's always something new to learn and actually consider as another useful tool.
I'm using Linux for about 30 years and apparently we were all wrong using cron for decades.
I haven't always been the biggest fan of systemd in some regards, but I will say that I mostly agree with this sentiment. I've almost completely quit using cron, and now favor systemd timers for scheduled jobs - at the "system" level anyway. I might still embed Quartz for scheduling that's scoped to a particular application or something.
Why? It's one of those fuzzy and somewhat hard to explain things. The systemd approach just maps more cleanly to my mental model of "how things should work" I guess. And maybe some of it is that I did indeed experience plenty of " Ambiguous $PATH settings make cron script execution difficult to predict" in the past, although it's not just that.
I won't sit here and claim that systemd timers are necessarily better than cron in any universal / objective sense. But they've won me over, for what it's worth.
We have used cron perfectly fine for decades and it served us well within its very clear limitations.
But now obviously we were so blind and wrong all this time and the only true solution is of course systemd.
Thank Lennart you degenerate apostates are finally starting to see the light of His glorious creation. Hallowed be thy systemd-journald.
Has it actually served you well? Because it hasn't served me well at all.
I am not the biggest fan of systemd, but today I will always reach for a systemd timer over cron simply due to the sheer amount of bad experiences I've had with cron. Hours upon hours wasted trying to troubleshoot crons that weren't working due to some stupid obscure issue, having to use dirty hacks to monitor for success or retry failed jobs.
A few years ago I was trying to run a very simple bash script with cron and the script just died halfway through for no reason. Nothing in logs, worked fine when run directly, but in cron it just stopped halfway through a loop. Never figured out the cause, just gave up and used a timer instead, which worked fine. Never touched cron again after that.
The ease and convenience of monitoring and troubleshooting alone are worth switching over.
Let me state once again: "within its very clear limitations".
Once you learn that env in cron is not same as in your shell and once you learn to redirect output to loggers - it works just fine.
It would be a lie to say that I never debugged cron and sure it's annoying.
> and the script just died halfway through for no reason
Unrelated to cron. Bad script.
Systemd will reign supreme for a millennium if the answer to every question or complaint about non-systemd tools is "you're holding it wrong".
As a user I'm kinda whatever about the tools because the answer to my complaints about systemd is also "you're holding it wrong."
I'm sympathetic, but "bad script" is an awful assertion.
We are all guilty of making bad scripts, bash is a disgusting degenerate language (and I love it). The way we learn to write good scripts is by writing bad scripts in enough amounts to get bitten by all the warts.
One thing I really love about cron, is that if you set up mail on the server (which: you should btw), then cron actually sends emails if it sees anything in stdout and stderr.
I am a dyed in the wool systemd non-believer, but I really do like the timers.
I don't agree that these are just limitations. The fundamental problem cron tries to solve is very simple: I want to run a program automatically at specific times. There are probably many features of systemd timers that can be considered niche or extraneous in solving this problem, but the ability to easily know when the program last ran and what its exit code and stderr output were is not one of them. I believe that if an alleged solution to this problem doesn't provide at least this, it's not really solving the problem.
> Unrelated to cron. Bad script
Again, worked fine when run manually, worked fine in a systemd timer. Pretty sure I still have it running today and it continues to work fine without ever failing.
So basically it took you decades to learn all the bugs, UX issues and problematic quirks and now you're complaning someone built something better? :)
I've converted all my crons to systemd timers+services over the past year but cant help but think it's sort of.. less tangible than cron
Like imagine trying to explain systemd timers and services and unit files to a beginner.
> Like imagine trying to explain systemd timers and services and unit files to a beginner.
I think it's... easier? Like "systemd is the place where your system manages all the processes it needs to run. Part of those processes can be run on a schedule, or on a timer, and you define them using this simple text file".
cron is easier for easy stuff ("just run this every 10 minutes") but harder for hard stuff ("run it every 8 hours but with randomized offset so not all machines at once do it, but also if machine was down when it should run, run it immediately").
It is also easier to debug as every job gets its own log rather than trying to write to system mailer nobody had set up with the job errors
This is a very good intro to systemd timers -- I think you convinced me to finally start using them. Love the "list-timers" thing as well. With cron, it never seemed easy to me to get a picture of all the cron jobs running on a box. I'd need to check crontab for all users, as well as /etc/cron.d/, as well as the daily/hourly/monthly directories.
And in fact I do have a use-case for needing to run something ~5 minutes after the system boots and then every ~12 hours onward from there. It's great that systemd timers has me covered!
After years of using orchestration tools like airflow and dagster so many lightbulbs have just lit up in my head.
I wish documentation for tools would explain their abstractions concepts in terms of its primitives.
Great post, thanks!
systemd is complex on first view, but after using it you didn't want to use anything else. It's handy to manage everything using systemctl
I hard disagree. Previously I could use what I knew about Linux commands to read logs and administer the systemm. Systemctl knowledge does not generalize.
I have done scheme all my life, which is why I prefer shepherd. Not only is it in a syntax that i can use elsewhere, I get completion in Emacs.
That and systemd having actually useful man pages.
Some people see it differently. As in: after using it they don't want to use it. I hate systemd with passion.
But that's because I'm old because obviously systemd-* is the only right way and everyone else who see things differently is a pundit.
This is such a modern view. People used to HATE systemd when it first came out, but I always liked it and knew people would eventually come around and its nice to see they finally did!
Some people are stubborn and refuse to see how obviously superior systemd is to the old ways. Me included.
I setup a few systemd timers last year and created https://systemd.guru to play with different options in OnCalendar expressions.
> Prime Time for a Timer Primer
It's pronounced, "primmer."
Oh I love them quite a lot! I use them to run all of our backup jobs, easy to set up and have never had an issue.
And you immediately lose the ability to do `crontab -l` on any server to know its scheduled tasks.
Now you get to look around the myriad of places where you can put systemd files, and figure out which ones are base services and which ones are custom, with no general convention to go about it. Nope.
If you had read the article, you would have seen its answer to this.
systemd list-timers
With —-all
I've been almost convinced by systemd (and have switched to using it), but God the syntax of those service files is so ugly ...
Never thought I'd see hackers saying INI format looked ugly of all things. It's basic, sure, but that's a good thing for something meant to be easily editable by hand from any editor. Otherwise, it's just key value pairs in named sections, how ugly can it be about that?
key-value pairs where the = cannot be surrounded by spaces, so I have to write
which fills me with sadnessWhitespace immediately before or after the equals sign is completely ignored by the parser. Its the standard INI format.
What? You absolutely can have spaces; most of mine look more like
Friend, you have changed my life
Is this one of those cases where at one point you had an error in the file and you figured it was down to spaces? You fixed that issue, it still didn't work but from that point you never thought to question the assumption.
I find myself doing this sort of thing all the time..
Somewhere in my head I had that spaces caused a syntax error, and the UI for systemd is not obvious when you first start using it ... so if it's working then leave well alone. I'll be making all of my .service files (not so many) human-readable in the near future!
That was (cough still is) ddclient for me.
TOML would look a lot more quiet, but I'm not sure if TOML would be a good fit
unit files barely have any nesting, so the INI-like format is already 90% of the way towards TOML, no?
There's definitely some weirdness to certain parts of systemd service files, but was a huge improvement over Upstart and the old SysV-style init scripts.
Over all I think Systemd get way to much criticism. You don't have to use all the parts, but if you care to go through the documentation you'll find interesting features such as journald log-shipping and systemd-machined which can manage containers and VMs.
As a passionate systemd hater I would say I do not agree. Cron syntax is worse.
Oh yes, because the well documented clean syntax of sys v init shell scripts was so nice.
If I never recall hacking in ulimit calls in the top of buggy shell scripts for crappy old services that done respect pam_limits it won’t be soon enough.
Hard disagree. Compared to an init script, with all its boilerplate, I'd take a systemd unit file.
Could have been worse.
Could have been YAML.
Could have been XML.
XML would have the advantage of having a grammar so we could validate the config files.
It would also make it much simpler to make good GUI editors for the files instead of the Notepad approach most unix config files take.
The systemd dialect of INI is actually pretty well-defined though.
https://www.freedesktop.org/software/systemd/man/latest/syst...
Since systemd is successfully parsing its INI files, and barks at you when you put weird shit into them, a grammar for them does exist as well.
XML is that wonderful format that gave us vulnerabilities like death by million laughs, up to a certain moment, you could MitM DTDs, and a whole slew of everything-XML stuff back when XML was like AI is today, none of which I miss today.
Oh, and remember times when programmers would argue whether argument order in XML files should be significant or not?
But XML books with their idealized XML future description did give me the same warm fuzzies as some intricate clockwork mechanism to a Victorian geek.
There are good GUI editors for XML?
XML - I see you’ve used macOS’ LaunchD, the system that inspired Systemd
To be honest, I think either of those would have been better ...
/me cowers in fear
Could have been better.
Could have been XML Property Lists.
ducks
This is why I like NixOS. Defining systemd services in it is very neat.
Does systemd ship with something to upgrade your cron jobs for you? That would be the friendly way. Write your old school cron jobs, and then a script that converts them to do things the systemd way, documenting its steps, i.e. I created this file and this is why. Friendly "I help you do things better" rather than standoffish "your way is obsolete, you need to do it our way". Oh wait. I get it. LLM agents can do exactly that for you can't they. Another way I'm behind the curve.
I have knocked together a systemd service or three based on google copypasta. But generally, for cron jobs, why make it complicated? One line in /etc/crontab and done. I generally call an encapsulation script that sets the right environment variables, uses absolute paths, captures stdout/stderr if required and so on. I just want the simplest possible way to launch that script on a schedule.
That would assume I like systemd in the first place.
Is there a way yet to force-trigger a timer? There wasn't the last time I used them, which I found to be super annoying for testing them.
It's covered in the article, you can simply start the unit that would be started by the timer.
Oh but it won't appear in the timer-specific logs, I guess...
I'm fully ready to drink the "just let systemd do all the things" kool-aid, but I would love to see some sort of introductory/tutorial info into some of the things it can do other than services - i.e. containers and timers. I know man pages exist, but it would be nice if there was more scannable intro out there.
btw that xz hack only effect systemd distros
> humble systemd
That the same cannot be said of its maintainer is another matter.
The main person in charge of Linux itself isn't considered the most humble but they make amazing products.
Linus is pretty humble tbh, he just expects that people don't throw shit over the fence.
There's another big feature: You're not relying on the time zone to which the server was set (like with cron) but can explicitly specify a time zone:
https://www.freedesktop.org/software/systemd/man/latest/syst...
Of course you can do this trivially in cron as well. It is what the CRON_TZ variable is for.
This is a GNU extension so not portable.
> stdout and stderr output often ends up in a black hole
Ain't that the truth. Literally every crontab I've written for the last 10 years has had this in it:
2>&1 | logger -t cron-WHATEVER
...and that does a pretty good job of capturing anything that the script emits and making it easy to grep for in syslog the following morning.
But I'm still amazed at how many crontabs I run across that don't capture any output at all.
Even better is systemd socket activation.
I design all my services expecting to receive sockets this way. It makes sandboxing easy as the service itself doesn't need network access to have a listening socket.
It's a shame docker never supported it. I feel like if they had got on board all those years ago there would be broad support across the software ecosystem for it and we wouldn't need half of these complicated iptables rules and proxies and service mesh. It would be a step towards a capability based system.
This is very interesting. I'm not sure what I'd use it for yet, but I imagine it could be useful for triggering ad hoc jobs over the network. Maybe have Home Assistant make a network call to kick off a daily back up when I leave the office at the end of a work day.
I believe its original motivation was just speeding up boot times by starting fewer services, even if you'd eventually want the service running. This was achieved in the past with xinetd, but systemd made the approach more popular for the masses.
inetd began to fall out of favour in the mid-late 90s as services became more heavyweight and startup times became longer (think of the initial crypto setup needed by sshd vs rsh/telnetd)
CPU speeds have increased & and i/o latency has decreased so much since then that startup times are generally imperceptible, so the pendulum has swung back to favouring socket activation.
The anti-systemd "traditionalists" never seem to acknowledge that history, though!
This is actually something that I like in systemd.
I am dealing with mostly non systemd system: BSD, Alpine, termux On BSD anacron works well, but I do not why I am always running into problems with the cronie anacron implementation. And it is very hard to debug.
I would really like a simple modern cron/anacron alternative.
Cronicle looked cool but it is node.js, a bit heavy and being replace now by their new product called xyOps anyway.
In decades of trying, I do not believe there was one time that I ever got a cron job to work properly in the first attempt. Systemd timers are a godsend.
You will love SystemD [0] timers until they fuck you over in an entirely inscrutable way and the SystemD maintainers don't care to either fix the problem or update the docs to warn of the shortcoming.
One of our customers called in with a production down incident caused by a full disk. We got a copy of the VM and took a look. Investigation revealed that / was full because /var/log was full and that our 'logrotate' timer unit that was scheduled to run once a day had run either exactly never or exactly once... I can't remember which. Further investigation revealed no difference in software load or configuration between this VM and a VM that had a functional logrotate timer unit. Exactly one VM out of hundreds of identical VMs at this site (and many multiples of that at other customer's sites) were affected by this. Advising the customer to clear out /var/log and reboot did not unstick 'logrotate', and none of the diagnostics or fixes we could find anywhere unstuck it. Once "systemd-crond" decided to never schedule this job ever again, it stuck to that decision.
After a lot of searching, we found an open bug report from a year or three prior where someone reported exactly the same symptoms and was scheduling a unit with pretty much the same set of unit configuration flags that we were using. The conversation from the core devs ran through the pattern that one gets used to seeing when one runs into SystemD bugs that are caused by extremely complex unanticipated interactions between parts of the project: "That's not a bug, only an idiot would want that to work.", "Oh, we don't document that that's not supposed to work?", "Wow, okay, yeah, I can see how that maybe should work. That it doesn't sure does seem weird.", "Having said that, I don't know if it's supposed to work, or if it's unsupported. Someone should really either document that or fix it."... and then the behavior is neither fixed nor documented. [1] Absent any actual explanation for the failure, we ended up swizzling the options in our 'logrotate' unit and praying that satisfied whatever gremlin arose from the depths to trouble our customer.
SystemD contains an enormous -and ever-growing- amount of accidental complexity, and has a set of core maintainers who are generally disinterested in either documenting the places where one or more complex systems bind together to cause stop-the-world problems or fixing the systems involved so that they don't bind up. It's a fine project until it's very, very suddenly not, and then you're absolutely SOL. If you're lucky, you can shuffle around what you're doing [2] and hope that avoids the problem. [3]
[0] Some folks use the spelling "SystemD" to mock the project. I use the spelling "SystemD" to distinguish between "the entire systemd project" and systemd(1). I do this because some folks will make a claim like "systemd is very, very small and self-contained. I don't understand why anyone would say otherwise.", but what they are actually saying is that systemd(1) is a fairly small program that doesn't do all that much when run as PID 1. It sucks minor amounts of ass that the project and the program it runs as PID 1 share the same name, but what can you do?
[1] No, I don't have a link to the open bug report. This was more than a year ago, so the bug ID has been long forgotten.
[2] The term of art for this practice is "wave a dead chicken at it".
[3] Plus, like, even disregarding most of the rest of my report... how in the hell do you design a cron that knows a job is scheduled to be run periodically, can tell you how long it has been since it last ran, but never manages to run it? To me, that's unforgivable. It's a "You had one job!"-tier cockup.
systemd is great all around. Don’t listen to the boomers complaining about it because their cheese was loved.
(typo, i think). cheese was moved?