I keep toying with the idea of writing a cron that implements a Poisson process. Say I give it a parameter of 3600; our `pcron` would ensure the jobs occur randomly but average out to once per hour, making the timing of the next run independent of the last via the memoryless property of the exponential distribution.
The next sleep interval would be calculated probably as as t = -\lambda \ln(U) (where U is a uniform random variable). This way you ensure that the probability of the job firing in the next 10 seconds is the same whether the last job finished an hour ago or just five seconds ago. But \lambda remains the average amount of time between jobs.
It’s compelling to me because it solves thundering herd problems at the architectural level, and also because it simply seems like a lot of fun to have to code very defensively against such chaos. Switching back to a deterministic schedule after surviving such chaos probably leads to a much more robust system overall.
> making the timing of the next run independent of the last via the memoryless property of the exponential distribution
Nit: you’re not relying on the memoryless property here but just plain old independent sampling. You’re right that memorylessness means that the elapsed time since the last job provides no information on when the job fires next, but this is orthogonal to the independence of the sleep intervals.
Learned once the hard way that it makes sense to use "flock" to prevent overlapping executions of frequently running jobs. Server started to slow down, my monitoring jobs started piling, causing server to slow down even more.
Sometimes certain containerized processes need to run according to a schedule, but maintainers also need a way to run them manually without the scheduled processing running or starting concurrently. A shared FS seems like the ”simplest thing that could possibly work” distribution method for locks intended for that purpose, but unfortunately not all cloud storage volumes are strongly consistent, even to the same user, and may take several ms for the lock to take hold.
Wouldn't a database give you better consistency guarantees in that case? NFS locking semantics are a lot more complicated than just a `SELECT .. FOR UPDATE`
Sure, but that would require a separate database for this one use case. Mixing infra concerns into an app db doesn’t sound kosher, either, and a shared volume is already available.
Seems easier to have a managed lockfile for each process, diligently checking that the lock has actually been acquired. Performance is not a concern anyway, as long as acquire takes just a few ms we’re golden.
If a file system implements lock/unlock functions precisely to the spec, it should be fully consistent for the file/directory that is being locked. Does not matter if the file system is local or remote.
In other words, it's not the author's problem. It's the problem of a particular storage that may decide to throw the spec out of the window. But even in an eventually consistent file system, the manufacturer is better off ensuring that the locking semantics is fully consistent as per the spec.
Great post. And if you want some control support for your cronjobs perl App::Cronjob[1] can provide features such has exclusive locking, so a job won't run if the previous run is still going, or provide a timeout, and some options for sending mail on success or failure
Does anyone maintain a programmatically accessible list of holidays for their company? Similar to the HOLIDAYS.txt in the article, but it would allow for things like “don’t run this the day before or during a company holiday.”
I work at a company with different holidays in certain countries, which would complicate things, and require something more structured than a list of dates. But having that accessible could be useful.
Has anyone tackled that, or come across a solution?
Not sure what do you mean. I.e., what exactly is supposed to be the tricky part. Yes, I've dealt with quite a few processes like that, but I never tried, or would ever want to to put this into crontab. In fact, I don't know how the author intended his article, but if you consider doing that for production, I strongly advice you not.
What you do instead, is you schedule the cronjob for the most generic case, e.g. each day. And if it does not need to run 3 days before holidays with crescent moon when wind is blowing from the south, it is just the part of business logic of the process, which you write in the any proper programming language that you prefer (or that the system is written in anyway).
Now, how do you manage the list itself depends on the details and I've done all sorts dirty things that one probably shouldn't do (cutting corners), but in the most flexible case it is just some CRUD-type page in your back-office system, with a real UI, and there is a person (usually in the bookkeeping department of the company) who has it among his responsibilities to maintain the schedule. You store it in some proper SQL database and cache it aggressively, so the the myriads of cronjobs don't bother it more than necessary.
I think this could be solved quite similar to the OP and better done with systemd. Spitballing, but I think the best thing to do would be to write the timer in a standard file but have the activation time be written in an override file. That way you can ensure you are just editing that file with your scraper (should be able to hit the API if it is something like a google calendar or outlook).
I think the systemd timer would give you the benefit here as you can write the time in varying formats. Timezones, UTC, local, or whatever. That should give you the structure you need, if I'm understanding your problem correctly.
While systemd has more boilerplate than cron I think it has a lot of advantages that make it worth it. Best to just have a skeleton of these jobs (I keep some in my dotfiles) and then you have it. Or have the LLM write it (ironically one of the few instances I'll advocate for letting the AI write the code). You can do everything in the article and so much more.
One just did it with code where all the processes had holiday.json which would be checked at each launch, if it was holiday, it would do no work and exit.
Other one is operator that would monitor if it was supposed to be a holiday and either change systemd or Kubernetes to suspend the jobs.
I'd recommend code over messing with the system, much more flexible.
I think it exists to power this easter egg?: https://github.com/alphagov/calendars/issues/678 The value I guess depends on whether its appropriate to put up decorations ie a 'celebratory' holiday. It looks like the only non-bunting holidays on there right now are Good Friday (which is more solemn) and Orangemens' Day (which is pretty sectarian), but apparently in the past the Queen's funeral was another non-bunting holiday: https://news.ycombinator.com/item?id=37789437
Ruby has https://github.com/bokmann/business_time but when I looked at it, custom code was needed to calculate holidays that were offset because they are on the weekend.
I had something come up recently that I think sounds similar. That project needs several time-sensitive jobs. When any one of them runs, the first thing it does is check a holidays.json file.
It parses the file using jq and compares its entries with the current time according to GNU date. At the root is the names of the jobs. Each job has its own list of holidays. Each of these holiday items in the job's respective list has keys for the display name of the holiday, the formatted date to compare to, and in a few cases the ISO day-of-week and a string containing a modulo arithmetic function (e.g. don't run the friday before Christmas, etc.).
Sorry, yes that means I call eval on that string and yes that means some of these are repeated in the same file under the arrays for the other jobs. Also, such lists will have to be maintained and the exact observed dates cannot always be known ahead of time beyond about a year since people can change their minds for various reasons (think bank holidays). Depending on your use case you may also want to define a start time and end time for a window of when this should or shouldn't run (i.e. business hours).
I don't know if that helps. I know it's hacky, but I don't think there's a nice way to handle things like "second monday after 4th of july, but if the 4th also happens to be monday then it should instead be the second tuesday". God help you if you also need to handle each holiday being observed in different timezones. At least at the end of the day none of this would be much code, just very terse code dense with meaning.
I would definitely recommend not putting complex logic like this in your cron definitions. Much more annoying to find and debug in the future. I prefer to write a short wrapper script that contains the test logic instead and track/version control it
This is great! I'm sure like a lot of programmers, I had been fulfilling the requirement for similar conditional logic by having a simple recurring cron job run other code or database queries with the conditional logic that this post demonstrates can be done directly in cron.
Isn't everyone here on systems with Systemd? Why isn't everyone using Systemd timers instead? A number of people mention locking, and AFAIK that's not an issue with timers.
I keep toying with the idea of writing a cron that implements a Poisson process. Say I give it a parameter of 3600; our `pcron` would ensure the jobs occur randomly but average out to once per hour, making the timing of the next run independent of the last via the memoryless property of the exponential distribution.
The next sleep interval would be calculated probably as as t = -\lambda \ln(U) (where U is a uniform random variable). This way you ensure that the probability of the job firing in the next 10 seconds is the same whether the last job finished an hour ago or just five seconds ago. But \lambda remains the average amount of time between jobs.
It’s compelling to me because it solves thundering herd problems at the architectural level, and also because it simply seems like a lot of fun to have to code very defensively against such chaos. Switching back to a deterministic schedule after surviving such chaos probably leads to a much more robust system overall.
> making the timing of the next run independent of the last via the memoryless property of the exponential distribution
Nit: you’re not relying on the memoryless property here but just plain old independent sampling. You’re right that memorylessness means that the elapsed time since the last job provides no information on when the job fires next, but this is orthogonal to the independence of the sleep intervals.
This sounds like a really fun idea.
Learned once the hard way that it makes sense to use "flock" to prevent overlapping executions of frequently running jobs. Server started to slow down, my monitoring jobs started piling, causing server to slow down even more.
We can also use systemd timer and it ensures there is no overlap.
https://avilpage.com/2024/08/guide-systemd-timer-cronjob.htm...
Have you tested how this behaves on eventually consistent cloud storage?
I'm confused, is EBS eventually consistent? I assume that it's strongly consistent as otherwise a lot of other linux things would break
If you're thinking about using NFS, why would you want to distribute your locks across other machines?
Why would anyone want a distributed lock?
Sometimes certain containerized processes need to run according to a schedule, but maintainers also need a way to run them manually without the scheduled processing running or starting concurrently. A shared FS seems like the ”simplest thing that could possibly work” distribution method for locks intended for that purpose, but unfortunately not all cloud storage volumes are strongly consistent, even to the same user, and may take several ms for the lock to take hold.
Wouldn't a database give you better consistency guarantees in that case? NFS locking semantics are a lot more complicated than just a `SELECT .. FOR UPDATE`
Sure, but that would require a separate database for this one use case. Mixing infra concerns into an app db doesn’t sound kosher, either, and a shared volume is already available.
Seems easier to have a managed lockfile for each process, diligently checking that the lock has actually been acquired. Performance is not a concern anyway, as long as acquire takes just a few ms we’re golden.
FWIW, it’s not NFS.
If a file system implements lock/unlock functions precisely to the spec, it should be fully consistent for the file/directory that is being locked. Does not matter if the file system is local or remote.
In other words, it's not the author's problem. It's the problem of a particular storage that may decide to throw the spec out of the window. But even in an eventually consistent file system, the manufacturer is better off ensuring that the locking semantics is fully consistent as per the spec.
Great post. And if you want some control support for your cronjobs perl App::Cronjob[1] can provide features such has exclusive locking, so a job won't run if the previous run is still going, or provide a timeout, and some options for sending mail on success or failure
[1]https://metacpan.org/pod/App::Cronjob https://metacpan.org/dist/App-Cronjob/view/bin/cronjob
Does anyone maintain a programmatically accessible list of holidays for their company? Similar to the HOLIDAYS.txt in the article, but it would allow for things like “don’t run this the day before or during a company holiday.”
I work at a company with different holidays in certain countries, which would complicate things, and require something more structured than a list of dates. But having that accessible could be useful.
Has anyone tackled that, or come across a solution?
At our company we have enough systems reliant on holiday dates that we have a Holiday system that emits events when there are changes.
This happens surprisingly often, given that religious dates change and there are holidays/closures for storms in some regions.
Not sure what do you mean. I.e., what exactly is supposed to be the tricky part. Yes, I've dealt with quite a few processes like that, but I never tried, or would ever want to to put this into crontab. In fact, I don't know how the author intended his article, but if you consider doing that for production, I strongly advice you not.
What you do instead, is you schedule the cronjob for the most generic case, e.g. each day. And if it does not need to run 3 days before holidays with crescent moon when wind is blowing from the south, it is just the part of business logic of the process, which you write in the any proper programming language that you prefer (or that the system is written in anyway).
Now, how do you manage the list itself depends on the details and I've done all sorts dirty things that one probably shouldn't do (cutting corners), but in the most flexible case it is just some CRUD-type page in your back-office system, with a real UI, and there is a person (usually in the bookkeeping department of the company) who has it among his responsibilities to maintain the schedule. You store it in some proper SQL database and cache it aggressively, so the the myriads of cronjobs don't bother it more than necessary.
I think this could be solved quite similar to the OP and better done with systemd. Spitballing, but I think the best thing to do would be to write the timer in a standard file but have the activation time be written in an override file. That way you can ensure you are just editing that file with your scraper (should be able to hit the API if it is something like a google calendar or outlook).
I think the systemd timer would give you the benefit here as you can write the time in varying formats. Timezones, UTC, local, or whatever. That should give you the structure you need, if I'm understanding your problem correctly.
While systemd has more boilerplate than cron I think it has a lot of advantages that make it worth it. Best to just have a skeleton of these jobs (I keep some in my dotfiles) and then you have it. Or have the LLM write it (ironically one of the few instances I'll advocate for letting the AI write the code). You can do everything in the article and so much more.
https://man.archlinux.org/man/systemd.time.7
I've been at several companies that have tried.
One just did it with code where all the processes had holiday.json which would be checked at each launch, if it was holiday, it would do no work and exit.
Other one is operator that would monitor if it was supposed to be a holiday and either change systemd or Kubernetes to suspend the jobs.
I'd recommend code over messing with the system, much more flexible.
The GOV.UK website has a list of UK bank holidays on it. If you add a .json to the end of the URL you get this, a thing of beauty.
https://www.gov.uk/bank-holidays.json
Neat, what does bunting mean in this context?
I think it exists to power this easter egg?: https://github.com/alphagov/calendars/issues/678 The value I guess depends on whether its appropriate to put up decorations ie a 'celebratory' holiday. It looks like the only non-bunting holidays on there right now are Good Friday (which is more solemn) and Orangemens' Day (which is pretty sectarian), but apparently in the past the Queen's funeral was another non-bunting holiday: https://news.ycombinator.com/item?id=37789437
Ruby has https://github.com/bokmann/business_time but when I looked at it, custom code was needed to calculate holidays that were offset because they are on the weekend.
I had something come up recently that I think sounds similar. That project needs several time-sensitive jobs. When any one of them runs, the first thing it does is check a holidays.json file.
It parses the file using jq and compares its entries with the current time according to GNU date. At the root is the names of the jobs. Each job has its own list of holidays. Each of these holiday items in the job's respective list has keys for the display name of the holiday, the formatted date to compare to, and in a few cases the ISO day-of-week and a string containing a modulo arithmetic function (e.g. don't run the friday before Christmas, etc.).
Sorry, yes that means I call eval on that string and yes that means some of these are repeated in the same file under the arrays for the other jobs. Also, such lists will have to be maintained and the exact observed dates cannot always be known ahead of time beyond about a year since people can change their minds for various reasons (think bank holidays). Depending on your use case you may also want to define a start time and end time for a window of when this should or shouldn't run (i.e. business hours).
I don't know if that helps. I know it's hacky, but I don't think there's a nice way to handle things like "second monday after 4th of july, but if the 4th also happens to be monday then it should instead be the second tuesday". God help you if you also need to handle each holiday being observed in different timezones. At least at the end of the day none of this would be much code, just very terse code dense with meaning.
I would definitely recommend not putting complex logic like this in your cron definitions. Much more annoying to find and debug in the future. I prefer to write a short wrapper script that contains the test logic instead and track/version control it
Good advice. You can also check in and version your crontabs (or timer units or whatnot) directly.
Embedding a test like that is something I've never considered - very cool.
These days I tend to use systemd timers on Linux though. Despite my love/hate relationship with systemd, timers and service files are really nice.
Also check out the 'chronic' command from moreutils. No more dev nulls.
This is great! I'm sure like a lot of programmers, I had been fulfilling the requirement for similar conditional logic by having a simple recurring cron job run other code or database queries with the conditional logic that this post demonstrates can be done directly in cron.
Isn't everyone here on systems with Systemd? Why isn't everyone using Systemd timers instead? A number of people mention locking, and AFAIK that's not an issue with timers.
Cool. Had no idea you could run commands inside a CRON expression.
Running a command is the main idea of cron. In this case, the author runs composite commands like:
Where 'test' is another shell command that returns 0 or 1. This is not a special cron syntax, it's just the inherent capability of the Unix shell.In any case, this whole approach is very clever and shows the beauty of The Unix Way.
I learned something cool about cron filtering and a nice api I didn't know existed - date.nager.at