Nice writeup. I work on native macOS utilities and have dealt with file monitoring quite a bit. kqueue is reliable but the per-file descriptor requirement can get tricky when you need to watch large directory trees.
For anyone looking at this space, FSEvents is the higher-level alternative Apple provides. It watches directory-level changes without needing a file descriptor per file, which scales better for broad monitoring. But kqueue gives you more granular control, like detecting attribute changes or renames specifically, which FSEvents sometimes lumps together.
In practice I've found a combination works well: FSEvents for broad directory watching to catch that something changed, then kqueue for targeted monitoring of specific files you care about.
That's the exciting thing about macOS development, right? There's always some funny riddles because Apple doesn't care to document stuff for 3rd parties (or at all).
There are benchmarks in there, and comparisons with all the other watchers out there I could find at the time I made it (see "Comparison with Similar Projects" at the very bottom of the readme).
It's a pretty long list. Tons of watchers out there with different design philosophies and shapes of problems they solve.
There are a lot of caveats to the filesystem monitoring APIs provided by kernels. Some projects (like facebook's watchman) take that as a kind of antagonism, and decide to fight back with layers and layers of fallbacks and distrust and rescans. That projects basically only makes sense as a daemon.
Other programs and libraries try to take that complexity and tame it by being super-focused on one platform or providing a lot of configuration options.
Some provide debouncing logic. This particular feature comes up from time to time, I believe, both for practical reasons and because over-reported events from the kernel subsystems, especially for some arcane events like moving a file across mount points can trigger a flurry of hard-to-associate events for the same path.
If you want to avoid dealing with under-documented filesystem event subsystems, you can also just make your own with ebpf. Especially for security-oriented systems, you'll find that the only (nearly) perfectly accurate filesystem event subsystem you can make, is the one you make from the ground up.
For a while I've been annoyed that esbuild, which is written in Go, eschews these APIs to detect changes in watch mode and instead continually polls the filesystem: https://github.com/evanw/esbuild/issues/1527#issuecomment-90.... It actually consumes quite a bit of battery, so I might fork it and apply this post's implementation!
This looks interesting and I might give it a try. I did look at FSEvents in the past but due to my needs in macOS app, I took a very niche (and not so fast) approach of running find | ls -lT | shasum spawned in a separate process to reliably tell if a folder has changed.
This style/format is really easy to follow along with, great blog writing. Separately I’ve been messing with some filesystem stuff so it’s interesting to me.
Love it. Are there any benchmarks available on how fast this is from file change, kernel event received, event dispatched? I would expect it to be really quick.
Nice writeup. I work on native macOS utilities and have dealt with file monitoring quite a bit. kqueue is reliable but the per-file descriptor requirement can get tricky when you need to watch large directory trees.
For anyone looking at this space, FSEvents is the higher-level alternative Apple provides. It watches directory-level changes without needing a file descriptor per file, which scales better for broad monitoring. But kqueue gives you more granular control, like detecting attribute changes or renames specifically, which FSEvents sometimes lumps together.
In practice I've found a combination works well: FSEvents for broad directory watching to catch that something changed, then kqueue for targeted monitoring of specific files you care about.
NSFilePresenter is an even higher level API and the one I typically recommend to people since it’s very easy to use
[flagged]
Can you treat kqueue as infallible? I've found FSEvents sometimes drops events at high volume (unless I'm misunderstanding how to use it).
That's the exciting thing about macOS development, right? There's always some funny riddles because Apple doesn't care to document stuff for 3rd parties (or at all).
Since there are comments below asking about benchmarks, others noting or expressing interest in other projects, I'll drop the one I made: https://github.com/e-dant/watcher (and also an ebpf experiment https://github.com/e-dant/bpf-fs-events)
There are benchmarks in there, and comparisons with all the other watchers out there I could find at the time I made it (see "Comparison with Similar Projects" at the very bottom of the readme).
It's a pretty long list. Tons of watchers out there with different design philosophies and shapes of problems they solve.
There are a lot of caveats to the filesystem monitoring APIs provided by kernels. Some projects (like facebook's watchman) take that as a kind of antagonism, and decide to fight back with layers and layers of fallbacks and distrust and rescans. That projects basically only makes sense as a daemon.
Other programs and libraries try to take that complexity and tame it by being super-focused on one platform or providing a lot of configuration options.
Some provide debouncing logic. This particular feature comes up from time to time, I believe, both for practical reasons and because over-reported events from the kernel subsystems, especially for some arcane events like moving a file across mount points can trigger a flurry of hard-to-associate events for the same path.
If you want to avoid dealing with under-documented filesystem event subsystems, you can also just make your own with ebpf. Especially for security-oriented systems, you'll find that the only (nearly) perfectly accurate filesystem event subsystem you can make, is the one you make from the ground up.
I love small personal projects like this, especially their writeups. I always have a hard time learning unless I am actually using it for something.
For more software in this domain see also the excellent entr https://eradman.com/entrproject/
kqueue is quite portable and works across all the BSDs.
The OpenBSD documentation for it is top notch, as usual. No idea about the rest (I suspect they’ve all converged at this point).
Kqueue originated in FreeBSD, and like most all of FreeBSD, it is very well documented.
Very nice :)
For a while I've been annoyed that esbuild, which is written in Go, eschews these APIs to detect changes in watch mode and instead continually polls the filesystem: https://github.com/evanw/esbuild/issues/1527#issuecomment-90.... It actually consumes quite a bit of battery, so I might fork it and apply this post's implementation!
This looks interesting and I might give it a try. I did look at FSEvents in the past but due to my needs in macOS app, I took a very niche (and not so fast) approach of running find | ls -lT | shasum spawned in a separate process to reliably tell if a folder has changed.
I personally like and use watchexec [1] to run commands on repeat.
Syntax example: `watchexec -r -e py uv run pytest -xvvs file.py`
[1]: https://github.com/watchexec/watchexec
This style/format is really easy to follow along with, great blog writing. Separately I’ve been messing with some filesystem stuff so it’s interesting to me.
Love it. Are there any benchmarks available on how fast this is from file change, kernel event received, event dispatched? I would expect it to be really quick.
In Linux, this is done with inotify.
Path units in systemd present inotify, but their use is somewhat constrained.
The incron utility is more flexible.
Would be neat if POSIX was reasonably kept up to date. Just saying.
[dead]