Ok so if we take this repo and then the headline from https://news.ycombinator.com/item?id=41085376, we could get an infinitely fast file system by just putting another SQLite filesystem on top.
I think I interviewed with that team (or tried to at least) - it had great concepts, but the limited leadership I met on that one seemed unlikely to actually get to product (I say with 20 years retrospection) - they were stuck on sql when it needed a directory, but the directory couldn't scale to handle filesystems. Both have since been fixed, but I think the idea is well dead in MS
I'm not a big fan of Windows, but I'm a bit sad that the project didn't go anywhere forever and then was cancelled. They had some pretty ambitious and interesting ideas for it.
E.g. the only public beta of Vista that had it enabled used the database as a backing store for Outlook contacts. Each contact was an object in the database. So were photos. And you could link - as I recall, I only read about it back then - people to photos. And then search on that data, of course. Like, "give me all emails, documents, and photos that have to do with person X, from 2022-2023".
But IIRC, the performance was atrocious. They might have been able to fix that, but instead it ended on the junk heap of unrealized ideas.
I remember those Longhorn betas - yes, it was coming apart under feature creep, but that only gave us more to play with and anticipate in the leaks.
A shame that MS is no longer in the mindset of ambition like this. Instead we get adverts and clickbait in the Start Menu, and the worst filesystem search on any OS I've experienced.
I feel like I need to inform you that the people doing the “recommendations” in the Windows start menu are not the same people who write filesystems.
The venn diagram of those two sets of people is two non-touching, non-intersecting circles.
There are interesting things going on, I’m sure, but we won’t see any of it until this AI baloney is behind us.
Nerds create a virtual moron that gets a large percentage of stuff wrong and execs demand that it be inserted into every product made immediately. An absolute genius move only an MBA can explain.
Sure, but that indicates institutional rot. If the people who write NTFS drivers (i.e. its devs who love Windows the most) can't push back on user hostility and short-sighted decisions are made constantly then we have a company that has drifted far from its strengths. It is just coasting on brand recognition. This is the kind of trend that is seldom reversed.
Why would you bother with this instead of just making an in memory drive? I remember mounting in memory drives even in MS-DOS back in the early 90s. I did it to move whatever game I wanted to play into the RAM Drive so my load times would be faster.
It turns out if you run SQLiteFS on SQLiteFS, performance increases exponentially, and as the number of SQLiteFS layers increases, seek and read times asymptotically approach zero. In other words, this is the world’s first O(0) file system.
At this rate, with only a few performance tweaks, SQLiteFS should be able to retrieve files before you even ask for them.
The problem is that it sends you files before you need them, which clogs up throughput as you try to figure out what the files are supposed to be for. It's more efficient to slow everything back down to NVMe speeds.
Jokes aside, if I understand correctly it's still assumed the performance increase from the sqlLite is down to the reduced number of OPEN and CLOSE calls. So if that's the case, then it will be as slow as however many sqllites you're running? Meaning it should in theory lose performance?!
They didn't specify what kind of update, but in this case no, I'm on zfs-2.2.3, which was released in February. I'd have to go check out when it was marked stable by Gentoo and I actually updated. (I do occasionally have to rebuild whatever version I'm on when upgrading kernel versions.)
For the bored, I was searching for 'database filesystem' and found one by Oracle. I gather it's a way to make editing BLOBs easier. That could make for some interesting recursive things.
It's "a file system interface placed on top of files and directories that are stored in database tables."[0]
Ok so if we take this repo and then the headline from https://news.ycombinator.com/item?id=41085376, we could get an infinitely fast file system by just putting another SQLite filesystem on top.
IMO in this fantasy it should go like (1-.35)^n, not 1-.35n. We must keep our harebrained schemes plausible.
There's a recursive CTE joke in here somewhere
I was just thinking the same thing.. Im supposed to be on holidays but damn it I need to test this.
Reminds me of Microsoft WinFS.
Which was essentially SQL Server as a filesystem.
Project was ultimately cancelled.
https://en.m.wikipedia.org/wiki/WinFS
I think I interviewed with that team (or tried to at least) - it had great concepts, but the limited leadership I met on that one seemed unlikely to actually get to product (I say with 20 years retrospection) - they were stuck on sql when it needed a directory, but the directory couldn't scale to handle filesystems. Both have since been fixed, but I think the idea is well dead in MS
I'm not a big fan of Windows, but I'm a bit sad that the project didn't go anywhere forever and then was cancelled. They had some pretty ambitious and interesting ideas for it.
E.g. the only public beta of Vista that had it enabled used the database as a backing store for Outlook contacts. Each contact was an object in the database. So were photos. And you could link - as I recall, I only read about it back then - people to photos. And then search on that data, of course. Like, "give me all emails, documents, and photos that have to do with person X, from 2022-2023".
But IIRC, the performance was atrocious. They might have been able to fix that, but instead it ended on the junk heap of unrealized ideas.
I remember those Longhorn betas - yes, it was coming apart under feature creep, but that only gave us more to play with and anticipate in the leaks.
A shame that MS is no longer in the mindset of ambition like this. Instead we get adverts and clickbait in the Start Menu, and the worst filesystem search on any OS I've experienced.
I feel like I need to inform you that the people doing the “recommendations” in the Windows start menu are not the same people who write filesystems.
The venn diagram of those two sets of people is two non-touching, non-intersecting circles.
There are interesting things going on, I’m sure, but we won’t see any of it until this AI baloney is behind us.
Nerds create a virtual moron that gets a large percentage of stuff wrong and execs demand that it be inserted into every product made immediately. An absolute genius move only an MBA can explain.
Sure, but that indicates institutional rot. If the people who write NTFS drivers (i.e. its devs who love Windows the most) can't push back on user hostility and short-sighted decisions are made constantly then we have a company that has drifted far from its strengths. It is just coasting on brand recognition. This is the kind of trend that is seldom reversed.
Why would you bother with this instead of just making an in memory drive? I remember mounting in memory drives even in MS-DOS back in the early 90s. I did it to move whatever game I wanted to play into the RAM Drive so my load times would be faster.
I use this today on my Raspberry Pis to load the logs and other active IO to memory and avoid wearing out the SD card.
Ok but why not just use a memory drive directly? What is gained by adding SQLite as a layer?
I am commenting on your "I remember mounting in memory drives even in MS-DOS back in the early 90s.". Just to say this this is still a thing today.
I also mounted drives in memory in early 90's :)
[dead]
Videogames have gotten so ridiculously huge nowadays, we’ll be at the prosumer/workstation cusp to make this work nowadays.
If SQLite is faster than a normal file system, how fast is SQLite running on “SQLite as a filesystem”?
It turns out if you run SQLiteFS on SQLiteFS, performance increases exponentially, and as the number of SQLiteFS layers increases, seek and read times asymptotically approach zero. In other words, this is the world’s first O(0) file system.
At this rate, with only a few performance tweaks, SQLiteFS should be able to retrieve files before you even ask for them.
The problem is that it sends you files before you need them, which clogs up throughput as you try to figure out what the files are supposed to be for. It's more efficient to slow everything back down to NVMe speeds.
Less humorously, if one were calculating a few files for compilation, or caching for a service, a SqliteFS using :memory: might be just the thing.
Jokes aside, if I understand correctly it's still assumed the performance increase from the sqlLite is down to the reduced number of OPEN and CLOSE calls. So if that's the case, then it will be as slow as however many sqllites you're running? Meaning it should in theory lose performance?!
Obviously adding redundant layers will reduce performance.
Will anyone mention the repo hasn't been updated on 4 years?
When have you last updated your filesystem?
A fair analogy would be: "when was upstream code of the filesystem was last updated?".
To that question, I doubt one would find a polular filesystem that was updated longer than 1 year ago (maybe I'm too liberal).
That's the on-disk format, you probably updated the actual driver code a lot more recently than that
They didn't specify what kind of update, but in this case no, I'm on zfs-2.2.3, which was released in February. I'd have to go check out when it was marked stable by Gentoo and I actually updated. (I do occasionally have to rebuild whatever version I'm on when upgrading kernel versions.)
What was in that update that you are worried about this missing?
Check it out: https://github.com/openzfs/zfs/releases
yesterday, zfs.
It works (on Linux)! Should add that I only got to do what it does after following the advice expressed in the error message:
It did not work using `sudo`For the bored, I was searching for 'database filesystem' and found one by Oracle. I gather it's a way to make editing BLOBs easier. That could make for some interesting recursive things.
It's "a file system interface placed on top of files and directories that are stored in database tables."[0]
[0] https://docs.oracle.com/en/database/oracle/oracle-database/1...
Related: SQLiteDAV - WebDAV server that maps an SQLite database to directories/files.
https://github.com/Airsequel/SQLiteDAV
I wrote this one, I tested with PostgresDB and Oracle , at time:
https://github.com/gbonacini/dbfs
Filesystems are DB like structures these days anyway.
Are they? From my experience from databases and some limited from filesystems I would disagree. Sure, there are some similarities but not that much.
> Filesystems are DB like structures these days anyway.
I think you're confusing the fact that an application uses data structures with that application being like a database.
Other way around. Applications use databases instead of the filesystem.
Where do you get that from? In my practice this was never true and still is not.