Do people have good experiences with LMDB, in terms of reliability? I've never used it in production, but I've read through the code and design documents for a database implementation class.
I remember some strange code (such as pushing return values 4k above the stack, with a comment like "this works as long as the caller doesn't use more than 4k of stack space before accessing the return value"), and the author also shared some unconventional opinions about undefined behavior (like "Compilers are deterministic, if I know what platform I'm compiling to then no behavior is undefined. And if compiler authors disagree, they are morons.")
But presumably it's thoroughly tested, so those aren't problems in practice? Would be really interested to hear from people who've actually used it. I've mainly stuck to SQLite instead.
Not amazing. In certain workloads I ran, once the db reached several hundred gb, writes would hang for longer and longer periods of time, eventually hours, while the db grew drastically in the background. https://news.ycombinator.com/item?id=30023623 seems to be the same issue, and it was serious enough that Shopify decided not to use lmdb.
And yes, I ensured there were no outstanding long lived readers, verified with mdb_stat -r. My workload used one transaction per read/write anyway (never needed larger atomicity). Once the db got into the bad state, running my program on it would almost immediately run into the issue again, so I really think the db is in a bad state such that most writes would cause it to hang, not related to how I do transactions. This workload would pretty consistently hit the issue once the db got to several hundred gb.
Issue #10236 on the OpenLDAP bug tracker might be the root cause, who knows. It's been marked CONFIRMED for years without a fix, while other similar issues are created.
This is extremely annoying. It seems workload dependent (other workloads I've run create absolutely massive lmdb dbs without this issue) and once it happens your only recourse is to make a new db and copy the contents over (thankfully reads still work fine on these borked dbs).
Other than that, though, it's great. Never in any case had actual data corruption, and reads and writes are extremely fast (until this issue happens)
Edit: fun fact, since shopify may have created Bolt in response to this bug, and then Bolt was the root cause of the 73-hour Roblox downtime in 2021, this bug may indirectly have caused one of the worst outages ever!
I've used LMDB in production for multi-terabyte databases, and we encountered the long-write time but found a solution.
The important idea is that LMDB offloads cache management almost completely to the OS. You have to become intimately familiar with the way that the page cache works and how to configure it.
That it keeps an infinite cache of malloc page allocations is annoying (the issue you referenced). I just removed that (after complaining on the mailing list about it). The performance advantage is probably negligible in many cases (since malloc implementations often already cache), while causing confusing memory usage behavior.
Idk, if it was your issue, but for long running write transactions it doesn't spill to disk. So you have all the changes being written to disk at the end of the transaction. One would think enabling write mapping fixes this, but it needs to mark all the pages as clean before commit, so same effect there.
I fixed this for 0.9 here https://github.com/uroni/hs5/tree/main/external/lmdb . Will have to investigate if it is improved with 1.0, or if I need to redo the changes.
Edit: Just noticed that the issue is about free list in the file. Never had a problem with that, but I also had to replace that MIDL structure with something more scalable for the spilling.
By the way, you're wrong on both points - the cache of page mallocs is not infinite, and it does spill dirty pages to disk when necessary. And the latter is what bounds the number of malloc'd pages.
I can't go into specifics, but I use LMDB for the commandline application I maintain for my employer. I also extended it into a web service for internal use. As long as you stick to the safe LMDB options, which are the default options, it's reliable. The documentation clearly outlines what safety guarantees you lose when you enable/disable certain options: http://www.lmdb.tech/doc/group__mdb.html#ga32a193c6bf4d7d5c5...
I had a situation where the web service's writes were slowing down to an unbearable crawl because the number of entries in the database were reaching tens of billions of entries. Thankfully, the users never experienced the slowness. The website stayed nice and fast, even though the background updates were extraordinarily slow. The issue was fixed by sharding the databases.
I use it as a session store for a computer music system. It has worked well for me as a way to read mutable (by any client) parameters during synthesis, clients will often read dozens of parameters during a block of computation (a relatively short window of time in the low milliseconds typically) without adding any noticeable overhead to the render time for each block.
Edit: I also tried using it for larger blobs of data (like audio) but ended up only storing a reference to shared memory for larger blocks, anything larger than IIRC 4k that can't be stored in a single node kills performance, but for small values it seems pretty great.
If you use the default mode MDB_SYNC then it's reliable but can be slow for writes. For max write performance you need MDB_NOSYNC (IIRC that's what the official benchmarks use) but then the whole database can be unrecoverable in case of crash. It has happened to me.
Sqlite in WAL mode will never lose all your data and performance can be configured vs durability by setting pragma synchronous to full or normal.
Be cautious if you're using large databases on iOS. At least until fairly recently, iOS doesn't page dirty mmaped pages back to disk and after enough churn the app will OOM.
IIRC: it is to leverage the OS page cache rather than having a separate buffer pool in user land. By default lmdb uses normal pwrite/fsync for the write path, but can optionally use a writable mapping and (presumably) msync.
How is pwrite/fsync any better than mmap/msync? Both go through the page cache and combine asynchronous writeback with forced flush. One theoretical advantage of pwrite might be that you can handle I/O errors, but I’d like to see a case where recovering from an I/O error makes sense (rather than just crashing the database, which SIGBUS would do anyway by default).
I think lmdb is mostly unusable, for many use cases. I switched to libmdbx, which fixes all the issues [2] I (and most sibling comments) ran into with lmdb.
Different level of abstraction. I don’t think it’s highlighted enough either - this latest (1.0) and the previous 0.9.x are mutually incompatible, requiring essentially a dump/restore. It is mentioned (I forget which file ottofmh), but should be a
> And if compiler authors disagree, they are morons
I remember arguing with Howard years ago on “C vs Rust”. He said that you don’t need Rust, you just have to be good at C programming, so I pointed out CVEs in LMDB attributed to his own bare hands… so there’s that.
I recently talked to Howard [1] about lies he was saying about Sanakirja, an LMDB-inspired disk allocator. That's always the same arguments: C is better than Rust for X, Y or Z reasons. While I reported a segfault just two weeks earlier... [2].
I love LMDB, we use it in Meilisearch (second most stared search engine on GitHub) [3] for about 7 years now. The main issues were related to write speed but we do a compaction of the database and write performances are way better after that. We never had any major DB corruption... I mean... other than when using it on Azure. Azure never works, that's expected, I suppose.
Actually, you should take a look at [1]. It's made in Rust, inspired by LMDB, and supports a cool feature that allows close to anything to be implemented: allocating any page you want to store anything you want. The BTree storage is optional and you can implement whatever storage system you want. When storing a value to disk, you can allocate pages and decide exactly how you plan to store the bytes, allowing you not to store the length of them or to split your data into multiple pages, etc.
It is a small amount of code so easy to integrate into an application.
It is really reliable except write performance in my experience.
Author of it writes very spicy stuff and sounds pretty rude.
I would recommend doing a prototype with real data scale and testing if it meets your requirements. The write performance can be really atrocious and It doesn't have a high performance potential because it is based on memmap.
Yeah, and I also added support for parallel read your own writes where you can write tons of entries and spawn multi children read-only transactions from your writes where transaction and read from them in parallel.
We use this in Meilisearch [1] to post-process cache for our most common prefixes i.e., "w" will match "work", "word"... and computing this requires doing large unions of the documents matching those words.
Being able to do it in parallel is necessary, especially when you have billions of entries to operate on.
Maybe rephrase this part - "It is read-only by default as this provides total immunity to corruption. Using read-write mode offers much higher write performance, but adds the possibility for stray application writes thru pointers to silently corrupt the database."
I generally do think read-write mode would offer higher write performance than read only as well :)
> The memory map can be used as a read-only or read-write map.
So presumably lmdb writes to the database using the `pwrite` syscall by default, but can optionally write via the mmap instead - if you are willing to accept the increased risk of accidental data corruption.
Yup! Same, I'm the maintainer of the main LMDB Rust wrapper [1] and I was maintaining heed and heed3 (because 1.0 was available from the mdb.master3 branch).
But now that it's LMDB 1.0, I need to find a better way to make it be the official one but I can't really rename heed3 into heed and heed into heed-0.9...
I've never understood the fascination some people have with mmap. Memory-mapped file IO is just a RAM cache combined with a hidden system call (a page fault) to fill the cache. You can do the same thing yourself by using O_DIRECT to fill regular anonymous memory. If you're feeling social, you can fill a mapped and shared memfd.
You can seal memfds too, which means that the "read-only" mode is easy to implement: just map your memfd for write, apply F_SEAL_FUTURE_WRITE, and share the memfd to anyone you want to have read-only access.
By doing your own O_DIRECT IO instead of relying on the kernel's defaults, you get a lot more control. You choose how much readahead to do; you choose your read-cluster size. You choose your cache eviction strategy. You choose when to write back.
BTW: O_DIRECT can also be done asynchronously using aio or io_uring. There's no such thing as an asynchronous page fault. And IO errors? Would you rather deal with EIO or SIGBUS?
Why would you want the kernel to do these things for you? It'll do a worse job: it has less information than you do and has to use blunt heuristics that work sort-of-good-enough for the whole world, not just your program.
And it's not any faster either. O_DIRECT is DMA. A page cache fill is also DMA. It's the same operation, spelled differently.
I use mmap with my SQLite database[1] because I have many concurrent SQLite connections (one per concurrent HTTP request) and I don't want each connection to have its own 2MB cache[2]. It's better that all the connections simply share the page cache.
Linux needs a way for userspace processes to participate in the kernel's shrinker system for reclaiming memory under pressure. Watching memory PSI is too coarse. MADV_FREE is too complicated and indiscriminate. You could imagine a notification FD, but then you've just reinvented PSI. You could imagine a synchronous signal, but everyone hates signals and won't couple any new functionality to them.
Shrinker-BPF attached to a memfd perhaps? A BPF shrinker could not only choose which pages to evict in a non-stupid way, but could notify userspace in some sane manner (e.g. setting a bitmask somewhere) that it's done so.
(Zero-fill as "notification" is insane and doesn't actually work because zero is a perfectly valid value in a lot of contexts.)
That’s why you do your own memory accounting in the database. Of course, that assumes you own the machine; for an “embedded” DB like LMDB something like PSI may be necessary.
Another possibility for reclaiming physical memory beside unused page decommit with MADV_FREE/MADV_DONTNEED (there might not be unused pages to decommit) is to manually page out cold anonymous pages with MADV_COLD/MADV_PAGEOUT (thanks Android). You can combine this with low swappiness so anonymous pages are unlikely to be paged out automatically when there are clean file-backed pages that can be reclaimed.
> I've never understood the fascination some people have with mmap.
Uncommonly used system calls give user-space programmers the sensation of learning something.
> Why would you want the kernel to do these things for you? It'll do a worse job: it has less information than you do and has to use blunt heuristics that work sort-of-good-enough for the whole world, not just your program.
Yes, you're opting into non-determinism you don't control. When resources get constrained and everything can't be in memory and someone asks you why the database sucks, all you'll be able to do is shrug. Anyone who builds critical systems would never rely on the kernel making decisions like this. Don't use LMDB for anything that matters.
This level of reasoning is insufficient when building reliable systems. The consequences of depending on the OS for page caching are different than the consequences of depending on it for device drivers, file systems, or scheduling.
Nonsense. The best you will ever do, even with full application knowledge and complete control of the machine, is an LRU cache replacement algorithm. But when you do it yourself you have to juggle the fine details of which indices to prioritize, and you will never get it perfect. If you're not running a dedicated machine, as soon as any other processes run all your careful tuning goes out the window.
Since LMDB manages multiple tables as a tree of trees, no fine tuning is needed. The internal paths to every hot page automatically take priority, regardless of which index or how large each index is. So a simpleminded LRU always makes optimal use of available cache, regardless of access pattern or other load on the system.
First let me just say that while it's possible to interpret my original comment as uniquely applying to LMDB (or databases with similar page cache designs), in practice it applies to all general purpose databases including PostgreSQL and SQLite. This is because all general purpose databases will eventually fall short when it comes to tweaking behavior to meet application specific requirements, customizations notwithstanding. So to the extent that one should not use LMDB for anything that matters, one should also not use PostgreSQL or SQLite for anything that matters. If that corollary appears false in your frame of reference, then my statement about LMDB should also be false.
For high-stakes applications, you will have to maintain your own database code (either original or derived from an existing database) and that database code will need its own page caching layer (or a patched kernel), a generic page caching system (whether in-kernel with mmap or out of kernel) will not do. I acknowledge most applications don't operate in this regime.
> The best you will ever do, even with full application knowledge and complete control of the machine, is an LRU cache replacement algorithm.
This is not true. Applications often have specific high-priority data which should always exist in memory. That may be a moot point because you can do mlock() with mmap(). If we focus only on general-purpose caching, then even in that case there are many alternatives to LRU. SIEVE and ARC are two notable alternatives that perform significantly better for certain data. An application developer should be able to experiment with different general purpose caching strategies for different types of data, mmap() does not afford this.
Thank you Mr. Chu for your contributions to the technology commons and humanity in general.
> The best you will ever do, even with full application knowledge and complete control of the machine, is an LRU cache replacement algorithm
First of all, even the kernel can do better than simple LRU. We have MGLRU now for example. That said, the kernel is at a structural disadvantage.
A general purpose eviction and prefetch algorithm is like an automatic transmission on a car. It can react only to what it's seen.
When you drive stick, you can react to what you can see on the road ahead of you. A database has a query plan. It can see the future as well as remember the past. It has more information than the kernel.
> So a simpleminded LRU always makes optimal use of available cache, regardless of access pattern or other load on the system
That cannot be true. If I have a random access pattern, LRU will perform no better than random. If I have a future-oracle, I can just evict what's most distant in my set of future accesses.
Regardless of whether you're right about the suitability of LRU for this or that workload, it's simply false, mathematically, from a computer science POV, that LRU is optimal.
And if you go around making confidently wrong claims like this, one must wonder about what else you are wrong. If you want to be disagreeable in public, fine: just make sure you have math on your side first.
In the time it takes for your query optimizer to dissect a query and "look ahead" LMDB would have already answered a million queries. You think your magical "future oracle" is zero cost? How many KLOCs is it? LMDB's hot paths fit entirely inside a CPU's L1 cache.
And that's adequate for casual programs. LMDB is big and serious enough to warrant the extra complexity (which, to be fair, is significant) of userspace buffer management. LMDB does the work once and all users benefit.
That said, having written my own buffer pool and paging, etc... in pure naive benchmarks ... it's actually kinda hard to beat mmap. And LMDB is really fast for what it is.
In real world workflows I think the story is more complicated. Especially under higher concurrency.
I read the linked post. You're not making a good argument.
The authors aren't arguing that a mmap database is worse because it's "more complex". They are arguing it must work with less information. You haven't refuted the original paper, but you have made me more skeptical of LMDB.
For example, you claim that applications "never" have control of memory. That's simply, again, false. We have explicit memory eviction and pinning operations. We even have VA-batched TLB shootdown IPIs via process_madvise. On some systems (AMD, soon Intel) we can do TLB invalidation without an IPI.
So no, you're just wrong in making the claim that you might as well use mmap because you can't control the memory lifecycle anyway. You absolutely can, and anyone reading this message can look up the relevant APIs for himself.
And you point to LMDB's benchmarks repeatedly as evidence you're right. That's not saying what you think it is. LMDB is fast despite being hobbled by vanilla kernel mmap. Yes, that means other databases are probably doing stupid things, but reverse stupidity is not intelligence.
You're dreaming. None of your explicit memory control operations mean anything in practice, because today everything runs in VMs with no actual control of the underlying hardware. Probably co-resident with an unknown number of other tenants.
As for what you claim the paper's authors were saying - I quoted their text verbatim. Your interpretation is not what they said.
They claimed using mmap safely is impossible, and using it correctly requires more complexity than a traditional DB design. The safety claim was already disproven by multiple researchers. To prove their second claim they would have had to produce a DB that did traditional buffer management and was simpler and more performant than using mmap. They never did any such thing, nor could they.
that could easily be trojan-horsed with links to malware if you are viewing it in a poorly secured setting (like public wifi), because you can't verify the origin. so the best we can say about the author is that we are getting inconsistent signals on how seriously they understand and implement security concerns. so better review that code carefully before use, rather than assuming their expertise from release notes.
Judging from this very release, where he implemented support for page-level checksums and encryption for LMBD, I assume the author knows a thing or two about encryption. He probably then deemed it unnecessary for this specific website.
Indeed, there's no need to use the doc website. There's nothing there that isn't embedded in the LMDB source code. All of the docs are generated from doxygen comments in the source.
Do people have good experiences with LMDB, in terms of reliability? I've never used it in production, but I've read through the code and design documents for a database implementation class.
I remember some strange code (such as pushing return values 4k above the stack, with a comment like "this works as long as the caller doesn't use more than 4k of stack space before accessing the return value"), and the author also shared some unconventional opinions about undefined behavior (like "Compilers are deterministic, if I know what platform I'm compiling to then no behavior is undefined. And if compiler authors disagree, they are morons.")
But presumably it's thoroughly tested, so those aren't problems in practice? Would be really interested to hear from people who've actually used it. I've mainly stuck to SQLite instead.
Not amazing. In certain workloads I ran, once the db reached several hundred gb, writes would hang for longer and longer periods of time, eventually hours, while the db grew drastically in the background. https://news.ycombinator.com/item?id=30023623 seems to be the same issue, and it was serious enough that Shopify decided not to use lmdb.
And yes, I ensured there were no outstanding long lived readers, verified with mdb_stat -r. My workload used one transaction per read/write anyway (never needed larger atomicity). Once the db got into the bad state, running my program on it would almost immediately run into the issue again, so I really think the db is in a bad state such that most writes would cause it to hang, not related to how I do transactions. This workload would pretty consistently hit the issue once the db got to several hundred gb.
Issue #10236 on the OpenLDAP bug tracker might be the root cause, who knows. It's been marked CONFIRMED for years without a fix, while other similar issues are created.
This is extremely annoying. It seems workload dependent (other workloads I've run create absolutely massive lmdb dbs without this issue) and once it happens your only recourse is to make a new db and copy the contents over (thankfully reads still work fine on these borked dbs).
Other than that, though, it's great. Never in any case had actual data corruption, and reads and writes are extremely fast (until this issue happens)
Edit: fun fact, since shopify may have created Bolt in response to this bug, and then Bolt was the root cause of the 73-hour Roblox downtime in 2021, this bug may indirectly have caused one of the worst outages ever!
I've used LMDB in production for multi-terabyte databases, and we encountered the long-write time but found a solution.
The important idea is that LMDB offloads cache management almost completely to the OS. You have to become intimately familiar with the way that the page cache works and how to configure it.
That it keeps an infinite cache of malloc page allocations is annoying (the issue you referenced). I just removed that (after complaining on the mailing list about it). The performance advantage is probably negligible in many cases (since malloc implementations often already cache), while causing confusing memory usage behavior.
Idk, if it was your issue, but for long running write transactions it doesn't spill to disk. So you have all the changes being written to disk at the end of the transaction. One would think enabling write mapping fixes this, but it needs to mark all the pages as clean before commit, so same effect there. I fixed this for 0.9 here https://github.com/uroni/hs5/tree/main/external/lmdb . Will have to investigate if it is improved with 1.0, or if I need to redo the changes.
Edit: Just noticed that the issue is about free list in the file. Never had a problem with that, but I also had to replace that MIDL structure with something more scalable for the spilling.
By the way, you're wrong on both points - the cache of page mallocs is not infinite, and it does spill dirty pages to disk when necessary. And the latter is what bounds the number of malloc'd pages.
LMDB 1.0 no longer uses a P_DIRTY flag, it no longer has to explicitly mark pages as clean.
FWIW I had this issue even with the MDB_NOSYNC flag so it shouldnt be force flushing to disk unless I'm out of ram or whatever
I can't go into specifics, but I use LMDB for the commandline application I maintain for my employer. I also extended it into a web service for internal use. As long as you stick to the safe LMDB options, which are the default options, it's reliable. The documentation clearly outlines what safety guarantees you lose when you enable/disable certain options: http://www.lmdb.tech/doc/group__mdb.html#ga32a193c6bf4d7d5c5...
I had a situation where the web service's writes were slowing down to an unbearable crawl because the number of entries in the database were reaching tens of billions of entries. Thankfully, the users never experienced the slowness. The website stayed nice and fast, even though the background updates were extraordinarily slow. The issue was fixed by sharding the databases.
I use it as a session store for a computer music system. It has worked well for me as a way to read mutable (by any client) parameters during synthesis, clients will often read dozens of parameters during a block of computation (a relatively short window of time in the low milliseconds typically) without adding any noticeable overhead to the render time for each block.
Edit: I also tried using it for larger blobs of data (like audio) but ended up only storing a reference to shared memory for larger blocks, anything larger than IIRC 4k that can't be stored in a single node kills performance, but for small values it seems pretty great.
If you use the default mode MDB_SYNC then it's reliable but can be slow for writes. For max write performance you need MDB_NOSYNC (IIRC that's what the official benchmarks use) but then the whole database can be unrecoverable in case of crash. It has happened to me.
Sqlite in WAL mode will never lose all your data and performance can be configured vs durability by setting pragma synchronous to full or normal.
There's a project to have LMDB as a backend for sqlite. It originates in the LMDB author's (Howard Chu) own work.
https://lumosql.org/src/lumosql/doc/trunk/README.md
Also https://www.actordb.com/ uses LMDB with a sql queries. (Not sure if sqlite or not)
It has been used successfully as the backend for OpenLDAP and Monero, at least.
Be cautious if you're using large databases on iOS. At least until fairly recently, iOS doesn't page dirty mmaped pages back to disk and after enough churn the app will OOM.
Isn’t that why the mapping is read-only by default?
Irrelevant, since LMDB doesn't dirty the pages in the mmap. The mmap is read-only.
Wow, really?
Then what’s the point of memory mapping in the fist place? Or do they suggest manual flush/sync actions for persistence.
IIRC: it is to leverage the OS page cache rather than having a separate buffer pool in user land. By default lmdb uses normal pwrite/fsync for the write path, but can optionally use a writable mapping and (presumably) msync.
However, some people think there are problems with this usage: (pdf warning) https://www.cidrdb.org/cidr2022/papers/p13-crotty.pdf
How is pwrite/fsync any better than mmap/msync? Both go through the page cache and combine asynchronous writeback with forced flush. One theoretical advantage of pwrite might be that you can handle I/O errors, but I’d like to see a case where recovering from an I/O error makes sense (rather than just crashing the database, which SIGBUS would do anyway by default).
I think lmdb is mostly unusable, for many use cases. I switched to libmdbx, which fixes all the issues [2] I (and most sibling comments) ran into with lmdb.
[1] https://github.com/Mithril-mine/libmdbx#improvements-beyond-...
We evaluated it but chose RocksDB instead
> stuck to SQLite instead
Different level of abstraction. I don’t think it’s highlighted enough either - this latest (1.0) and the previous 0.9.x are mutually incompatible, requiring essentially a dump/restore. It is mentioned (I forget which file ottofmh), but should be a
*HIGHLIGHT* in the CHANGES.
I believe at least one of the two official Minecraft implementations use it for their map/save format.
> And if compiler authors disagree, they are morons
I remember arguing with Howard years ago on “C vs Rust”. He said that you don’t need Rust, you just have to be good at C programming, so I pointed out CVEs in LMDB attributed to his own bare hands… so there’s that.
I recently talked to Howard [1] about lies he was saying about Sanakirja, an LMDB-inspired disk allocator. That's always the same arguments: C is better than Rust for X, Y or Z reasons. While I reported a segfault just two weeks earlier... [2].
I love LMDB, we use it in Meilisearch (second most stared search engine on GitHub) [3] for about 7 years now. The main issues were related to write speed but we do a compaction of the database and write performances are way better after that. We never had any major DB corruption... I mean... other than when using it on Azure. Azure never works, that's expected, I suppose.
[1]: https://mastodon.social/@hyc/116838499082046918 [2]: https://bugs.openldap.org/show_bug.cgi?id=10522 [3]: https://github.com/meilisearch/meilisearch
Unfortunately, yeah.
> The main issues were related to write speed but we do a compaction of the database and write performances are way better after that
I'll ping you if I ever get around to rewriting a faster kv-store in Rust :)
Actually, you should take a look at [1]. It's made in Rust, inspired by LMDB, and supports a cool feature that allows close to anything to be implemented: allocating any page you want to store anything you want. The BTree storage is optional and you can implement whatever storage system you want. When storing a value to disk, you can allocate pages and decide exactly how you plan to store the bytes, allowing you not to store the length of them or to split your data into multiple pages, etc.
[1]: https://pijul.org/posts/2021-02-06-rethinking-sanakirja
It is a small amount of code so easy to integrate into an application.
It is really reliable except write performance in my experience.
Author of it writes very spicy stuff and sounds pretty rude.
I would recommend doing a prototype with real data scale and testing if it meets your requirements. The write performance can be really atrocious and It doesn't have a high performance potential because it is based on memmap.
New features in LMDB 1.0 include:
- support for incremental backup
- support for page-level checksums and encryption
- support for DB on raw block devices
- support for 2-phase commit
- support for page sizes up to 64KB
plus other minor additions to the API.
Yeah, and I also added support for parallel read your own writes where you can write tons of entries and spawn multi children read-only transactions from your writes where transaction and read from them in parallel.
We use this in Meilisearch [1] to post-process cache for our most common prefixes i.e., "w" will match "work", "word"... and computing this requires doing large unions of the documents matching those words.
Being able to do it in parallel is necessary, especially when you have billions of entries to operate on.
[1]: https://github.com/meilisearch/meilisearch
Maybe rephrase this part - "It is read-only by default as this provides total immunity to corruption. Using read-write mode offers much higher write performance, but adds the possibility for stray application writes thru pointers to silently corrupt the database."
I generally do think read-write mode would offer higher write performance than read only as well :)
The context is in the sentence before your quote:
> The memory map can be used as a read-only or read-write map.
So presumably lmdb writes to the database using the `pwrite` syscall by default, but can optionally write via the mmap instead - if you are willing to accept the increased risk of accidental data corruption.
https://www.meilisearch.com/docs/resources/internals/storage
Bummer. I'm the maintainer of the Python bindings.
I have to figure out how to support both versions now...
Yup! Same, I'm the maintainer of the main LMDB Rust wrapper [1] and I was maintaining heed and heed3 (because 1.0 was available from the mdb.master3 branch).
But now that it's LMDB 1.0, I need to find a better way to make it be the official one but I can't really rename heed3 into heed and heed into heed-0.9...
[1]: https://github.com/meilisearch/heed
heed09 and heed10 ? :)
I've never understood the fascination some people have with mmap. Memory-mapped file IO is just a RAM cache combined with a hidden system call (a page fault) to fill the cache. You can do the same thing yourself by using O_DIRECT to fill regular anonymous memory. If you're feeling social, you can fill a mapped and shared memfd.
You can seal memfds too, which means that the "read-only" mode is easy to implement: just map your memfd for write, apply F_SEAL_FUTURE_WRITE, and share the memfd to anyone you want to have read-only access.
By doing your own O_DIRECT IO instead of relying on the kernel's defaults, you get a lot more control. You choose how much readahead to do; you choose your read-cluster size. You choose your cache eviction strategy. You choose when to write back.
BTW: O_DIRECT can also be done asynchronously using aio or io_uring. There's no such thing as an asynchronous page fault. And IO errors? Would you rather deal with EIO or SIGBUS?
Why would you want the kernel to do these things for you? It'll do a worse job: it has less information than you do and has to use blunt heuristics that work sort-of-good-enough for the whole world, not just your program.
And it's not any faster either. O_DIRECT is DMA. A page cache fill is also DMA. It's the same operation, spelled differently.
I use mmap with my SQLite database[1] because I have many concurrent SQLite connections (one per concurrent HTTP request) and I don't want each connection to have its own 2MB cache[2]. It's better that all the connections simply share the page cache.
[1]: https://sqlite.org/pragma.html#pragma_mmap_size
[2]: https://sqlite.org/pragma.html#pragma_cache_size
with mmap you also don't have to worry about committing too much system memory, if another application needs it it will start evicting your cache.
You're right about that.
Linux needs a way for userspace processes to participate in the kernel's shrinker system for reclaiming memory under pressure. Watching memory PSI is too coarse. MADV_FREE is too complicated and indiscriminate. You could imagine a notification FD, but then you've just reinvented PSI. You could imagine a synchronous signal, but everyone hates signals and won't couple any new functionality to them.
Shrinker-BPF attached to a memfd perhaps? A BPF shrinker could not only choose which pages to evict in a non-stupid way, but could notify userspace in some sane manner (e.g. setting a bitmask somewhere) that it's done so.
(Zero-fill as "notification" is insane and doesn't actually work because zero is a perfectly valid value in a lot of contexts.)
That’s why you do your own memory accounting in the database. Of course, that assumes you own the machine; for an “embedded” DB like LMDB something like PSI may be necessary.
Another possibility for reclaiming physical memory beside unused page decommit with MADV_FREE/MADV_DONTNEED (there might not be unused pages to decommit) is to manually page out cold anonymous pages with MADV_COLD/MADV_PAGEOUT (thanks Android). You can combine this with low swappiness so anonymous pages are unlikely to be paged out automatically when there are clean file-backed pages that can be reclaimed.
> I've never understood the fascination some people have with mmap.
Uncommonly used system calls give user-space programmers the sensation of learning something.
> Why would you want the kernel to do these things for you? It'll do a worse job: it has less information than you do and has to use blunt heuristics that work sort-of-good-enough for the whole world, not just your program.
Yes, you're opting into non-determinism you don't control. When resources get constrained and everything can't be in memory and someone asks you why the database sucks, all you'll be able to do is shrug. Anyone who builds critical systems would never rely on the kernel making decisions like this. Don't use LMDB for anything that matters.
You're already depending on the OS for many other things. Depending on it for page caching is just one more thing.
This level of reasoning is insufficient when building reliable systems. The consequences of depending on the OS for page caching are different than the consequences of depending on it for device drivers, file systems, or scheduling.
Nonsense. The best you will ever do, even with full application knowledge and complete control of the machine, is an LRU cache replacement algorithm. But when you do it yourself you have to juggle the fine details of which indices to prioritize, and you will never get it perfect. If you're not running a dedicated machine, as soon as any other processes run all your careful tuning goes out the window.
Since LMDB manages multiple tables as a tree of trees, no fine tuning is needed. The internal paths to every hot page automatically take priority, regardless of which index or how large each index is. So a simpleminded LRU always makes optimal use of available cache, regardless of access pattern or other load on the system.
First let me just say that while it's possible to interpret my original comment as uniquely applying to LMDB (or databases with similar page cache designs), in practice it applies to all general purpose databases including PostgreSQL and SQLite. This is because all general purpose databases will eventually fall short when it comes to tweaking behavior to meet application specific requirements, customizations notwithstanding. So to the extent that one should not use LMDB for anything that matters, one should also not use PostgreSQL or SQLite for anything that matters. If that corollary appears false in your frame of reference, then my statement about LMDB should also be false.
For high-stakes applications, you will have to maintain your own database code (either original or derived from an existing database) and that database code will need its own page caching layer (or a patched kernel), a generic page caching system (whether in-kernel with mmap or out of kernel) will not do. I acknowledge most applications don't operate in this regime.
> The best you will ever do, even with full application knowledge and complete control of the machine, is an LRU cache replacement algorithm.
This is not true. Applications often have specific high-priority data which should always exist in memory. That may be a moot point because you can do mlock() with mmap(). If we focus only on general-purpose caching, then even in that case there are many alternatives to LRU. SIEVE and ARC are two notable alternatives that perform significantly better for certain data. An application developer should be able to experiment with different general purpose caching strategies for different types of data, mmap() does not afford this.
Thank you Mr. Chu for your contributions to the technology commons and humanity in general.
> The best you will ever do, even with full application knowledge and complete control of the machine, is an LRU cache replacement algorithm
First of all, even the kernel can do better than simple LRU. We have MGLRU now for example. That said, the kernel is at a structural disadvantage.
A general purpose eviction and prefetch algorithm is like an automatic transmission on a car. It can react only to what it's seen.
When you drive stick, you can react to what you can see on the road ahead of you. A database has a query plan. It can see the future as well as remember the past. It has more information than the kernel.
> So a simpleminded LRU always makes optimal use of available cache, regardless of access pattern or other load on the system
That cannot be true. If I have a random access pattern, LRU will perform no better than random. If I have a future-oracle, I can just evict what's most distant in my set of future accesses.
Regardless of whether you're right about the suitability of LRU for this or that workload, it's simply false, mathematically, from a computer science POV, that LRU is optimal.
And if you go around making confidently wrong claims like this, one must wonder about what else you are wrong. If you want to be disagreeable in public, fine: just make sure you have math on your side first.
In the time it takes for your query optimizer to dissect a query and "look ahead" LMDB would have already answered a million queries. You think your magical "future oracle" is zero cost? How many KLOCs is it? LMDB's hot paths fit entirely inside a CPU's L1 cache.
The OS handles all of that transparently, without requiring any additional code. I think that is the draw.
And that's adequate for casual programs. LMDB is big and serious enough to warrant the extra complexity (which, to be fair, is significant) of userspace buffer management. LMDB does the work once and all users benefit.
Obligatory: https://db.cs.cmu.edu/mmap-cidr2022/
Consensus says "don't do it" ...
That said, having written my own buffer pool and paging, etc... in pure naive benchmarks ... it's actually kinda hard to beat mmap. And LMDB is really fast for what it is.
In real world workflows I think the story is more complicated. Especially under higher concurrency.
Obligatory "that paper is garbage" https://www.symas.com/post/are-you-sure-you-want-to-use-mmap...
I read the linked post. You're not making a good argument.
The authors aren't arguing that a mmap database is worse because it's "more complex". They are arguing it must work with less information. You haven't refuted the original paper, but you have made me more skeptical of LMDB.
For example, you claim that applications "never" have control of memory. That's simply, again, false. We have explicit memory eviction and pinning operations. We even have VA-batched TLB shootdown IPIs via process_madvise. On some systems (AMD, soon Intel) we can do TLB invalidation without an IPI.
So no, you're just wrong in making the claim that you might as well use mmap because you can't control the memory lifecycle anyway. You absolutely can, and anyone reading this message can look up the relevant APIs for himself.
And you point to LMDB's benchmarks repeatedly as evidence you're right. That's not saying what you think it is. LMDB is fast despite being hobbled by vanilla kernel mmap. Yes, that means other databases are probably doing stupid things, but reverse stupidity is not intelligence.
You're dreaming. None of your explicit memory control operations mean anything in practice, because today everything runs in VMs with no actual control of the underlying hardware. Probably co-resident with an unknown number of other tenants.
As for what you claim the paper's authors were saying - I quoted their text verbatim. Your interpretation is not what they said.
They claimed using mmap safely is impossible, and using it correctly requires more complexity than a traditional DB design. The safety claim was already disproven by multiple researchers. To prove their second claim they would have had to produce a DB that did traditional buffer management and was simpler and more performant than using mmap. They never did any such thing, nor could they.
"we don't have to pay back any vulture capitalists"! a good one
Like 8 years ago I encountered this package, I recall reading that it was designed by someone who programmed components for the Space Shuttles.
I was very impressed.
HTTP ?? Com’on man
it is just a link to documentation
that could easily be trojan-horsed with links to malware if you are viewing it in a poorly secured setting (like public wifi), because you can't verify the origin. so the best we can say about the author is that we are getting inconsistent signals on how seriously they understand and implement security concerns. so better review that code carefully before use, rather than assuming their expertise from release notes.
If you're downloading binaries from a plaintext documentation site, I think that's on you.
TLS certs are freeeeee
Judging from this very release, where he implemented support for page-level checksums and encryption for LMBD, I assume the author knows a thing or two about encryption. He probably then deemed it unnecessary for this specific website.
Cryptography engineers are not excluded from being lazy sysadmins.
What do you mean "lazy"? I thought you said TLS certs were free. Do you mean they cost something after all? Time, for example?
Anyway, of course in case you feel the website is a risk, you should refrain from using it. Safety comes first.
Indeed, there's no need to use the doc website. There's nothing there that isn't embedded in the LMDB source code. All of the docs are generated from doxygen comments in the source.
Obligatory https://doesmysiteneedhttps.com/
Another interesting LMDB fork: https://github.com/datalevin/dlmdb
[flagged]