PgBouncer is entirely optional and it's not always the right choice. If you have a classical app (non serverless) and you can maintain a connection pool from your app, then I recommend avoiding pgbouncer.
The benefits of pgbouncer mostly come from irregular client connections (too many, too much churn). If you don't have that problem, go direct to postgres.
I'm exploring replacing pgbouncer with an alternative (maybe home grown) at the moment. Mostly for multi-tenancy and HA reasons. Pgbouncer has been good for us, but it's limited in how we can deploy it in a multi-tenant environment.
Also some people with use cases that shouldn't need PgBouncer end up thinking they do because something else is misconfigured. This was brought up in a parent of https://news.ycombinator.com/item?id=49019695; FastAPI recommends dep-injecting transactions into your HTTP handlers. Ties up all your connections and creates idle xact spam. There are valid reasons to use PgBouncer, this isn't one.
Even on serverless platforms like Heroku, I've been fine giving each worker a pool such that max_workers * pool_size < max_connections.
And the only comment in this whole thread, who argues for a similar architectural alternative...is the ONLY one in the whole thread down voted. HN continue to excel in technical chops...
Article title is pure click-bait. PgBouncer adds complexity. If you need it, you need it. If you don't need it, then you added complexity for nothing.
> since neither IBM nor Oracle is a service that any self-respecting person not part of an enterprise sales cycle would actually use
Author needs a serious ego check. There are legitimate engineering reasons to pick IBM Cloud (like if you need to support Z mainframes, which you will sometimes need if you sell to those enterprise folk) as well as Oracle Cloud (they built datacenters in cities that are not served by other cloud providers and can thus offer the lowest latency). These reasons may not be common, but they're certainly legitimate.
It's an ironic claim because the Oracle cloud has great support for the Oracle Database, which doesn't require "bouncers" or equivalent. Its support for server-side connection pooling, client side load balancing (SCAN) and horizontal scaling means it offers exactly what the author wants - a single URL that just magically works and scales to any amount of work or connections cheaply.
Yet he says simultaneously that Postgres hasn't improved in a decade, but also no self-respecting person would use a database that fixes all the problems he identified. Right!
Disclosure: work part time in the Oracle DB group.
Slightly off topic, but a friend of mine worked on Microsoft SQL Server a ~decade ago, and I recall all sorts of conversations on types of optimizations they were doing that sounded truly magical compared to other databases at the time. There was a meaningful frustration on the team that their features weren't getting as much hype as the databases du jour.
I think it's very relevant to consider that "enterprise" adjacent databases may not support the licensing you need at scale - but that doesn't mean they don't have great engineering and research teams that are solving really difficult challenges, and those challenges may be highly relevant to your workflows. Go into things with an open mind, if not an open wallet!
Yeah, that's a common sentiment. I regularly see people describe features commercial RDBMS' have shipped for years as if they were unmapped frontiers in computer science. Awareness of their capabilities is very low; I was once there too and remember being taken by surprise when I realized how far ahead of open source they truly are.
IMO startups can get edge by exploiting this information asymmetry. Spending a bit more to solve all your DB problems and buy productivity is a no brainer as they have VC funding but not enough time. A single bad DB outage can be the difference between beating a competitor or losing to them. Ditto for slowly shipping a feature because your senior dev is trying to implement their own message queue engine or other random thing that comes out of the box in other RDBMS engines.
The costs depend what you compare it to. People tend to overestimate it. Cost multipler in Azure is very roughly about 4x, it seems (caveat: am not a cloud pricing expert, comparisons may vary wildly). That doesn't include the cost of bouncers and other hacks that increase the Postgres cost, so it's artificially generous to PG.
If you want better features on the Postgres side then you might look at AlloyDB in Google Cloud which is only 2x cheaper on compute but where storage is actually ~3x more expensive!
The extra money buys you a lot. Not only far more features but you can provision a smaller database because the Oracle DB burst scales in response to load. You are only charged for the extra you use so you can provision for normal load without padding extra for emergencies or peaks. It's a genuine cluster that scales up writes more or less indefinitely without sharding if you design your schema right, that's synchronous multi-write master scaling too so its simple for apps. You don't face OpenAI style problems where the single Postgres master reaches its limits and the whole thing breaks requiring app redesigns. And you aren't just paying for an idle replica: all the capacity you buy can be used for queries. It also uses more efficient algorithms e.g. better MVCC with no autovacuuming problems. And a gazillion other things.
So I think you can easily argue that value delivered is much greater than 2x-4x. The capability gap is much larger than 4x. Especially if you're the sort of startup where a bored dev might start citing Postgres' limitations to justify inventing their own DB infra - if that happens you'll never recover the cost difference, Oracle will always be cheaper.
This happens because clouds don't charge databases at licensing+labor value+margin, prices are set at what the market will bear.
Depends what you mean by people. Outside of the startup space you'll find commercial RDBMS everywhere, especially OracleDB. Not many are running banks or hospitals on Postgres.
This question will get more interesting responses if it was qualified as:
"Does anyone run Postgres without PgBouncer for non-trivial workloads?"
Because, as we can see from the comments so far, lots of people are going to say you don't need it for your blog that gets 10 hits a month.
I've personally never heard of anyone not using PGbouncer, or some connection pooling proxy, for reasonably concurrent workloads. PG's process-per-connection architecture almost requires it. Otherwise even a small connection storm will wreak havoc on your server.
Not all non-trivial workloads are web-scale. There are plenty of on-premise applications out there that have at most hundreds or thousands of concurrent users, and the connections come from a bunch of fat spring boot servers that handle most of the pooling by themselves.
Yes, I've done a variety of typical, nontrivial workloads on Postgres for about a decade and have never used PgBouncer. Though I can imagine use cases where it'd make sense.
PGBouncer isn't as useful if you have seriously long-running transactions. It can’t do much of anything with those. Sure, you can give it a pool of 1000 and your Postgres instance a pool of 100, but you’re just moving who is going to say, “sorry, the database can’t handle your request right now.”
It’s not a silver bullet.
If you have more connections than Postgres can handle on the hardware it’s on, but with a bit of buffer it’ll be able to burn them down: great.
If you have long-held connections with many transactions that PGBouncer can interleave: great.
If you have connections whose transactions are longer than a reasonable timeout, well, your optimization princess is in another castle.
The main Question is: why do you allow clients to connect to your database server, it should be limited to a server which could actually serve the data in a format the client can just render without any logic client side.
I feel like there are a lot of use cases where I’d opt for SQLite and a lot of use cases where I’d opt for Postgres + PgBouncer. I’m curious what kinds of features push towards using Postgres alone over SQLite.
The setup cost is basically zero anyway. Add apt repo, apt-get install the correct version. Easy to run different versions at the same time too. Upgrading is annoying.
Concurrency, data types, scalability, centralization, or replication push for Postgres. The push against PgBouncer is that you don't need it, unless you do. If you have an app-level connection pool, you probably don't need PgBouncer.
Python: absolutely necessary due to the amount of processes and various deployments to run an application once it grows.
Java: never felt the need even on quite big apps. As it's much easier to share a connection pool locally, it's not as many single connections across the whole app.
Your high level buckets are languages but the constraints you list are usage patterns.
You can build applications in any language that have many short-lived transactions in single connections or a few huge, blocking one, or anything in between.
The former case is a good one for PGBouncer (it can interleave transactions) and the latter isn’t (it can’t do much about your three minute long BEGIN…COMMIT). Neither has to do with the language.
Languages have affordances. At the extreme ends are PHP, and Java or Go. PHP runs a separate logical process on every request which can't share resources with any other request. Almost everyone using Go is writing a long-running server process because that's how the libraries are designed. Almost everyone using Java is writing a long-running process or a module for one, because Java startup times are obscene. Java also has a very convenient synchronization primitive.
They addressed this though? I suspect you are unfamiliar with the GIL in python. The reason for a language distinction is because, as OP says: "As it's much easier to share a connection pool locally".
This may change vaguely soon, but right now you typically scale python apps by starting multiple python processes, while for Java you can just add threads. Python processes can't share a thread pool among all of them, while Java can.
> I suspect you are unfamiliar with the GIL in python.
Sadly, I’m deeply familiar. What made Evan Phoenix’s Rubinius work so exciting in ~2012 was getting rid of the GIL and seeing what was “fixed” by parallelism and what was not.
You make a good point. I didn’t read the parent comment that way but you’re right about how you scale python with the GIL vs JVM.
Totally agree. Postgres core should include connection pooling by default, and PgBouncer is probably the natural path to get there.
With PgBouncer addressing one of its biggest historical pain points - prepared statement support in transaction mode, in our managed Postgres offering, we’re increasingly seeing customers use the PgBouncer connection string by default for mose use-cases without running into any hiccups. That wouldn’t necessarily have been the case a few years ago.
PgBouncer is also battle-tested, widely validated, and offers a (surprising) level of configurability. You could also run a peered setup and make it multi-threaded, which is something I didn’t expect when I first came to know about it. https://news.ycombinator.com/item?id=48872874
> since neither IBM nor Oracle is a service that any self-respecting person not part of an enterprise sales cycle would actually use
Heh, snarky.
I guess (almost) everyone uses PgBouncer because they want to use a setup that will need the scaling needs of most users from the get-go, to avoid wasting support time.
Personally, I run without it due to a fairly small scale - I don't need more than about 128-256 connections max and for me a dedicated connection pooler (other than what's sometimes used app-side) would just add complexity.
At the same time, one could totally reasonably make the argument that if almost everyone uses it, then it SHOULD quite possibly be a built in feature, instead of a separate component - such a tighter integration would most likely bring the overall complexity down.
I’d argue this is not the right question. Obviously people use Postgres without PgBouncer. If you include non-production in the mix (CI/CD), most connections probably avoid it.
But because PgBouncer is so common, the question should probably be, why isn’t connection pooling part of Postgres out of the box? I think this might be the more interesting question.
If you have a connection pooler in your application, and the DB is only used for this application, you don't need an external pool like PgBouncer. That is probably a pretty common scenario, and typical web frameworks include a connection pool anyway.
An additional complication is that more aggressive pooling methods have side effects that you must know and prevent in your application. They're not safe to use out of the box.
The need for a connection pool is a side effect of the heavy process-based PostgreSQL connections. And I would suspect that this will change at some point in the not so near future, so that users don't have to think about this part this much.
It's a little more nuanced than that. At my company we run Kubernetes workloads where we have 20-30 or sometimes more pods connecting to a single Postgres instance, each pod handling hundreds of concurrent requests. If each pod hangs on to pooled connections for more than a few seconds, then you end up with quite a few Postgres processes and a lot of memory usage and process churn; we mitigate that by having a relatively short TTL on the pool, so idle ones get reaped relatively quickly. But any idle connection not used by a pod can't be used by a different pod. A connection pooler lets all the pods share more connections to Postgres, reducing in less wastage.
SRE here, that's a ton of pods and sounds like it's language or architecture if you are serving up so few requests per pod.
We have Kubernetes here too for Java monoliths and we have 8 Pods serving more than that. Since Java has connection pooling, the overhead has never been enough to justify overhead of Pgbouncer for this Java app.
> If you have a connection pooler in your application, and the DB is only used for this application, you don't need an external pool like PgBouncer.
But this is only true if you have no more than a few running instances of your application. So it feels like the cases where you must have Postgres (over an alternative like SQLite) but can't justify PgBouncer are very narrow.
I see many comments comparing PgBouncer with application connection poolers without addressing the conceptual difference between them. Here it is:
1. Most application connection poolers follow a first-in-first-out (FIFO) algorithm, which is simple enough to implement and is enough to make sure the application always has a connection available to connect to the database. It optimizes low latency, and works great from the point of view from the application. The problem is that it has few mechanisms to remove redundant connections, since the application is constantly keeping them all "warm".
2. PgBouncer and very few external poolers follow the inverse idea – last-in-first-out (LIFO), and they optimize for reducing the number of connections that reach Postgres, thus improving its throughput. The idea might seem crazy at first – the last connection used is the first one to be picked up again – but this algorithm automatically removes excess connections, which will get cold and get closed.
When starting a new application, option (1) is enough, but as it scales up enough, at some time it is recommended to use (2), since having hundreds of open connections to Postgres is bad for performance if you can use PgBouncer or similar to cut it by 90%. Postgres' process-per-connection design works much better when there are fewer connections reaching it.
If you don’t use serverless but instead a few (vertically scaling) servers, and your ORM / query builder supports pooling (all node libraries I’ve used have a pooler)…
Having a setup with just a simple docker deploy, running a monolith, not using pgbouncer so you can use LISTEN/NOTIFY to implement your own job queue:
I wouldn't do it without pgbouncer. Asking for trouble when one day connections exceeds. It's just that you start with "oh I'll manage the pool from my app" and then you're stuck with either putting things into the app or tuning the pool for the other side-programs you need from the app.
Every single connection to Postgres is a new process, which requires a fork and new memory allocation (at least 10MB plus whatever you need for your query).
PgBouncer opens a pool of connections and then reuses them each time a client asks for a connection. This reduces latency (no more fork) and overhead (reuse memory).
The issue is language ecosystems that don't use client side connection pooling because they're single threaded (node, Python). So scaling up the number of web server threads means scaling the number of Postgres processes, which are expensive.
How many running instances do you need though? e.g. Scala web frameworks should be able to do thousands of RPS on a single core without the application developer really trying to optimize anything, and I always hear that even Ruby, Python, etc. are also fast enough to be IO bound so you should just need 2 copies for redundancy, right? Then give each like 8-16 connections.
That's fine but doesn't address the issue that PgBouncer does. Your application connection pool can multiplex all the connections needed in one application. PgBouncer can multiplex the connections across all applications (whether different apps or many instances of the same app).
If you handle database requests naively, every request to the database may open its own connection. This is a super simple approach but will exceed the amount of connections the database can or wants to handle concurrently.
One solution is to increase your app complexity and introduce a layer that manages connection pooling or queuing.
Or you can just keep your app naive and simple and put pgbouncer transparently in front of your DB. Even for multiple apps, so instead of every app increasing in complexity, reimplementing connection handling, you just have pgbouncer.
Clickbait post, already made the rounds on in other sites and was mercilessly torn to shreds. The answer is that millions do in production. PgBouncer is only needed for stateless backends, and even then, only under specific circumstances.
What pgbouncer does is indeed core functionality. Compare Postgres to MySQL and sql server, where analogous standalone connection pools are rarely used. The fundamental reason pgbouncer needs to exist is Postgres’ utterly retrograde design. Other examples: xid wraparound, conflict with recovery, lack of undo space.
I mean… my self hosted Postgres with its ca 15 active connections certainly doesn't use PgBouncer, and it doesn't need to. But that was presumably not the intended scope of the question?
Then again, people forget you can just run your own Postgres (or anything really).
of course, the vast, vast majority of PostgreSQL users outside of managed cloud hosting are not using pgbouncer. pgbouncer introduces complexities into the database conversation (transaction-level pooling interacting with the prepared statement cache is a long recurring nightmare for us at sqlalchemy) that often not worth the complexity for small local installations.
this article seems to be talking about commercial cloud managed PG services, which yes, those absolutely need to support connection pooling and of course they're going to use pgbouncer.
DataDirect is not free but I guess we are discussing technical merits here. For them pooling happens in the connectivity layer and exposes proper pool controls such as minimum and maximum pool size, connection lifetime behavior, and optional connection state reset.
More importantly, it has capabilities that PgBouncer is not designed to provide. It can maintain alternate PostgreSQL servers, retry connections, randomize connection attempts across primary/alternate servers, and has explicit failover modes. The application retains a real PostgreSQL session while the driver handles connection reuse and failure handling.
That matters because of PgBouncer big technical compromise that is transaction pooling.
This breaks the assumption that one client connection is the PostgreSQL backend session.
Consequently, several session scoped PostgreSQL features do not work normally in transaction pooling. PgBouncer own compatibility table, lists some of the limitations:
https://www.pgbouncer.org/features.html
DataDirect does not need to solve that particular problem because its architecture does not perform that same transaction level back end swapping.
We run asyncpg directly against Neon Postgres without PgBouncer —
works well for low-to-medium concurrency. Neon's serverless connection
pooling handles a lot of what PgBouncer would, so the overhead didn't
seem worth it for our use case.
Transaction mode has one real footgun: SET statements are session-scoped, but in transaction mode your session gets reassigned between transactions. Easy to miss until you're chasing a mysterious search_path bug at 2am.
psycopg3 prepares statements by default now, which breaks in transaction mode unless you explicitly opt out.
For apps with a persistent server process and an in-app pool the external bouncer is mostly ceremony. The math changes with serverless: no persistent process means no persistent pool, so a dedicated pooler starts pulling its weight.
The overhead of process-per-connection in Postgres is still the biggest hurdle for mobile-heavy workloads where you might have thousands of intermittent clients. Brandur is right that modern hardware makes the context switching less of a bottleneck, but the memory pressure from large work_mem settings on each backend process remains a real risk. I'm curious if anyone here has successfully moved to a pure built-in connection pool in v14+ without seeing a regression in latency during traffic spikes.
(I work on the postgres proxy layer at Neon)
PgBouncer is entirely optional and it's not always the right choice. If you have a classical app (non serverless) and you can maintain a connection pool from your app, then I recommend avoiding pgbouncer.
The benefits of pgbouncer mostly come from irregular client connections (too many, too much churn). If you don't have that problem, go direct to postgres.
I'm exploring replacing pgbouncer with an alternative (maybe home grown) at the moment. Mostly for multi-tenancy and HA reasons. Pgbouncer has been good for us, but it's limited in how we can deploy it in a multi-tenant environment.
Also some people with use cases that shouldn't need PgBouncer end up thinking they do because something else is misconfigured. This was brought up in a parent of https://news.ycombinator.com/item?id=49019695; FastAPI recommends dep-injecting transactions into your HTTP handlers. Ties up all your connections and creates idle xact spam. There are valid reasons to use PgBouncer, this isn't one.
Even on serverless platforms like Heroku, I've been fine giving each worker a pool such that max_workers * pool_size < max_connections.
And the only comment in this whole thread, who argues for a similar architectural alternative...is the ONLY one in the whole thread down voted. HN continue to excel in technical chops...
https://news.ycombinator.com/item?id=49319988
https://news.ycombinator.com/item?id=49320460
Article title is pure click-bait. PgBouncer adds complexity. If you need it, you need it. If you don't need it, then you added complexity for nothing.
> since neither IBM nor Oracle is a service that any self-respecting person not part of an enterprise sales cycle would actually use
Author needs a serious ego check. There are legitimate engineering reasons to pick IBM Cloud (like if you need to support Z mainframes, which you will sometimes need if you sell to those enterprise folk) as well as Oracle Cloud (they built datacenters in cities that are not served by other cloud providers and can thus offer the lowest latency). These reasons may not be common, but they're certainly legitimate.
It's an ironic claim because the Oracle cloud has great support for the Oracle Database, which doesn't require "bouncers" or equivalent. Its support for server-side connection pooling, client side load balancing (SCAN) and horizontal scaling means it offers exactly what the author wants - a single URL that just magically works and scales to any amount of work or connections cheaply.
https://docs.oracle.com/en/database/oracle/oracle-database/2...
Yet he says simultaneously that Postgres hasn't improved in a decade, but also no self-respecting person would use a database that fixes all the problems he identified. Right!
Disclosure: work part time in the Oracle DB group.
Slightly off topic, but a friend of mine worked on Microsoft SQL Server a ~decade ago, and I recall all sorts of conversations on types of optimizations they were doing that sounded truly magical compared to other databases at the time. There was a meaningful frustration on the team that their features weren't getting as much hype as the databases du jour.
I think it's very relevant to consider that "enterprise" adjacent databases may not support the licensing you need at scale - but that doesn't mean they don't have great engineering and research teams that are solving really difficult challenges, and those challenges may be highly relevant to your workflows. Go into things with an open mind, if not an open wallet!
Yeah, that's a common sentiment. I regularly see people describe features commercial RDBMS' have shipped for years as if they were unmapped frontiers in computer science. Awareness of their capabilities is very low; I was once there too and remember being taken by surprise when I realized how far ahead of open source they truly are.
IMO startups can get edge by exploiting this information asymmetry. Spending a bit more to solve all your DB problems and buy productivity is a no brainer as they have VC funding but not enough time. A single bad DB outage can be the difference between beating a competitor or losing to them. Ditto for slowly shipping a feature because your senior dev is trying to implement their own message queue engine or other random thing that comes out of the box in other RDBMS engines.
The costs depend what you compare it to. People tend to overestimate it. Cost multipler in Azure is very roughly about 4x, it seems (caveat: am not a cloud pricing expert, comparisons may vary wildly). That doesn't include the cost of bouncers and other hacks that increase the Postgres cost, so it's artificially generous to PG.
If you want better features on the Postgres side then you might look at AlloyDB in Google Cloud which is only 2x cheaper on compute but where storage is actually ~3x more expensive!
The extra money buys you a lot. Not only far more features but you can provision a smaller database because the Oracle DB burst scales in response to load. You are only charged for the extra you use so you can provision for normal load without padding extra for emergencies or peaks. It's a genuine cluster that scales up writes more or less indefinitely without sharding if you design your schema right, that's synchronous multi-write master scaling too so its simple for apps. You don't face OpenAI style problems where the single Postgres master reaches its limits and the whole thing breaks requiring app redesigns. And you aren't just paying for an idle replica: all the capacity you buy can be used for queries. It also uses more efficient algorithms e.g. better MVCC with no autovacuuming problems. And a gazillion other things.
So I think you can easily argue that value delivered is much greater than 2x-4x. The capability gap is much larger than 4x. Especially if you're the sort of startup where a bored dev might start citing Postgres' limitations to justify inventing their own DB infra - if that happens you'll never recover the cost difference, Oracle will always be cheaper.
This happens because clouds don't charge databases at licensing+labor value+margin, prices are set at what the market will bear.
I do not think it is a mystery why people favor Postgres over OracleDB, is it?
Depends what you mean by people. Outside of the startup space you'll find commercial RDBMS everywhere, especially OracleDB. Not many are running banks or hospitals on Postgres.
This question will get more interesting responses if it was qualified as:
"Does anyone run Postgres without PgBouncer for non-trivial workloads?"
Because, as we can see from the comments so far, lots of people are going to say you don't need it for your blog that gets 10 hits a month.
I've personally never heard of anyone not using PGbouncer, or some connection pooling proxy, for reasonably concurrent workloads. PG's process-per-connection architecture almost requires it. Otherwise even a small connection storm will wreak havoc on your server.
Not all non-trivial workloads are web-scale. There are plenty of on-premise applications out there that have at most hundreds or thousands of concurrent users, and the connections come from a bunch of fat spring boot servers that handle most of the pooling by themselves.
Yes, I've done a variety of typical, nontrivial workloads on Postgres for about a decade and have never used PgBouncer. Though I can imagine use cases where it'd make sense.
Good question.
PGBouncer isn't as useful if you have seriously long-running transactions. It can’t do much of anything with those. Sure, you can give it a pool of 1000 and your Postgres instance a pool of 100, but you’re just moving who is going to say, “sorry, the database can’t handle your request right now.”
It’s not a silver bullet.
If you have more connections than Postgres can handle on the hardware it’s on, but with a bit of buffer it’ll be able to burn them down: great.
If you have long-held connections with many transactions that PGBouncer can interleave: great.
If you have connections whose transactions are longer than a reasonable timeout, well, your optimization princess is in another castle.
The main Question is: why do you allow clients to connect to your database server, it should be limited to a server which could actually serve the data in a format the client can just render without any logic client side.
I feel like there are a lot of use cases where I’d opt for SQLite and a lot of use cases where I’d opt for Postgres + PgBouncer. I’m curious what kinds of features push towards using Postgres alone over SQLite.
All of my personal apps use Postgres because:
- types are lovely. We love types. SQLite’s default of non-strict typing is, to me, bananas.
- SELECT DISTINCT ON is my ride-or-die
- most importantly, I’m very comfortable in Postgres and the setup cost is basically zero (like SQLite) because Claude does it.
The setup cost is basically zero anyway. Add apt repo, apt-get install the correct version. Easy to run different versions at the same time too. Upgrading is annoying.
Concurrency, data types, scalability, centralization, or replication push for Postgres. The push against PgBouncer is that you don't need it, unless you do. If you have an app-level connection pool, you probably don't need PgBouncer.
Python: absolutely necessary due to the amount of processes and various deployments to run an application once it grows.
Java: never felt the need even on quite big apps. As it's much easier to share a connection pool locally, it's not as many single connections across the whole app.
Your high level buckets are languages but the constraints you list are usage patterns.
You can build applications in any language that have many short-lived transactions in single connections or a few huge, blocking one, or anything in between.
The former case is a good one for PGBouncer (it can interleave transactions) and the latter isn’t (it can’t do much about your three minute long BEGIN…COMMIT). Neither has to do with the language.
Languages have affordances. At the extreme ends are PHP, and Java or Go. PHP runs a separate logical process on every request which can't share resources with any other request. Almost everyone using Go is writing a long-running server process because that's how the libraries are designed. Almost everyone using Java is writing a long-running process or a module for one, because Java startup times are obscene. Java also has a very convenient synchronization primitive.
They addressed this though? I suspect you are unfamiliar with the GIL in python. The reason for a language distinction is because, as OP says: "As it's much easier to share a connection pool locally".
This may change vaguely soon, but right now you typically scale python apps by starting multiple python processes, while for Java you can just add threads. Python processes can't share a thread pool among all of them, while Java can.
> I suspect you are unfamiliar with the GIL in python.
Sadly, I’m deeply familiar. What made Evan Phoenix’s Rubinius work so exciting in ~2012 was getting rid of the GIL and seeing what was “fixed” by parallelism and what was not.
You make a good point. I didn’t read the parent comment that way but you’re right about how you scale python with the GIL vs JVM.
Totally agree. Postgres core should include connection pooling by default, and PgBouncer is probably the natural path to get there.
With PgBouncer addressing one of its biggest historical pain points - prepared statement support in transaction mode, in our managed Postgres offering, we’re increasingly seeing customers use the PgBouncer connection string by default for mose use-cases without running into any hiccups. That wouldn’t necessarily have been the case a few years ago.
PgBouncer is also battle-tested, widely validated, and offers a (surprising) level of configurability. You could also run a peered setup and make it multi-threaded, which is something I didn’t expect when I first came to know about it. https://news.ycombinator.com/item?id=48872874
> since neither IBM nor Oracle is a service that any self-respecting person not part of an enterprise sales cycle would actually use
Heh, snarky.
I guess (almost) everyone uses PgBouncer because they want to use a setup that will need the scaling needs of most users from the get-go, to avoid wasting support time.
Personally, I run without it due to a fairly small scale - I don't need more than about 128-256 connections max and for me a dedicated connection pooler (other than what's sometimes used app-side) would just add complexity.
At the same time, one could totally reasonably make the argument that if almost everyone uses it, then it SHOULD quite possibly be a built in feature, instead of a separate component - such a tighter integration would most likely bring the overall complexity down.
I’d argue this is not the right question. Obviously people use Postgres without PgBouncer. If you include non-production in the mix (CI/CD), most connections probably avoid it.
But because PgBouncer is so common, the question should probably be, why isn’t connection pooling part of Postgres out of the box? I think this might be the more interesting question.
If you have a connection pooler in your application, and the DB is only used for this application, you don't need an external pool like PgBouncer. That is probably a pretty common scenario, and typical web frameworks include a connection pool anyway.
An additional complication is that more aggressive pooling methods have side effects that you must know and prevent in your application. They're not safe to use out of the box.
The need for a connection pool is a side effect of the heavy process-based PostgreSQL connections. And I would suspect that this will change at some point in the not so near future, so that users don't have to think about this part this much.
It's a little more nuanced than that. At my company we run Kubernetes workloads where we have 20-30 or sometimes more pods connecting to a single Postgres instance, each pod handling hundreds of concurrent requests. If each pod hangs on to pooled connections for more than a few seconds, then you end up with quite a few Postgres processes and a lot of memory usage and process churn; we mitigate that by having a relatively short TTL on the pool, so idle ones get reaped relatively quickly. But any idle connection not used by a pod can't be used by a different pod. A connection pooler lets all the pods share more connections to Postgres, reducing in less wastage.
SRE here, that's a ton of pods and sounds like it's language or architecture if you are serving up so few requests per pod.
We have Kubernetes here too for Java monoliths and we have 8 Pods serving more than that. Since Java has connection pooling, the overhead has never been enough to justify overhead of Pgbouncer for this Java app.
Our workloads are served by Go processes, so quite lean. But the workloads are relatively CPU-bound and memory-hungry.
> If you have a connection pooler in your application, and the DB is only used for this application, you don't need an external pool like PgBouncer.
But this is only true if you have no more than a few running instances of your application. So it feels like the cases where you must have Postgres (over an alternative like SQLite) but can't justify PgBouncer are very narrow.
Not everything is a SaaS application. There's a lot of software where you only run one or two application servers.
I see many comments comparing PgBouncer with application connection poolers without addressing the conceptual difference between them. Here it is:
1. Most application connection poolers follow a first-in-first-out (FIFO) algorithm, which is simple enough to implement and is enough to make sure the application always has a connection available to connect to the database. It optimizes low latency, and works great from the point of view from the application. The problem is that it has few mechanisms to remove redundant connections, since the application is constantly keeping them all "warm".
2. PgBouncer and very few external poolers follow the inverse idea – last-in-first-out (LIFO), and they optimize for reducing the number of connections that reach Postgres, thus improving its throughput. The idea might seem crazy at first – the last connection used is the first one to be picked up again – but this algorithm automatically removes excess connections, which will get cold and get closed.
When starting a new application, option (1) is enough, but as it scales up enough, at some time it is recommended to use (2), since having hundreds of open connections to Postgres is bad for performance if you can use PgBouncer or similar to cut it by 90%. Postgres' process-per-connection design works much better when there are fewer connections reaching it.
Yes. Sometimes PostgreSQL is overkill for a low traffic, simple application. PgBouncer would be even more overkill and add unnecessary complexity.
If you don’t use serverless but instead a few (vertically scaling) servers, and your ORM / query builder supports pooling (all node libraries I’ve used have a pooler)…
Having a setup with just a simple docker deploy, running a monolith, not using pgbouncer so you can use LISTEN/NOTIFY to implement your own job queue:
https://www.dbos.dev/blog/postgres-listen-notify-scalability
This gives me warm fuzzy feelings, also making me relatively cloud-agnostic in the process, even though devops is not my strong point.
Most projects I do don’t need something more complex or vendor locked-in than this.
I guarantee you the vast majority of RDS instances aren't using RDS Proxy. For one, it's rather expensive.
For sure. I mean: many – if not most – people?
In my case, with Go, I have always relied on http://github.com/jackc/pgx pool, which works quite well, especially with the binary protocol.
I use the Ecto connection pool when using Elixir/Phoenix.
Yes, I don't always need it. Depends on the service architecture.
I wouldn't do it without pgbouncer. Asking for trouble when one day connections exceeds. It's just that you start with "oh I'll manage the pool from my app" and then you're stuck with either putting things into the app or tuning the pool for the other side-programs you need from the app.
Yes. I've never even heard of PgBouncer.
This is the first time I've heard pgbouncer mentioned in 15 years. I didn't know it was still around.
Me too. I even opened the page and I'm not sure of what's the problem being solved.
Every single connection to Postgres is a new process, which requires a fork and new memory allocation (at least 10MB plus whatever you need for your query).
PgBouncer opens a pool of connections and then reuses them each time a client asks for a connection. This reduces latency (no more fork) and overhead (reuse memory).
If it's designed well, your application also opens a pool of connections and re-uses them each time the code needs a DB connection.
The issue is language ecosystems that don't use client side connection pooling because they're single threaded (node, Python). So scaling up the number of web server threads means scaling the number of Postgres processes, which are expensive.
This solves the latency but not the overhead--you'll end up with a full pool of connections for each running instance of your application.
How many running instances do you need though? e.g. Scala web frameworks should be able to do thousands of RPS on a single core without the application developer really trying to optimize anything, and I always hear that even Ruby, Python, etc. are also fast enough to be IO bound so you should just need 2 copies for redundancy, right? Then give each like 8-16 connections.
If I'm implementing my own connection pool, I'll certainly make the number of connections configurable.
That's fine but doesn't address the issue that PgBouncer does. Your application connection pool can multiplex all the connections needed in one application. PgBouncer can multiplex the connections across all applications (whether different apps or many instances of the same app).
If you handle database requests naively, every request to the database may open its own connection. This is a super simple approach but will exceed the amount of connections the database can or wants to handle concurrently.
One solution is to increase your app complexity and introduce a layer that manages connection pooling or queuing.
Or you can just keep your app naive and simple and put pgbouncer transparently in front of your DB. Even for multiple apps, so instead of every app increasing in complexity, reimplementing connection handling, you just have pgbouncer.
PgBouncer? Don't even know er.
Clickbait post, already made the rounds on in other sites and was mercilessly torn to shreds. The answer is that millions do in production. PgBouncer is only needed for stateless backends, and even then, only under specific circumstances.
A whole bunch of Cloudflare clients run Postgres (self hosted) without pgbouncer (they use hyperdrive).
What pgbouncer does is indeed core functionality. Compare Postgres to MySQL and sql server, where analogous standalone connection pools are rarely used. The fundamental reason pgbouncer needs to exist is Postgres’ utterly retrograde design. Other examples: xid wraparound, conflict with recovery, lack of undo space.
I currently run without pgbouncer, but I have at most 200 connections. 50 of them are just replication. I didn't want the complexity.
Yes
I’m curius about pgcat, is there anyone here using it ?
We don't maintain it anymore. I'm working on pgdog[1] fulltime now. We've had several successful pgcat -> pgdog migrations now.
[1] https://github.com/pgdogdev/pgdog
Yes.
I mean… my self hosted Postgres with its ca 15 active connections certainly doesn't use PgBouncer, and it doesn't need to. But that was presumably not the intended scope of the question?
Then again, people forget you can just run your own Postgres (or anything really).
The answer is: yes.
I use postgres. I've never heard of pgbouncer. So the answer is an easy Yes.
Saved you a click.
of course, the vast, vast majority of PostgreSQL users outside of managed cloud hosting are not using pgbouncer. pgbouncer introduces complexities into the database conversation (transaction-level pooling interacting with the prepared statement cache is a long recurring nightmare for us at sqlalchemy) that often not worth the complexity for small local installations.
this article seems to be talking about commercial cloud managed PG services, which yes, those absolutely need to support connection pooling and of course they're going to use pgbouncer.
pgdog.
Are you kidding me? Have you heard about Data Direct?
https://docs.progress.com/bundle/datadirect-postgresql-odbc-...
DataDirect is not free but I guess we are discussing technical merits here. For them pooling happens in the connectivity layer and exposes proper pool controls such as minimum and maximum pool size, connection lifetime behavior, and optional connection state reset.
More importantly, it has capabilities that PgBouncer is not designed to provide. It can maintain alternate PostgreSQL servers, retry connections, randomize connection attempts across primary/alternate servers, and has explicit failover modes. The application retains a real PostgreSQL session while the driver handles connection reuse and failure handling.
That matters because of PgBouncer big technical compromise that is transaction pooling. This breaks the assumption that one client connection is the PostgreSQL backend session.
Consequently, several session scoped PostgreSQL features do not work normally in transaction pooling. PgBouncer own compatibility table, lists some of the limitations: https://www.pgbouncer.org/features.html
DataDirect does not need to solve that particular problem because its architecture does not perform that same transaction level back end swapping.
Yes, except for serverless backend, you normally don't need it.
We run asyncpg directly against Neon Postgres without PgBouncer — works well for low-to-medium concurrency. Neon's serverless connection pooling handles a lot of what PgBouncer would, so the overhead didn't seem worth it for our use case.
Transaction mode has one real footgun: SET statements are session-scoped, but in transaction mode your session gets reassigned between transactions. Easy to miss until you're chasing a mysterious search_path bug at 2am.
psycopg3 prepares statements by default now, which breaks in transaction mode unless you explicitly opt out.
For apps with a persistent server process and an in-app pool the external bouncer is mostly ceremony. The math changes with serverless: no persistent process means no persistent pool, so a dedicated pooler starts pulling its weight.
The overhead of process-per-connection in Postgres is still the biggest hurdle for mobile-heavy workloads where you might have thousands of intermittent clients. Brandur is right that modern hardware makes the context switching less of a bottleneck, but the memory pressure from large work_mem settings on each backend process remains a real risk. I'm curious if anyone here has successfully moved to a pure built-in connection pool in v14+ without seeing a regression in latency during traffic spikes.