How to Make Postgres Queues Scale: Three Lessons from DBOS

The conventional wisdom is that Postgres-based queues don't scale: serious workloads supposedly demand specialized systems like RabbitMQ with Celery or Redis with BullMQ. There's some basis for this view — thousands of workers polling a queue table at once create contention and wear down indexes. Yet DBOS engineers Qian Li and Peter Kraft have shown that, configured correctly, Postgres holds up, reaching 30,000 workflow executions per second across thousands of servers. They distilled their experience into three lessons.

Lesson one: SKIP LOCKED

The first problem is contention over the same tasks. If every worker uses a simple query to pick the N oldest tasks in the queue, they all see the same rows and try to grab them simultaneously. Only one can claim a task; the rest retry in vain. The solution has long been built into Postgres — locking constructs. A query with FOR UPDATE SKIP LOCKED locks the selected rows and skips those already locked by other workers. That way each worker takes its own batch of tasks without conflicts. The authors wryly note that SKIP LOCKED is one of those old Postgres tricks that gets rediscovered again and again. Without it, the ceiling is around 100 tasks per second.

Lesson two: transaction isolation levels

The next bottleneck: past a thousand tasks per second, most operations failed with serialization errors and required retries. The culprit turned out to be the REPEATABLE READ isolation level, originally chosen for the sake of global constraints of the form “no more than N processes at once across all workers.” That mode requires a consistent snapshot of the database, but under high contention it becomes expensive: overlapping changes cause transactions to be rolled back. The key observation is that the largest queues rarely use global control, preferring local limits (“no more than 10 processes per worker”), which need no coordination between workers. So the isolation level was made conditional: queues with global control stay on REPEATABLE READ, while the rest switch to READ COMMITTED, which eliminates serialization errors entirely.

Lesson three: indexes aren't free

After this, contention all but disappeared, but at loads above 8,000 tasks per second CPU usage climbed — because of inefficient indexes. The index for selecting tasks didn't return them in the needed order, so Postgres had to sort them by time afterward. On top of that, every status change required updating all the indexes, and autovacuum then had to clear out the stale entries. The solution was selectivity: the main index was reworked to sort tasks by priority and time up front, and turned into a partial index — maintained only for processes in the ENQUEUED status. The same technique was applied to the observability indexes, for example the index on the parent process identifier.

Together, these optimizations sharply reduced CPU load and made it possible to push the queues past 30,000 tasks per second — roughly 80 billion per month.

Source: Hacker News