OrioleDB beta15 and beta16: stability, testing, and replication hardening

The beta15 and beta16 cycles together cover roughly four months of work on OrioleDB. Where previous releases introduced headline features such as non-B-tree index support, tablespaces, and undo-based rewind, these two releases are deliberately focused on stability, correctness, and test coverage. The goal is simple: make every existing feature work reliably under real-world conditions, especially replication, recovery, and concurrent workloads.

What is OrioleDB?

OrioleDB is a PostgreSQL storage extension that implements a custom Table Access Method as a drop-in replacement for the default heap storage engine. It is designed to address scalability bottlenecks in PostgreSQL's buffer manager and WAL subsystem, enabling better utilization of modern multi-core CPUs and high-performance storage.

By rethinking core components such as MVCC, page caching, and checkpoints, OrioleDB improves throughput in transactional workloads without altering PostgreSQL's user-facing behavior.

Why a stability-focused stretch?

After several feature-heavy cycles, the OrioleDB team shifted focus to hardening the engine against edge cases exposed by broader adoption and more aggressive testing. The combined beta15+beta16 work contains numerous bug fixes spanning recovery, physical and logical replication, checkpointing, undo management, bridge indexes, page pool, and B-tree internals, plus separate fixes for various races during concurrent operations. Rather than adding new surface area, these releases make the existing surface area solid.

A new approach to testing

The most impactful change in this period is the dramatic expansion of test coverage. Instead of relying on existing and new OrioleDB-specific tests, we adopted a fundamentally different strategy: use PostgreSQL itself as the test source and predictor.

Running PostgreSQL's own tests on OrioleDB tables

A major milestone is running the standard PostgreSQL regression test suite with OrioleDB as the default table access method. This means the same INSERT, SELECT, UPDATE, MERGE, COPY, window functions, CTEs, partitioning, grouping sets, domain types, row types, generated columns, and dozens of other SQL features tested by the PostgreSQL community are now continuously validated against OrioleDB storage.

The tests are filtered by a purpose-built diff-filtering tool that accounts for known, expected differences in OrioleDB's output: slightly different plan shapes, storage-level details, and similar cosmetic divergences. The result is a continuous signal: if a standard PostgreSQL behavior breaks under OrioleDB, CI catches it immediately.

PostgreSQL's isolation tests are included as well, verifying that concurrent transaction behavior (eval-plan-qual and friends) matches expectations when the storage layer is OrioleDB. PostgreSQL's own recovery tests have likewise been brought into the CI pipeline, exercising restart, crash-recovery, and replay paths that earlier OrioleDB-specific suites only touched indirectly.

This single change surfaced more compatibility issues in a few weeks than months of dedicated testing. Many of the fixes in this period: hashing, generated columns, domain types, TOAST overflow handling, ALTER TABLE edge cases, contradictory NULL scan keys, SAOP/IN scans on PKs with reordered columns were discovered by simply running PostgreSQL's own tests and reading the diffs.

Verifying replication correctness automatically

Running tests on a single primary node is necessary but not sufficient. As with any Postgres cluster, everything under the Oriole storage engine must also replicate correctly: every row written on the primary must appear identically on the replica after recovery.

We introduced automated primary/replica data comparison as part of the CI process. After running the full regression and isolation test suites, the pipeline now runs all tests with a replica attached, exercising the replication path alongside the primary path, compares the replica's data and meta state to the primary's data, table by table, and fails if any divergence is found.

This catches a class of edge case bugs that are invisible on a single node: incorrect WAL records for DDL operations, bridge index state propagation anomalies, undo cleanup races that can corrupt replica-side pages, and tablespace metadata not surviving a replica restart. Several of the replication fixes shipped in these releases were found exclusively through this mechanism.

A complementary test verifies that after a replica is stopped and restarted, its indexes are rebuilt correctly and queries return the same results as on the primary, targeting the subtle class of bugs where in-memory state diverges from on-disk state during recovery.

Crash consistency testing with dm-log-writes

We also introduced crash consistency testing built on the Linux dm-log-writes device-mapper target. The target records every block-level write and FLUSH/FUA boundary issued to the underlying device, producing a precise log of what reached storage and in what order. The test harness replays the log up to each flush point, mounts the resulting image, and starts OrioleDB on it, simulating a crash at every legal recovery point.

This exercises the parts of the engine that synthetic tests rarely reach: WAL durability boundaries, checkpoint atomicity, and the interaction between fsync ordering and undo cleanup. A bug that only manifests when power is lost between two specific writes is otherwise nearly impossible to reproduce; with dm-log-writes, it becomes a deterministic, debuggable failure.

Broader CI infrastructure

Beyond new test categories, the CI infrastructure itself was substantially expanded:

• The CI matrix now covers PostgreSQL 16, 17, and 18 as first-class targets. PG18 support involved a non-trivial amount of code work (skip-array scan keys, virtual generated columns, default-locale initialization, the new tuple_get_transaction_info table-AM hook), all validated against the same regression and isolation suites.
• A nightly cron pipeline builds and publishes Docker images for every supported Postgres major version, allowing users to run daily benchmarks and integration tests in addition to full OrioleDB CI matrix, surfacing flakiness and slow regressions long before they reach a release branch. Also, users can pull bleeding-edge Docker builds without waiting for a tagged release.
• Injection points are enabled for the pg_tests check type, allowing deterministic reproduction of races that previously required luck and a slow CI runner to hit.
• TAP test logs and core dumps are now collected as CI artifacts on failure, eliminating the "can't reproduce, log is gone" cycle that had previously slowed down post-mortems.
• The valgrind GDB integration (vgdb) is now wired up, so memory issues caught under valgrind can be inspected interactively instead of guessed at from a stack trace.

Recovery and logical replication hardening

Replication and recovery received the most attention in this period. The fixes span nearly every layer of the recovery stack: WAL replay, undo management, bridge indexes, checkpointing, recovery xmin tracking, and process coordination.

Several fixes address process-coordination issues during parallel recovery: signal handler registration for recovery workers, correct cleanup during process exit, prevention of deadlocks between the background writer and signal barriers, and proper wakeup of the startup process when recovery workers finish their work. The recovery leader now allocates the xmin queue exclusively, and globals are saved and restored across the drainer's checkpoint-stack walk so that drained fast-path-aborted state is torn down fully. Together, these eliminate a class of intermittent hangs that were difficult to reproduce but could stall replication.

Undo management

The undo subsystem is central to OrioleDB's MVCC implementation, and its interaction with replication is one of the most complex parts of the engine. Several classes of undo-related issues were addressed:

• Undo retain tracking was overhauled to correctly account for recovery retain locations. Previously, the undo cleaner could reclaim records that a replica-side scan still needed, causing "snapshot too old" errors. The retain logic now covers parallel index builds and background writer cycles as well.
• Recovery xmin invariants (runXmin >= globalXmin) are now enforced and explicitly tested on the secondary, closing a class of races where fast-path-aborted oxids left stale entries in the xmin queue and pinned the floor below where it belonged.
• Concurrent undo cleanup during row-lock chain walks is now tolerated gracefully instead of triggering assertion failures.
• Memory leaks in the undo chain traversal path were fixed, addressing a slow memory growth issue in long-running read workloads.
• The undo cleanup is now called regularly in a bgwriter loop, reducing storage footprint after long-running transactions. Also, it punches holes in partial undo/oxid/rewind files instead of leaving them on disk until the whole files become unused.

Bridge index operations

Bridge indexes, the mechanism that allows non-OrioleDB index types (GiST, SP-GiST, GIN, etc.) on OrioleDB tables, were a significant source of replication issues in earlier releases. Bridge index ADD, DROP, UPDATE, partitioning, vacuum, and WAL replay are now robust, including graceful handling of WAL records that reference bridge indexes that no longer exist on the replica. Additional fixes addressed a page_item_rollback assertion on same-transaction rollback, an out-of-bounds access during parallel vacuum on bridged indexes, and a missing bridge_ctid access on non-leaf tuples.

Logical replication

OrioleDB now participates in PostgreSQL's xidless commit/origin code path, so logical replication subscribers consuming changes from OrioleDB tables behave identically to subscribers consuming from heap tables. Logical replication insert conflicts are now reported to pg_stat_subscription_stats, restoring parity with PostgreSQL's own observability for subscription health. Replication of REINDEX TABLE for tables with secondary indices was also fixed.

Checkpoint and eviction reliability

The checkpoint and page eviction paths were hardened against edge cases that could leave corrupt metadata on disk:

• Eviction of trees that had never been fully checkpointed no longer leaves invalid .map files.
• Compressed trees correctly save evicted data in all cases.
• Checkpoint handling of temporary trees tolerates failures instead of crashing the postmaster.
• Root page eviction for tables with CTID primary keys was corrected.
• Sync-queue draining is now generalized and applied to oSysTreesLock so that the checkpointer no longer blocks indefinitely waiting on oTablesMetaLock.
• An infinite clock-algorithm loop in the page pool when the pool was fully exhausted was fixed.

B-tree internals

Several long-standing B-tree race conditions and waiter-chain bugs were resolved:

• A leftPageSpaceLeft assertion that fired when merge_waited_tuples rejected a waiter is gone; the merge path was rewritten with a global-budget gate and a split dry-run.
• The unlock_page_internal() wakeup loop was hardened against a rare missed-wakeup pattern.
• follow_rightlink() now honors BTREE_PAGE_FIND_TRY_LOCK instead of unconditionally blocking.
• A stale parent chunk pointer in IMAGE mode find_page was fixed.
• A waiter-insert path that reused a stale CSN now assigns a fresh one.
• A dangling indexRelation pointer in custom scan state, and an index-only-scan crash on secondary indexes, were both fixed.

Several memory leaks along these scan and index paths were also tracked down and fixed: in the bitmap-scan path, in orioledb_amgettuple's fill_itup/fill_hitup helpers, and in o_exec_bitmap_fetch().

New functionality

While the focus of this period is stability, two notable user-facing additions landed:

• SERIALIZABLE isolation: instead of rejecting SERIALIZABLE outright on OrioleDB tables, a new orioledb.serializable GUC selects the policy, with a table_lock mode that takes a heavyweight ExclusiveLock on every OrioleDB table touched by a serializable transaction as the default. Applications that previously had to either avoid SERIALIZABLE or fall back to heap tables now have a supported path.
• Backend-local page pool for temporary tables: a new LocalPagePool, sized by the orioledb_temp_buffers GUC, keeps temp-table pages in per-backend memory instead of routing them through the shared buffer pool. Temp data is visible only to the owning backend, so the previous routing wasted shared buffers, took cross-backend locks on every access, and forced defensive checks on hot paths. With local pages, temp-table workloads no longer compete with regular tables for shared buffer space and skip the locking entirely; temp B-trees also use a backend-local SharedRootInfo hash and free-extent list, decoupling them from the global checkpoint counter.

SQL compatibility improvements

Running PostgreSQL's regression tests against OrioleDB surfaced a number of compatibility gaps that were fixed:

• Hashing: OrioleDB switched from a custom hashing approach to using PostgreSQL's native hash functions (resolved via the equality operator's hash support). This fixed incorrect query results for indexes on name columns, composite types, and other non-trivial data types.
• TOAST: Oversized STORAGE MAIN attributes are now forced to out-of-line storage, matching PostgreSQL's behavior. Index builds on compressed toasted data no longer fail.
• Generated columns: ALTER TABLE ... DROP EXPRESSION and SET EXPRESSION are now handled correctly, and generated column behavior (both stored and virtual) matches PostgreSQL's own test expectations. ALTER TABLE rewrites involving virtual generated columns were fixed.
• Domain types: Columns with domain types and NULL missing-attribute values no longer crash.
• DDL: Table compression settings survive TRUNCATE, database size calculation now includes OrioleDB tables, NOT NULL is properly enforced during ALTER TABLE rewrite while honoring NOT VALID, and NOT NULL constraints stored in pg_constraint are now handled.
• Scan keys: Contradictory NULL scan keys on PG16 no longer return wrong results; SAOP/IN scans on primary keys whose table column order differs from the index column order are now correct.
• pg_amcheck support: OrioleDB tables can now be verified by the standard PostgreSQL pg_amcheck tool, integrating structural checks into existing operational workflows.

Diagnostic and observability improvements

Debugging storage engine issues requires good introspection tools. Several were added:

• orioledb_undo_size() reports undo log sizes, useful for monitoring undo growth.
• A new orioledb.replay_until_lsn GUC allows pausing WAL replay at a specific LSN, invaluable for reproducing and debugging recovery issues.
• A list_stuck.sh helper now dumps the stuck-page header and waiter chain on demand, replacing a previous workflow of manual GDB introspection.

Code quality

Beyond fixes, this period invested in making the codebase more maintainable:

• The WAL container parsing logic was refactored into a dedicated wal_reader module common to recovery, applying during streaming replication and logical decoding.
• Table and index descriptor lifetimes are now tracked via ResourceOwner, preventing use-after-free bugs from concurrent invalidation.
• Cached FmgrInfo lifetimes for comparators, hash, and exclusion operators are now pinned to descrCxt, eliminating a class of cache-stale fmgr lookups.
• Extensive comments were added to document non-obvious behaviour of different parts of the code.

Upgrading

OrioleDB beta16 is based on PostgreSQL 17.9/16.13, with PostgreSQL 18 supported in CI. These are newer, but also time-proven versions.

The recommended way to get started:

docker run -d --name orioledb -p 5432:5432 orioledb/orioledb

Looking ahead

The combined beta15+beta16 work represents a deliberate investment in the foundation. By running PostgreSQL's own test suite against OrioleDB, comparing primary and replica data in CI, exercising crash consistency at the block-write level, and systematically hardening recovery and replication, the project is building the confidence necessary for a production-ready release. The work in these cycles, less visible but no less important than new features, is what turns a promising storage engine into a reliable one.

We're also working on integrating SQLsmith, a randomized SQL query generator, into CI, so that OrioleDB is continuously exercised by query shapes no human would think to write. Combined with the regression, isolation, replication, and crash-consistency suites already in place, this will close one of the last remaining gaps in our coverage.

We encourage users to test the latest beta against their workloads and report any issues on GitHub.