1. As I began with answering the LLM’s questions, “How will the harness programmatically assert that memory resides on the stack (space_) for 200 bytes and on the heap (new char[]) for > 200 bytes without breaking encapsulation” was one of them. I didn’t know how to answer the question.

   Here’s what I learnt:

   In the case where the memory resides in stack, the start_ pointer, being a part of the LookupKey object, will reside within memory occupied by that object. A dynamic allocation on the other hand, will not.

   Hence, to verify whether memory resides in the inline buffer (space_) or heap without exposing private members, the solution is to compare key.mem()(the function that returns start_) against the memory address range of the LookupKey instance itself(&key marks the beginning of the LookupKey object, while &key + 1 marks its end):

1. As I progressed, I was encountered with the most fundamental question: why are test cases even written? What does passing all test cases imply? On the other hand, what does failing a few imply about our code?

   Test harnesses are a reflection of the constraints we already define and the outcomes we guarantee in Stage 1. Test cases are written to

   1. Immobiliize the architectural contract i.e. ensure that the code handles every case it claims it can.

   2. Verify whether the code guarantees deterministic failure routing rather than undefined behaviour by Intentionally forcing catastrophic environmental states.

   Verify every part of code except for, obviously, initializations. Conditions are hotspots to check.

2. What does a volatile benchmark accumulator mean?

   In benchmark routines, compilers actively optimize away code if they detect that a value is computed or read but never used.

   If you write a loop to sweep across memory purely to trigger AddressSanitizer (ASan) bounds checks:

   for (size_t i = 0; i < key.size(); ++i) {
       char c = key.mem()[i]; 
   

   An optimizing compiler (-O2 or -O3) sees that c is never read afterwards. It removes the loop entirely as dead code. Because the memory access loop gets eliminated, ASan never actually scans the bytes at runtime, making the sanitizer test useless.

   A volatile benchmark accumulator prevents this dead-code elimination:

   volatile char sink = 0;
   for (size_t i = 0; i < key.size(); ++i) {
       sink = key.mem()[i]; 
   }
   

   The volatile keyword tells the C++ compiler that writing to sink has visible side effects outside the program's observable state. The compiler is legally forced to emit instructions that execute every single memory read and write, ensuring ASan intercepts every byte access during the test sweep.

3. A very small learning but one worth mentioning: when writing code, we often change the names of our variables where we realise it’s required, but fail to implement that name change throughout the code. Hence it is important to replace obsolete nomenclature with what is the latest.

Before I describe the bug, you should know the meaning of the following keywords: index, ART and checkpoint.

Note: Checkpointing has a slightly more specific meaning in the current context; when you execute INSERT, rows go into memory and a Write-Ahead Log (WAL). Once the WAL hits a size threshold (default: 16 MiB), DuckDB triggers an automatic checkpoint. The checkpoint flushes memory buffers to disk and resets the log so disk space doesn't grow infinitely.

Core Issue

When inserting data into DuckDB, checkpointing occurs. The bug in #24848 is that DuckDB re-writes the entire ART index from scratch on every automatic checkpoint, rather than only writing the newly inserted or modified keys (delta).

As a table grows from 100,000 rows to 3,000,000 rows, saving the index changes from a tiny task into an O(N) operation. Every few seconds, the engine freezes to re-serialize millions of unchanged index keys.

The issue reporter proved this behavior by running a synthetic benchmark:

• Without a unique index, insert performance is flat.

• Disabling auto-checkpoints removes the periodic latency spikes entirely.

• Re-enabling default auto-checkpoints causes per-batch latency to spike linearly with total table row count.

Root Cause

In src/execution/index/art/art.cpp and storage checkpoint handlers, calling ART::Serialize traverses the entire radix tree from root to leaves regardless of whether nodes were modified during the current transaction window.

A potential optimization could be introducing dirty-subtree tracking:

1. Flag ART nodes as "dirty" when modified during writes.

2. During StorageManager::CreateCheckpoint, bypass clean subtrees and append only dirty nodes/blocks to storage.

3. Clear dirty flags post-checkpoint.

I'm currently working on a reproducible benchmark and investigating DuckDB's ART checkpointing path. I'll publish a follow-up covering the reproduction, root-cause analysis, and a patch if the investigation supports a viable fix.

Despite we had proven that the reintegration of ClearUpdates() stopped the OOM, my first decision was to inspect three source files: column_data_checkpointer.cpp, column_data.cpp, and standard_column_data.cpp. Why? Because the passing of the 35 assertions of the test established that the intervention prevents OOM from occuring. It did not although, establish that clearing update state was safe across every checkpoint path, transaction state, and concurrent access pattern. There’s several more reasons which will be mentioned later in the blog.

The central function was ColumnDataCheckpointer::FinalizeCheckpoint():

void ColumnDataCheckpointer::FinalizeCheckpoint() {
    for (idx_t i = 0; i < checkpoint_states.size(); i++) {
        auto &state = checkpoint_states[i].get();

        if (!has_changes[i]) {
            state.SetUnchanged();
            WritePersistentSegments(state);
        }

        state.original_column.ClearUpdates();
    }
}

Now,

1. Unnecessary strings were being copied into the StringHeap, then retained longer than necessary because the update state was not being cleared until checkpoint finalization. But since changes in a column can also be ineffective, the update state—as opposed to the original state stored in the data member—need not persist in every case.

2. ColumnData::Checkpoint() and StandardColumnData::Checkpoint() both invoke Checkpoint() followed by FinalizeCheckpoint(). However, both contain early returns that can bypass finalization. We therefore needed to trace both paths to verify which states reach the cleanup loop.

3. ClearUpdates() and UpdateInternal() acquire update_lock, but HasChanges(start_row, end_row) accesses updates without that mutex. We thus needed to verify whether the mutex inside ClearUpdates() is sufficient to establish safety or not.

I ran the following commands to investigate the above questions and insights:

2. Designing a regression test

I ran three searches:

The intention behind this was to examine DuckDB's existing test conventions; DuckDB uses its SQLLogicTest framework, which has its own syntax and conventions, such as how DuckDB expresses repeated updates, checkpoint operations, expected results, and persistent-database behavior.

Each search targeted a different aspect of our problem:

The searches led me to four particularly useful files:

• test/sql/storage/update/nop_updates.test_slow

• test/sql/storage/checkpoint_with_pending_updates.test_slow

• test/sql/storage/test_repeated_checkpoint.test_slow

• test/sql/upsert/upsert_basic.test

Based on the knowledge acquired from above operations, a new test was created at:

test/sql/storage/update/upsert_delete_checkpoint_memory.test_slow

It used a file-backed database with a 384 MB memory limit and a 1 MB checkpoint threshold. The test created a table with a primary key, a VARCHAR payload, and a deletion flag. It inserted 300,000 initial rows containing strings of approximately 385–387 bytes.

Each of ten rounds then performed two upserts against 100,000 keys, followed by deleting rows whose deletion flag was true. The final expected row count was 285,000.

The intent was to exercise the same combination of long strings, no-op updates, deletes, and checkpointing that characterized the original failure.

3. The first test run failed with an internal error

The first execution of the custom SQLLogicTest did not produce the expected out-of-memory error. It failed during the first upsert with:

The stack trace passed through RowGroupCollection::Update() and the physical update pipeline.

To investigate, the source was searched for GetUpdateStatistics().

RowGroup::Update() contained a dereference of the returned pointer:

MergeStatistics(column.index, *col_data.GetUpdateStatistics());

StandardColumnData::GetUpdateStatistics() could return null when neither the column's update statistics nor its validity column's update statistics existed.

This made the failure path worth examining, particularly given the earlier experiment in which immediate update-state cleanup had caused a similar error. However, the exact cause of this first SQLLogicTest failure was not conclusively established.

4. Disabling cleanup reproduced the memory regression

To determine whether ClearUpdates() affected the test outcome, the cleanup call was temporarily disabled in column_data_checkpointer.cpp.

A backup of the patched file was retained:

cp src/storage/table/column_data_checkpointer.cpp \
  /tmp/column_data_checkpointer.cpp.with_clear_updates

After disabling the call, the test was rebuilt and executed again. This time, it failed with the expected memory exhaustion:

As can be seen, the failure occurred during the upsert at line 30 of the custom test.

The original source was then restored:

cp /tmp/column_data_checkpointer.cpp.with_clear_updates \
  src/storage/table/column_data_checkpointer.cpp

After rebuilding unittest, the test passed:

The sequence was significant. Disabling cleanup reproduced OOM, while restoring cleanup and rebuilding yielded a passing test. This was strong experimental evidence that update-state cleanup affected the memory behavior.

It did not establish that the cleanup intervention was the best upstream fix, nor did it explain the exact cause of the earlier internal error.

5. Comparing the known-good and first-bad revisions

The investigation also prepared separate worktrees for the two revisions defining the regression boundary:

• Known-good parent: 763e942790c6cbc627ac401d38ce87dadec20514

• First-bad candidate: 8d61c6fb35de65a2cc98c8405ba3caaa5348deb7

The custom regression test was copied into both worktrees so that the test could be run without changing the original working tree:

cp test/sql/storage/update/upsert_delete_checkpoint_memory.test_slow \
  /tmp/duckdb-good-check/test/sql/storage/update/

cp test/sql/storage/update/upsert_delete_checkpoint_memory.test_slow \
  /tmp/duckdb-bad-check/test/sql/storage/update/

The recorded output from the unpatched first-bad revision showed the expected OOM failure. The patched test passed in the original working tree.

The supplied terminal record does not conclusively establish the final test outcome on the known-good parent. That comparison must therefore remain unconfirmed rather than being presented as a completed three-way validation.

This distinction mattered because a convincing regression test should distinguish the known-good implementation, the failing implementation, and the implementation with the proposed fix.

6. Inspecting the upstream fix changed the direction of the investigation

The next important development came from getting to know that an upstream resolution associated with issue #25162 already existed in PR #26291.

The relevant source change was in src/storage/table/update_segment.cpp, in UpdateSegment::Update().

Before the fix, the VARCHAR statistics path copied strings into the update segment's string heap before the effective-update filtering had completed. Thus, strings could be copied even when the operation ultimately had no effective updates.

The upstream change moved that copying until after the effective-update filter. If the update count was zero, the function returned before performing the unnecessary string allocations.

This addressed the mechanism at its source:

• The experimental ClearUpdates() intervention released update state during checkpoint finalization.

• The upstream fix prevented unnecessary string copies from being allocated in the first place.

The distinction was important for both engineering correctness and this blog. The cleanup intervention provided useful evidence about the update-state lifecycle, but the upstream implementation solved the problem at an earlier point in the operation.

7. Building the upstream revision in a persistent worktree

To validate the accepted implementation independently, the PR was fetched into a separate Git reference and checked out in a dedicated worktree.

The initial worktree lived under /tmp/duckdb-pr26291. The build was interrupted, and after returning to WSL the temporary directory was no longer available. Subsequent commands accidentally ran in the original ~/duckdb checkout, still at the first-bad revision. The focused upstream test then reported: No test cases matched.

That result meant the test had not run; it was not a failure of the upstream implementation.

The problem was resolved by recreating the worktree under the home directory:

~/duckdb-pr26291

The checkout was verified at commit:

b66f6070de3f115dd34cd3e21340223041f4728e

The build was then completed in that persistent worktree.

8. The focused upstream regression test passed

The upstream test was run with:

build/release/test/unittest \
  "test/sql/storage/update/noop_string_update_memory.test"

The result was:

All tests passed (4 assertions in 1 test case)

The focused test created a table containing a 65,536-byte string, checkpointed it, performed a no-op update, and checked the allocator measurement against a small threshold.

This test targeted the unnecessary allocation more directly than the ten-round workload. Passing it established that the upstream implementation satisfied its own regression assertion.

9. The original file-backed reproducer passed on the upstream build

The original repro.sql was copied into the upstream worktree. The database and WAL were removed before running it, ensuring that the experiment began with a fresh file-backed database:

cp ~/duckdb/repro.sql ./repro.sql

rm -f test.db test.db.wal output.txt

./build/release/duckdb test.db < repro.sql > output.txt 2>&1
rc=$?

echo "Exit status: $rc"

The recorded exit status was zero, and the output contained:

All ten rounds completed, and 285,000 rows remained after each round.

This was the first direct confirmation that the upstream-fixed build could execute the original workload from start to finish without the previous OOM failure.

Neighbouring tests were also run:

• nop_updates.test_slow: passed, 9 assertions.

• upsert_basic.test: passed, 154 assertions.

• nop_update_tpch.test_slow: skipped because TPC-H was required.

The skipped test was not counted as a pass.

10. Replacing the integer MiB measurement with raw bytes

The original reproducer reported allocator usage by dividing the byte count by 1,048,576 and casting the result to an integer. The output showed allocator_mib = 0 in all ten rounds.

That measurement was ambiguous: integer conversion can hide nonzero usage below one MiB.

Rather than modifying the original script, a separate copy was created: repro-pr26291-raw.sql

A Python script replaced the ten measurement queries. It required exactly ten matches before writing the file, preserving the rest of the workload. The revised query reported:

• allocator_rows: the number of rows returned for the ALLOCATOR tag.

• allocator_bytes: the reported raw byte total.

• rows: the table's row count.

The instrumented reproducer was run again against a fresh file-backed database on the upstream build.

The reported measurements were:

The workload again exited successfully and printed completed all rounds without failing.

The ALLOCATOR tag was present at every sample. Its reported usage was zero in eight rounds and 16 KiB in two rounds. The important observation was that the readings did not increase steadily across rounds, unlike the historical pattern that rose from approximately 80 MiB to more than 220 MiB before OOM.

11. Preserving the experiment and restoring the original working tree

After verifying the upstream implementation, the ClearUpdates() experiment was preserved separately:

cd ~/duckdb

git diff --binary > /tmp/clear-updates-experiment.patch

cp test/sql/storage/update/upsert_delete_checkpoint_memory.test_slow \
  /tmp/upsert_delete_checkpoint_memory.test_slow

The three modified source files were restored, and the custom regression test was removed from the original repository. The subsequent status output showed only untracked investigation artifacts, such as reproducer scripts, logs, the Parquet file, and supporting shell scripts.

These files were left in place rather than being removed indiscriminately, preserving the evidence needed for the investigation and technical write-up.

Conclusion

The investigation progressed from auditing checkpoint finalization and update-state ownership to creating a reproducer, testing a cleanup intervention, examining the accepted upstream change, and validating that change against both a focused regression test and the original workload.

The evidence supports the following conclusions:

1. The original workload reproducibly accumulated allocator memory and eventually exhausted the configured memory limit on the first-bad implementation.

2. Disabling ClearUpdates() reproduced OOM in the custom SQLLogicTest, while restoring cleanup and rebuilding yielded a passing test.

3. The upstream fix moved string copying after ineffective updates had been filtered out, preventing the unnecessary allocations at their source.

4. The upstream focused regression test passed, and the original file-backed reproducer completed all ten rounds.

5. The raw-byte measurements showed no steadily increasing allocator usage in the upstream run.

6. The temporary cleanup intervention was preserved for reference but removed from the working tree; the final account distinguishes it from the implementation merged upstream.

The final successful experiment was not merely a passing query. It combined a passing focused regression test, successful completion of the original ten-round workload, stable logical row counts, and raw-byte measurements that did not exhibit the original accumulating pattern.

I started by putting in the following commands serially into the gdb:

(gdb) set disassembly-flavor intel
(gdb) disassemble /r 'duckdb::ColumnData::~ColumnData()'

It returned: ❌️ No symbol table is loaded. Use the "file" command.

I first turned off the debugging-info download, and then loaded the DuckDB executable into GDB:

(gdb) set debuginfod enabled off
(gdb) file ./build/release/duckdb

Following was the error that occured:

As can be seen, GDB suffered an internal failure and aborted. To solve this, I ran the following command:

Here’s each keyword explained as well as the effect it has:

• DEBUGINFOD_URLS : Is an environment variable. An environment variable is a named setting that a program can read when it runs. Since nothing is directly adjacent to the ‘=’ symbol, the value of that variable is an empty string.

This prevents GDB from receiving any debuginfod server URLs through this variable. Debuginfod is a service that can provide debugging information, such as debug symbols, over the internet.

• -q: Only suppresses GDB's introductory message and some startup information.

• -nx: Normally, GDB may read configuration commands from files such as .gdbinit.

This option tells GDB not to read those initialization files and hence reduces the chance that existing GDB configurations interfere with your debugging session.

• -iex 'set debuginfod enabled off': The -iex tells GDB to execute a GDB command('set debuginfod enabled off') early in its startup process. 'set debuginfod enabled off' effectively instructs GDB not to use debuginfod to retrieve debugging information.

• --args: informs GDB of the following executable and its command-line arguments.

The investigation began with a source-level comparison between the parent of the first-bad commit, 763e942790c6cbc627ac401d38ce87dadec20514, and 8d61c6fb35de65a2cc98c8405ba3caaa5348deb7.

I initially inspected three files:

• src/storage/table/column_checkpoint_state.cpp

• src/storage/table/column_data_checkpointer.cpp

• src/storage/table/standard_column_data.cpp using the command:

git diff 8d61c6fb35de65a2cc98c8405ba3caaa5348deb7^ 8d61c6fb35de65a2cc98c8405ba3caaa5348deb7 -- src/storage/table/column_checkpoint_state.cpp

for column_checkpoint_state.cpp and so on.

They were inspected first because they felt like a sensible starting point as checkpointing was the step just prior to the deallocation step which was removed in the first bad commit. The following were the changes in each of the files:

What changed in ColumnCheckpointState?

The old column_data member was replaced by a reference named original_column, together with an optional non-owning parent_state pointer and an owning shared_ptr<ColumnData> named result_column.

This reflected a substantial architectural change. Rather than always checkpointing by moving segments out of the original column and rebuilding its segment tree, the newer design could construct or obtain a result column separately.

Methods that previously wrote to new_tree were changed to append through GetResultColumn().GetSegmentTree(). The checkpoint code also began distinguishing the original column from the column being produced by the checkpoint.

Note: original_column and result_column are not necessarily two distinct objects. In the unchanged-column case, SetUnchanged() assigns original_column.shared_from_this() to result_column. The result pointer can therefore refer to the original object. The diagram showing separate columns should be understood as illustrating the case where a new result column is created, not every checkpoint.

What changed in ColumnDataCheckpointer?

The checkpointing code increasingly used GetResultColumn() for operations on the resulting column, while retaining access to original_column for the original data and metadata.

The most significant difference appeared in FinalizeCheckpoint().

In the parent implementation, finalization handled the existing segments, reset the compression function, moved the new segments back into the column's tree, and called ClearUpdates().

In the first-bad implementation, that finalization logic had largely disappeared. The remaining logic handled the unchanged-column case by calling SetUnchanged() and WritePersistentSegments().

This was the first strong lead: the checkpoint design had changed, and cleanup that previously occurred at finalization had been removed.

What changed in StandardColumnData?

The validity data changed from an embedded object to a shared-pointer-based representation, and the validity checkpoint state gained a parent_state relationship.

This made the new checkpoint-state hierarchy clearer, but the pointer changes alone were not enough to explain the memory regression. The more consequential question was how the result column was installed into the live row group and what happened to the original column's update state.

They were not, however, the whole story. Further inspection of RowGroup::WriteToDisk() and ColumnData was needed to understand how the checkpoint state related to the live column and who owned the relevant memory.

2. Reproducing the regression under controlled conditions

The next step was to establish that the regression was real and to identify a repeatable boundary.

The parent revision, 763e942, passed the established reproduction. The first-bad commit, 8d61c6f, failed under the constrained-memory workload.

The workload used a 384 MiB memory limit, a 1 MiB checkpoint threshold, and a table containing 300,000 rows. Each payload was generated using a long VARCHAR string:

repeat('x',384) || (i%997)::VARCHAR

Each round performed two 100,000-row upserts using ON CONFLICT ... DO UPDATE, followed by a deletion of rows marked for deletion. The workload then recorded allocator memory and row counts.

The failing file-backed runs showed this pattern:

The failing allocation varied between runs: some reports showed a 16 MiB allocation failure, while another recorded a 256 KiB block failure. The exact allocation size was not perfectly repeatable, but the broader regression was.

For a more granular view of memory growth during individual operations, see repro_auto_trace.sql. This diagnostic variant records allocator usage after each upsert, deletion, and explicit checkpoint, helping distinguish memory growth during DML from changes observed after checkpointing.

Correcting the database-mode mistake

An important correction was distinguishing in-memory from file-backed execution.

When I used ./build/release/duckdb test.db < repro.sql, all the rounds were passed; OOM did not occur:

This is so because the above command does not use a database file. The canonical file-backed command was: ./build/release/duckdb test.db < repro.sql > output.txt 2>&1, which when I used, OOM occured in 10/10 runs.

The fact that changing only the execution harness changes whether the OOM occurs means the workload is timing/execution-sensitive. That was evidence of execution sensitivity, but not proof of the cause. Later redirected runs passed after the cleanup fix, so redirection itself cannot be treated as the established explanation.

3. Building the ownership model

Source inspection established the main ownership relationships.

A RowGroup holds shared pointers to its columns. ColumnData owns its segment tree and its updates pointer, which is a unique_ptr<UpdateSegment>. The update segment in turn owns its update-related state, including a StringHeap.

The checkpoint state has a different role:

• original_column is a non-owning ColumnData&.

• result_column is a shared_ptr<ColumnData>.

• parent_state is a non-owning link to another checkpoint state.

• The checkpoint state acts as temporary bookkeeping during checkpointing.

The change in RowGroup::WriteToDisk() was particularly relevant. The first-bad implementation added an AssignSharedPointer() call that replaced a row group's column pointer with the result column's pointer.

This led to a useful mental model: there is a live column, a temporary checkpoint state, and potentially a separately constructed result column. The checkpoint state itself does not automatically own the original column simply because it stores a reference to it.

(The first row group diagram represents the RowGroup dependencies in the good design).

At this stage, the architecture suggested that the new checkpoint path might be leaving state behind. It did not yet establish which retained state accounted for the memory increase.

4. Investigating whether the original column survived checkpointing

The next experiments focused on the lifetime of a specific UpdateSegment.

An early attempt to inspect arbitrary UpdateInternal() and WriteToDisk() calls produced different this addresses. Those observations could not be connected to one another. An update call on one ColumnData object was not evidence about the lifetime of another column seen in an earlier checkpoint.

To avoid that mistake, I restarted the experiment and tracked one exact UpdateSegment address from construction. I then installed a conditional breakpoint on its destructor.

During the bad run:

1. The tracked object was constructed through ColumnData::UpdateInternal().

2. Checkpoint finalization occurred while the object was still alive.

3. The workload progressed through the allocator-growth rounds and eventually reported OOM.

4. The tracked object's destructor breakpoint fired afterward, through destruction of its owning ColumnData and subsequent row-group cleanup.

A separate assembly inspection showed the destructor loading the UpdateSegment pointer from an offset within ColumnData and calling the update segment's destructor. In the observed case, that pointer matched the tracked address.

This was much stronger than merely noticing a constructor and a destructor breakpoint somewhere in the run. It connected one specific object's lifetime to the failure.

I thus concluded that the tracked UpdateSegment survived checkpoint activity and was destroyed only after the OOM had been reported in that run.

It did not, by itself, establish which member of the surviving object accounted for the accumulated memory.

5. Testing the first cleanup hypothesis

The source diff had revealed that ClearUpdates() had been removed from finalization. It was natural to test whether restoring it fixed the workload.

ClearUpdates() resets the owning updates pointer. As a result, destroying the UpdateSegment also destroys its members and their owned state.

To isolate checkpoint effects from the DML workload, I also used repro_checkpoint_isolation.sql. It suppresses ordinary automatic checkpointing with a 10 GB threshold and explicitly measures allocator memory before and after checkpoints. This was a diagnostic experiment with a different checkpoint configuration, not a replacement for the canonical regression reproducer.

There was also a separate experiment that reset updates immediately after updates->Update().

That approach caused an internal error because subsequent processing expected the updates object to remain available.

The early reset was therefore removed. The cleanup needed to be investigated at checkpoint finalization, not forced into the update method immediately after an update call.

6. Investigating the wrong allocation path: UpdateInfo

The next question was whether the retained memory came from allocated UpdateInfo entries.

The workload definitely executed UpdateSegment::Update(). GDB backtraces established the path from the update operator through DataTable::Update(), RowGroup::Update(), and ColumnData::UpdateInternal():

I initially instrumented UpdateInfo allocation using a live_update_info_bytes counter. That counter remained zero. The corresponding allocation instruction was not reached.

The reason became clear from the execution order in UpdateSegment::Update():

1. Run the statistics-update callback.

2. Return early if that callback produces a zero count.

3. Otherwise, determine the effective updates.

4. Return early again if there are no effective updates.

5. Only then initialize the update information.

GDB showed the effective-update function returning zero in the observed path, so InitializeUpdateInfo() was bypassed.

Consequently, the zero UpdateInfo counter was not evidence that the update operation did no work. It was evidence that I had been instrumenting the wrong allocation mechanism.

I thus went ahead to examine what happens before the effective-update check.

7. Discovering the VARCHAR string-heap path

The source revealed the decisive ordering.

For VARCHAR, GetStatisticsUpdateFunction() selects UpdateStringStatistics(). In this function, non-inline strings are copied through:

segment->GetStringHeap().AddBlob(str)

That happens during the statistics phase, before UpdateSegment::Update() determines that an update is ineffective and returns.

The payload strings in the reproduction were around 385–387 bytes long, so they were non-inline strings. The callback therefore had a path to copy substantial string payload even when the later effective-update check returned zero.

The next measurement hence targeted this specific path, rather than UpdateInfo.

Instrumentation difficulties and fixes

The string-heap counter took several iterations to instrument correctly:

• String code was initially inserted into TemplatedUpdateNumericStatistics(), causing compile failures when the compiler instantiated it for numeric types. The numeric template was restored.

• Some code was outside the relevant loop, so str, idx, and not_null_count were out of scope. The instrumentation was moved inside the existing UpdateStringStatistics() branches.

• The per-segment string_heap_bytes counter was initially declared at namespace scope. It was moved into the UpdateSegment class so each object had its own count.

• GDB function calls were interrupted by other threads hitting checkpoint breakpoints. The checkpoint breakpoint was disabled around the counter read; scheduler locking was left off so the multithreaded workload could continue normally.

• A function address calculated before a rebuild became stale. Calling it returned an impossible counter value, which was discarded. The symbol offset was checked with nm -C and the runtime address recalculated using the current executable mapping.

These were instrumentation errors, not evidence about the original bug.

The positive measurements

Once the counter and call path were working, GDB stopped directly in UpdateStringStatistics(). The backtrace showed the function being called through UpdateSegment::Update() on the real update path.

After an invocation, the live string-payload counter read 792,292 bytes.

Later in the same failing workload, it rose to:

• 114,132,226 bytes, approximately 108.8 MiB;

• 189,575,562 bytes, approximately 180.8 MiB;

• 187,641,118 bytes, approximately 178.9 MiB near the OOM.

The allocator had reached 226 MiB by round 3, followed by the allocation failure.

These were not perfectly aligned per-round samples, and the counter measured payload bytes rather than the allocator's full memory usage. Nevertheless, the counter's growth was close in scale to the workload's approximately 77 MB of input string payload per round and strongly supported the retained-string-memory mechanism.

8. The decisive checkpoint-cleanup control experiment

With the string-heap mechanism identified, I returned to the cleanup hypothesis and applied a controlled intervention to the first-bad commit.

The intervention restored the ClearUpdates() declaration and implementation and called:

state.original_column.ClearUpdates();

at the end of each ColumnDataCheckpointer::FinalizeCheckpoint() state-processing iteration.

I did not restore the old segment-movement implementation, the compression reset, or the old new_tree movement logic. The intervention was intentionally narrow.

The cleanup method was restored as:

updates.reset() under the existing update lock.

The call was placed after the conditional that handles unchanged columns, so cleanup ran for each checkpoint state. The earlier experimental reset inside ColumnData::UpdateInternal() was removed.

Instrumented result

The file-backed workload completed all ten rounds. Allocator memory was:

The live string-payload counter was zero at the checkpoint sample points in that cleanup-enabled run. A zero before and after a particular checkpoint is not by itself proof that the checkpoint released memory within that invocation. The useful combined result is that the workload completed and allocator usage remained bounded instead of following the earlier accumulation trajectory.

This was the strongest causal experiment: the same reproduction failed with the original first-bad checkpoint cleanup and completed when the cleanup was restored at finalization.

9. Removing instrumentation and validating the final patch

Once the cleanup-control result was obtained, I removed the temporary measurement code from update_segment.cpp and update_segment.hpp. The final diff then contained only three source files and seven insertions:

1. Add the ClearUpdates() declaration in column_data.hpp.

2. Implement ColumnData::ClearUpdates() in column_data.cpp.

3. Call state.original_column.ClearUpdates() in ColumnDataCheckpointer::FinalizeCheckpoint().

git diff --check produced no output. A separate diff check confirmed that the temporary instrumentation files no longer had changes. ColumnData::UpdateInternal() was also inspected to ensure that the invalid early-reset experiment was gone.

After a successful rebuild without instrumentation, I ran the exact file-backed reproduction ten times. Each run used a fresh database and WAL, and the harness required both a zero exit status and the final completion message while rejecting any OOM message.

All ten runs passed:

This is the final successful experiment to report. It is stronger than a passing run with temporary instrumentation because the patch was tested after that instrumentation had been removed.

Conclusion

The investigation has identified a specific cleanup regression and a targeted change that prevents memory accumulation in the reproducer. The fix is promising, but the investigation is not complete: a permanent regression test and broader validation against DuckDB's test suite remain outstanding.

In the next article, I'll document the patch, the reasoning behind the change, the design of the regression test, and the results of testing it against the relevant DuckDB code paths. I'll also document any failures encountered and the further changes needed before considering the contribution ready for an upstream pull request.

The investigation revealed how an update could copy string data into an UpdateSegment's StringHeap before DuckDB determined that the proposed update had no effect. Because the early return occurred after those allocations, and the UpdateSegment was no longer explicitly cleared during checkpoint finalization, the copied string data could remain allocated across checkpoint cycles.

Tracing this sequence narrowed the investigation to the code responsible for the memory accumulation and provided a concrete direction for understanding the regression. The following traces the sequence of events that caused the memory accumulation, introducing each relevant concept alongside the code path where it becomes necessary:

ColumnData is the BlockHandle of DuckDB columns in a given table i.e. it’s the in-memory representation of a column.

It has 2 members which are of relevance: data and updates. data is the existing column data, organized into segments. updates is a smart pointer(unique_ptr) to an UpdateSegment.

Now what is UpdateSegment? When DuckDB processes the updates which are to be made to a column, it can create an UpdateSegment object to manage that work.

• For this investigation, the important fact is that UpdateSegment contains a StringHeap.

What is a StringHeap? Strings are processed in the form of batches. DuckDB represents individual string values using an object called string_t. Short strings are stored directly inside the string_t object (inline), whereas longer strings are represented using metadata, including their length, and a pointer to character bytes stored elsewhere. StringHeap is a separate object that stores copies of these character bytes(given that the copy fulfills certain conditions), allowing them to remain valid independently of the original storage.

• Because ColumnData owns updates through a unique_pt, destroying or resetting that pointer destroys UpdateSegment, consequently destroying its string heap.

• When an upsert arrives, DuckDB

  1. Enters the update path i.e. it executes UpdateSegment::Update(),

     ColumnData::UpdateInternal()
         ↓
     UpdateSegment::Update()
     

  and then,

1. Calls its statistics-update callback.

What is a callback? A callback is a function, a lambda, a function object, or any other callable mechanism in C++ passed as a parameter into another function so that the receiving function can invoke it at a point of its choosing.

For VARCHAR values, UpdateStringStatistics() copies non-inlined strings into StringHeap through AddBlob().

StringStats::Update(stats.statistics, str);

if (!str.IsInlined()) {
    update_data[idx] = segment->GetStringHeap().AddBlob(str);
}

Conceptually, AddBlob() copies the string bytes into storage managed by the heap and returns a string representation referring to the copied data.

• After the statistics callback finishes, DuckDB checks whether the proposed values actually differ from the existing values. The code calls:

  count = get_effective_updates(
      update_format, ids, count, sel, base_data, vector_offset
  );
  
  if (count == 0) {
      return;
  }
  

  When the proposed values are the same as the existing values, the count = 0. This implies no effective update. Consequently, the function returns before InitializeUpdateInfo() is reached i.e. before creating UpdateInfo entries. This is relevant as the initialisation of UpdateInfo objects could have been a possible cause of the memory accumulation and consequently, the OOM.

  Here is the full sequence:

  

  The key is that step 3 has already allocated and copied the string before step 5 rejects the update. Returning early does not undo those copies.

• Before the first bad commit, checkpoint finalization called ClearUpdates(), whose implementation was:

  void ColumnData::ClearUpdates() {
      lock_guard<mutex> update_guard(update_lock);
      updates.reset();
  }
  

  Here, updates is a unique_ptr<UpdateSegment>. Resetting it destroyes the owned UpdateSegment, including its StringHeap. That released the heap's allocations.

• In the first bad commit, the previous ClearUpdates() call disappeared from checkpoint finalization. Consequently, the UpdateSegment and its heap could survive checkpointing instead of being reclaimed there. As subsequent updates processed more strings, additional character bytes could accumulate in that surviving heap.

I’ll soon publish the investigation notes behind this article, documenting the experiments I ran, the failures I encountered, and the reasoning behind each debugging decision.

That will be followed by a detailed account of the contribution process itself: tracing the issue to its root cause, developing the patch, designing and running regression tests, validating the fix, and working toward an upstream pull request.

The following are the 5 things I learned, plus a bonus tip, while completing Stage 2:

• Inquire into the purpose of each fn and variable that the LLM tells you to define.

When writing a fn from scratch, we can end up endlessly rewriting the function signature and still write code that doesn't fulfil its intended role. This vagueness stems from not clearly knowing that role. Hence, clarify the scope of the role that is to be fulfilled by each of the functions and variables designated by the LLM.

Consider yourself neither superior nor inferior to the LLM; it is capable both of making mistakes and providing us with valuable methods. Question each of its decisions.

• Before defining fns or variables, check the accessibility label you’re putting them under(private, public).

In the members we define, we must be thoughtful with respect to who must have access and what must remain private.

• Take a pen and paper, and state the logic of the mechanisms being used in your fns.

Rewriting code can become messy. Code must only be written as a reflection of the logic you have already cleared out on paper. This is an insight only understood upon implementing it first-hand.

• Aim at reducing redundancy(for example: avoid recalculation and copies by defining variables that store the calculated parameters and passing values by reference, respectively).

I suppose this is intelligence only built after accumulating some experience.

• Correctness trumps performance on edge cases.

Validity is more innate of a requirement than speed.

The above five points can be summarised into one robust rule: be more thoughtful with the smallest of the decisions you make; be Socratic. One may use these insights as a checklist when writing code for their own projects.

Bonus tip:

• Shallow copying of an owning raw pointer that points to a heap-allocated resource will result in a double-free error upon destruction. Hence, the copy assignment and the copy constructor need to be deleted for such instances.

LookupKey(std::string &userkey, uint64_t sequence_no)
        {
            //make base128 varint for the length of the userkey, put into space_
            //kstart_ += size in bytes of the varint
            //in the remaining space put userkey and trailer using memcpy
            //end_ = kstart_ + decoded_varint + 8;
        }

To this, the LLM (which I’m using in the 6-Stage process) responded with:

If userkey.size() exceeds this capacity, memcpy will trigger stack corruption. The constructor must verify the required memory footprint. If it exceeds 200 bytes, it must dynamically allocate memory (new char[]), update start_ to point to the heap, and implement ~LookupKey() to delete[] the memory if a dynamic allocation occurred.

But wouldn’t that cause heap fragmentation and consequently cache misses? We could have put a length limit when taking in the UserKey, but that would be invalid in cases where the key has a fixed length, such as a phone number. But aren’t cache misses a concern when the CPU has to consecutively retrieve memory? LookupKey, being a temporary search object, won’t be retrieved much I guess. Is that why it’s okay for us to dynamically allocate memory?

This is why it is justified for our code to use heap allocations for arbitrarily large keys:

• First of all, when it comes to edge cases, correctness must trump performance, as validity is more innate of a requirement than speed.

• And my guess was correct—because LookupKey is destroyed at the exit of the Get() call, any dynamically allocated memory block is immediately returned to the thread-local allocator cache. It does not linger in memory or degrade the long-term memory layout of the LSM-tree.

• Also, a single 200+ byte key already spans multiple 64-byte CPU cache lines regardless of whether it resides on the stack or heap. Cache hits are hence inevitably impossible.

• Lastly, SkipList traversal requires O(log N) key comparisons per search. Memory allocation for LookupKey executes exactly once per Get() call. For large keys, sequentially reading and comparing(using memcmp) hundreds of bytes across dozens of node evaluations consumes more CPU cycles than a single dynamic allocation. Hence, the bottleneck for large key queries shifts from the memory allocation overhead to the memcmp byte-by-byte comparison overhead in the SkipList search traversal.

git bisect start
git bisect good v1.4.4
git bisect bad v1.5.2

I used the following automated evaluation script (bisect_test.sh) to detect the exact commit where the memory threshold violation was taking place:

#!/bin/bash
make RELEASE_BUILD=1 > /dev/null 2>&1
rm -f test.db test.db.wal
./build/release/duckdb test.db < repro.sql > output.txt 2>&1
grep -q "Out of Memory Error" output.txt
if [ $? -eq 0 ]; then
    exit 1
else
    exit 0
fi

• /dev/null is a special Linux file that accepts data and immediately discards it. 2>&1 means “stderr (2) → wherever stdout (1) is currently going”. Since stdout is already going to /dev/null, stderr also gets sent there. The result of that line is that you won't see the build output, even if the build fails.

• grep -q "Out of Memory Error" output.txt searches output.txt for Out of Memory Error. -q means quiet mode; it only prints whether the matching line was found, rather than the line itself.

• $? stands for the exit status of the immediately preceding command. grep returns 0 when it finds the given text, and 1 otherwise. OOM occurrence means that the test failed; hence, if the OOM message wasn't found, the script exits with 0, meaning the test passed.

Using grep’s exit status as a signal for git bisect run ./bisect_test.sh, the automated bisect identified the first commit satisfying the test harness's definition of ‘bad’. I subsequently examined the harness itself and identified an important limitation: it validated only the presence of the expected OOM message, not whether the build and DuckDB execution actually succeeded(more on this in the P.S. section).

Using the reproduction script given on the issue page, I was able to reproduce the error:

PS:

There is one risk to the evaluation script that came to my mind after the reproduction of the OOM: since the if conditions in the script only check grep’s status, and not DuckDB's actual exit status, there could be a scenario where "Out of Memory Error" isn't present, but the line ./build/release/duckdb test.db < repro.sql > output.txt 2>&1 fails for some completely different reason, and the script returns 0 and tells git bisect that the commit is good simply because of the absence of the aforementioned OOM keywords.

First is being a purist, i.e. renouncing LLMs altogether and scraping documentation, subreddits, YouTube, and other such platforms for knowledge. As I have experienced, this can get very time-consuming, especially for a beginner, i.e. someone who is not familiar with this method of acquiring the required knowledge, and can become stressful when one is not able to get the answers to their questions even after spending that much time.

Second is being unhealthily reliant on AI; for example, asking for the complete implementation code from AI, reverse engineering it to understand what is happening, and mistaking the sequence of these actions for ‘building’. Such a person cannot even be considered a programmer, given that they haven’t opened a code editor, encountered a single error, or even run one of their ‘built’ implementation files.

Personally, I perceive AI as ultimately a source of thoughts, one which can take both the form of a teacher and, most importantly, an interrogator. I thus want AI to play those exact roles—question my every decision to ensure that each one is well thought out, and acquaint me with concepts or syntax which I do not already know, which are required to build the project.

Thus, driven by the desire to be thorough but not primal, I use the following 6-stage process to navigate the building:

This makes the entire process hands-on and makes sure I do not bypass the deepest, nastiest depths that programming can expose one to, and instead forces my own analytical thinking to find a way out.

Before we describe the problem, I would like to share some keywords that are necessary to understand it.

• Memory Leak: Occurs when a program allocates RAM, finishes using it, but forgets to deallocate it. The unreferenced RAM remains marked as "in use" indefinitely.

• Database Index: Linear searching is expensive for a large database. A database index is an additional data structure(which mirrors the database's contents) maintained alongside a database to make specific data lookups faster. B-tree, hash table, skip list, etc. are possible implementations.

  DuckDB uses an index structure called an Adaptive Radix Tree (ART).

• Upsert (ON CONFLICT ... DO UPDATE): A database command instructing: "Insert this new row. If a row with the same primary key already exists, overwrite the old row instead."

• Version Tracking & Undo Buffers: When a database modifies or overwrites a row, it creates temporary copies (versions) of the old data in RAM so concurrent read operations are not interrupted.

• Checkpointing: The background process that takes modified data sitting in fast RAM, writes it permanently to Disk, and deallocates the temporary RAM blocks used during the modifications.

So, here’s the issue:

Normal Operation (DuckDB v1.4.4)

1. An ON CONFLICT ... DO UPDATE query executes.

2. The database allocates RAM to hold updated row versions and update the index tree.

3. A Checkpoint triggers: modified rows are written to disk.

4. The database deallocates the obsolete version buffers and old index nodes from RAM.

5. Result: Total memory usage stays flat at ~50 MiB over millions of operations.

Defective Operation (DuckDB v1.5.x)

1, 2 and 3 execute the same as DuckDB v1.4.4.

1. The Bug: A code omission in v1.5.x skips the deallocation step for the temporary nodes used by the index as scaffolding and version-tracking structures created during conflict resolution.

2. Result: The system holds onto ~6 MiB of dead RAM per update cycle. Memory consumption rises continuously until RAM is exhausted, causing an Out-Of-Memory (OOM) process crash.

Downstream Mitigation (SpiceAI):

A third-party project using DuckDB observed these OOM crashes on v1.5.x. They changed their configuration file to bundle DuckDB v1.4.4 instead. This circumvents the bug for their users but does not repair the code.

Upstream Resolution (DuckDB Engine):

The mitigation made it clear that the actual defect resides inside DuckDB's C++ source code between version tags v1.4.4 and v1.5.2. Fixing it requires identifying the missing deallocation call in the checkpoint or index vacuum logic and adding a pointer reset, an explicit delete, or a missing traversal loop to free orphaned nodes.

I will shortly document all the details regarding the investigation and, hopefully, the execution of the solution to this issue. Until then, stay tuned!

With the build successful, I began the process of reproducing the double free.

Creating a Reproducible DuckDB Workload

To prevent the system from reproducing ~900MB of data during each test, I generated the output.parquet file and stored it in the DuckDB folder. I made a file named repro.sql, which, copied from the original, contained the following:

SET memory_limit = '6.4g';

FROM 'output.parquet'
SELECT
    FIRST(columns(*)),
    LIST(value ORDER BY day)
GROUP BY id
ORDER BY id, day;

Configuring AddressSanitizer for Targeted Diagnostics

I experimented with:

ASAN_OPTIONS=verbosity=1

This generated large amounts of runtime diagnostic noise, including:

failed to intercept ...
Unregistered root region ...
poisoning/unpoisoning ...

These are ASan runtime diagnostics, not proof of the target bug; the ASan output required a change.This is the configuration I finally chose to go with:

ASAN_OPTIONS='symbolize=1:halt_on_error=1:abort_on_error=1'

Capturing and Verifying ASan Test Results

Further, I established the following shell sequence:

set -o pipefail
./build/debug/duckdb < repro.sql 2>&1 | tee asan-run-01.log
echo "exit code: ${PIPESTATUS[0]}"

• set -o pipefail tells Bash to treat the whole pipeline as failed if any command in that pipeline fails.

• < takes the contents of the RHS and connects them to LHS's stdin.

• > takes the contents of the RHS and connects them to LHS's stdout.

• The keyword tee does two things simultaneously:

  1. It shows what’s on the other side of the pipe( | ) on the terminal.

  2. Store the same into a file mentioned after tee.

• echo "exit code: ${PIPESTATUS[0]}" gives the exit status of the first command(./build/debug/duckdb < repro.sql 2>&1) in the pipeline.

The following flowchart summarises what this pipeline does:

Part 2 — Establishing the Failure Conditions

Testing Memory Limits

I tried setting 6.4GB as the memory limit, but was unable to reproduce even the OOM. I lowered it down a notch, and OOM was successfully reproduced:

Varying Memory Limits and Thread Counts

Though even now, I was unable to reproduce the double-free error. I thus decided to experiment by varying the number of threads and the memory limit, ranging from 1 to 20 threads and 512MB to 4GB(since 6.4GB was the verified upper limit).

I executed the variations for 512MB and 1GB:

• memory_limit = ‘512MB’

  

  memory_limit = ‘1GB’

  

Repeating Configurations Instead of Testing Once

Then I remembered a segment of the issue that was submitted on GitHub:

I, on the other hand, was running each configuration only once.

Automating the Reproduction Matrix

To run each configuration multiple times would require manually creating and executing each iteration. This would be time-consuming. I automated the process using simple for loops in a shell script(here’s a link to the code).

Since 512MB and 1GB were already manually gone through, I chose to execute only the case of 2 and 4GB:

Part 3 — Evaluating the Reproduction Results

The Double-Free Remained Unreproduced

As can be seen, unlike the OOM exception, I couldn’t reproduce the double-free. There were one of the two choices I could make from this point onwards:

1. Build an earlier version of DuckDB and try on that version all the tests that I did on the current one.

2. Abandon this issue as a whole and find and examine another one.

Evaluating the Upstreaming Path

Reproducing the issue against the previous DuckDB version would require rebuilding the older environment and repeating the configuration work from scratch.

More importantly, the issue was posted a year ago; the current DuckDB codebase has since moved forward. Perhaps the bug existed in the affected release, but the relevant cleanup behaviour has changed since then. Hence, perpetually testing only the latest source would not necessarily reproduce the historical issue.

Also, a patch developed against the older revision would therefore need to be reworked against the current version before it could serve as a meaningful upstream contribution.

Decision: Moving to a New PR Candidate

Given the cost of reproducing the historical environment and the additional work required to carry any resulting fix forward, I decided to stop pursuing this issue and find a new PR candidate.

P.S.

While performing the ASan tests manually, the cursor would hang upon executing my pipeline sequence. To check whether DuckDB was still executing, I would run the following command in another WSL terminal:

ps aux | grep '[d]uckdb'

There is a question that would very commonly arise in the minds of anybody looking at the above command: why are square brackets ([ ]) involved in it?

The square brackets are a small regex trick. [d] is a bracket expression matching the character d, so [d]uckdb still matches duckdb. The difference is that the grep process itself contains the literal pattern [d]uckdb in its command line. It therefore does not contain the contiguous string duckdb, preventing grep from matching itself.

It is a tiny trick, but useful enough to remember ^_^.

The MSYS2 CLANG64 Attempt

I have been using an MSYS2 CLANG64 terminal for my LSM tree building and compiling, so I made the mistake of using the same terminal for building DuckDB.

The initial attempt was to build DuckDB with sanitizers using the following command:

CC=clang CXX=clang++ GEN=ninja BUILD_SANITIZERS=1 make debug

The build encountered several toolchain and compatibility problems, including:

lld: error: unknown argument: --exclude-libs

and POSIX-related compilation problems such as:

getpid undeclared

DuckDB's sanitizer build configuration thus exposed compatibility problems when I tried to use it from the MSYS2 CLANG64 terminal.

Another option was to shift the project into a native WSL2/Ubuntu environment, so I chose to do that. Further, I installed the required toolchain using the following command:

sudo apt update
sudo apt install -y build-essential cmake ninja-build clang llvm git

The AddressSanitizer Runtime Problem

After moving to WSL2, the sanitizer build reached the linking stage but failed to locate AddressSanitizer runtime libraries:

cannot find ... libclang_rt.asan_static.a
cannot find ... libclang_rt.asan.a
cannot find ... libclang_rt.asan_cxx.a

I initially thought that the Clang/ASan installation might be incomplete. But they were eventually found. I ran a standalone ASan program successfully with:

clang++ -fsanitize=address /tmp/asan_test.cpp -o /tmp/asan_test

This established that:

• Clang was installed correctly.

• compiler-rt was installed.

• AddressSanitizer itself worked.

• The failure was specifically related to where DuckDB's Clang build expected the sanitizer runtime libraries to be located.

I decided to discontinue that problematic Clang configuration for a change; GCC's sanitizer runtime was used instead of the problematic Clang/LLVM runtime lookup. That fixed the aforementioned issue.

Part 2 — Diagnosing "No Space Left on Device"

When I tried to build now, it failed with:

Fatal error: 
... .o: No space left on device

Verifying Available Disk Space

At first, this appeared to be a conventional disk-space problem. I checked the available Windows filesystem space using:

df -h /mnt/c

It showed:

Filesystem Size Used Avail Use% Mounted on 
C:\        476G 341G  135G  72%     /mnt/c

So there were approximately 135 GB of free space on C:. That made a conventional “disk is full” explanation unlikely.

Testing the Filesystem

To test this directly, I then attempted to write a 1 GB file to the exact filesystem containing the DuckDB source:

dd if=/dev/zero \ 
of="/mnt/c/Users/Akshat Kumar/duckdb/test_space.tmp" \ 
bs=1M count=1024

It reported:

1024+0 records in 
1024+0 records out 
1073741824 bytes (1.1 GB, 1.0 GiB) copied

So it’s clear that WSL could successfully allocate and write another 1 GiB on /mnt/c. Why did it then show “No space left on device”?

Moving the Build to the Native Linux Filesystem

The problem was this:

My WSL build was still located on the Windows filesystem. So WSL had to translate Linux filesystem operations into Windows filesystem operations.

Every operation such as open(), close(), stat(), mkdir(), rename(), write(), fsync() has to cross that boundary.

A large C++ build(such as that of DuckDB) performs huge numbers of these operations, because every compilation unit produces intermediate files, object files, dependency information, temporary files, etc. Ninja was similarly doing many of them concurrently.

The fix was to build on the native Linux filesystem. Doing this completely circumvented the Windows filesystem translation layer, which was previously involved.

An expected conflict arose as a result. CMake detected that the existing CMakeCache.txt belonged to a different build location/environment; the old build directory had previously been generated from the Windows environment. CMake therefore had cached Windows paths such as:

C:/Users/Akshat Kumar/duckdb/build/debug

while the new WSL environment referred to the project through paths such as:

/home/akshat/duckdb

I hence removed the stale generated build directory using rm -rf build/debug and regenerated it from the WSL environment.

Part 3 — Reducing Ninja Concurrency

I initiated the build using:

GEN=ninja BUILD_SANITIZERS=1 make debug

CMake configured successfully:

Configuring done
Generating done
Build files have been written to:
/home/akshat/duckdb/build/debug

Ninja then appeared to remain at:

[0/425] Building CXX object.../ub_duckdb_expression_binders.cpp.o

It didn’t move forward. To track the status of the build, i.e. whether it was progressing or whether it had frozen, I tried to open other terminals, but for some reason, they would hang the moment I would attempt to enter WSL using commands like wsl and wsl -d Ubuntu.

Why the Build Appeared to Hang

Upon examining the terminal, I saw this amongst the lines generated during the build:

-- Build workers: compile=12

-- Build workers: compile=12

The build repeatedly struggled around ub_duckdb_expression_binders.cpp.o because the large DuckDB C++ translation units, debugging information, and sanitizer instrumentation were making each compiler process memory-intensive.

Reducing Compile Workers with Ninja

Instead of continuing through the Makefile wrapper, I invoked the generated Ninja build directly with reduced concurrency:

cd ~/duckdb/build/debug
ninja -j 2

This restricted Ninja to two concurrent build jobs rather than the previous 12 compile workers.

The build then completed successfully.

Before I describe the problem, I would like to acquaint you with some keywords that are included in the description of the issue, which I feel are quite significant for understanding it.

• Stack Unwinding

  Say a function runs and causes an exception. To find a handler for the error, the C++ runtime goes backwards line by line until it reaches one for that exception(if no handler exists, the program terminates). As it unwinds, it destroys the already-constructed local/automatic objects whose scopes are being exited. This process is called stack unwinding. Their destructors then release the resources those objects own, such as memory, file handles, locks, etc.

  

  Eg:

  void foo() {
      MyObject obj;
      do_something();   // throws an exception
  }
  

  C++ walks backward through the chain of function calls, destroying the local objects that have already been constructed, until it finds code that catches the exception.

  

• OOM(Out-of-Memory)

  An OOM condition occurs when a program requests more memory than the system or the program's configured memory limit allows it to use.

• Double-free

  This exception occurs when a destructor attempts to free memory that has already been freed.

So, here’s the issue:

• The provided SQL query groups data and builds lists of values. DuckDB stores this in a temporary dictionary structure called a GroupedAggregateHashTable.

• The query intentionally generates excessive data, violating the artificial 6.4GB memory constraint. This triggers an OOM. Stack unwinding begins, and destructors are run to free all active pointers before the operation fully aborts.

• During this emergency teardown sequence, the destructor for the aggregate states (the lists and strings) executes a free operation on its memory. Due to a structural flaw in how DuckDB handles OOM aborts, a secondary cleanup path also attempts to free those exact same pointers. This executes the fatal double-free.

The solution is simply the following:

In C++, attempting to free a nullptr is safely ignored by the runtime. Setting the pointer to null will thus sever the connection to the reclaimed memory and neutralise the double-free hazard. That is the core change that I will effectively implement to fix the issue.

I will shortly document the entire process of the execution of my solution. Until then, stay tuned!

Initially, when I ran my test harness with just two threads, each with one insert function, it compiled and ran successfully. I then increased the number to 10000 insertions per thread to see its performance, attempting to witness a potential hardware collision. And I did. It gave me a long output, the summary of which was:

SUMMARY: ThreadSanitizer: data race src/skiplist.h:37 in SkipList<int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::Insert(int const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
==================
ThreadSanitizer: reported 4 warnings

I checked line 37 and eventually understood what the problem was—candidate.value not being an atomic variable meant it was capable of being written to simultaneously by two distinct writing threads.

The fix is to make the value variable atomic.

Firstly, I tried:

wsl

It showed:

Windows Subsystem for Linux has no installed distributions.

You can resolve this by installing a distribution with the instructions below:

Use 'wsl.exe --list --online' to list available distributions and 'wsl.exe --install <Distro>' to install.

I tried:

wsl --list --online
wsl --install -d Ubuntu

It would tell me to reboot to see the changes, but it would show the earlier (”Windows subsystem.. ..to install.”) output. Upon GPT’s suggestion, I put:

wsl --status

It showed:

Default Version: 2
WSL1 is not supported with your current machine configuration.
Please enable the "Windows Subsystem for Linux" optional component to use WSL1.
WSL2 is unable to start since virtualization is not enabled on this machine.
Please ensure the "Virtual Machine Platform" optional component is enabled and virtualization is turned on in your computer's firmware settings.

Enable "Virtual Machine Platform" by running: wsl.exe --install --no-distribution" 

I put:

wsl --install --no-distribution

This required another reboot. After that, I checked whether my Virtualization was enabled(Go to Task Manager→Performance→CPU). Thankfully, it was enabled(if it’s not, you need to enable virtualisation in your BIOS/UEFI).

The next two commands really revealed what was inhibiting the wsl installation. For wsl to install, both of the following commands..

Get-WindowsOptionalFeature -Online -FeatureName VirtualMachinePlatform

Get-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux

..must return Status: Enabled among its outputs. But in my case, for the latter one, it returned Status: Disabled. To enable it, I had to run:

dism.exe /online /enable-feature /featurename:Microsoft-Windows-Subsystem-Linux /all /norestart

The operation completed successfully. I rebooted the pc as per GPT’s suggestion. Finally, for the last time, when I put in:

wsl --install -d Ubuntu

It started its proper installation and asked for my Linux username and password.

Ownership:

Here are some premises that helped me understand the ownership aspect of the components better.

• X owning Y implies that the destruction of the former will cause the destruction of the latter. X having access to Y means X has access to the allowed members of Y.

• optional_ptr is a DuckDB-defined pointer. It is a nullable, non-owning ptr.

• The members memory_p and &memory serve the role of giving ownership and access to a BlockMemory, respectively, to BlockHandle. memory is an alias, not a second object.

Proceeding with the deconstruction, there were many questions I encountered. After much clarification and rigorous questioning, the following is what my understanding crystallised into:

Who other than BlockHandle can own a BlockMemory object?

From our current knowledge of the system, only BlockHandle, and possibly other objects that acquire a copy of memory_p, can own a BlockMemory object.

What is a FileBuffer? What is the difference between a FileBuffer object and a Block?

A block is a fixed-size unit/region that can represent ephemeral/intermediate data or data from persistent storage.

When DuckDB needs, say, Block X, it cannot directly operate on the disk bytes efficiently. It loads that block into RAM. That chunk of RAM containing the loaded bytes needs an object to represent it and provide other objects with functions to interact with this chunk. That object is a FileBuffer.

Summary:

A block is a fixed-size unit/region that can represent an ephemeral/ intermediate data or data from persistent storage. A FileBuffer is an object representing the RAM buffer that holds the block's bytes when that block is loaded into memory, along with metadata and operations related to that buffer.

What is the difference between FileBuffer and BufferHandle? If FileBuffer is already the representative of the buffer in RAM occupied by a block, what is BufferHandle for?

When a reader requests access to a block, that access is granted using an optional_ptr to the FileBuffer containing the block. This ptr is called BufferHandle. Pinning increases the reader count, while unpinning decreases it.

When a reader requests access, why is a shared_ptr to the BlockHandle given along with the optional ptr(BufferHandle) required to access the FileBuffer which holds the block?

Readers, using the part of BufferHandle that contains a shared_ptr to BlockHandle, collectively own the BlockHandle. The collective lifetime of the BufferHandles thus corresponds with that of BlockHandle and consequently, FileBuffer or the block itself.

Lifetime:

The following are some premises that I uncovered in the process of deconstructing these components:

• block_ids are given to blocks based on whether or not they represent regions from persistent storage or intermediate/short-lived data generated during active query execution (e.g., overflow data from a massive sort or join operation). A threshold is defined(MAXIMUM_BLOCK). Blocks with block_ids less than this threshold represent the previous type, while those with IDs greater than the threshold represent the latter.

• Readers are granted access to a block using BufferHandle(an optional pointer is a part of the BufferHandle object). BufferHandle also has a shared pointer to BlockHandle. The collective lifetime of the BufferHandles thus corresponds with that of BlockHandle and consequently, FileBuffer or the block itself.

  Assumption: BufferHandles are the only strong owners of a BlockHandle.

• CanUnload() is a function that determines whether a block can or cannot be unloaded.

  Its conditions are:

  1. The block must currently be loaded.

  2. The reader count must be zero.

  3. If block_id > MAXIMUM_BLOCK, the block is only flagged as unloadable if there exists a temporary directory to store this block. Otherwise, it returns false.

• When something needs to read or write to a specific block, a BufferHandle is provided to this fn or user. This loads/pins it, which increments the reader count. Unpinning decrements it and further triggers a check of the reader count. If it becomes zero, this block is pushed into the eviction queue. When memory pressure occurs, CanUnload() is run on this block. If this returns true, it may then be unloaded.

A few more questions that I encountered in the process of this deconstruction:

Is getting access to a FileBuffer the same as getting access to the block which resides in that buffer?

GetFileBuffer() itself does not magically give you a high-level "Block object." It gives you the underlying byte storage. The code that knows the block's structure interprets those bytes as the block's actual contents.

Is a decrement in the reader count the sole consequence and purpose of unpinning? Same question for Pinning(is the increment the sole purpose?).

Pinning loads the block into RAM(if not already loaded), increments the reader count, and returns a BufferHandle to the requester. Unpinning reverses all of this.

The caller identifies the block through its BlockHandle; it asks the BufferManager to pin that block. If the block's contents are not currently in RAM, DuckDB first gets a piece of RAM for that block, and then reads the block's bytes from disk into that RAM.

Alongside this question, there were many others which I encountered. Here are my answers to all of them:

What is the difference between the roles and capabilities of BufferPool and BufferManager?

BufferManager manages buffers; BufferPool provides/manages the underlying memory resource and the mechanisms needed to make that management possible.

BufferManager = higher-level buffer operations(receptionist & executioner)

• allocate memory

• pin blocks

• unpin blocks

• coordinate loading/freeing

• interact with other DuckDB components

BufferPool = underlying memory-resource management

• tracks the available memory capacity

• tracks memory consumption

• participates in eviction

• reclaims memory when necessary

• provides the memory resource that the BufferManager operates against

What is a blockid? What is a BlockHandle? How does it differ from a BlockMemoryobject?

A blockid is simply an identifier for a block.

A BlockHandle is the C++ object that holds all the information DuckDB needs to ‘handle’ or manage block 42.

Eg:

A persistent block can have a block_id without currently having a BlockHandle.

It can exist even when the block's contents are not currently in RAM.

A BlockMemory object on other hand only stores information about the memory state(pinned, unpinned etc) and the location of the block in the RAM(i.e. which buffer). A BlockMemory object is a component i.e. subset of a BlockHandleobject.

What is a BlockManager?

An instance of BlockManagerowns and manages a collection of BlockHandleobjects. Here, ownership of a block means “This BlockManager keeps track of this handle”, or that the block belongs to this BlockManager's storage context. Multiple blocks can have the same blockid, but not the same manager, therefore to refer to a block one needs 1. the instance of BlockManager owning that block and 2. the blockid. Neither of them can individually address a block.

What’s the difference between TryGetBlock() and RegisterBlock()?

Does RegisterBlock() fill every field of a BlockHandle once it creates it?

No, RegisterBlock() itself isn't manually filling every field of a BlockHandle. It constructs the BlockHandle, and the BlockHandle's implementation determines what information it contains.

Where does the usage of weak_ptr come in the picture of blockid, b_handle, b_manager etc if anywhere?

The BlockManager keeps a weak_ptr reference to the blockhandles that correspond to that instance.

What is the BufferHandle?

BufferHandle gives you temporary access to the RAM buffer that currently contains the block's data, not ownership/management of the block itself.

What does expire and lock do or return wrt a weak ptr?

lock() returns nullptr if the object doesn’t exist anymore and a shared_ptr to the object if it does, whereas expire() gives boolean value for the same.

Here's what I understood about the code.

The two core objects that are repeatedly mentioned:

• BufferPool: Provides/manages the underlying memory resource and the mechanisms needed to let the memory receptionist (BufferManager) use it.

• BlockMemory: Knows the status of blocks with respect to the relevant aspects. Different BlockMemory objects contain metadata about different blocks of memory.

Blocks are pinned when a user or a local function requests them, and are unpinned when they are no longer needed. An unpinned block gets placed in the eviction queue but not unloaded. They are evicted only under memory pressure.

The following is the trajectory of a freshly evicted(or dequeued) candidate:

Discard — a newer entry exists.

Leave block — the entry is current, but A is being used again.

Proceeding with the analysis of this file, I came across several implementation questions. This is my attempt at answering each of them:

What are eviction queues?

They are waiting queues for entries to be removed. Eviction is just one of the ways of deleting an object.

What are weak and shared ptrs?

A shared_ptr keeps a count of the number of owners(all the shared_ptrs pointing to the object) of the object to which it points. When this no. drops to 0, the object is destroyed.

A weak_ptr tracks (only) the lifetime of an object owned by some shared_ptr using functions like lock() and expire().

“BufferPool starts removing entries from these queues and tries to unload them.” What’s the different between removing and unloading?

Why does the eviction queue have a hierarchy according to which eviction is conducted?

There is a certain a cost to evicting entities. The eviction queue is thus divided into 3 based on how much they free at the cost of how much time is spend at the act of doing so.

In the case of a dead node, why would removing the old copy be expensive? What does ‘this queue entry is obsolete’ mean? Which one of the two is a dead node? What are sequence numbers?

Say a node A is evicted, and then taken out for use again, then evicted again. Thus the queue now has 2 entries of A. The older one(which is called the dead node) needs to be removed.

Searching for an entry mid-queue is expensive. Instead, a hash table called page directory hashes serial no.s to entries in the queue. This way, when A reappears in the eviction queue, it increments its hash value to 2. Later, when the older value reaches the tail, the sequence no.s of the dead node and the current hashed value of A are compared. If they match, the block is unloaded. If not, it is simply discarded, as the inequality indicates that the block at the tail is not its latest entry.

What does purging a queue mean?

Purging involves taking a batch out of the eviction queue, discarding dead nodes in bulk, and then putting it back into the queue. The newer batch becomes the newest entries; they are not removed and put back into the queue from the tail side.

Does the system constantly check every block to see which one it can push into the eviction queue?

No; once the reader count drops to zero, the function CanUnload() checks other conditions. A block is pushed to the eviction queue only when all the conditions hold.