DuckDB PR2 — How a Missing ClearUpdates() Retained StringHeap Allocations

After a week of investigating DuckDB's memory leak, I traced the accumulation to a specific change in checkpoint finalization: the removal of a call to ClearUpdates().

The investigation revealed how an update could copy string data into an UpdateSegment's StringHeap before DuckDB determined that the proposed update had no effect. Because the early return occurred after those allocations, and the UpdateSegment was no longer explicitly cleared during checkpoint finalization, the copied string data could remain allocated across checkpoint cycles.

Tracing this sequence narrowed the investigation to the code responsible for the memory accumulation and provided a concrete direction for understanding the regression. The following traces the sequence of events that caused the memory accumulation, introducing each relevant concept alongside the code path where it becomes necessary:

ColumnData is the BlockHandle of DuckDB columns in a given table i.e. it’s the in-memory representation of a column.

It has 2 members which are of relevance: data and updates. data is the existing column data, organized into segments. updates is a smart pointer(unique_ptr) to an UpdateSegment.

Now what is UpdateSegment? When DuckDB processes the updates which are to be made to a column, it can create an UpdateSegment object to manage that work.

• For this investigation, the important fact is that UpdateSegment contains a StringHeap.

What is a StringHeap? Strings are processed in the form of batches. DuckDB represents individual string values using an object called string_t. Short strings are stored directly inside the string_t object (inline), whereas longer strings are represented using metadata, including their length, and a pointer to character bytes stored elsewhere. StringHeap is a separate object that stores copies of these character bytes(given that the copy fulfills certain conditions), allowing them to remain valid independently of the original storage.

• Because ColumnData owns updates through a unique_pt, destroying or resetting that pointer destroys UpdateSegment, consequently destroying its string heap.

• When an upsert arrives, DuckDB

  1. Enters the update path i.e. it executes UpdateSegment::Update(),

     ColumnData::UpdateInternal()
         ↓
     UpdateSegment::Update()
     

  and then,

1. Calls its statistics-update callback.

What is a callback? A callback is a function, a lambda, a function object, or any other callable mechanism in C++ passed as a parameter into another function so that the receiving function can invoke it at a point of its choosing.

For VARCHAR values, UpdateStringStatistics() copies non-inlined strings into StringHeap through AddBlob().

StringStats::Update(stats.statistics, str);

if (!str.IsInlined()) {
    update_data[idx] = segment->GetStringHeap().AddBlob(str);
}

Conceptually, AddBlob() copies the string bytes into storage managed by the heap and returns a string representation referring to the copied data.

• After the statistics callback finishes, DuckDB checks whether the proposed values actually differ from the existing values. The code calls:

  count = get_effective_updates(
      update_format, ids, count, sel, base_data, vector_offset
  );
  
  if (count == 0) {
      return;
  }
  

  When the proposed values are the same as the existing values, the count = 0. This implies no effective update. Consequently, the function returns before InitializeUpdateInfo() is reached i.e. before creating UpdateInfo entries. This is relevant as the initialisation of UpdateInfo objects could have been a possible cause of the memory accumulation and consequently, the OOM.

  Here is the full sequence:

  

  The key is that step 3 has already allocated and copied the string before step 5 rejects the update. Returning early does not undo those copies.

• Before the first bad commit, checkpoint finalization called ClearUpdates(), whose implementation was:

  void ColumnData::ClearUpdates() {
      lock_guard<mutex> update_guard(update_lock);
      updates.reset();
  }
  

  Here, updates is a unique_ptr<UpdateSegment>. Resetting it destroyes the owned UpdateSegment, including its StringHeap. That released the heap's allocations.

• In the first bad commit, the previous ClearUpdates() call disappeared from checkpoint finalization. Consequently, the UpdateSegment and its heap could survive checkpointing instead of being reclaimed there. As subsequent updates processed more strings, additional character bytes could accumulate in that surviving heap.

I’ll soon publish the investigation notes behind this article, documenting the experiments I ran, the failures I encountered, and the reasoning behind each debugging decision.

That will be followed by a detailed account of the contribution process itself: tracing the issue to its root cause, developing the patch, designing and running regression tests, validating the fix, and working toward an upstream pull request.