After deconstructing the bufferpool, I was off to analysing the block_manager.cpp(here's a link to it). The question at the core of this component of the buffer system was "How is a disk block requested?”.
The caller identifies the block through its BlockHandle; it asks the BufferManager to pin that block. If the block's contents are not currently in RAM, DuckDB first gets a piece of RAM for that block, and then reads the block's bytes from disk into that RAM.
Alongside this question, there were many others which I encountered. Here are my answers to all of them:
What is the difference between the roles and capabilities of
BufferPoolandBufferManager?
BufferManager manages buffers; BufferPool provides/manages the underlying memory resource and the mechanisms needed to make that management possible.
BufferManager = higher-level buffer operations(receptionist & executioner)
allocate memory
pin blocks
unpin blocks
coordinate loading/freeing
interact with other DuckDB components
BufferPool = underlying memory-resource management
tracks the available memory capacity
tracks memory consumption
participates in eviction
reclaims memory when necessary
provides the memory resource that the BufferManager operates against
What is a
blockid? What is aBlockHandle? How does it differ from aBlockMemoryobject?
A blockid is simply an identifier for a block.
A BlockHandle is the C++ object that holds all the information DuckDB needs to ‘handle’ or manage block 42.
Eg:
A persistent block can have a block_id without currently having a BlockHandle.
It can exist even when the block's contents are not currently in RAM.
A BlockMemory object on other hand only stores information about the memory state(pinned, unpinned etc) and the location of the block in the RAM(i.e. which buffer). A BlockMemory object is a component i.e. subset of a BlockHandleobject.
What is a
BlockManager?
An instance of BlockManagerowns and manages a collection of BlockHandleobjects. Here, ownership of a block means “This BlockManager keeps track of this handle”, or that the block belongs to this BlockManager's storage context. Multiple blocks can have the same blockid, but not the same manager, therefore to refer to a block one needs 1. the instance of BlockManager owning that block and 2. the blockid. Neither of them can individually address a block.
What’s the difference between TryGetBlock() and RegisterBlock()?
Does RegisterBlock() fill every field of a BlockHandle once it creates it?
No, RegisterBlock() itself isn't manually filling every field of a BlockHandle. It constructs the BlockHandle, and the BlockHandle's implementation determines what information it contains.
Where does the usage of weak_ptr come in the picture of blockid, b_handle, b_manager etc if anywhere?
The BlockManager keeps a weak_ptr reference to the blockhandles that correspond to that instance.
What is the
BufferHandle?
BufferHandle gives you temporary access to the RAM buffer that currently contains the block's data, not ownership/management of the block itself.
What does expire and lock do or return wrt a weak ptr?
lock() returns nullptr if the object doesn’t exist anymore and a shared_ptr to the object if it does, whereas expire() gives boolean value for the same.
About the author
- Low-level C++ engineer specializing in database storage architectures. Currently deconstructing analytical query engines and implementing LSM write paths under strict ASan, TSan, and memory-ordering constraints.
Comments
Join the discussionNo comments yet. Be the first to comment.
Deconstructing DuckDB
A systematic, mechanical deconstruction of the DuckDB buffer manager and storage internals. This series covers low-level component analyses—spanning block allocation, buffer pools, eviction policies, and handle lifecycles—directly from the source code. Followed by the complete engineering process of diagnosing edge cases, isolating system constraints, and auditing upstream pull request candidates for merge.
