Fix storage leak from concurrent checkpoint and optimistic write

[dentiny]

Hi team, I found the following SQL seems to suffer storage leak (already added into regression test)

statement ok
PRAGMA disable_checkpoint_on_shutdown

statement ok
SET checkpoint_threshold = '1TB'

statement ok
SET checkpoint_on_detach = 'disabled'

# Force row groups to flush to disk immediately, so that the WAL stores block pointers (WriteRowGroupData).
statement ok
SET write_buffer_row_group_count = 1

statement ok
ATTACH '{TEST_DIR}/concurrent_checkpoint_block_leak.db' AS db (BLOCK_SIZE 16384, ROW_GROUP_SIZE 20480)

statement ok
CREATE TABLE db.t (i BIGINT, s VARCHAR)

statement ok
INSERT INTO db.t SELECT i, repeat(md5(i::VARCHAR), 4) FROM range(50000) tt(i)

statement ok
CHECKPOINT db

# An open write transaction prevents the vacuum lock, so the checkpoint below is a concurrent checkpoint.
statement ok con_holder
BEGIN TRANSACTION

statement ok con_holder
INSERT INTO db.t VALUES (-2, 'holder')

# Make the main WAL non-empty so the checkpoint creates a checkpoint WAL.
statement ok
INSERT INTO db.t VALUES (-1, 'trigger')

statement ok
SET debug_checkpoint_sleep_ms = 3000

# Thread 0: checkpoint sleeps after WALStartCheckpoint, before writing the header.
# Thread 1: optimistic insert that commits into the checkpoint WAL during that window.
concurrentloop threadid 0 2

onlyif threadid=0
statement ok
CHECKPOINT db

onlyif threadid=1
statement ok
SELECT sleep_ms(500)

onlyif threadid=1
statement ok
INSERT INTO db.t SELECT i, repeat(md5(i::VARCHAR), 4) FROM range(50000, 150000) tt(i)

endloop

statement ok
SET debug_checkpoint_sleep_ms = 0

statement ok con_holder
ROLLBACK

# Reattach without checkpointing: the former checkpoint WAL is replayed on top of the concurrent checkpoint.
statement ok
DETACH db

statement ok
ATTACH '{TEST_DIR}/concurrent_checkpoint_block_leak.db' AS db

query I
SELECT COUNT(*) FROM db.t
----
150001

statement ok
SET debug_verify_blocks = true

statement ok
CHECKPOINT db

statement ok
DROP TABLE db.t

statement ok
CHECKPOINT db

which gets error message

CHECKPOINT db;
================================================================================
IO Error: Checkpoint failed for database "db". The database has been invalidated. Original error: Block verification failed - blocks "251, 252,..." were not found as being used OR marked as free
Max block: 1306
Block usage: 747, 744, 743, 742, 741...
Free list: 0, 1, 2, 3, 4...
Multi-use blocks: 804, 747, 744, 743, 742...
Newly used blocks:

In this test, I tried to craft a testing scenario where,

• During an active checkpoint, there's large insertion ongoing and attempted optimistic writes
• These writes are written directly to DuckDB file, their in-memory blocks are marked as newly used ones
• The write transaction commits, these blocks are removed outside of newly used ones, which indicates ref count 1
• On checkpoint write, these blocks are not recorded as free blocks, and they live in the WAL file

In the normal code path, where there's no concurrent checkpoint and optimistic writes

• Optimistic write marks blocks as newly used ones, with WAL entries referencing to these blocks
• On transaction commit and later checkpoint, these blocks are not recorded as free block, with WAL file deleted
• On database restart, blocks are recorded with in use with reference count 1

But when there're concurrent checkpoint and optimistic write,

• Blocks are still recorded as in use (same as above), but WAL entries are retained (.wal.checkpoint)
• On database restart, blocks are double recorded with reference count 2 (one comes from DuckDB file header, another comes from WAL entries) => storage leak

Because WAL recording is definitely needed, the easiest fix, is to record these blocks as free blocks in header.

---

Note

High Risk
Changes WAL/block-manager lifecycle during concurrent checkpoints and optimistic commit revert paths, which are correctness-critical for persistence and can cause leaks or corruption if wrong.

Overview
Fixes a storage block leak when optimistic writes commit into the checkpoint WAL during an in-flight concurrent checkpoint.

Checkpoint WAL row-group blocks are no longer marked checkpointed at WriteRowGroupData time. They are tracked in pending_checkpoint_blocks until WALFinishCheckpoint calls MarkPendingBlocksAsCheckpointed after the checkpoint header is written, so the header’s free/in-use state matches what the replayed WAL will reference. Reverted optimistic commits remove their blocks from that pending set via RemovePendingCheckpointBlocks in RevertCommit.

Adds two slow regression tests: concurrent checkpoint + successful optimistic insert (detach/reattach + debug_verify_blocks), and the same race with a forced commit failure followed by a successful insert.

^{Reviewed by Cursor Bugbot for commit 85b53de. Bugbot is set up for automated code reviews on this repo. Configure here.}