EmberKV

An append-only key-value store you can read all the way down to the bytes.

EmberKV is a small C storage project about a specific question: what should a database do when its final write stops halfway through?

The log uses explicit little-endian fields, per-record CRC-32 checksums, tombstones, and disk synchronization. Startup rebuilds a hash index from the log. An incomplete final record can be recovered into a new file; a checksum failure is rejected rather than silently treated as a recoverable tail.

Build and use

GCC or Clang, with POSIX or MinGW runtime APIs:

gcc -std=gnu99 -O2 -Wall -Wextra -Wpedantic -Werror ember.c -o ember
./ember notes.ekv init
./ember notes.ekv put config/theme graphite
./ember notes.ekv put config/retries 3
./ember notes.ekv get config/theme
./ember notes.ekv list config/
./ember notes.ekv delete config/retries
./ember notes.ekv check
./ember notes.ekv compact notes-compact.ekv

On Windows, build with -o ember.exe and run ./ember.exe. make is optional.

$ ./ember notes.ekv list config/
config/retries  1 bytes
config/theme   8 bytes

$ ./ember notes.ekv check
records=3 live=1 valid_bytes=117 tail=clean

get writes exact value bytes, without an added newline. Use put-file KEY FILE for binary data. Missing keys return exit code 1; other failures return 2. check returns 3 for an incomplete final record.

Recovery is explicit

./ember damaged.ekv check
./ember damaged.ekv recover recovered.ekv
./ember recovered.ekv check

Recovery copies the last verified value of every live key into a new database. It never truncates the original. Both compaction and recovery refuse existing output paths, including the source itself.

Input condition	Result
Complete checksummed record	Apply it to the index
EOF between complete records	Clean log
EOF partway through final header or payload	Mark incomplete; allow explicit recovery
Complete record with wrong checksum	Fail; do not guess which bytes are trustworthy
Invalid magic, flags or lengths	Fail; refuse repair

Mutating commands refuse an incomplete log. compact requires a clean log; recover also accepts an incomplete final record. Successful writes flush the C stream and call fsync on POSIX or _commit on Windows.

Test the failure paths

python tests/test_ember.py ./ember

The black-box suite compares randomized operations against a Python dictionary, checks binary and empty values, verifies compaction, and truncates a final record at every byte boundary. It also injects a checksum error and verifies that recovery refuses it.

These are deterministic file-fault tests, not a claim of power-loss certification. Linux CI additionally runs AddressSanitizer and UndefinedBehaviorSanitizer.

Deliberate limits

• One process at a time. There is no process lock. Concurrent access, including a reader during a write, is unsupported.
• One operation per record; no multi-key transactions.
• Keys are nonempty NUL-free strings. Values are arbitrary bytes. Each field is capped at 16 MiB.
• The live index and values are rebuilt in memory on every command. Startup scales with log size; compaction reduces replay work.
• File offsets use C long; on 32-bit-offset platforms, stay below 2 GiB.
• CRC-32 detects accidental damage; it does not authenticate data or protect against malicious modification.
• The file is synced, but its parent directory is not. Initial creation is not guaranteed durable across every filesystem and power-loss scenario.
• Failed output writes can leave an incomplete destination. Keep the source until you have checked the new file.

See the binary format. MIT licensed.