Harden core JSON decoding, singleton init, and startup arg parsing against faults by linmajia · Pull Request #264 · microsoft/rDSN

HX Lin (linmajia) · 2026-07-02T08:19:38Z

Summary

Core-runtime robustness hardening found via libfiu fault injection, plus the
submodule pointer bump for the meta-server hardening in
linmajia/rDSN.dist.service (see companion PR). Focus: untrusted/corrupt input and
allocation failures should surface as recoverable errors, not terminate() /
dsn_coredump / SIGABRT.

Changes

1. JSON decoders report malformed input instead of aborting

json_helper.h: the string_tokenizer and value decoders (gpid, rpc_address,
etc.) now raise exceptions on malformed input, which the top-level decode() already
converts to a clean failure — instead of dassert / dsn_coredump on corrupt or
untrusted data.

2. Reject unresolvable / malformed `rpc_address` in JSON

from_string_ipv4() returns true even when the host can't be resolved (yielding a
non-invalid 0.0.0.0:port). The decoder now rejects a false return or a zero
IP as malformed, so a corrupt secondaries/primary value can't slip past the later
is_invalid() invariant checks.

3. Exception-safe singleton initialization

singleton.h: a throwing constructor during lazy init no longer leaves a corrupt
half-initialized instance. Added singleton.test.cpp unit tests that simulate
exceptions thrown during new T().

4. Reject malformed `-app_list` index instead of aborting startup

main.cpp: parsing the name@index instance index used throwing std::stoi
(uncaught std::invalid_argument / std::out_of_range → terminate() on a bad
-app_list; also silently accepted a garbage tail like 3xyz). Now uses the
codebase's non-throwing lexical_cast_integer<int>() with a clear error and graceful
exit.

5. macOS build fix

disk_engine.cpp: remove an unused lambda capture that broke the macOS
-Werror,-Wunused-lambda-capture build.

6. Submodule bump

src/plugins_ext/rDSN.dist.service: a564c61 → ca52e98 .

…c dist service submodule The meta server parses ZooKeeper-stored app/partition state through json_forwarder<T>::decode(blob, t), which is written to return false on parse failure via a try/catch. But the string_tokenizer signalled malformed input with dassert(false) (expect_token, walk_until) and empty input with a constructor dassert(pos < length) -- both process aborts that the wrapper's catch(...) cannot catch (the constructor even runs before the try). So corrupt or empty remote-storage JSON aborted the meta server. - Relax the constructor assert to pos <= length so an empty / fully-consumed buffer is tolerated (the first expect_token then throws cleanly). - Replace the malformed-input dassert(false) in expect_token x2 and walk_until with throw std::runtime_error, routing through the exception channel decode() already handles. - Guard peek_next() against a 1-byte over-read past the buffer end on input ending right after '{' / '[' (return '\0' sentinel). Every json decoder in the tree goes through this wrapper, so they all become abort-safe. Also advance the rDSN.dist.service submodule pointer to the meta server hardening commit (dump write-error, remote-storage JSON decode, partition_count validation).

…re build json_helper.h: make the gpid and rpc_address JSON value decoders report malformed input through exceptions instead of aborting. An earlier change hardened the tokenizer, but json_decode(gpid&) still dassert'd sscanf()==2 and json_decode(rpc_address&) ignored from_string_ipv4()'s result, so a corrupt remote-storage value aborted the meta server (gpid) or was silently stored as an invalid address and aborted later at initialize_node_state (rpc_address) -- both defeating json_forwarder<>::decode()'s try/catch that sync_apps_from_remote_storage() relies on. gpid now throws on a bad format; rpc_address accepts the legitimate "invalid address" marker (so an unset address, e.g. a partition with no primary, still round-trips) and throws on any other unparseable string. singleton.h: release the init spinlock on every exit path. If T's constructor threw (e.g. std::bad_alloc under memory pressure), instance() skipped the _l.store(0) unlock and leaked the lock forever: later instance() calls spin, and a re-entrant call on the same thread -- the failure is logged through the logger, itself a singleton -- self-deadlocks. Add a small RAII guard so _l is released on both normal return and exception unwinding, turning the failure into a clean, retryable throw. disk_engine.cpp: drop an unused [this] capture from a local lambda in process_write(). It tripped -Werror,-Wunused-lambda-capture on macOS/clang; GCC has no such warning, so the Linux build passed and only the macOS build broke. Bump rDSN.dist.service submodule (meta-state log replay + dump close hardening).

…hrow tests json_decode(rpc_address&) previously trusted rpc_address::from_string_ipv4()'s return value, but that function returns true even when the host cannot be converted: assign_ipv4() stores dsn_ipv4_from_host(host), which yields 0 for a malformed/unresolvable host (e.g. "999.999.999.999:12345") or an empty host (":12345"). The result was a non-invalid 0.0.0.0:port address that slipped past the downstream is_invalid() checks. Reject at the decode boundary when the decoded address did not resolve to a concrete IPv4 (ip() == 0); a genuine serialized node address is never 0.0.0.0, and the "invalid address" unset marker is still accepted by the earlier early-return. Also add src/core/src/singleton.test.cpp covering the singleton constructor-throw path: verify that when new T() throws, instance() propagates the exception (fail-stop), the internal init lock is released (no deadlock), no half-built instance is published, and a subsequent instance() call succeeds.

Points the submodule at b6f630e, which replaces %ld/%lx with PRId64/PRIx64 (and fixed-width casts) when logging 64-bit file offsets and int64_t control flags in dump_file.h and meta_service.cpp, so they are not truncated or mis-read on Windows (LLP64, where long is 32-bit).

Pull in submodule 79e9875, which converts fatal dasserts on recoverable ZooKeeper / remote-storage errors into retries or a graceful startup failure, so an allocation failure inside the ZooKeeper C client no longer crashes the meta server. Affected: server_state.cpp (init_app_partition_node, do_app_create, do_app_drop, on_update_configuration_on_remote_reply) and zookeeper_session::attach().

run_apps() parsed the numeric instance index of the -app_list name@index argument with std::stoi(), which throws std::invalid_argument or std::out_of_range on a non-numeric or overflowing value. The throw is uncaught in this frame, so a single malformed index (for example -app_list replica@abc) aborts the whole process via std::terminate() instead of reporting a clear error. std::stoi also silently accepted a trailing garbage tail (stoi("3xyz") == 3). Use the standard non-throwing helper dsn::utils::lexical_cast_integer<int>() (already used at 30+ sites, including untrusted HTTP-header parsing) and fail startup cleanly with a clear stderr diagnostic. Valid input is unchanged; malformed, overflowing and garbage-tail indices are now rejected gracefully.

…rage errors Point the submodule at 4e384da, which narrows the round-11 meta-server retry logic so that permanent remote-storage errors (ERR_OBJECT_NOT_FOUND / ERR_INVALID_PARAMETERS / ERR_INCONSISTENT_STATE) fail-stop instead of retrying once per second forever, while transient errors (ERR_TIMEOUT / ERR_ZOOKEEPER_OPERATION) keep retrying as before.

…_sync test Picks up the rDSN.dist.service fix that deletes the /meta_test/apps ZooKeeper root at the start of the meta.state_sync test, making it robust to stale state left by a previous aborted run (which otherwise causes app_mapper_compare to abort at misc.cpp:187).

The meta-server hardening and state_sync test changes have been squash-merged into rDSN.dist.service master as commit ca52e98 (microsoft#32). Point the submodule at that master commit instead of the now-merged fix-branch tip. No content change: ca52e98 carries the same six files (dump_file.h, meta_service.cpp, meta_state_service_simple.cpp, server_state.cpp, zookeeper_session.cpp, state_sync_test.cpp).

HX Lin added 9 commits July 2, 2026 07:46

HX Lin (linmajia) merged commit 0adf0f2 into microsoft:master Jul 2, 2026
2 checks passed

HX Lin (linmajia) deleted the fix branch July 2, 2026 08:46

Navigation Menu

Harden core JSON decoding, singleton init, and startup arg parsing against faults - #264

Harden core JSON decoding, singleton init, and startup arg parsing against faults#264
HX Lin (linmajia) merged 9 commits into
microsoft:masterfrom
linmajia:fix

HX Lin (linmajia) commented Jul 2, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

HX Lin (linmajia) commented Jul 2, 2026

Summary

Changes

1. JSON decoders report malformed input instead of aborting

2. Reject unresolvable / malformed rpc_address in JSON

3. Exception-safe singleton initialization

4. Reject malformed -app_list index instead of aborting startup

5. macOS build fix

6. Submodule bump

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

2. Reject unresolvable / malformed `rpc_address` in JSON

4. Reject malformed `-app_list` index instead of aborting startup