Summary
DeclarativePartitionFactory claims to create "a retriever per thread" in its docstring, but it holds and reuses a single retriever instance for every partition. With the default concurrency_level of a declarative source, all worker threads share that one retriever - and for HTTP-based retrievers that means one shared requests.Session and unlocked mutable state used concurrently from multiple threads.
Code refs (v7.25.1)
airbyte_cdk/sources/declarative/stream_slicers/declarative_partition_generator.py:44-47 - docstring: the factory exists "in order to prevent the stream instance from being shared between threads" by creating "a retriever per thread".airbyte_cdk/sources/declarative/stream_slicers/declarative_partition_generator.py:50 - the factory stores ONE retriever instance and passes the same object into every DeclarativePartition (:91-93 calls self._retriever.read_records(...) per partition, potentially from N worker threads at once).- For custom retrievers built around
airbyte_cdk/sources/streams/http/http_client.py: one shared requests.Session (:126-180) and the unlocked _request_attempt_count dict (:137, mutated at :329-334) are then shared across threads.
Impact
- Any
CustomRetriever following the documented contract gets concurrent read_records calls on one instance without any warning - easy to write thread-unsafe connector code that works in unit tests (single-threaded) and misbehaves in production (default concurrency 5+).
- Shared-session symptoms are environment-dependent (connection-pool contention, retry-counter races), which makes them hard to attribute.
Expected
Either the implementation should match the docstring (instantiate a retriever per partition/thread, e.g. deep-copy or factory callback), or the docstring and the CustomRetriever documentation should state explicitly that one instance is shared across worker threads and implementations must be thread-safe.
Precedent
Found while reviewing airbytehq/airbyte#80306 (source-freshdesk ticket_activities custom retriever): 5 worker threads issue concurrent read_records through one HttpClient/requests.Session instance.
Summary
DeclarativePartitionFactoryclaims to create "a retriever per thread" in its docstring, but it holds and reuses a single retriever instance for every partition. With the defaultconcurrency_levelof a declarative source, all worker threads share that one retriever - and for HTTP-based retrievers that means one sharedrequests.Sessionand unlocked mutable state used concurrently from multiple threads.Code refs (v7.25.1)
airbyte_cdk/sources/declarative/stream_slicers/declarative_partition_generator.py:44-47- docstring: the factory exists "in order to prevent the stream instance from being shared between threads" by creating "a retriever per thread".airbyte_cdk/sources/declarative/stream_slicers/declarative_partition_generator.py:50- the factory stores ONE retriever instance and passes the same object into everyDeclarativePartition(:91-93callsself._retriever.read_records(...)per partition, potentially from N worker threads at once).airbyte_cdk/sources/streams/http/http_client.py: one sharedrequests.Session(:126-180) and the unlocked_request_attempt_countdict (:137, mutated at:329-334) are then shared across threads.Impact
CustomRetrieverfollowing the documented contract gets concurrentread_recordscalls on one instance without any warning - easy to write thread-unsafe connector code that works in unit tests (single-threaded) and misbehaves in production (default concurrency 5+).Expected
Either the implementation should match the docstring (instantiate a retriever per partition/thread, e.g. deep-copy or factory callback), or the docstring and the
CustomRetrieverdocumentation should state explicitly that one instance is shared across worker threads and implementations must be thread-safe.Precedent
Found while reviewing airbytehq/airbyte#80306 (source-freshdesk
ticket_activitiescustom retriever): 5 worker threads issue concurrentread_recordsthrough oneHttpClient/requests.Sessioninstance.