From 693187d3e0e7dcb4d75d63c7b207e363a8cd10de Mon Sep 17 00:00:00 2001 From: Dana Powers Date: Tue, 9 Jun 2026 07:17:33 -0700 Subject: [PATCH 1/6] docs: Updates to changelog; align README and docs/index.rst --- CHANGES.md | 5 ++++- README.rst | 17 +++++++++----- docs/changelog.rst | 5 ++++- docs/index.rst | 55 +++++++++++++++++++++++++++------------------- 4 files changed, 52 insertions(+), 30 deletions(-) diff --git a/CHANGES.md b/CHANGES.md index 4393a4eee..36f9e418e 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -25,6 +25,7 @@ This is a major release with significant changes to kafka-python internals to si * Make IncompatibleBrokerVersion a subclass of UnsupportedVersionError (#2924) * Eliminate NoBrokersAvailableError (#2942) * KafkaProtocolError is not retriable (#2941) +* Prefer raised Exceptions to assert / AssertionError (#3042) ### Old Networking Stack Removal * Remove kafka.client_async / kafka.conn legacy modules (#2918) @@ -85,7 +86,7 @@ A new JSON-schema-based dynamic protocol generator now replaces the legacy hand- ### Dynamic Protocol Classes Protocol classes are now generated from the upstream Apache Kafka JSON schemas. -* Dynamic protocol classes using upstream json schemas (#2727, #2745, #2779, #2782, #2787, #2810) +* Dynamic protocol classes using upstream json schemas (#2727, #2745, #2779, #2782, #2787, #2810, #3037) * Add .pyi type annotation stubs for generated protocol classes (#2784) * Migrate all internal usage to new protocol classes (#2764, #2765, #2766, #2767, #2768, #2772) * Refactor treatment of versioned ApiMessage classes (#2739) @@ -153,6 +154,7 @@ All consumer network I/O now flows through the shared kafka.net IO thread; `cons * Consumer: use background thread for all network io; drop HeartbeatThread (#2965) * Consumer: send all requests from net io thread (#2980) * Consumer: simplify poll() with fetcher.fetch_records (#2960) +* Consumer: sleep in poll() if timeout, no records, and no fetchable partitions (#3039) * Consumer: drop poll loop optimizations for pending offset resets and rejoins (#2959) * Consumer: `_update_fetch_positions` -> `_refresh_committed_offsets`; dont poll in position() (#2958) * Consumer: convert fetcher reset_offsets/send_list_offsets_requests to async def @@ -293,6 +295,7 @@ Small quality-of-life additions to the public API surface. * Support context manager interface for consumer/producer/admin (#2969) * Make IncompatibleBrokerVersion a subclass of UnsupportedVersionError (#2924) * Add OffsetSpec / IsolationLevel to kafka imports (#2898) +* Errors: subclasses for RetriableError and InvalidMetadataError (#3041) ## Fixes diff --git a/README.rst b/README.rst index cf839a70f..3d10279c6 100644 --- a/README.rst +++ b/README.rst @@ -1,5 +1,5 @@ -Kafka Python client ------------------------- +kafka-python +############ .. image:: https://img.shields.io/badge/kafka-4.3--0.8-brightgreen.svg :target: https://kafka-python.readthedocs.io/en/master/compatibility.html @@ -23,9 +23,9 @@ to manage. kafka-python can also be used as a simple alternative to the apache kafka admin scripts, which require an installed/compatible jvm. A simple CLI interface for -admin commands is provided as `kafka-python admin` / `python -m kafka.admin`. +admin commands is provided as ``kafka-python admin`` / ``python -m kafka.admin``. -Users looking to add more raw throughput can pip install `crc32c` as +Users looking to add more raw throughput can ``pip install crc32c`` as an optional dependency, offloading one of the most CPU intensive subsystems to an optimized C library. @@ -33,11 +33,18 @@ to an optimized C library. .. code-block:: bash pip install kafka-python + # callable as module or as cli-script kafka-python admin -b localhost:9092 cluster describe + + # Create a topic with the admin cli python -m kafka.admin -b localhost:9092 topics create -t foo-topic + + # Produce messages echo "foo message" | python -m kafka.producer -b localhost:9092 -t foo-topic - python -m kafka.consumer -b localhost:9092 -C auto_offset_reset=earliest -g foo-group -t foo-topic + + # Consume messages + python -m kafka.consumer -b localhost:9092 -C auto_offset_reset=earliest -C consumer_timeout_ms=1000 -g foo-group -t foo-topic What's New in 3.0 diff --git a/docs/changelog.rst b/docs/changelog.rst index e16ab0461..7b353a9eb 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -35,6 +35,7 @@ Error Hierarchy * Make IncompatibleBrokerVersion a subclass of UnsupportedVersionError (#2924) * Eliminate NoBrokersAvailableError (#2942) * KafkaProtocolError is not retriable (#2941) +* Prefer raised Exceptions to assert / AssertionError (#3042) Old Networking Stack Removal ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -109,7 +110,7 @@ Dynamic Protocol Classes ^^^^^^^^^^^^^^^^^^^^^^^^ Protocol classes are now generated from the upstream Apache Kafka JSON schemas. -* Dynamic protocol classes using upstream json schemas (#2727, #2745, #2779, #2782, #2787, #2810) +* Dynamic protocol classes using upstream json schemas (#2727, #2745, #2779, #2782, #2787, #2810, #3037) * Add .pyi type annotation stubs for generated protocol classes (#2784) * Migrate all internal usage to new protocol classes (#2764, #2765, #2766, #2767, #2768, #2772) * Refactor treatment of versioned ApiMessage classes (#2739) @@ -188,6 +189,7 @@ All consumer network I/O now flows through the shared IO thread; ``poll()`` no l * Consumer: use background thread for all network io; drop HeartbeatThread (#2965) * Consumer: send all requests from net io thread (#2980) * Consumer: simplify poll() with fetcher.fetch_records (#2960) +* Consumer: sleep in poll() if timeout, no records, and no fetchable partitions (#3039) * Consumer: drop poll loop optimizations for pending offset resets and rejoins (#2959) * Consumer: _update_fetch_positions -> _refresh_committed_offsets; dont poll in position() (#2958) * Consumer: convert fetcher reset_offsets/send_list_offsets_requests to async def @@ -352,6 +354,7 @@ Small quality-of-life additions to the public API surface. * Support context manager interface for consumer/producer/admin (#2969) * Make IncompatibleBrokerVersion a subclass of UnsupportedVersionError (#2924) * Add OffsetSpec / IsolationLevel to kafka imports (#2898) +* Errors: subclasses for RetriableError and InvalidMetadataError (#3041) Fixes ----- diff --git a/docs/index.rst b/docs/index.rst index dc1dc3e11..907db2deb 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -19,22 +19,32 @@ kafka-python kafka-python is a pure-python client library for Apache Kafka, the distributed stream processing engine. It has no external dependencies and no Cython/C/rust core, making installation across a wide variety of environments simple and easy -to manage. Users looking to add more raw throughput can pip install `crc32c` as -an optional dependency, offloading one of the most CPU intensive subsystems -to an optimized C library. +to manage. kafka-python can also be used as a simple alternative to the apache kafka admin scripts, which require an installed/compatible jvm. A simple CLI interface for -admin commands is provided as `kafka-python admin` / `python -m kafka.admin`. +admin commands is provided as ``kafka-python admin`` / ``python -m kafka.admin``. + +Users looking to add more raw throughput can ``pip install crc32c`` as +an optional dependency, offloading one of the most CPU intensive subsystems +to an optimized C library. + -.. code:: bash +.. code-block:: bash pip install kafka-python + # callable as module or as cli-script kafka-python admin -b localhost:9092 cluster describe + + # Create a topic with the admin cli python -m kafka.admin -b localhost:9092 topics create -t foo-topic + + # Produce messages echo "foo message" | python -m kafka.producer -b localhost:9092 -t foo-topic - python -m kafka.consumer -b localhost:9092 -C auto_offset_reset=earliest -g foo-group -t foo-topic + + # Consume messages + python -m kafka.consumer -b localhost:9092 -C auto_offset_reset=earliest -C consumer_timeout_ms=1000 -g foo-group -t foo-topic What's New in 3.0 @@ -61,14 +71,14 @@ for API and configuration details. The consumer iterator returns ConsumerRecords, which are simple namedtuples that expose basic message attributes: topic, partition, offset, key, and value: -.. code:: python +.. code-block:: python from kafka import KafkaConsumer consumer = KafkaConsumer('my_favorite_topic') for msg in consumer: print (msg) -.. code:: python +.. code-block:: python # join a consumer group for dynamic partition assignment and offset commits from kafka import KafkaConsumer @@ -76,7 +86,7 @@ that expose basic message attributes: topic, partition, offset, key, and value: for msg in consumer: print (msg) -.. code:: python +.. code-block:: python # manually assign the partition list for the consumer from kafka import TopicPartition @@ -84,7 +94,7 @@ that expose basic message attributes: topic, partition, offset, key, and value: consumer.assign([TopicPartition('foobar', 2)]) msg = next(consumer) -.. code:: python +.. code-block:: python # Deserialize msgpack-encoded values consumer = KafkaConsumer(value_deserializer=msgpack.loads) @@ -120,45 +130,45 @@ KafkaProducer The class is intended to operate as similarly as possible to the official java client. See `KafkaProducer `_ for more details. -.. code:: python +.. code-block:: python from kafka import KafkaProducer producer = KafkaProducer(bootstrap_servers='localhost:1234') for _ in range(100): producer.send('foobar', b'some_message_bytes') -.. code:: python +.. code-block:: python # Block until a single message is sent (or timeout) future = producer.send('foobar', b'another_message') result = future.get(timeout=60) -.. code:: python +.. code-block:: python # Block until all pending messages are at least put on the network # NOTE: This does not guarantee delivery or success! It is really # only useful if you configure internal batching using linger_ms producer.flush() -.. code:: python +.. code-block:: python # Use a key for hashed-partitioning producer.send('foobar', key=b'foo', value=b'bar') -.. code:: python +.. code-block:: python # Serialize json messages import json producer = KafkaProducer(value_serializer=lambda v: json.dumps(v).encode('utf-8')) producer.send('fizzbuzz', {'foo': 'bar'}) -.. code:: python +.. code-block:: python # Serialize string keys producer = KafkaProducer(key_serializer=str.encode) producer.send('flipflap', key='ping', value=b'1234') -.. code:: python +.. code-block:: python # Compress messages producer = KafkaProducer(compression_type='gzip') @@ -220,8 +230,8 @@ Optimized CRC32 Validation Kafka uses CRC32 checksums to validate messages. kafka-python includes a pure python implementation for compatibility. To improve performance for high-throughput applications, kafka-python will use `crc32c` for optimized native code if installed. -See `Install `_ for installation instructions and -https://pypi.org/project/crc32c/ for details on the underlying crc32c lib. +See `Install `_ for installation instructions. +See https://pypi.org/project/crc32c/ for details on the underlying crc32c lib. Protocol @@ -229,10 +239,9 @@ Protocol A secondary goal of kafka-python is to provide an easy-to-use protocol layer for interacting with kafka brokers via the python repl. This is useful for -testing, probing, and general experimentation. The protocol support is -leveraged to enable a :meth:`~kafka.KafkaClient.check_version()` -method that probes a kafka broker and -attempts to identify which version it is running (0.8.0 to 2.6+). +testing, probing, and general experimentation. In version 3.0 the protocol +layer was re-written to generate encoder/decoder classes using json message +definitions imported directly from the Apache Kafka project source. Debugging From cf409d80a365a3f0fc90c5909ba76ba3422ffff3 Mon Sep 17 00:00:00 2001 From: Dana Powers Date: Tue, 9 Jun 2026 07:24:29 -0700 Subject: [PATCH 2/6] Reorg shields; drop downloads+implementation --- README.rst | 14 +++++--------- docs/index.rst | 14 +++++--------- 2 files changed, 10 insertions(+), 18 deletions(-) diff --git a/README.rst b/README.rst index 3d10279c6..b81a3fe43 100644 --- a/README.rst +++ b/README.rst @@ -1,19 +1,15 @@ kafka-python ############ +.. image:: https://img.shields.io/pypi/v/kafka-python.svg + :target: https://pypi.org/project/kafka-python .. image:: https://img.shields.io/badge/kafka-4.3--0.8-brightgreen.svg :target: https://kafka-python.readthedocs.io/en/master/compatibility.html -.. image:: https://img.shields.io/pypi/pyversions/kafka-python.svg - :target: https://pypi.python.org/pypi/kafka-python + .. image:: https://img.shields.io/badge/license-Apache%202-blue.svg :target: https://github.com/dpkp/kafka-python/blob/master/LICENSE -.. image:: https://img.shields.io/pypi/dw/kafka-python.svg - :target: https://pypistats.org/packages/kafka-python -.. image:: https://img.shields.io/pypi/v/kafka-python.svg - :target: https://pypi.org/project/kafka-python -.. image:: https://img.shields.io/pypi/implementation/kafka-python - :target: https://github.com/dpkp/kafka-python/blob/master/pyproject.toml - +.. image:: https://img.shields.io/pypi/pyversions/kafka-python.svg + :target: https://pypi.python.org/pypi/kafka-python kafka-python is a pure-python client library for Apache Kafka, the distributed diff --git a/docs/index.rst b/docs/index.rst index 907db2deb..050b293ed 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -1,19 +1,15 @@ kafka-python ############ +.. image:: https://img.shields.io/pypi/v/kafka-python.svg + :target: https://pypi.org/project/kafka-python .. image:: https://img.shields.io/badge/kafka-4.3--0.8-brightgreen.svg :target: https://kafka-python.readthedocs.io/en/master/compatibility.html -.. image:: https://img.shields.io/pypi/pyversions/kafka-python.svg - :target: https://pypi.python.org/pypi/kafka-python + .. image:: https://img.shields.io/badge/license-Apache%202-blue.svg :target: https://github.com/dpkp/kafka-python/blob/master/LICENSE -.. image:: https://img.shields.io/pypi/dw/kafka-python.svg - :target: https://pypistats.org/packages/kafka-python -.. image:: https://img.shields.io/pypi/v/kafka-python.svg - :target: https://pypi.org/project/kafka-python -.. image:: https://img.shields.io/pypi/implementation/kafka-python - :target: https://github.com/dpkp/kafka-python/blob/master/pyproject.toml - +.. image:: https://img.shields.io/pypi/pyversions/kafka-python.svg + :target: https://pypi.python.org/pypi/kafka-python kafka-python is a pure-python client library for Apache Kafka, the distributed From 5f975bc3e8858afb0f9308a6aee3bb201a953e70 Mon Sep 17 00:00:00 2001 From: Dana Powers Date: Tue, 9 Jun 2026 11:08:22 -0700 Subject: [PATCH 3/6] more changes --- CHANGES.md | 9 +++++++++ docs/changelog.rst | 11 +++++++++++ 2 files changed, 20 insertions(+) diff --git a/CHANGES.md b/CHANGES.md index 36f9e418e..7066428de 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -11,6 +11,11 @@ This is a major release with significant changes to kafka-python internals to si * KIP-679: Update Producer defaults -- enable_idempotence=True, acks='all' (#3013) * KIP-735: Increase default consumer session_timeout_ms from 10s to 45s (#3030) * Rename api_version_auto_timeout_ms -> bootstrap_timeout_ms; default 30s (#3028) +* Producer: Remove deprecation warning for buffer_memory config (now raises) (#3047) + +### Abstract Interface Changes +* Serializer: Pass headers to serialize/deserialize (#3046) +* Partitioner: pass both key/value, serialized and unserialized, to partition() (#3045) ### Admin API Changes * Admin: change response shapes to simple dicts (#2883) @@ -288,6 +293,9 @@ The CLI adds shared parser config, SASL/SSL connection support across all subcom * admin cli: catch AttributeError and print_help() (#2880) * admin cli: fix describe_configs (#2875) +### Consumer CLI +* consumer cli does not require group_id (#3044) + ## Compatibility / Misc Small quality-of-life additions to the public API surface. @@ -296,6 +304,7 @@ Small quality-of-life additions to the public API surface. * Make IncompatibleBrokerVersion a subclass of UnsupportedVersionError (#2924) * Add OffsetSpec / IsolationLevel to kafka imports (#2898) * Errors: subclasses for RetriableError and InvalidMetadataError (#3041) +* Helper classes: DefaultSerializer and JsonSerializer (#3046) ## Fixes diff --git a/docs/changelog.rst b/docs/changelog.rst index 7b353a9eb..9cf126fb9 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -18,6 +18,12 @@ Default Configuration Changes * KIP-679: Update Producer defaults -- enable_idempotence=True, acks='all' (#3013) * KIP-735: Increase default consumer session_timeout_ms from 10s to 45s (#3030) * Rename api_version_auto_timeout_ms -> bootstrap_timeout_ms; default 30s (#3028) +* Producer: Remove deprecation warning for buffer_memory config (now raises) (#3047) + +Abstract Interface Changes +^^^^^^^^^^^^^^^^^^^^^^^^^^ +* Serializer: Pass headers to serialize/deserialize (#3046) +* Partitioner: pass both key/value, serialized and unserialized, to partition() (#3045) Admin API Changes ^^^^^^^^^^^^^^^^^ @@ -346,6 +352,10 @@ Admin CLI * admin cli: catch AttributeError and print_help() (#2880) * admin cli: fix describe_configs (#2875) +Consumer CLI +^^^^^^^^^^^^ +* consumer cli does not require group_id (#3044) + Compatibility / Misc -------------------- @@ -355,6 +365,7 @@ Small quality-of-life additions to the public API surface. * Make IncompatibleBrokerVersion a subclass of UnsupportedVersionError (#2924) * Add OffsetSpec / IsolationLevel to kafka imports (#2898) * Errors: subclasses for RetriableError and InvalidMetadataError (#3041) +* Helper classes: DefaultSerializer and JsonSerializer (#3046) Fixes ----- From 6bf0fd0d2212ef0a46a93e533213097de197deb1 Mon Sep 17 00:00:00 2001 From: Dana Powers Date: Tue, 9 Jun 2026 15:28:29 -0700 Subject: [PATCH 4/6] KIP-664 --- CHANGES.md | 9 ++++++--- docs/changelog.rst | 9 +++++++-- 2 files changed, 13 insertions(+), 5 deletions(-) diff --git a/CHANGES.md b/CHANGES.md index 7066428de..f45b50b00 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -14,7 +14,7 @@ This is a major release with significant changes to kafka-python internals to si * Producer: Remove deprecation warning for buffer_memory config (now raises) (#3047) ### Abstract Interface Changes -* Serializer: Pass headers to serialize/deserialize (#3046) +* Serializer/Deserializer: Pass headers to serialize/deserialize (#3046) * Partitioner: pass both key/value, serialized and unserialized, to partition() (#3045) ### Admin API Changes @@ -123,6 +123,7 @@ Protocol classes are now generated from the upstream Apache Kafka JSON schemas. * Store in-flight request headers only for protocol parser (#2723) * Debug log send/recv bytes from protocol parser (#2707) * Adjust protocol debug logging; add KAFKA_PYTHON_PROTOCOL_DEBUG_LOG (#2719) +* CoordinatorType enum (GROUP/TRANSACTION/SHARE) (#3049) ## Broker Version Check @@ -234,7 +235,7 @@ Split KafkaAdminClient into focused mixin classes (cluster, topics, configs, gro ### Refactor and Async Migration The admin client interface remains sync but wraps a fully-async internal api (does not support asyncio yet). Adds cached coordinator lookups and a mixin structure to separate logical resource groups. * Admin: split into functional mixin classes (#2873, #2877, #2882) -* Admin: convert request paths to async; cache coordinator_ids (#2851, #2862, #2863, #2866, #2867, #2870, #2871) +* Admin: convert request paths to async; cache coordinator_ids (#2851, #2862, #2863, #2866, #2867, #2870, #2871, #3050) * Admin: refactor `_send_request_to_controller` error handling (#2751) ### KIP Support @@ -259,13 +260,15 @@ The admin client interface remains sync but wraps a fully-async internal api (do * Admin: Expand partitions api support (#2890) * Admin: Cleanup alter_partition_reassignments (#3002) - ### Groups * Admin: include latest offsets and lag in list_group_offsets; add reset_group_offsets (#2897) * Admin: support group state/type filters for list_groups (#2910) * Admin: add extended group reset options (#2911) * Admin: Dont return MemberToRemove as key in remove_group_members dict result (#2893) +### Transactions +* Admin: implement KIP-664 hanging-transaction tooling (#3051) + ### Log Dirs * Admin: alter_log_dirs (#2912) * Admin: Fix describe_log_dirs, accept optional topics/brokers (#2881) diff --git a/docs/changelog.rst b/docs/changelog.rst index 9cf126fb9..7f3dee722 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -22,7 +22,7 @@ Default Configuration Changes Abstract Interface Changes ^^^^^^^^^^^^^^^^^^^^^^^^^^ -* Serializer: Pass headers to serialize/deserialize (#3046) +* Serializer/Deserializer: Pass headers to serialize/deserialize (#3046) * Partitioner: pass both key/value, serialized and unserialized, to partition() (#3045) Admin API Changes @@ -152,6 +152,7 @@ Helpers and Debugging * Store in-flight request headers only for protocol parser (#2723) * Debug log send/recv bytes from protocol parser (#2707) * Adjust protocol debug logging; add KAFKA_PYTHON_PROTOCOL_DEBUG_LOG (#2719) +* CoordinatorType enum (GROUP/TRANSACTION/SHARE) (#3049) Broker Version Check -------------------- @@ -284,7 +285,7 @@ Refactor and Async Migration The admin client interface remains sync but wraps a fully-async internal api (does not support asyncio yet). Adds cached coordinator lookups and a mixin structure to separate logical resource groups. * Admin: split into functional mixin classes (#2873, #2877, #2882) -* Admin: convert request paths to async; cache coordinator_ids (#2851, #2862, #2863, #2866, #2867, #2870, #2871) +* Admin: convert request paths to async; cache coordinator_ids (#2851, #2862, #2863, #2866, #2867, #2870, #2871, #3050) * Admin: refactor `_send_request_to_controller` error handling (#2751) KIP Support @@ -320,6 +321,10 @@ Groups * Admin: add extended group reset options (#2911) * Admin: Dont return MemberToRemove as key in remove_group_members dict result (#2893) +Transactions +^^^^^^^^^^^^ +* Admin: implement KIP-664 hanging-transaction tooling (#3051) + Log Dirs ^^^^^^^^ * Admin: alter_log_dirs (#2912) From a23241b38f5a33f6a5a6922479b977f5a26cf039 Mon Sep 17 00:00:00 2001 From: Dana Powers Date: Tue, 9 Jun 2026 15:56:54 -0700 Subject: [PATCH 5/6] fixup commit usage formatting --- docs/usage.rst | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/docs/usage.rst b/docs/usage.rst index 0b3b89cf4..f5367637a 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -111,24 +111,21 @@ KafkaConsumer # Manually commit offsets (disable auto-commit) consumer = KafkaConsumer('my-topic', - group_id='my-group', - enable_auto_commit=False, - bootstrap_servers=['localhost:9092'], - ) + group_id='my-group', + enable_auto_commit=False, + bootstrap_servers=['localhost:9092']) for message in consumer: # process message process_message(message) # TopicPartition for this record tp = TopicPartition(message.topic, message.partition) - # Commit the next offset to consume (message.offset + 1) + # Note: When committing offsets manually, commit the next offset the consumer + # should read. For example, after successfully processing a message at + # offset 42, commit offset 43. consumer.commit({ tp: OffsetAndMetadata(message.offset + 1, '', -1) }) - # When committing offsets manually, commit the next offset the consumer - # should read. For example, after successfully processing a message at - # offset 42, commit offset 43. - # consume earliest available messages, don't commit offsets KafkaConsumer(auto_offset_reset='earliest', enable_auto_commit=False) From bf98cddceda93c3d37d52f1b81d5f4ba170b43d1 Mon Sep 17 00:00:00 2001 From: Dana Powers Date: Wed, 10 Jun 2026 10:26:55 -0700 Subject: [PATCH 6/6] updates; add connection api_version clamp; add DescribeCluster KIP-700 --- CHANGES.md | 15 ++++++++------- docs/changelog.rst | 17 +++++++++-------- 2 files changed, 17 insertions(+), 15 deletions(-) diff --git a/CHANGES.md b/CHANGES.md index f45b50b00..416760721 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -48,6 +48,7 @@ Complete refactor of the networking layer using a bespoke event-loop supporting * kafka.net.manager: Add call_soon(coro) and run(coro) for sync/async bridge (#2862) * kafka.net.manager: Bootstrap is sync/blocking (#2919) * connection: short-circuit send/recv when closed (#2967) +* Clamp broker_version to user-supplied api_version (#3052) ### Transports and Proxies * Default SSLContext -> PROTOCOL_TLS_CLIENT; minimum version TLS 1.2 (#2807) @@ -78,12 +79,12 @@ Defensive checks throughout the kafka.net event loop and transport stack: improv * kafka.net.selector: Use threading.Lock() to detect concurrent access to poll() (#2945) * kafka.net.selector: Track pending tasks to prevent gc before completion (#2950) * kafka.net.selector: Support reschedule(when, task); idempotent unschedule (#2939) -* kafka.net: Raise RuntimeError on concurrent access to net.poll or wakeup() (#2938) -* kafka.net: Raise RuntimeError on run/call_at/call_soon_threadsafe after closed (#2971) -* kafka.net: Catch unhandled exceptions in IO thread (#2970) -* kafka.net: Improve error handling on sock read/write (#2995) +* kafka.net.selector: Raise RuntimeError on concurrent access to net.poll or wakeup() (#2938) +* kafka.net.selector: Raise RuntimeError on run/call_at/call_soon_threadsafe after closed (#2971) +* kafka.net.selector: Catch unhandled exceptions in IO thread (#2970) +* kafka.net.selector: Improve error handling on sock read/write (#2995) +* kafka.net.selector: Check locks in `_poll_once`; add net.drain() (#2949) * kafka.net.transport: Close connection on socket write error (#2973) -* kafka.net: Check locks in _poll_once; add net.drain() (#2949) ## Protocol @@ -238,13 +239,13 @@ The admin client interface remains sync but wraps a fully-async internal api (do * Admin: convert request paths to async; cache coordinator_ids (#2851, #2862, #2863, #2866, #2867, #2870, #2871, #3050) * Admin: refactor `_send_request_to_controller` error handling (#2751) -### KIP Support +### Batch Protocol Support * KIP-699: FindCoordinatorRequest v4 -- multi-group support (#3025) * KIP-709: OffsetFetch v8 -- use batch interface when available (#3024) ### New Cluster and Quorum APIs * Admin: describe_metadata_quorum (#2914) -* Admin: cluster features describe/update (#2908) +* Admin: cluster features describe/update (#2908, #3053) * Admin: cluster get_broker_version_data / api_versions (#2903) ### Configs diff --git a/docs/changelog.rst b/docs/changelog.rst index 7f3dee722..953b79172 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -62,6 +62,7 @@ Async IO Substrate * kafka.net.manager: Add call_soon(coro) and run(coro) for sync/async bridge (#2862) * kafka.net.manager: Bootstrap is sync/blocking (#2919) * connection: short-circuit send/recv when closed (#2967) +* Clamp broker_version to user-supplied api_version (#3052) Transports and Proxies ^^^^^^^^^^^^^^^^^^^^^^ @@ -100,12 +101,12 @@ Defensive checks throughout the kafka.net event loop and transport stack: improv * kafka.net.selector: Use threading.Lock() to detect concurrent access to poll() (#2945) * kafka.net.selector: Track pending tasks to prevent gc before completion (#2950) * kafka.net.selector: Support reschedule(when, task); idempotent unschedule (#2939) -* kafka.net: Raise RuntimeError on concurrent access to net.poll or wakeup() (#2938) -* kafka.net: Raise RuntimeError on run/call_at/call_soon_threadsafe after closed (#2971) -* kafka.net: Catch unhandled exceptions in IO thread (#2970) -* kafka.net: Improve error handling on sock read/write (#2995) +* kafka.net.selector: Raise RuntimeError on concurrent access to net.poll or wakeup() (#2938) +* kafka.net.selector: Raise RuntimeError on run/call_at/call_soon_threadsafe after closed (#2971) +* kafka.net.selector: Catch unhandled exceptions in IO thread (#2970) +* kafka.net.selector: Improve error handling on sock read/write (#2995) +* kafka.net.selector: Check locks in _poll_once; add net.drain() (#2949) * kafka.net.transport: Close connection on socket write error (#2973) -* kafka.net: Check locks in _poll_once; add net.drain() (#2949) Protocol -------- @@ -288,15 +289,15 @@ The admin client interface remains sync but wraps a fully-async internal api (do * Admin: convert request paths to async; cache coordinator_ids (#2851, #2862, #2863, #2866, #2867, #2870, #2871, #3050) * Admin: refactor `_send_request_to_controller` error handling (#2751) -KIP Support -^^^^^^^^^^^ +Batch Protocol Support +^^^^^^^^^^^^^^^^^^^^^^ * KIP-699: FindCoordinatorRequest v4 -- multi-group support (#3025) * KIP-709: OffsetFetch v8 -- use batch interface when available (#3024) New Cluster and Quorum APIs ^^^^^^^^^^^^^^^^^^^^^^^^^^^ * Admin: describe_metadata_quorum (#2914) -* Admin: cluster features describe/update (#2908) +* Admin: cluster features describe/update (#2908, #3053) * Admin: cluster get_broker_version_data / api_versions (#2903) Configs