Skip to content

feat(kafka-connect): reset connector using REST API, remove custom resetter #646

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
disrupted merged 25 commits into from
Jul 29, 2026
+286 −810
Merged

feat(kafka-connect): reset connector using REST API, remove custom resetter #646

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
72c94ed
feat(kafka-connect): reset connector using REST API, remove custom re…
disrupted Apr 9, 2026
8e076ba
test(pipeline): update snapshots
disrupted Jul 1, 2026
b11835e
test: update
disrupted Jul 1, 2026
3c06aa4
chore(dprint): ignore changelog
disrupted Jul 1, 2026
46d4527
test(word-count): update snapshot
disrupted Jul 1, 2026
f245297
Merge remote-tracking branch 'origin/main' into kafka-connect-reset
disrupted Jul 1, 2026
a2d4966
feat(kafka-connect): implement dry-run for reset connector
disrupted Jul 1, 2026
3754421
test: update
disrupted Jul 1, 2026
e429e8c
test: cleanup leftover resources
disrupted Jul 2, 2026
bda7370
fix(kafka-connect): handle connector does not exist on stop
disrupted Jul 28, 2026
acf2aa8
feat(kafka-connect): temporarily deploy missing connector to reset
disrupted Jul 28, 2026
87ea315
test: improve isolation
disrupted Jul 28, 2026
12a1a69
ci: update uv
disrupted Jul 28, 2026
5e40560
fix(dprint): ignore changelog formatting
disrupted Jul 28, 2026
3fc0542
ci: set main Python and uv version as repository variables
disrupted Jul 28, 2026
6e46061
Merge branch 'update-uv' into kafka-connect-reset
disrupted Jul 28, 2026
cb46981
Merge branch 'main' into kafka-connect-reset
disrupted Jul 28, 2026
e1c3f26
Merge branch 'main' into kafka-connect-reset
disrupted Jul 28, 2026
8b07134
chore(dev): report missing Python type annotations (#651)
disrupted Jul 28, 2026
8dcb20d
Merge remote-tracking branch 'origin/main' into kafka-connect-reset
disrupted Jul 28, 2026
b0e25b4
test: add missing return type annotations
disrupted Jul 28, 2026
42bf1d0
refactor(kafka-connector): destroy after reset
disrupted Jul 29, 2026
e96a6bf
refactor(kafka-connect): improve API response status checks
disrupted Jul 29, 2026
32cec46
test(kafka-connect): add coverage for reset offset API
disrupted Jul 29, 2026
811b80f
refactor(kafka-connector): remove offset_topic attribute
disrupted Jul 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions .../resources/pipeline-components/dependencies/defaults_pipeline_component_dependencies.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,11 +13,9 @@ kafka-connector.yaml:
- from_.yaml
- to.yaml
- config-kafka-connector.yaml
- resetter_values.yaml
kafka-sink-connector.yaml: []
kafka-source-connector.yaml:
- from_-kafka-source-connector.yaml
- offset_topic-kafka-source-connector.yaml
kubernetes-app.yaml:
- prefix.yaml
- from_.yaml
Expand Down
7 changes: 0 additions & 7 deletions docs/docs/resources/pipeline-components/dependencies/kpops_structure.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,8 +37,6 @@ kpops_components_fields:
- to
- config
- state
- resetter_namespace
- resetter_values
kafka-sink-connector:
- name
- enabled
Expand All @@ -47,8 +45,6 @@ kpops_components_fields:
- to
- config
- state
- resetter_namespace
- resetter_values
kafka-source-connector:
- name
- enabled
Expand All @@ -57,9 +53,6 @@ kpops_components_fields:
- to
- config
- state
- resetter_namespace
- resetter_values
- offset_topic
kubernetes-app:
- name
- enabled
Expand Down
4 changes: 0 additions & 4 deletions docs/docs/resources/pipeline-components/dependencies/pipeline_component_dependencies.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,20 +23,16 @@ kafka-connector.yaml:
- from_.yaml
- to.yaml
- config-kafka-connector.yaml
- resetter_values.yaml
kafka-sink-connector.yaml:
- prefix.yaml
- from_.yaml
- to.yaml
- config-kafka-connector.yaml
- resetter_values.yaml
kafka-source-connector.yaml:
- prefix.yaml
- from_-kafka-source-connector.yaml
- to.yaml
- config-kafka-connector.yaml
- resetter_values.yaml
- offset_topic-kafka-source-connector.yaml
kubernetes-app.yaml:
- prefix.yaml
- from_.yaml
Expand Down
4 changes: 0 additions & 4 deletions docs/docs/resources/pipeline-components/kafka-connector.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,3 @@
# Full documentation on connectors: https://kafka.apache.org/documentation/#connectconfigs
config: # required
tasks.max: 1
# Overriding Kafka Connect Resetter Helm values. E.g. to override the
# Image Tag etc.
resetter_values:
imageTag: "1.2.3"
4 changes: 0 additions & 4 deletions docs/docs/resources/pipeline-components/kafka-sink-connector.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,3 @@
# Full documentation on connectors: https://kafka.apache.org/documentation/#connectconfigs
config: # required
tasks.max: 1
# Overriding Kafka Connect Resetter Helm values. E.g. to override the
# Image Tag etc.
resetter_values:
imageTag: "1.2.3"
7 changes: 0 additions & 7 deletions docs/docs/resources/pipeline-components/kafka-source-connector.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,10 +27,3 @@
# Full documentation on connectors: https://kafka.apache.org/documentation/#connectconfigs
config: # required
tasks.max: 1
# Overriding Kafka Connect Resetter Helm values. E.g. to override the
# Image Tag etc.
resetter_values:
imageTag: "1.2.3"
# offset.storage.topic
# https://kafka.apache.org/documentation/#connect_running
offset_topic: offset_topic
11 changes: 0 additions & 11 deletions docs/docs/resources/pipeline-components/pipeline.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -248,10 +248,6 @@
# Full documentation on connectors: https://kafka.apache.org/documentation/#connectconfigs
config: # required
tasks.max: 1
# Overriding Kafka Connect Resetter Helm values. E.g. to override the
# Image Tag etc.
resetter_values:
imageTag: "1.2.3"
# Kafka source connector
- type: kafka-source-connector # required
name: kafka-source-connector # required
Expand Down Expand Up @@ -281,13 +277,6 @@
# Full documentation on connectors: https://kafka.apache.org/documentation/#connectconfigs
config: # required
tasks.max: 1
# Overriding Kafka Connect Resetter Helm values. E.g. to override the
# Image Tag etc.
resetter_values:
imageTag: "1.2.3"
# offset.storage.topic
# https://kafka.apache.org/documentation/#connect_running
offset_topic: offset_topic
# Base Kubernetes App
- type: kubernetes-app
name: kubernetes-app # required
Expand Down
3 changes: 0 additions & 3 deletions docs/docs/resources/pipeline-components/sections/offset_topic-kafka-source-connector.yaml
View file
Open in desktop

This file was deleted.

4 changes: 0 additions & 4 deletions docs/docs/resources/pipeline-components/sections/resetter_values.yaml
View file
Open in desktop

This file was deleted.

4 changes: 0 additions & 4 deletions docs/docs/resources/pipeline-defaults/defaults-kafka-connector.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,3 @@ kafka-connector:
# Full documentation on connectors: https://kafka.apache.org/documentation/#connectconfigs
config: # required
tasks.max: 1
# Overriding Kafka Connect Resetter Helm values. E.g. to override the
# Image Tag etc.
resetter_values:
imageTag: "1.2.3"
3 changes: 0 additions & 3 deletions docs/docs/resources/pipeline-defaults/defaults-kafka-source-connector.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,3 @@
kafka-source-connector:
# The source connector has no `from` section
# from:
# offset.storage.topic
# https://kafka.apache.org/documentation/#connect_running
offset_topic: offset_topic
7 changes: 0 additions & 7 deletions docs/docs/resources/pipeline-defaults/defaults.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -172,10 +172,6 @@ kafka-connector:
# Full documentation on connectors: https://kafka.apache.org/documentation/#connectconfigs
config: # required
tasks.max: 1
# Overriding Kafka Connect Resetter Helm values. E.g. to override the
# Image Tag etc.
resetter_values:
imageTag: "1.2.3"
# Kafka sink connector
#
# Child of: KafkaConnector
Expand All @@ -187,9 +183,6 @@ kafka-sink-connector:
kafka-source-connector:
# The source connector has no `from` section
# from:
# offset.storage.topic
# https://kafka.apache.org/documentation/#connect_running
offset_topic: offset_topic
# Base Kubernetes App
#
# Parent of: HelmApp
Expand Down
64 changes: 0 additions & 64 deletions docs/docs/schema/defaults.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1397,23 +1397,6 @@
"title": "Prefix",
"type": "string"
},
"resetter_namespace": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "Kubernetes namespace in which the Kafka Connect resetter shall be deployed",
"title": "Resetter Namespace"
},
"resetter_values": {
"$ref": "#/$defs/HelmAppValues",
"description": "Overriding Kafka Connect resetter Helm values, e.g. to override the image tag etc."
},
"state": {
"anyOf": [
{
Expand Down Expand Up @@ -1548,23 +1531,6 @@
"title": "Prefix",
"type": "string"
},
"resetter_namespace": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "Kubernetes namespace in which the Kafka Connect resetter shall be deployed",
"title": "Resetter Namespace"
},
"resetter_values": {
"$ref": "#/$defs/HelmAppValues",
"description": "Overriding Kafka Connect resetter Helm values, e.g. to override the image tag etc."
},
"state": {
"anyOf": [
{
Expand Down Expand Up @@ -1635,42 +1601,12 @@
"title": "Name",
"type": "string"
},
"offset_topic": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "`offset.storage.topic`, more info: https://kafka.apache.org/documentation/#connect_running",
"title": "Offset Topic"
},
"prefix": {
"default": "${pipeline.name}-",
"description": "Pipeline prefix that will prefix every component name. If you wish to not have any prefix you can specify an empty string.",
"title": "Prefix",
"type": "string"
},
"resetter_namespace": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "Kubernetes namespace in which the Kafka Connect resetter shall be deployed",
"title": "Resetter Namespace"
},
"resetter_values": {
"$ref": "#/$defs/HelmAppValues",
"description": "Overriding Kafka Connect resetter Helm values, e.g. to override the image tag etc."
},
"state": {
"anyOf": [
{
Expand Down
47 changes: 0 additions & 47 deletions docs/docs/schema/pipeline.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1353,23 +1353,6 @@
"title": "Prefix",
"type": "string"
},
"resetter_namespace": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "Kubernetes namespace in which the Kafka Connect resetter shall be deployed",
"title": "Resetter Namespace"
},
"resetter_values": {
"$ref": "#/$defs/HelmAppValues",
"description": "Overriding Kafka Connect resetter Helm values, e.g. to override the image tag etc."
},
"state": {
"anyOf": [
{
Expand Down Expand Up @@ -1440,42 +1423,12 @@
"title": "Name",
"type": "string"
},
"offset_topic": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "`offset.storage.topic`, more info: https://kafka.apache.org/documentation/#connect_running",
"title": "Offset Topic"
},
"prefix": {
"default": "${pipeline.name}-",
"description": "Pipeline prefix that will prefix every component name. If you wish to not have any prefix you can specify an empty string.",
"title": "Prefix",
"type": "string"
},
"resetter_namespace": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "Kubernetes namespace in which the Kafka Connect resetter shall be deployed",
"title": "Resetter Namespace"
},
"resetter_values": {
"$ref": "#/$defs/HelmAppValues",
"description": "Overriding Kafka Connect resetter Helm values, e.g. to override the image tag etc."
},
"state": {
"anyOf": [
{
Expand Down
2 changes: 0 additions & 2 deletions docs/docs/user/core-concepts/defaults.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,6 @@ KPOps has a very efficient way of dealing with repeating settings which manifest

An important mechanic of KPOps is that `defaults` set for a component apply to all components that inherit from it.

It is possible, although not recommended, to add settings that are specific to a component's subclass. An example would be configuring `offset_topic` under `kafka-connector` instead of `kafka-source-connector`.

### Configuration

KPOps allows using multiple default values. The `defaults.yaml` (or `defaults_<env>.yaml`) files can be distributed across multiple files. These will be picked up by KPOps and get merged into a single `pipeline.yaml` file.
Expand Down
Loading
Loading