Skip to content

Handle graceful shutdown for IoP service quadlets #113

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
brantleyr wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Handle graceful shutdown for IoP service quadlets #113

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ac21971
Handle graceful shutdown for IoP service quadlets
brantleyr Oct 8, 2026
37692d9
puppet-lint: fix arrow alignment in Service hashes
brantleyr Oct 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 12 additions & 4 deletions manifests/core_host_inventory.pp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -209,6 +209,9 @@
'Image' => $image,
'ContainerName' => 'iop-core-host-inventory',
'Network' => 'iop-core-network',
# Allow the worker to finish in-flight message processing and close its Kafka
# client before Podman escalates to SIGKILL (default container stop timeout is 10s).
'StopTimeout' => '45',
'Exec' => './inv_mq_service.py',
'Environment' => [
'KAFKA_BOOTSTRAP_SERVERS=PLAINTEXT://iop-core-kafka:9092',
Expand All @@ -225,8 +228,9 @@
],
},
'Service' => {
'Environment' => 'REGISTRY_AUTH_FILE=/etc/foreman/registry-auth.json',
'Restart' => 'on-failure',
'Environment' => 'REGISTRY_AUTH_FILE=/etc/foreman/registry-auth.json',
'Restart' => 'on-failure',
'SuccessExitStatus' => '143',

@vkrizan vkrizan Oct 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: the number can be replaced by the signal name.

@vkrizan vkrizan Oct 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Never mind, this is a combination of sigterm and graceful shutdown.

https://www.dash0.com/guides/kubernetes-exit-code-143-a-practical-guide

},
'Install' => {
'WantedBy' => ['multi-user.target', 'default.target'],
Expand Down Expand Up @@ -257,6 +261,9 @@
'Image' => $image,
'ContainerName' => 'iop-core-host-inventory-api',
'Network' => 'iop-core-network',
# Must exceed gunicorn's graceful_timeout (30s) so workers drain before Podman
# escalates to SIGKILL (default container stop timeout is 10s).
'StopTimeout' => '45',
'Exec' => 'python run_gunicorn.py',
'Environment' => [
'KAFKA_BOOTSTRAP_SERVERS=iop-core-kafka:9092',
Expand All @@ -275,8 +282,9 @@
],
},
'Service' => {
'Environment' => 'REGISTRY_AUTH_FILE=/etc/foreman/registry-auth.json',
'Restart' => 'on-failure',
'Environment' => 'REGISTRY_AUTH_FILE=/etc/foreman/registry-auth.json',
'Restart' => 'on-failure',
'SuccessExitStatus' => '143',
},
'Install' => {
'WantedBy' => ['multi-user.target', 'default.target'],
Expand Down
3 changes: 3 additions & 0 deletions manifests/core_ingress.pp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,9 @@
'Image' => $image,
'ContainerName' => 'iop-core-ingress',
'Network' => 'iop-core-network',
# Give the bounded HTTP/telemetry shutdown (~10s in-app) headroom before Podman
# escalates to SIGKILL (default container stop timeout is 10s).
'StopTimeout' => '20',
'Volume' => '/var/tmp/insights-archives:/var/tmp/insights-archives:rw,Z',
'Environment' => [
'INGRESS_VALID_UPLOAD_TYPES=advisor,compliance,qpc,rhv,tower,leapp-reporting,xavier,playbook,playbook-sat,malware-detection,tasks',
Expand Down
5 changes: 3 additions & 2 deletions manifests/core_kafka.pp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,8 +62,9 @@
],
},
'Service' => {
'Environment' => 'REGISTRY_AUTH_FILE=/etc/foreman/registry-auth.json',
'Restart' => 'on-failure',
'Environment' => 'REGISTRY_AUTH_FILE=/etc/foreman/registry-auth.json',
'Restart' => 'on-failure',
'SuccessExitStatus' => '143',
},
'Install' => {
'WantedBy' => ['multi-user.target', 'default.target'],
Expand Down
15 changes: 15 additions & 0 deletions manifests/service_vulnerability.pp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -223,6 +223,9 @@
'Image' => $image,
'ContainerName' => 'iop-service-vuln-taskomatic',
'Network' => 'iop-core-network',
# The scheduler stops with wait=True, blocking until an in-progress job finishes.
# Give a running job room to complete before Podman escalates to SIGKILL (default 10s).
'StopTimeout' => '30',
'Exec' => '/engine/entrypoint.sh taskomatic',
'Volume' => $socket_volume,
'Environment' => [
Expand Down Expand Up @@ -271,6 +274,9 @@
'Image' => $image,
'ContainerName' => 'iop-service-vuln-grouper',
'Network' => 'iop-core-network',
# Give the service time to finish in-flight Kafka message handling and close its
# aiokafka clients before Podman escalates to SIGKILL (default stop timeout is 10s).
'StopTimeout' => '30',
'Exec' => '/engine/entrypoint.sh grouper',
'Volume' => $socket_volume,
'Environment' => [
Expand Down Expand Up @@ -323,6 +329,9 @@
'Image' => $image,
'ContainerName' => 'iop-service-vuln-listener',
'Network' => 'iop-core-network',
# Give the service time to finish in-flight Kafka message handling and close its
# aiokafka clients before Podman escalates to SIGKILL (default stop timeout is 10s).
'StopTimeout' => '30',
'Exec' => '/engine/entrypoint.sh listener',
'Volume' => $socket_volume,
'Environment' => [
Expand Down Expand Up @@ -376,6 +385,9 @@
'Image' => $image,
'ContainerName' => 'iop-service-vuln-evaluator-recalc',
'Network' => 'iop-core-network',
# Give the service time to finish in-flight Kafka message handling and close its
# aiokafka clients before Podman escalates to SIGKILL (default stop timeout is 10s).
'StopTimeout' => '30',
'Exec' => '/engine/entrypoint.sh evaluator',
'Volume' => $socket_volume,
'Environment' => [
Expand Down Expand Up @@ -429,6 +441,9 @@
'Image' => $image,
'ContainerName' => 'iop-service-vuln-evaluator-upload',
'Network' => 'iop-core-network',
# Give the service time to finish in-flight Kafka message handling and close its
# aiokafka clients before Podman escalates to SIGKILL (default stop timeout is 10s).
'StopTimeout' => '30',
'Exec' => '/engine/entrypoint.sh evaluator',
'Volume' => $socket_volume,
'Environment' => [
Expand Down
Loading