Skip to content

Handle graceful shutdown for IoP service quadlets #926

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
evgeni merged 1 commit into from
Oct 12, 2026
Merged

Handle graceful shutdown for IoP service quadlets #926

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions src/roles/iop_ingress/tasks/main.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@
name: iop-core-ingress
image: iop-ingress.image
state: quadlet
# Give the bounded in-app HTTP/telemetry shutdown headroom before SIGKILL.
stop_timeout: 20
env:
INGRESS_VALID_UPLOAD_TYPES: "advisor,compliance,qpc,rhv,tower,leapp-reporting,xavier,playbook,playbook-sat,malware-detection,tasks"
INGRESS_KAFKABROKERS: "iop-core-kafka:9092"
Expand Down
6 changes: 6 additions & 0 deletions src/roles/iop_inventory/tasks/main.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,8 @@
name: iop-core-host-inventory
image: iop-host-inventory.image
state: quadlet
# Give the worker time to finish and close its Kafka client before SIGKILL.
stop_timeout: 45
command: make run_inv_mq_service
network: "{{ foreman_core_network_container_networks }}"
env:
Expand All @@ -89,6 +91,7 @@
PartOf=foreman.target
[Service]
Restart=on-failure
SuccessExitStatus=143

@vkrizan vkrizan Oct 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this standard to handle the 143 like this? Should the foreman services benefit from this as well?

cc @ekohl @evgeni

@evgeni evgeni Oct 9, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd prefer not to handle it like this, but I've also certainly seen Foreman things end up in "failed" state with 143.

Is having a sufficiently sized stop_timeout not sufficient to avoid being SIGTERM'ed?

@vkrizan vkrizan Oct 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I came across this article that explains the 143 a bit https://www.dash0.com/guides/kubernetes-exit-code-143-a-practical-guide

@evgeni evgeni Oct 9, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interestingly, https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html#SuccessExitStatus= suggests that SIGTERM should already be considered successful. I'll need to dig that more on Monday.

@vkrizan vkrizan Oct 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

143 is not SIGTERM

@vkrizan vkrizan Oct 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are reviewing container exit codes after a deployment or node drain and see 143. If your monitoring alerts on it, you might think something failed. Exit code 143 is not an error. It is 128 plus signal 15 (SIGTERM), and it means the container’s PID 1 process received SIGTERM and exited voluntarily. This is exactly what docker stop is designed to do.

https://www.netdata.cloud/guides/docker/docker-exit-code-143/

This seems to be some container specific signal reporting.

@evgeni evgeni Oct 9, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, it's sigterm plus container shenanigans. gotcha.

[Install]
WantedBy=default.target foreman.target

Expand All @@ -97,6 +100,8 @@
name: iop-core-host-inventory-api
image: iop-host-inventory.image
state: quadlet
# Exceed gunicorn's graceful timeout so workers drain before SIGKILL.
stop_timeout: 45
command: python run_gunicorn.py
network: "{{ foreman_core_network_container_networks }}"
env:
Expand All @@ -121,6 +126,7 @@
PartOf=foreman.target
[Service]
Restart=on-failure
SuccessExitStatus=143
[Install]
WantedBy=default.target foreman.target

Expand Down
1 change: 1 addition & 0 deletions src/roles/iop_kafka/tasks/main.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@
PartOf=foreman.target
[Service]
Restart=on-failure
SuccessExitStatus=143
[Install]
WantedBy=default.target foreman.target

Expand Down
10 changes: 10 additions & 0 deletions src/roles/iop_vulnerability/tasks/main.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,8 @@
name: iop-service-vuln-taskomatic
image: iop-vulnerability-engine.image
state: quadlet
# Let an in-progress scheduled job finish before Podman escalates to SIGKILL.
stop_timeout: 30
quadlet_dir: /etc/containers/systemd
network: "{{ foreman_core_network_container_networks }}"
command: "/engine/entrypoint.sh taskomatic"
Expand Down Expand Up @@ -143,6 +145,8 @@
name: iop-service-vuln-grouper
image: iop-vulnerability-engine.image
state: quadlet
# Allow in-flight Kafka message handling and client cleanup to finish before SIGKILL.
stop_timeout: 30
quadlet_dir: /etc/containers/systemd
network: "{{ foreman_core_network_container_networks }}"
command: "/engine/entrypoint.sh grouper"
Expand Down Expand Up @@ -181,6 +185,8 @@
name: iop-service-vuln-listener
image: iop-vulnerability-engine.image
state: quadlet
# Allow in-flight Kafka message handling and client cleanup to finish before SIGKILL.
stop_timeout: 30
quadlet_dir: /etc/containers/systemd
network: "{{ foreman_core_network_container_networks }}"
command: "/engine/entrypoint.sh listener"
Expand Down Expand Up @@ -219,6 +225,8 @@
name: iop-service-vuln-evaluator-recalc
image: iop-vulnerability-engine.image
state: quadlet
# Allow in-flight Kafka message handling and client cleanup to finish before SIGKILL.
stop_timeout: 30
quadlet_dir: /etc/containers/systemd
network: "{{ foreman_core_network_container_networks }}"
command: "/engine/entrypoint.sh evaluator"
Expand Down Expand Up @@ -257,6 +265,8 @@
name: iop-service-vuln-evaluator-upload
image: iop-vulnerability-engine.image
state: quadlet
# Allow in-flight Kafka message handling and client cleanup to finish before SIGKILL.
stop_timeout: 30
quadlet_dir: /etc/containers/systemd
network: "{{ foreman_core_network_container_networks }}"
command: "/engine/entrypoint.sh evaluator"
Expand Down
Loading