Skip to content

Handle graceful shutdown for IoP service quadlets - #113

Open
brantleyr wants to merge 2 commits into
theforeman:masterfrom
brantleyr:graceful-shutdown
Open

brantleyr wants to merge 2 commits into
theforeman:masterfrom
brantleyr:graceful-shutdown

Conversation

@brantleyr

@brantleyr brantleyr commented Oct 8, 2026 •

Copy link
Copy Markdown

Problem

When IoP services are stopped (e.g. systemctl stop foreman.target) several units end up failed: some because they exit on SIGTERM (143), which systemd treats as a failure, and some because they are force-killed (137) before they finish.

Change

  • SuccessExitStatus=143 for services that exit on SIGTERM (kafka, host-inventory worker and API).
  • StopTimeout headroom for services that may be mid-work at shutdown: vulnerability consumers and taskomatic (30s), host-inventory (45s), ingress (20s).

Testing

Verified on a Foreman development instance running IoP: after the change systemctl stop foreman.target leaves no failed units and no SIGKILL escalation; services stop cleanly and the target restarts.

Related

Part of the IoP graceful-shutdown fix:

Add SuccessExitStatus=143 for services that exit on SIGTERM (kafka,
host-inventory worker and API) so a clean SIGTERM stop is not reported
as failed, and add StopTimeout headroom for services that may be
mid-work at shutdown (vulnerability consumers and taskomatic, 30s;
host-inventory, 45s; ingress, 20s) so they stop cleanly instead of
being force-killed.
@brantleyr

Copy link
Copy Markdown
Author

@vkrizan @pfreyburg ready for your review.

'Service' => {
'Environment' => 'REGISTRY_AUTH_FILE=/etc/foreman/registry-auth.json',
'Restart' => 'on-failure',
'SuccessExitStatus' => '143',

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: the number can be replaced by the signal name.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Never mind, this is a combination of sigterm and graceful shutdown.

https://www.dash0.com/guides/kubernetes-exit-code-143-a-practical-guide

@vkrizan

vkrizan commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Note, that SIGTERM is 15.

@ehelms

ehelms commented Oct 12, 2026

Copy link
Copy Markdown
Member

@brantleyr Can you fix the validation errors?

@brantleyr

Copy link
Copy Markdown
Author

@brantleyr Can you fix the validation errors?

@ehelms Should be fixed now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants