Skip to content

Fixes #39229 - Add foreman_request_timeout setting to control Net::HTTP read timeout - #936

Open
pablomh wants to merge 1 commit into
theforeman:developfrom
pablomh:fix-foreman-request-timeout
Open

pablomh wants to merge 1 commit into
theforeman:developfrom
pablomh:fix-foreman-request-timeout

Conversation

@pablomh

@pablomh pablomh commented Apr 13, 2026 •

Copy link
Copy Markdown
Contributor

Problem

ForemanRequest uses a bare Net::HTTP instance whose read_timeout defaults to Ruby's hardcoded 60s. Under high-concurrency registration load this causes 500 errors when Foreman takes longer than 60s to process POST /register.

Fix

Introduces :foreman_request_timeout and documents it in settings.yml.example, then applies it to http.read_timeout in ForemanRequest#http_init.

subscription-manager's own client-side server_timeout already defaults to 180s, so smart-proxy cutting the connection at 60s was strictly bad — it turned a slow-but-recoverable request into a hard failure. The setting now defaults to 180s when unset, matching subscription-manager, instead of silently inheriting Ruby's unrelated 60s default. :foreman_request_timeout: 0 is kept as an escape hatch back to that 60s behavior.

Validated with an isolated A/B in a lab environment, ramping concurrent registrations up, on both an RPM/puppet-based deployment and a containerized (foremanctl) deployment: raising the timeout to 180s reduced failures in the moderate-to-high concurrency range, though it doesn't help at the very top end where failures come from genuine backend saturation rather than this timeout.

Fixes https://projects.theforeman.org/issues/39229

@pablomh

pablomh commented Apr 17, 2026

Copy link
Copy Markdown
Contributor Author

CI failure: rake 13.4.2 regression (not related to this PR)

All test jobs fail with rake_test_loader: version unknown and exit code 1. The test runner never actually runs any tests.

Root cause: gem 'rake' in bundler.d/test.rb has no version constraint and no Gemfile.lock is committed, so CI resolves the latest version from rubygems.org. Rake 13.4.2 (released April 16) changed Rake::TestTask#option_list to append -v to the test command when verbose = true — previously this flag was not passed to the test runner. The new -v causes test-unit 3.7.7 to fail silently.

This affects all new PRs on smart-proxy, not just this one. PR #935 passed because its bundle cache still had rake 13.3.1.

Diff in lib/rake/testtask.rb (13.3.1 → 13.4.2):

# 13.3.1 — option_list never adds -v
def option_list
  (ENV["TESTOPTS"] || ENV["TESTOPT"] || ENV["TEST_OPTS"] || ENV["TEST_OPT"] || @options || "")
end

# 13.4.2 — now adds -v when verbose is true
def option_list(verbose: @verbose)
  opts = ENV["TESTOPTS"] || ENV["TESTOPT"] || ENV["TEST_OPTS"] || ENV["TEST_OPT"] || @options || ""
  if verbose && !opts.split.include?("-v")
    opts = opts.empty? ? "-v" : "#{opts} -v"
  end
  opts
end

Fix options:

  • Pin gem 'rake', '< 13.4' in bundler.d/test.rb (workaround)
  • Report the regression upstream to ruby/rake
  • Or fix the test-unit interaction with -v if the issue is on that side

@ekohl ekohl left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense, but 1 minor comment inline.

Comment thread lib/proxy/request.rb Outdated
@pablomh

pablomh commented May 7, 2026

Copy link
Copy Markdown
Contributor Author

Friendly ping.

@pablomh
pablomh force-pushed the fix-foreman-request-timeout branch from 8071686 to f4925d8 Compare June 28, 2026 15:21
@pablomh

pablomh commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

Ping.

@pablomh
pablomh force-pushed the fix-foreman-request-timeout branch from f4925d8 to 64d7652 Compare September 18, 2026 09:42
…TP read timeout

ForemanRequest uses a bare Net::HTTP instance whose read_timeout defaults to
Ruby's hardcoded 60s. Under high-concurrency registration load this causes
500 errors when Foreman takes longer than 60s to process POST /register.

Introduces :foreman_request_timeout (documented in settings.yml.example) and
applies it to http.read_timeout in ForemanRequest#http_init.

subscription-manager's default server_timeout is 180s, so the setting now
defaults to 180s when unset instead of silently inheriting Ruby's unrelated
60s Net::HTTP default. Set it to 0 to opt back into that 60s default.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
@pablomh
pablomh force-pushed the fix-foreman-request-timeout branch from 64d7652 to 4b30a6b Compare September 18, 2026 09:56
@pablomh

pablomh commented Sep 28, 2026

Copy link
Copy Markdown
Contributor Author

I've set the default to 180s to match subscription-manager's server_timeout.

@jeremylenz jeremylenz left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code & logic LGTM (did not test)

@pablomh
pablomh requested a review from ekohl September 30, 2026 05:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants