Relay v0.40.0 causing 100% CPU spike

[mrcave]

After upgrading to v0.40.0 we're seeing CPU spontaneously maxing out on PHP workers and staying that way indefinitely.

This is on a QA/DEV server where traffic doesn't explain the anomaly, and only change made was the relay upgrade to 0.40.

@michael-grunder what else would you need to help diagnose/troubleshoot?

[szepeviktor]

Interesting that 1 PHP process causes 200% load.

[jdreesen]

It's PHP-FPM (PHP FastCGI Process Manager) which can spawn multiple PHP processes.

[michael-grunder]

What will be initially useful is

1. The output from phpinfo()
2. The values you are setting in relay.ini
3. A callstack (e.g. perf top) showing what Relay is doing when it's spinning

If I'm not mistaken you have had this same issue in several versions of Relay in a row surviving large-scale redesigns of the internals. There must be somehing unique about this setup or server.

One thing I know would cause this problem is if an fpm worker was killed was kill -9ed and you are using spinlocks for the cache. Relay's mutex based cache locks are robust and can detect that a worker is dead while holding the lock. The spinlocks however are much lower level so they'll just spin.

Edit: I am also going to rework our spinlocks to also be robust so Relay will just crash rather than spin.

[michael-grunder]

Fix is merged. It'll be in nightly builds tomorrow. I am going to keep running our fuzz testing suite and tag soon

[mrcave]

@michael-grunder great will look into deploying this to our QA env this week. I see 0.50 is tagged now - is the fix in that version?

[michael-grunder]

Yep, fix is in. On Linux you can pin signal handlers to a thread so the fix should solve the instability but actually make Relay a bit more effieent.

[mrcave]

@michael-grunder we're two weeks in on 0.50 deployment in our QA env and have seen no problems with CPU spiking. We'll be promoting to STG and PROD in the next few weeks. I think this may have finally fixed the problem 🤞

[michael-grunder]

Awesome to hear!