Skip to content

Switch production deployment to gunicorn#855

Open
EvanDietzMorris wants to merge 2 commits into
masterfrom
deployment-upgrades
Open

Switch production deployment to gunicorn#855
EvanDietzMorris wants to merge 2 commits into
masterfrom
deployment-upgrades

Conversation

@EvanDietzMorris

@EvanDietzMorris EvanDietzMorris commented Jun 11, 2026

Copy link
Copy Markdown

Unless there is another helm chart living somewhere else, we're running Django's development server (manage.py runserver) in production. That is not recommended by Django and can cause serious performance issues:

  • It's just one python process with one GIL. CPU intensive tasks block every other process.
  • It doesn't have great restart and recycling capabilities, if it gets to a completely hung state there's no recourse and the best case scenario is k8s kills it and spins up an entirely new instance with hard down time in between

Switching to gunicorn with workers solves a lot of those issues:

  • Workers have their own python processes with their own GIL which enables real concurrency that can scale by adding more cpu & workers (cpu hogging tasks only block their own worker)
  • Workers can crash and restart and are automatically recycled periodically intentionally, while other workers are still processing requests
  • Better logging and understanding of these restarts

If we adopt this, we'll want to add the new gunicorn values to our ncats-values files, and it's strongly recommended to tune the number of workers according to the amount of CPU we have allocated. Comments in the chart show where/how to do that.

Along with this, it is highly recommended that we double check the settings not committed to this repo (deploy/configs/settings.py) and make sure Django is not being run in DEBUG mode for CI, test, or prod, and probably we don't need DJANGO_LOG_LEVEL=DEBUG there either. It might be nice to use environment variables configurable from helm values for these to make it easy to set them per-maturity-level.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well done! You have taken the first step into a larger world

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants