Managing Django’s Queue

With the Django 5.2 prerelease cycle now in play, eyes can turn to what might come in for Django 6.0. Top of that list is the interface part of django-tasks, the proposal to bring background task handling into Django.

If you’ve run any queuing system for any length of time, you’ll know that they don’t like being run at anywhere near capacity. You think it’s your queue — Oh it’s Celery’s fault again — but it’s not: it’s just math. I’ll quote Dan Slimmon’s classic article on the topic, The Most Important Thing to Understand About Queues:

As you approach maximum throughput, average queue size – and therefore average wait time – approaches infinity.

If you run your queue hot, it backs up fatally. (I’m not going to explain that. Go read the article. But it’s true.)