Flower + Celery on Django: How I Found Silently Failing Tasks on Tabeer.ai

The Problem That Started This

On Tabeer.ai, some transactional emails — verification links, password resets — just weren't showing up for a subset of users. Not all of them, not consistently, and nothing was throwing an obvious error anyone was watching. Digging in, the root cause was Celery tasks failing outright, with no monitoring layer surfacing that a task had failed at all — the only trace was in worker logs you'd have to already know to go grep. A user not getting an email and a developer having no idea a task crashed are the same failure from two different vantage points, and closing that gap is the entire point of what I set up.

Worth being precise about what this fixes and what it doesn't: Flower gives you visibility into a task failing, retrying, or timing out. It does not fix a task that succeeds from Celery's point of view but whose actual effect fails downstream — a send_mail() call that returns cleanly but gets rejected later by the receiving mail server, for instance, is invisible to Flower too, because as far as Celery is concerned, that task worked. Flower closes the "silent crash" gap, not the "downstream success/failure of what the task did" gap. Both matter; they're different problems with different fixes.

Install

pip install flower

Standard, no surprises. One version pin worth being explicit about in requirements.txt if you're touching a real production deploy pipeline, same as any other dependency:

flower>=2.1.0

Running It as a systemd Service — the Gotcha

Flower needs to run continuously, same as a Celery worker. The natural move is a systemd unit matching the existing worker/beat services:

[Unit]
Description=Tabeer.ai Celery Flower (monitoring)
After=network.target redis-server.service

[Service]
User=ubuntu
Group=ubuntu
WorkingDirectory=/opt/tabeerdotai/backend
ExecStart=/opt/tabeerdotai/backend/.venv/bin/celery -A config flower --address=127.0.0.1 --port=5555 --basic_auth=admin:somepassword
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

That works — but it puts the password in plaintext in a file at /etc/systemd/system/, readable by anyone with sudo on the box. Moving it to an env file is one line of config away in theory:

EnvironmentFile=/opt/tabeerdotai/backend/flower.env
ExecStart=/opt/tabeerdotai/backend/.venv/bin/celery -A config flower --basic_auth=${FLOWER_BASIC_AUTH}

This does not work as written, and the failure mode is quiet: Celery starts fine, but the actual basic-auth credential Flower receives is the literal eight-character string ${FLOWER_BASIC_AUTH} — because systemd's ExecStart execs the binary directly. It never goes through a shell, so it never does variable expansion. EnvironmentFile correctly populates the process environment; ExecStart just never reads from it inline like that.

The actual fix is routing the whole command through a shell explicitly:

EnvironmentFile=/opt/tabeerdotai/backend/flower.env
ExecStart=/bin/sh -c '/opt/tabeerdotai/backend/.venv/bin/celery -A config flower --address=127.0.0.1 --port=5555 --basic_auth=${FLOWER_BASIC_AUTH} --max_tasks=2000'

/bin/sh -c '...' gives you a shell that does expand ${FLOWER_BASIC_AUTH} before Celery ever sees the argument — and the credential itself lives only in flower.env (chmod 600, outside version control), not in a world-readable unit file.

# flower.env
FLOWER_BASIC_AUTH=admin:<strong generated password>

Bind to 127.0.0.1 only — Flower should never be reachable directly from the internet on its own port; it goes through a reverse proxy, next.

Exposing It: a Subdomain, Not a Path

The obvious move is tabeer.ai/flower/. Skip it — Flower's --url_prefix option for subpath-serving has real, documented friction with its static assets and websocket connection unless configured exactly right. A dedicated subdomain sidesteps the whole problem and, if you already have a wildcard TLS cert (*.yourdomain.com), costs nothing extra to set up:

server {
    listen 80;
    server_name flower.tabeer.ai;
    return 301 https://flower.tabeer.ai$request_uri;
}

server {
    listen 443 ssl;
    http2 on;
    server_name flower.tabeer.ai;
    ssl_certificate     /etc/ssl/certs/tabeer_origin.pem;
    ssl_certificate_key /etc/ssl/private/tabeer_origin.key;

    location / {
        proxy_pass http://127.0.0.1:5555;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

Then one DNS record — flower.tabeer.ai → the server's IP, proxied through Cloudflare for the same DDoS/edge protection the main domain gets — and it's live.

Verifying Auth Actually Works (Don't Skip This)

Basic auth misconfigured silently is worse than no auth at all, because it looks protected. Three checks, every time:

curl -o /dev/null -w '%{http_code}\n' http://127.0.0.1:5555/                       # expect 401
curl -o /dev/null -w '%{http_code}\n' http://admin:wrong@127.0.0.1:5555/           # expect 401
curl -o /dev/null -w '%{http_code}\n' http://admin:<real>@127.0.0.1:5555/          # expect 200

All three, not just the last one — confirming no creds and wrong creds both fail is what actually proves the auth is enforced, not just present in the config.

The Resource Cost, Measured

This runs on a small, memory-constrained EC2 instance — the same one where I'd already skipped running Grafana Alloy for exactly this reason. Flower is much lighter, but not free: available memory dropped from roughly 208MB to 188MB, swap usage rose from ~151MB to ~352MB after starting it. Real overhead, worth knowing before assuming "just a monitoring tool" is resource-neutral — it isn't, though on this box it settled and stayed stable rather than climbing.

What It's Actually For

Task-level visibility that didn't exist before: which tasks failed, their arguments, their traceback, whether they retried and how many times, per-worker throughput. The next time an email or a background job goes missing, the question changes from "did something break, and where" to "open Flower, look at the failed task, read the traceback" — turning a user-facing mystery into a five-minute diagnosis, which is the entire point of adding observability tooling in the first place.

If you're running Celery in production without task-level monitoring, or want a second look at how your background jobs are actually behaving, get in touch.