By Shahid MalikI added distributed tracing to Tabeer.ai's Django backend using Grafana Cloud's Application Observability — per-endpoint latency, error rates, database query spans, Celery task spans, all without touching a single view function, via OpenTelemetry's zero-code auto-instrumentation. The setup itself is well documented. The decision that actually mattered — and the one worth writing about — was choosing not to run the piece of infrastructure the default setup guide recommends.
The Default Path: Run a Local Collector
Grafana's own setup wizard, and most OpenTelemetry guides generally, point you toward running Grafana Alloy — a local collector that receives telemetry from your app and forwards it to Grafana Cloud (or wherever). It's genuinely good software: it batches, retries, can route to multiple backends, and decouples your app from directly depending on an external endpoint's availability.
It's also a permanent, always-on process. On a normal server, that's a rounding error. On the EC2 instance Tabeer.ai actually runs on — 908MB of total RAM, an instance size chosen because the project doesn't yet have the traffic or budget to justify more — it's not a rounding error. That same day, before this integration, the box had been OOM-killing Nuxt builds repeatedly under normal load. Adding a new always-on process with a 50-150MB+ baseline footprint, for a single Django app that doesn't need Alloy's multi-service routing, was an easy call to skip.
What I Did Instead
The OpenTelemetry Python SDK doesn't require a local collector at all. It can export OTLP directly to Grafana Cloud's gateway over HTTPS:
pip install "opentelemetry-distro[otlp]"
opentelemetry-bootstrap -a install
bootstrap inspects what's actually installed in your environment and adds matching instrumentors — for this project, that pulled in Django, psycopg2, Celery, and Redis instrumentation automatically, no manual selection needed.
Then, zero code changes to the app itself — the whole thing runs through the opentelemetry-instrument wrapper, configured entirely via environment variables:
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-prod-eu-west-2.grafana.net/otlp
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Basic%20<base64 of stackID:token>
OTEL_SERVICE_NAME=tabeer-backend
OTEL_TRACES_SAMPLER=traceidratio
OTEL_TRACES_SAMPLER_ARG=0.2
Two things worth flagging for anyone doing this on Python specifically: the Basic Auth credentials are base64(<Grafana stack ID>:<glc_ API token>) — and the space between Basic and the token needs to be percent-encoded as %20 in that env var specifically, a Python-SDK-specific parsing quirk Grafana's own quickstart calls out (most other language SDKs don't need this). And traceidratio at 0.2 — 20% sampling — was a deliberate choice, not the default; full sampling adds real per-request overhead, and on a box this size, that overhead is worth avoiding for statistically-representative-but-not-exhaustive trace data.
The Bug This Approach Surfaced
opentelemetry-instrument initializes its auto-instrumentation before your application code — including Django's own settings — is ever imported. This project's wsgi.py sets DJANGO_SETTINGS_MODULE via os.environ.setdefault(...), the standard Django pattern, which normally works because gunicorn importing wsgi.py is the first thing that touches Django settings.
With opentelemetry-instrument in front of gunicorn, that's no longer true — the Django instrumentor touches django.conf.settings during its own setup, before wsgi.py ever runs. Django's settings object is a lazy singleton that resolves exactly once, on first access. It resolved against an unset DJANGO_SETTINGS_MODULE, landed on a broken default state, and every single request started failing with AttributeError: module 'django.conf.global_settings' has no attribute 'ROOT_URLCONF' — permanently, for the life of the process, even after wsgi.py's fallback ran moments later.
The fix is one line, once you know where to look: set DJANGO_SETTINGS_MODULE explicitly in the process environment, so it exists before opentelemetry-instrument starts, rather than depending on the app's own runtime fallback. Cheap fix. Expensive to find without knowing auto-instrumentation runs earlier than you'd assume.
What It Actually Cost
Measured, not estimated: gunicorn's combined memory footprint (2 workers) went from roughly 85-115MB to about 195MB after instrumentation — 80-100MB of real, ongoing overhead. On a 908MB box, available memory dropped from roughly 260-300MB free to about 190MB free under light load. I watched it for several minutes post-deploy: zero restarts, no new errors, memory held steady rather than climbing. Stable — but it meaningfully eats into an already-thin margin.
The Actual Opinion
If you're running infrastructure that's resource-constrained — a small EC2 instance, a budget VPS, anything sized for "this project doesn't have the traffic yet to justify more" — default to the cloud-hosted version of whatever tool you need, not the self-hosted one, unless self-hosting is the actual point (cost at scale, data residency, a specific compliance requirement). The instinct to self-host everything because it's "more control" or marginally cheaper per month is exactly backwards when your actual constraint is RAM, not money: a self-hosted collector, database, or cache you have to run yourself is one more process competing for the same fixed pool of memory as your actual application, and one more thing that needs babysitting when it inevitably needs a restart or a resource bump.
This isn't a one-off call I made for Tabeer.ai. It's the same reasoning I apply to mediodconsulting.com and any other project where the hosting budget is deliberately lean: pay for the managed/cloud version of infrastructure tooling — observability, error tracking, session recording, whatever it is — and keep the server itself doing exactly one job, running the actual application. The moment self-hosting a piece of infrastructure means your app server starts fighting itself for memory, you've made the "cheaper" choice more expensive than the subscription would have been.
If you're deciding between a self-hosted and managed setup for observability, error tracking, or anything else on a resource-constrained server, get in touch — it's usually a five-minute call to figure out which side of that line you're actually on.