Deploy an OpenTelemetry observability stack on Railway
Deploy an OpenTelemetry Collector with Prometheus, Tempo, Loki, and Grafana on Railway, connect your app, and confirm alerts reach your webhook.
When a request slows down or fails on Railway, you need its metrics, logs, and trace side by side to see why. This stack puts all three in Grafana and alerts you when errors climb.
This guide walks you through deploying an OpenTelemetry observability stack on Railway from a template. You’ll generate demo traffic, follow one request from its metrics to its logs and trace in Grafana, then connect your own app and test alerts.
TL;DR:
- Deploy the template and set
ALERT_WEBHOOK_URLto your receiver’s URL. - Run the demo traffic and follow a request from its metrics to its logs and trace in Grafana.
- Point your app’s OpenTelemetry variables at the Collector’s private address, with its credentials.
- Run the error-rate test and confirm that a firing and a resolved notification reach your webhook.
The OpenTelemetry Observability Starter includes:
| Component | What it does |
|---|---|
| OpenTelemetry Collector | Receives and forwards application telemetry |
| Prometheus | Stores metrics |
| Tempo | Stores distributed traces |
| Loki | Stores logs |
| Grafana | Displays dashboards and sends alerts |
| Two Python demo services | Generate requests and telemetry you can explore |
The template deploys seven services and five persistent volumes. Grafana is public; the other services communicate over Railway’s private network.
The source repository holds the service configuration, demo apps, and end-to-end tests.
The Collector receives telemetry from the demo services and your app and passes it to Prometheus, Tempo, and Loki. Grafana queries all three and sends alerts to your webhook.
Do you need this stack if Railway already has built-in observability?
Not always. Railway has built-in logs, metrics, and tracing, and tracing needs no backend to run. Railway’s Instrument an app with OpenTelemetry guide covers that route.
Deploy this stack when you want metrics, traces, and logs together in Grafana, alerts you define, retention you configure, or a Collector as the intake for your apps. If the built-in tools answer your questions, you don’t need it.
What do you need before you start?
You need room for five volumes in your Railway workspace and a webhook receiver for Grafana’s alerts.
- A Railway workspace with capacity for five volumes. Hobby supports this template; Trial’s three-volume limit does not. An existing project needs room for five additional volumes.
- A webhook receiver that accepts Grafana’s JSON notifications and Bearer-token authentication. Use HTTPS if it runs outside your Railway project.
A Slack incoming-webhook URL won’t work directly with this template’s webhook format and authentication.
You’ll run the demo commands in Railway’s browser Console, so you don’t need a local CLI or SSH key.
1. Deploy the template
- Open the template deployment page and select your workspace.
- Under Grafana, select Configure and set
ALERT_WEBHOOK_URLto your receiver’s URL. Select Save Config, then Deploy. - Wait for all seven services to become healthy.
- In Grafana’s service variables, find
ALERT_WEBHOOK_TOKEN. Configure your receiver to accept it as a Bearer token. If your receiver supplies its own token, set Grafana’s variable to that value and redeploy Grafana.
To deploy into an existing project, open its target environment and choose New → Template. Keep the application and Collector in the same Railway project and environment.
Railway may rename services to avoid collisions. Use the actual deployed names throughout this guide.
2. Open Grafana
Open the public domain listed in Grafana’s Railway service. In that service’s Variables tab, retrieve both generated login values:
- Username:
GF_SECURITY_ADMIN_USER - Password:
GF_SECURITY_ADMIN_PASSWORD
Sign in with those values, then go to Dashboards → Application and telemetry pipeline. You’ll find request rate, error rate, latency, application logs, and Collector health panels.
The application panels use the demo’s metrics. When you connect your own app, you’ll adapt those queries to its metric names.
3. Generate traffic and explore it
In Railway, open the demo API service’s Console tab. Wait for a connected shell, then paste:
python - <<'PY'
import json
import time
import urllib.error
import urllib.request
for i in range(120):
route = "/slow" if i % 10 == 0 else "/work"
with urllib.request.urlopen("http://127.0.0.1:8080" + route, timeout=10) as response:
result = json.load(response)
if i == 0:
print("Trace ID:", result["trace_id"])
time.sleep(0.5)
try:
urllib.request.urlopen("http://127.0.0.1:8080/error", timeout=10)
except urllib.error.HTTPError as error:
if error.code != 500:
raise
print("Controlled error status:", error.code)
PYThe command prints a trace ID and an intentional 500 response. The other requests should succeed. Type exit when it finishes.
In Grafana:
- Set the dashboard to Last 15 minutes and refresh. Allow a few export and scrape cycles for data to appear.
- Expand an application log row and select View trace.
- Open a successful
/workor/slowtrace to see spans from both demo services. - From a span, select Explore the logs for this in split view to return to its logs.
You can also use Explore to run these queries. Select the matching data source and use Code mode for Prometheus and Loki.
| Data source | Query |
|---|---|
| Prometheus | sum by (service) (rate(demo_requests_total[1m])) |
| Prometheus | histogram_quantile(0.95, sum by (le, service) (rate(demo_request_duration_seconds_bucket[1m]))) |
| Loki | {service_name=~"demo-api|demo-downstream"} |
| Loki | {service_name="demo-api"} | trace_id="YOUR_TRACE_ID" |
| Tempo | Paste the printed trace ID into a trace-ID search |
Keep the time range on the period when you generated traffic. Request rates return to zero once traffic stops.
4. Connect your own application
Install your language’s OpenTelemetry SDK, OpenTelemetry Protocol (OTLP) HTTP exporters, and framework instrumentation. Follow the OpenTelemetry language guide for your app.
Add these variables to your application service, replacing Collector if Railway assigned a different service name:
OTEL_SERVICE_NAME=my-app
OTEL_EXPORTER_OTLP_ENDPOINT=http://${{Collector.RAILWAY_PRIVATE_DOMAIN}}:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTLP_USERNAME=demo
OTLP_PASSWORD=${{Collector.OTLP_PASSWORD}}The username and password are convenience variables used by this template. Your app must convert them into an authorization header. In Python, run this before creating OTLP exporters:
import base64
import os
username = os.environ["OTLP_USERNAME"]
password = os.environ["OTLP_PASSWORD"]
credential = base64.b64encode(f"{username}:{password}".encode()).decode()
os.environ["OTEL_EXPORTER_OTLP_HEADERS"] = f"Authorization=Basic {credential}"For other languages, use the SDK’s supported header configuration. Keep credentials out of logs and browser code.
If you turned on Railway’s built-in tracing for this service, Railway also sets its own OTEL_* variables. An OTEL_EXPORTER_OTLP_ENDPOINT you set yourself takes precedence, so spans go to the Collector instead of the Traces page.
The endpoint configuration alone doesn’t instrument your app, and the template doesn’t collect container stdout. Export structured OTLP logs from within active spans to preserve log-to-trace correlation.
Then send requests through your app and search for its service name in Tempo and Loki and its metrics in Prometheus.
5. Test alert delivery
The template includes two alerts:
- Demo API high error rate fires on more than 5% HTTP 500 responses, with more than five estimated requests in a one-minute window, sustained for 20 seconds.
- Collector unavailable fires when the Collector scrape fails or its scrape series is absent for 30 seconds.
To test the error-rate alert, reopen the demo API’s Console and run:
python - <<'PY'
import time
import urllib.error
import urllib.request
for route, seconds in [("/error", 90), ("/work", 120)]:
print("Sending", route, "requests for", seconds, "seconds", flush=True)
until = time.monotonic() + seconds
while time.monotonic() < until:
try:
with urllib.request.urlopen("http://127.0.0.1:8080" + route, timeout=10):
if route == "/error":
raise RuntimeError("Expected the demo to return HTTP 500")
except urllib.error.HTTPError as error:
if route != "/error" or error.code != 500:
raise
time.sleep(0.5)
print("Traffic finished; check firing and resolved notifications.")
PYIn Grafana, open Alerting → Alert rules, select List, and expand Observability. The error-rate alert should move from Pending to Firing, then return to Normal after healthy traffic clears the error window.
During this test, check your receiver for a firing notification followed by a resolved notification for Demo API high error rate. Both should have the same alert fingerprint. If either is missing, check Grafana’s delivery errors and your receiver’s logs.
The contact point uses Grafana’s generic webhook format. Provider-specific integrations may need a different contact point or an adapter.
What should you check if data or alerts don’t show up?
Grafana shows no data for your app
Check that the dashboard’s time range covers your requests, and allow a few export and scrape cycles. Then check the app’s variables: the endpoint should use the Collector’s private domain on port 4318, the protocol should be http/protobuf, and OTEL_EXPORTER_OTLP_HEADERS should carry the Basic authorization header. If Railway renamed the Collector, replace Collector in the reference variables with its deployed name.
Logs are missing or don’t link to traces
The template doesn’t collect container stdout, so your app must export structured OTLP logs. Export them from within active spans to keep log-to-trace correlation.
The error-rate alert doesn’t fire
The rule needs more than 5% HTTP 500 responses and more than five estimated requests in a one-minute window, sustained for 20 seconds. Run the full test script in step 5, which sends errors for 90 seconds and then healthy traffic for 120 seconds.
The alert fires but your receiver gets nothing
Check Grafana’s delivery errors and your receiver’s logs. The receiver must accept Grafana’s JSON notifications with the ALERT_WEBHOOK_TOKEN value as a Bearer token. A Slack incoming-webhook URL won’t work directly with this template.
Do you need the whole stack to host Grafana?
Not if you already store your metrics, logs, or traces elsewhere. A Grafana-only deployment may be enough. For example, this community-maintained Grafana template provides a single service with a volume at /var/lib/grafana. It’s a separate template from the seven-service stack this guide tests.
After signing in, go to Connections → Data sources → Add new data source, choose your backend, enter its URL and credentials, then select Save & test. Use a private address for a backend in the same Railway project and environment, or an authenticated HTTPS endpoint for an external service. Grafana’s data source documentation covers the options.
You can also add data sources to this starter. Its three bundled data sources are provisioned from files; change their configuration in your copy of the repository.
How long does the stack keep data?
Prometheus retains metrics for seven days or until its 2 GB block-retention limit removes older data. Tempo and Loki are configured for 72-hour retention. Monitor disk usage: retention settings don’t guarantee that the volumes can hold that much data at every ingestion rate.
What does the stack cost?
At Railway’s $0.15 per GB/month for volume storage, storage alone costs at most 5 × 5 GB × $0.15 = $3.75 a month, assuming all five volumes stay full. Railway bills for storage actually used, so empty capacity doesn’t incur that full amount.
CPU, RAM, and egress add to that total. As of November 2026, Railway charges $20 per vCPU/month, $10 per GB of RAM/month, and $0.05 per GB of network egress. Hobby costs $5/month and includes $5 of resource usage; usage above that is billed separately. Use your project’s usage metrics to estimate your workload.
Is the stack ready for production?
It’s a single-instance starter you operate. Railway doesn’t allow replicas on services with volumes, and those services have a short outage when redeployed, even with a healthcheck, as Railway’s volume limitations describe. Schedule upgrades and variable changes for a quiet window. Before you rely on the stack in production, plan for backups, capacity, upgrades, and recovery. The template doesn’t include high availability or recovery from lost volumes.
In a 24-hour test on November 17–18, 2026, the template handled 1 request per second: 86,400 successful requests, with no errors or skipped requests. It also passed deployment from a fresh Hobby account with no template configuration changes. Validation covered metrics, traces, logs, Grafana log-to-trace navigation, firing and resolved alerts, stored-data persistence after service replacement, and queued trace/log recovery after backend outages. The validation coverage lists details and limits.
Grafana stores its own configuration, such as users, data sources, and dashboards, in SQLite on its persistent volume. Grafana recommends PostgreSQL or MySQL for production, so back up the volume or move that database before you rely on Grafana in production.
How do you customize the stack?
For configuration changes, start with the source repository. Alert rules and contact points are provisioned from files, so customize them through your own copy of the source.
To make your own copy, eject the services from the template repository in each service’s settings.
To use a custom domain, follow Railway’s domain setup instructions and update Grafana’s GF_SERVER_ROOT_URL to match. To install plugins, set GF_PLUGINS_PREINSTALL, as Grafana’s plugin configuration describes.
Next steps
These Railway guides and docs cover tracing, incidents, alerts, and volumes: