Deploy Grafana 13 LGTM Observability Stack

Deploy and Host Grafana LGTM with Railway

LGTM is Grafana's open-source observability stack: Loki for logs, Grafana for dashboards, Tempo for traces and Mimir for metrics, here replaced by single-node Prometheus. This community template deploys Grafana, Loki, Tempo and Prometheus plus Grafana Alloy as a single OpenTelemetry endpoint, with logs and traces stored in Railway Buckets and the datasources linked to each other.

About Hosting Grafana LGTM

A self-hosted LGTM stack needs more than four containers: Loki and Tempo need object storage and retention, Prometheus needs a way to receive pushed metrics, applications need one place to send OpenTelemetry data, and Grafana needs datasources that link a slow trace to its logs and metrics. This template wires that up on Railway. Alloy receives OTLP from your services over the private network, or from anywhere through a token-protected public endpoint, and routes traces to Tempo, logs to Loki and metrics to Prometheus. Loki and Tempo keep their data in Railway Buckets, Tempo generates request-rate and service-graph metrics from your traces, and Grafana opens with everything provisioned.

Common Use Cases

• Collect traces, logs and metrics from apps on Railway with any OpenTelemetry SDK, using one endpoint.
• Jump from a slow or failing trace to the logs and request metrics of that service, and back.
• See service dependencies and error rates in a service graph built from traces.
• Scrape /metrics endpoints of your Railway services and build dashboards and alerts on them.
• Receive telemetry from apps hosted outside Railway through the token-protected OTLP endpoint.

Dependencies for Grafana LGTM Hosting

• Grafana 13.2.3
• Loki 3.7.8 (single binary) with a Railway Bucket
• Tempo 3.1.0 (monolithic mode) with a Railway Bucket
• Prometheus v3.15.0
• Grafana Alloy v1.20.1 (OpenTelemetry collector)

Deployment Dependencies

• Grafana documentation: https://grafana.com/docs/grafana/latest/
• Loki documentation: https://grafana.com/docs/loki/latest/
• Tempo documentation: https://grafana.com/docs/tempo/latest/
• Prometheus documentation: https://prometheus.io/docs/introduction/overview/
• Grafana Alloy documentation: https://grafana.com/docs/alloy/latest/
• OpenTelemetry SDK configuration: https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/
• Railway Buckets: https://docs.railway.com/storage-buckets

Implementation Details

First login

1. Deploy the template. No input is required.
2. Open the grafana public domain and log in as admin with GF_SECURITY_ADMIN_PASSWORD from the grafana variables (applied on first start; change it later in Grafana).
3. Open Explore and pick Prometheus, Loki or Tempo. The stack's own metrics are there from the start.

Sending data from services in this project

Add to your service:

• OTEL_EXPORTER_OTLP_ENDPOINT = ${{alloy.OTEL_EXPORTER_OTLP_ENDPOINT}} (OTLP/HTTP on port 4318; gRPC is on port 4317)
• OTEL_EXPORTER_OTLP_PROTOCOL = http/protobuf
• OTEL_SERVICE_NAME = {your-service-name}

Sending data from outside Railway

• OTEL_EXPORTER_OTLP_ENDPOINT = https://{alloy-public-domain} (OTLP/HTTP only)
• OTEL_EXPORTER_OTLP_HEADERS = Authorization=Bearer%20{OTLP_PUBLIC_TOKEN} (most SDKs decode %20 to a space; if yours does not, use a literal space)

Requests without the token are rejected with 401. Rotate the token by changing OTLP_PUBLIC_TOKEN on alloy and redeploying.

Other ways in

• Prometheus scraping: set EXTRA_SCRAPE_TARGETS on prometheus to a comma-separated list such as ${{api.RAILWAY_PRIVATE_DOMAIN}}:8080 (services exposing /metrics).
• Prometheus remote write: ${{prometheus.PROMETHEUS_REMOTE_WRITE_URL}}.
• Loki push API for log shippers (Promtail-compatible clients, Vector, Fluent Bit): ${{loki.LOKI_PUSH_URL}}.

Retention: Loki LOKI_RETENTION_PERIOD (31 days), Tempo TEMPO_RETENTION (14 days), Prometheus PROMETHEUS_RETENTION_TIME (15 days) and PROMETHEUS_RETENTION_SIZE (4 GB).

Scaling: this is a single-node stack. Scale vertically: give Loki and Tempo more memory as ingest grows, and Prometheus more memory and volume as series grow. Alloy is stateless; raise ALLOY_MEMORY_LIMIT with its memory. Grafana uses SQLite, so keep one replica.

Pinning and upgrades: each version is pinned in services/{service}/Dockerfile. To upgrade, change the FROM tag in that Dockerfile, read the release notes (Loki and Tempo major versions can change configuration), and redeploy the service. Data in the buckets and volumes is kept.

Resources: about 0.6 GB RAM idle (around $7/month). A setup with 5 to 15 instrumented services typically uses about 2.5 GB RAM and 30 GB of bucket storage (around $35/month). Hobby plan works; Pro once Prometheus needs more than 5 GB of volume.

Why Deploy Grafana LGTM on Railway?

Railway runs Grafana, Loki, Tempo, Prometheus and Alloy as one project with private networking to the services you monitor, volumes and S3 buckets for storage, generated secrets and usage-based billing. Your telemetry stays in your own project, and instrumenting a service is a matter of referencing one variable.