Deploy and Host MLflow Agent Observability on Railway

About Hosting

This template deploys two services on Railway: a self-hosted MLflow tracking & tracing server (the trace-ingestion backend) and a FastAPI demo agent instrumented with the MLflow Tracing SDK. The demo agent is wired to the MLflow server automatically and validates the whole pipeline from the first deploy — click deploy, wait for the build, then watch traces appear in the MLflow UI.

The MLflow server is open by design so any of your own agents can push traces without an API key. Use the tracking REST API (via the mlflow-tracing SDK) or the OpenTelemetry ingestion endpoint at /v1/traces (with Content-Type: application/x-protobuf or application/json plus the X-MLflow-Experiment-Id header) — LangChain.js, Vercel AI SDK, Claude Agent SDK and any plain OTel exporter can report in. SQLite metadata and trace artifacts persist on a /mlruns volume; swap in Railway Postgres and S3 once you outgrow the bundled setup.

Why Deploy

Zero-config observability: the included demo agent captures LLM calls, tool calls and agent handoffs end-to-end on the very first deploy.
Keyless trace ingestion: no auth to wire up before your own agents can report traces; the server's security middleware accepts any Railway host and any CORS origin by default.
Bundled reference implementation: a working OpenAI Agents SDK app that shows exactly how to instrument a FastAPI service with mlflow-tracing.
Reliability-tuned: trace logging fails fast and never blocks agent requests (HTTP timeouts shortened, retries capped) so a slow trace backend can't stall your agents.

Common Use Cases

Inspect every LLM call, token count and latency for agents you run on Railway.
Centralize traces from multiple services (Python, JS, any OTel-speaking SDK) in one self-hosted MLflow UI.
Use the demo agent as a starting point for instrumenting your own FastAPI or OpenAI Agents SDK services.

Dependencies for

mlflow-server: Python 3.12 with the full MLflow package, SQLite metadata store and an artifact store rooted at the /mlruns volume.
agent-app: Python 3.12 with FastAPI, the OpenAI Agents SDK and the mlflow-tracing package, pre-configured to report to the MLflow server's Railway public domain (HTTPS).

Deployment Dependencies

GitHub: both services build from this repository's mlflow-server/ and agent-app/ directories using Railway's Dockerfile builder.
Railway volume: attach a volume to mlflow-server mounted at /mlruns to persist experiments, runs and traces across redeploys.
OpenAI-compatible API (optional): to run the demo agent, add OPENAI_API_KEY (official OpenAI API) or OPENAI_COMPATIBLE_ENDPOINT (e.g. Groq, OpenRouter, vLLM, Ollama) to agent-app after deploying.
Scaling (optional): Railway Postgres for the metadata store and S3 for artifacts when you exceed the bundled SQLite/volume setup.