Deploy and Host TensorFlow Serving on Railway

Built for an LLM agent to deploy and use. There is nothing to fill in: the API key is generated at deploy time, every request except the health probe needs it, a sample model answers from the first minute, and the contract below lists every URL and variable name with one script that checks health, auth and a prediction.

TensorFlow Serving is Google's production model server: it loads TensorFlow SavedModels from disk and answers prediction requests over REST and gRPC, with model versioning and hot reload. This template runs TensorFlow Serving 2.21 behind a small wrapper that puts a bearer key in front of the REST API, serves every model found on the volume, and caps the thread pools to the container's CPU quota.

About Hosting TensorFlow Serving

This template runs one service, built from the official tensorflow/serving 2.21.0 image (pinned by digest) in the public wrapper bon5co/tf-serving-railway, with the volume at /models. The stock image has no authentication at all, so a public URL would let anyone run inference on your CPU. In this template a Caddy proxy listens on the Railway port and forwards to TensorFlow Serving only when the request carries Authorization: Bearer plus the value of API_KEY; anything else gets 401. Only /healthz is open, and it reports the state of the first model.

Models live on the volume as /models/NAME/VERSION/saved_model.pb with VERSION a number. On a first boot with an empty volume the wrapper seeds the sample model half_plus_two (it returns x/2 + 2) so the API answers straight away. Set MODEL_URL to a .tar.gz of numeric version folders to fetch your own model once into /models/MODEL_NAME, or copy a model folder onto the volume: the model list is rescanned every 10 seconds and new folders are loaded without a restart. Thread pools are sized from the container's CPU quota, not the host's core count.

The gRPC port 8500 is open on the private network only: other services in the project reach it at tensorflow-serving.railway.internal:8500, and it has no authentication, so it is not exposed publicly.

Agent contract

Public URL: https://, on the tensorflow-serving service (set TFS_URL to it, no trailing slash)
Private URL: REST http://tensorflow-serving.railway.internal:8080 (key required), gRPC tensorflow-serving.railway.internal:8500 (no key, private network only)
Credentials: the key is in API_KEY (on the tensorflow-serving service)
Auth: send Authorization: Bearer API_KEY on every REST request except /healthz; a missing or wrong key gets 401
Health: GET /healthz returns 200 with no credentials once the first model is loaded

# run with TFS_URL and API_KEY (tensorflow-serving service); needs curl
curl -s -o /dev/null -w 'health %{http_code}\n' "$TFS_URL/healthz"
curl -s -o /dev/null -w 'anonymous %{http_code}\n' "$TFS_URL/v1/models/half_plus_two"
curl -s -o /dev/null -w 'wrong key %{http_code}\n' -H 'Authorization: Bearer wrong' "$TFS_URL/v1/models/half_plus_two"
curl -s -H "Authorization: Bearer $API_KEY" "$TFS_URL/v1/models/half_plus_two" | tr -d ' \n'; echo
curl -s -H "Authorization: Bearer $API_KEY" -X POST \
  -d '{"instances":[1.0,2.0,5.0]}' "$TFS_URL/v1/models/half_plus_two:predict" | tr -d ' \n'; echo

A fresh deploy prints health 200, anonymous 401, wrong key 401, a model status with state AVAILABLE, and {"predictions":[2.5,3.0,4.5]}. To serve your own model, set MODEL_URL and MODEL_NAME and redeploy, then call /v1/models/MODEL_NAME:predict the same way.

Why Deploy TensorFlow Serving on Railway?

Railway is a singular platform to deploy your infrastructure stack. Railway will host your infrastructure so you don't have to deal with configuration, while allowing you to vertically and horizontally scale it.

By deploying TensorFlow Serving on Railway, you are one step closer to supporting a complete full-stack application with minimal burden. Host your servers, databases, AI agents, and more on Railway.

Agent-ready - generated key, a contract block and one working script; no setup step.
Closed by default - the upstream image has no auth; here every prediction call needs the key.
Works on first boot - a sample model is seeded, so you can test the whole path before uploading anything.
State survives redeploys - models and versions live on the volume, and the key stays the same after every redeploy.

Common Use Cases

Model inference API - serve a trained SavedModel to your app over REST without writing a web service around it.
Agent tool backend - give an AI agent a private classifier, embedding or ranking model it can call with one header.
Versioned rollouts - drop a new numeric version folder on the volume and TensorFlow Serving switches to it without downtime.

Dependencies for TensorFlow Serving Hosting

A volume at /models (included in this template).

Deployment Dependencies

tensorflow/serving - upstream project
bon5co/tf-serving-railway - the wrapper image this template deploys, built from tensorflow/serving 2.21.0