---
title: "Deploy TensorFlow Serving | (Just Updated) ML Model Inference API, Bearer Key From Boot"
description: "TensorFlow Serving model inference REST API with generated bearer key"
category: "AI/ML"
url: https://railway.com/deploy/tensorflow-serving-or-just-updated-ml-mo
---

# Deploy TensorFlow Serving | (Just Updated) ML Model Inference API, Bearer Key From Boot

TensorFlow Serving model inference REST API with generated bearer key

**[Deploy TensorFlow Serving | (Just Updated) ML Model Inference API, Bearer Key From Boot on Railway](https://railway.com/template/tensorflow-serving-or-just-updated-ml-mo)**

Machine-readable deploy manifest (JSON, validated by TemplateCI): https://railway.com/deploy/tensorflow-serving-or-just-updated-ml-mo/manifest.json

- **Creator:** SuperSlowSloth
- **Category:** AI/ML

## Template content

### tensorflow-serving

- **Image:** ghcr.io/bon5co/tf-serving-railway:2.21.0@sha256:811b5000d9f1be70080f3782d64c12ca0bb67c40b394fed4633f76389434991a
- **Health check:** /healthz
- **Public domain:** Yes

## Documentation

# Deploy and Host TensorFlow Serving on Railway

**Built for an LLM agent to deploy and use.** There is nothing to fill in: the API key is generated at deploy time, every request except the health probe needs it, a sample model answers from the first minute, and the contract below lists every URL and variable name with one script that checks health, auth and a prediction.

TensorFlow Serving is Google's production model server: it loads TensorFlow SavedModels from disk and answers prediction requests over REST and gRPC, with model versioning and hot reload. This template runs TensorFlow Serving 2.21 behind a small wrapper that puts a bearer key in front of the REST API, serves every model found on the volume, and caps the thread pools to the container's CPU quota.

## About Hosting TensorFlow Serving

This template runs one service, built from the official **tensorflow/serving** `2.21.0` image (pinned by digest) in the public wrapper [bon5co/tf-serving-railway](https://github.com/bon5co/tf-serving-railway), with the volume at `/models`. The stock image has no authentication at all, so a public URL would let anyone run inference on your CPU. In this template a Caddy proxy listens on the Railway port and forwards to TensorFlow Serving only when the request carries `Authorization: Bearer` plus the value of `API_KEY`; anything else gets 401. Only `/healthz` is open, and it reports the state of the first model.

Models live on the volume as `/models/NAME/VERSION/saved_model.pb` with `VERSION` a number. On a first boot with an empty volume the wrapper seeds the sample model `half_plus_two` (it returns x/2 + 2) so the API answers straight away. Set `MODEL_URL` to a `.tar.gz` of numeric version folders to fetch your own model once into `/models/MODEL_NAME`, or copy a model folder onto the volume: the model list is rescanned every 10 seconds and new folders are loaded without a restart. Thread pools are sized from the container's CPU quota, not the host's core count.

The gRPC port 8500 is open on the private network only: other services in the project reach it at `tensorflow-serving.railway.internal:8500`, and it has no authentication, so it is not exposed publicly.

### Agent contract

- Public URL: `https://`, on the `tensorflow-serving` service (set `TFS_URL` to it, no trailing slash)
- Private URL: REST `http://tensorflow-serving.railway.internal:8080` (key required), gRPC `tensorflow-serving.railway.internal:8500` (no key, private network only)
- Credentials: the key is in `API_KEY` (on the `tensorflow-serving` service)
- Auth: send `Authorization: Bearer API_KEY` on every REST request except `/healthz`; a missing or wrong key gets 401
- Health: `GET /healthz` returns 200 with no credentials once the first model is loaded

```bash
# run with TFS_URL and API_KEY (tensorflow-serving service); needs curl
curl -s -o /dev/null -w 'health %{http_code}\n' "$TFS_URL/healthz"
curl -s -o /dev/null -w 'anonymous %{http_code}\n' "$TFS_URL/v1/models/half_plus_two"
curl -s -o /dev/null -w 'wrong key %{http_code}\n' -H 'Authorization: Bearer wrong' "$TFS_URL/v1/models/half_plus_two"
curl -s -H "Authorization: Bearer $API_KEY" "$TFS_URL/v1/models/half_plus_two" | tr -d ' \n'; echo
curl -s -H "Authorization: Bearer $API_KEY" -X POST \
  -d '{"instances":[1.0,2.0,5.0]}' "$TFS_URL/v1/models/half_plus_two:predict" | tr -d ' \n'; echo
```

A fresh deploy prints `health 200`, `anonymous 401`, `wrong key 401`, a model status with state `AVAILABLE`, and `{"predictions":[2.5,3.0,4.5]}`. To serve your own model, set `MODEL_URL` and `MODEL_NAME` and redeploy, then call `/v1/models/MODEL_NAME:predict` the same way.

## Why Deploy TensorFlow Serving on Railway?

Railway is a singular platform to deploy your infrastructure stack. Railway will host your infrastructure so you don't have to deal with configuration, while allowing you to vertically and horizontally scale it.

By deploying TensorFlow Serving on Railway, you are one step closer to supporting a complete full-stack application with minimal burden. Host your servers, databases, AI agents, and more on Railway.

- **Agent-ready** - generated key, a contract block and one working script; no setup step.
- **Closed by default** - the upstream image has no auth; here every prediction call needs the key.
- **Works on first boot** - a sample model is seeded, so you can test the whole path before uploading anything.
- **State survives redeploys** - models and versions live on the volume, and the key stays the same after every redeploy.

## Common Use Cases

- **Model inference API** - serve a trained SavedModel to your app over REST without writing a web service around it.
- **Agent tool backend** - give an AI agent a private classifier, embedding or ranking model it can call with one header.
- **Versioned rollouts** - drop a new numeric version folder on the volume and TensorFlow Serving switches to it without downtime.

## Dependencies for TensorFlow Serving Hosting

- A volume at `/models` (included in this template).

### Deployment Dependencies

- [tensorflow/serving](https://github.com/tensorflow/serving) - upstream project
- [bon5co/tf-serving-railway](https://github.com/bon5co/tf-serving-railway) - the wrapper image this template deploys, built from tensorflow/serving 2.21.0


## Similar templates

- [Chat Chat](https://railway.com/deploy/-WWW5r) — Chat Chat, your own unified chat and search to AI platform.
- [Weaviate](https://railway.com/deploy/weaviate-1) — Vector database with hybrid search and an API key-protected API
- [Möbius v0.1 Personal AI OS](https://railway.com/deploy/mobius-v01-personal-ai-os) — Self-hosted AI agent that builds and runs apps for you, with memory.

Open this page in a browser: https://railway.com/deploy/tensorflow-serving-or-just-updated-ml-mo