---
title: "Deploy Florence-2 | (Just Updated) Vision API: Image Captioning, OCR, Object Detection, API Key Pre-Set"
description: "Microsoft Florence-2 vision API: image captioning, OCR, object detection"
category: "AI/ML"
url: https://railway.com/deploy/florence-2-or-just-updated-vision-api-im
---

# Deploy Florence-2 | (Just Updated) Vision API: Image Captioning, OCR, Object Detection, API Key Pre-Set

Microsoft Florence-2 vision API: image captioning, OCR, object detection

**[Deploy Florence-2 | (Just Updated) Vision API: Image Captioning, OCR, Object Detection, API Key Pre-Set on Railway](https://railway.com/template/florence-2-or-just-updated-vision-api-im)**

Machine-readable deploy manifest (JSON, validated by TemplateCI): https://railway.com/deploy/florence-2-or-just-updated-vision-api-im/manifest.json

- **Creator:** SuperSlowSloth
- **Category:** AI/ML

## Template content

### florence

- **Image:** ghcr.io/bon5co/florence-railway:1.0.0@sha256:60fe69f0130518ae3bcffe3f1d2eb8711ea5da33c3c7bf3393440806ddf3b101
- **Health check:** /health
- **Public domain:** Yes

## Documentation

# Deploy and Host Florence-2 Vision API (Image Captioning, OCR, Object Detection) on Railway

Microsoft Florence-2 is an open vision foundation model with MIT weights: it describes an image, reads the text in it, finds and labels objects, and outlines a named thing, all from one small model that runs on CPU. This template serves Florence-2 base and large behind one authenticated JSON API: post an image, get back a caption, the text, boxes or a polygon. The weights are already inside the image, an API key is generated for you at deploy, and no GPU is needed.

## About Hosting Florence-2

This template runs the Florence-2 checkpoints `florence-community/Florence-2-base` (0.23B) and `florence-community/Florence-2-large` (0.77B) with `transformers` 5.23.0 and PyTorch on CPU behind a small FastAPI server. The native `transformers` classes are used, so no remote model code is executed. `POST /v1/caption` returns a `short`, `detailed` or `long` caption. `POST /v1/ocr` returns the text of the image, and with `boxes=true` one quadrilateral per line. `POST /v1/detect` returns labelled boxes for the objects in the image (`mode` `objects`, `dense` or `proposals`), or the box of a phrase you send as `phrase`. `POST /v1/segment` returns the polygon outline of the thing named in `phrase`. Send the image as a multipart `file` or as a base64 `image` in a JSON body; pick `model` (default `florence-2-large`) and `num_beams` (1 to 5) per request. `GET /v1/models` lists the models.

The weights are baked into the image, so the first request is answered from disk and no model download hides inside a request that Railway's edge would abandon after five minutes. The service sizes its PyTorch thread pool from the container's own CPU quota instead of the host's core count, handles one request at a time so memory stays bounded, and refuses an image above 20 MB or 50 million pixels.

**Measured on Railway, 8 vCPU, 8 GB container:** a caption took **3.3 s** on Florence-2 large and **1.1 s** on base (a `long` caption 5.9 s and 2.0 s); object detection, phrase grounding and segmentation took **3.1 to 3.8 s** on large and **1.0 to 1.3 s** on base; OCR of a full 1600 x 2000 page of 45 text lines took **23.7 s** on large and **8.8 s** on base, and with line boxes **43.6 s** and **16.0 s**. The slowest request measured, a dense page with line boxes at `num_beams=5` on large, took **52.7 s**, far inside the five minute edge limit. A 4000 x 3000 JPEG of 9 MB captioned in 4.1 s end to end against 3.4 s for an 800 x 500 image (the model's own time was the same, 3.2 s; the extra is upload and decode). The default model loads in about 7 s after the container starts, and the process peak resident memory after all of those requests read **5.5 GB** (`peak_rss_mb` on `/health`). Quality is the model's own and was not benchmarked here. One timing outlier is not explained: a segmentation of a named phrase on the base model took 24 s on both runs, against 3.3 s on large, so use large for `/v1/segment`.

### Agent contract

- Public URL: `https://RAILWAY_PUBLIC_DOMAIN` (the generated domain of the `florence` service)
- Private URL: `http://florence.railway.internal:8080`
- Credentials: the `API_KEY` variable of the `florence` service
- Auth: send `Authorization: Bearer $API_KEY` on every route except `GET /health`; a call without the key or with a wrong one answers `401`
- Health: `GET /health` returns 200 once the model is loaded (503 while it loads)

```bash
# needs: RAILWAY_PUBLIC_DOMAIN, API_KEY (a variable of the florence service); also curl.
# signup: none - the API has no user accounts, only the API_KEY bearer
R="https://$RAILWAY_PUBLIC_DOMAIN"
n=0; until [ "$(curl -s -o /dev/null -w '%{http_code}' "$R/health")" = 200 ]; do n=$((n+1)); [ "$n" -le 60 ] || break; sleep 5; done
curl -s -o /dev/null -w "health: %{http_code}\n" "$R/health"
# a 320x120 PNG (804 bytes) of the words HELLO RAILWAY, sent as base64 JSON
B='iVBORw0KGgoAAAANSUhEUgAAAUAAAAB4AgMAAAD+oMNNAAAADFBMVEX///+5ublFRUUAAAADU1edAAACrElEQVR42u2WP24TQRhH3yRjHJTCUxiEjMChR4KeSDs3wBSJEEpBQZWGI9i5QY6Q2inwCWCWE8ANLAQRSC4mEsXKHu9QzNq7dog3QkrFvGrnt943+3n+LUQikUgkEolEIpHIbdI4+McH50Cfbe+9I9FA6wuA+OTHAC3vfwHNOTAP93LYzaBraM6hDzwbAGxt7KejeVxc3j8FKcpbBqQEZZACzAB65qpwenj4utp+92MrHQBcHJq3q8IUlFgKxz1IzNWSfwOVkkUO3XFoNTPoftDLkvua7kdIQro7RsypLVlM4Wc7XM8kqJEuS9YoA0lIXRsxqxc2XBABeLEqtNBKl8LZDo2sXrgzAd8ohEDiVCnUaKvRIc232BnXC5UFn5btFaEisRpfpCd6b3QD4SgMJ4Dw8KIidOBBuCIdaWXqhYHwvwmHwMtKb+xbvUxtrx+EcuMbApgw+RoZwlWETgnnFK5IJ0+eUhWKA9An7HouFSvLIXDv4M0pjUlVKAVOCVukruFWhcPNZXeG+SuwlCPkwYXHLSnM7l7yl5KnR8yuUW5pIzvDytqTwjolLSHNGa8I823ow+x8bTwWNX99/n6g2/cqm5mX0nqJpUgHRV8bR9mWg8yZQpUt8EjrpTSL1JgbCMPj4ZeZQgmxpyvbjQFl1tKaN1RAspxDCdheKZR4qVhL64UiWUwTEnDVWWpIMevpRmHWZrEr4SGBytpLlSFtraebhTMZtjAWiw/XLl//JZhkPd0s9HfgwWQ57xzVxWw1XAZhdYnLK6fmOcfHXPQ7Q6ZHJ98fnZ0tShbFrgidIdMjEmC/kl5zjOYk3vtxy3uf8dD7nHCmCNccA3PmhHstB92MIg0nUe08vDB8W5T/WVLujoBzYB1r6a19OUQikUgkEolEIpFIJBKJ/Gf8Acx/Dk3nzWEoAAAAAElFTkSuQmCC'
curl -s -o /dev/null -w "write without key: %{http_code}\n" -X POST "$R/v1/ocr" -H "Content-Type: application/json" -d "{\"image\":\"$B\"}"
curl -s -o /dev/null -w "write wrong key: %{http_code}\n" -X POST "$R/v1/ocr" -H "Content-Type: application/json" -H "Authorization: Bearer not-the-key" -d "{\"image\":\"$B\"}"
curl -s -X POST "$R/v1/ocr" -H "Content-Type: application/json" -H "Authorization: Bearer $API_KEY" -d "{\"image\":\"$B\"}" \
  | grep -o '"text":"[^"]*"' | head -1
```

A fresh deploy prints `health: 200`, `write without key: 401`, `write wrong key: 401` and `"text":"HELLO RAILWAY"`, the words in the test image. To caption a local file, locate a phrase and outline it:

```bash
curl -X POST "https://$RAILWAY_PUBLIC_DOMAIN/v1/caption" -H "Authorization: Bearer $API_KEY" -F file=@photo.jpg -F detail=detailed
curl -X POST "https://$RAILWAY_PUBLIC_DOMAIN/v1/detect"  -H "Authorization: Bearer $API_KEY" -F file=@photo.jpg -F "phrase=the red car"
curl -X POST "https://$RAILWAY_PUBLIC_DOMAIN/v1/segment" -H "Authorization: Bearer $API_KEY" -F file=@photo.jpg -F "phrase=the red car"
```

A bad image answers `422` with the reason, an unknown `model` answers `400`, and a request body over 20 MB answers `413`. The API never fetches a URL for you: it only reads the bytes you send, so it cannot be pointed at an address inside your network.

## Why Deploy Florence-2 Vision API on Railway?

Railway is a singular platform to deploy your infrastructure stack. Railway will host your infrastructure so you don't have to deal with configuration, while allowing you to vertically and horizontally scale it.

By deploying Florence-2 on Railway, you are one step closer to supporting a complete full-stack application with minimal burden. Host your servers, databases, AI agents, and more on Railway.

- **Four vision jobs, one small model** — captioning, OCR, detection and segmentation without separate services for each.
- **Your images stay in your container** — photos, scans and screenshots are read by your own service, not sent to a hosted vision API.
- **No GPU bill and no per-call fee** — a caption takes about three seconds on 8 vCPU and you pay for the container, not the call.
- **Open weights, commercial use** — the Florence-2 weights are MIT licensed.
- **Private by default** — a generated bearer key guards every route but the health check, and the container refuses to start without one.

## Common Use Cases

- **Alt text and image search** — caption product photos and screenshots for accessibility and search indexes.
- **Document and screenshot OCR** — pull the text, with line positions, out of a scan, a receipt or a UI screenshot.
- **Vision tool for AI agents** — give an LLM agent one authenticated request that describes an image, finds an object in it or reads its text.

## Dependencies for Florence-2 Hosting

- A service with **8 GB of RAM** (peak resident memory measured at 5.5 GB after requests on both models; the default large model alone loads at 4.7 GB). Free and Trial plans cannot run it.
- No volume: the service keeps no state and the models are in the image.

### Deployment Dependencies

- [microsoft/Florence-2-large](https://huggingface.co/microsoft/Florence-2-large) and [Florence-2-base](https://huggingface.co/microsoft/Florence-2-base) — the models, converted for `transformers` as [florence-community/Florence-2-large](https://huggingface.co/florence-community/Florence-2-large) and [florence-community/Florence-2-base](https://huggingface.co/florence-community/Florence-2-base)
- [huggingface/transformers](https://github.com/huggingface/transformers) — the inference library, Apache-2.0, version 5.23.0
- [bon5co/florence-railway](https://github.com/bon5co/florence-railway) — the Railway wrapper image this template deploys, Apache-2.0
- Weights licence: the Hugging Face model card of `florence-community/Florence-2-large` declares `license: mit`, and so does the card of `florence-community/Florence-2-base`. MIT allows commercial use and requires keeping the copyright and licence notice. Attribution: Florence-2, copyright Microsoft, licensed MIT.


## Similar templates

- [Chat Chat](https://railway.com/deploy/-WWW5r) — Chat Chat, your own unified chat and search to AI platform.
- [Weaviate](https://railway.com/deploy/weaviate-1) — Vector database with hybrid search and an API key-protected API
- [Möbius Personal AI OS](https://railway.com/deploy/mobius-v01-personal-ai-os) — Self-hosted AI agent that builds and runs apps for you, with memory.

Open this page in a browser: https://railway.com/deploy/florence-2-or-just-updated-vision-api-im
