
Deploy Florence-2 | (Just Updated) Vision API: Image Captioning, OCR, Object Detection, API Key Pre-Set
Microsoft Florence-2 vision API: image captioning, OCR, object detection
florence
Just deployed
Deploy and Host Florence-2 Vision API (Image Captioning, OCR, Object Detection) on Railway
Microsoft Florence-2 is an open vision foundation model with MIT weights: it describes an image, reads the text in it, finds and labels objects, and outlines a named thing, all from one small model that runs on CPU. This template serves Florence-2 base and large behind one authenticated JSON API: post an image, get back a caption, the text, boxes or a polygon. The weights are already inside the image, an API key is generated for you at deploy, and no GPU is needed.
About Hosting Florence-2
This template runs the Florence-2 checkpoints florence-community/Florence-2-base (0.23B) and florence-community/Florence-2-large (0.77B) with transformers 5.23.0 and PyTorch on CPU behind a small FastAPI server. The native transformers classes are used, so no remote model code is executed. POST /v1/caption returns a short, detailed or long caption. POST /v1/ocr returns the text of the image, and with boxes=true one quadrilateral per line. POST /v1/detect returns labelled boxes for the objects in the image (mode objects, dense or proposals), or the box of a phrase you send as phrase. POST /v1/segment returns the polygon outline of the thing named in phrase. Send the image as a multipart file or as a base64 image in a JSON body; pick model (default florence-2-large) and num_beams (1 to 5) per request. GET /v1/models lists the models.
The weights are baked into the image, so the first request is answered from disk and no model download hides inside a request that Railway's edge would abandon after five minutes. The service sizes its PyTorch thread pool from the container's own CPU quota instead of the host's core count, handles one request at a time so memory stays bounded, and refuses an image above 20 MB or 50 million pixels.
Measured on Railway, 8 vCPU, 8 GB container: a caption took 3.3 s on Florence-2 large and 1.1 s on base (a long caption 5.9 s and 2.0 s); object detection, phrase grounding and segmentation took 3.1 to 3.8 s on large and 1.0 to 1.3 s on base; OCR of a full 1600 x 2000 page of 45 text lines took 23.7 s on large and 8.8 s on base, and with line boxes 43.6 s and 16.0 s. The slowest request measured, a dense page with line boxes at num_beams=5 on large, took 52.7 s, far inside the five minute edge limit. A 4000 x 3000 JPEG of 9 MB captioned in 4.1 s end to end against 3.4 s for an 800 x 500 image (the model's own time was the same, 3.2 s; the extra is upload and decode). The default model loads in about 7 s after the container starts, and the process peak resident memory after all of those requests read 5.5 GB (peak_rss_mb on /health). Quality is the model's own and was not benchmarked here. One timing outlier is not explained: a segmentation of a named phrase on the base model took 24 s on both runs, against 3.3 s on large, so use large for /v1/segment.
Agent contract
- Public URL:
https://RAILWAY_PUBLIC_DOMAIN(the generated domain of theflorenceservice) - Private URL:
http://florence.railway.internal:8080 - Credentials: the
API_KEYvariable of theflorenceservice - Auth: send
Authorization: Bearer $API_KEYon every route exceptGET /health; a call without the key or with a wrong one answers401 - Health:
GET /healthreturns 200 once the model is loaded (503 while it loads)
# needs: RAILWAY_PUBLIC_DOMAIN, API_KEY (a variable of the florence service); also curl.
# signup: none - the API has no user accounts, only the API_KEY bearer
R="https://$RAILWAY_PUBLIC_DOMAIN"
n=0; until [ "$(curl -s -o /dev/null -w '%{http_code}' "$R/health")" = 200 ]; do n=$((n+1)); [ "$n" -le 60 ] || break; sleep 5; done
curl -s -o /dev/null -w "health: %{http_code}\n" "$R/health"
# a 320x120 PNG (804 bytes) of the words HELLO RAILWAY, sent as base64 JSON
B='iVBORw0KGgoAAAANSUhEUgAAAUAAAAB4AgMAAAD+oMNNAAAADFBMVEX///+5ublFRUUAAAADU1edAAACrElEQVR42u2WP24TQRhH3yRjHJTCUxiEjMChR4KeSDs3wBSJEEpBQZWGI9i5QY6Q2inwCWCWE8ANLAQRSC4mEsXKHu9QzNq7dog3QkrFvGrnt943+3n+LUQikUgkEolEIpHIbdI4+McH50Cfbe+9I9FA6wuA+OTHAC3vfwHNOTAP93LYzaBraM6hDzwbAGxt7KejeVxc3j8FKcpbBqQEZZACzAB65qpwenj4utp+92MrHQBcHJq3q8IUlFgKxz1IzNWSfwOVkkUO3XFoNTPoftDLkvua7kdIQro7RsypLVlM4Wc7XM8kqJEuS9YoA0lIXRsxqxc2XBABeLEqtNBKl8LZDo2sXrgzAd8ohEDiVCnUaKvRIc232BnXC5UFn5btFaEisRpfpCd6b3QD4SgMJ4Dw8KIidOBBuCIdaWXqhYHwvwmHwMtKb+xbvUxtrx+EcuMbApgw+RoZwlWETgnnFK5IJ0+eUhWKA9An7HouFSvLIXDv4M0pjUlVKAVOCVukruFWhcPNZXeG+SuwlCPkwYXHLSnM7l7yl5KnR8yuUW5pIzvDytqTwjolLSHNGa8I823ow+x8bTwWNX99/n6g2/cqm5mX0nqJpUgHRV8bR9mWg8yZQpUt8EjrpTSL1JgbCMPj4ZeZQgmxpyvbjQFl1tKaN1RAspxDCdheKZR4qVhL64UiWUwTEnDVWWpIMevpRmHWZrEr4SGBytpLlSFtraebhTMZtjAWiw/XLl//JZhkPd0s9HfgwWQ57xzVxWw1XAZhdYnLK6fmOcfHXPQ7Q6ZHJ98fnZ0tShbFrgidIdMjEmC/kl5zjOYk3vtxy3uf8dD7nHCmCNccA3PmhHstB92MIg0nUe08vDB8W5T/WVLujoBzYB1r6a19OUQikUgkEolEIpFIJBKJ/Gf8Acx/Dk3nzWEoAAAAAElFTkSuQmCC'
curl -s -o /dev/null -w "write without key: %{http_code}\n" -X POST "$R/v1/ocr" -H "Content-Type: application/json" -d "{\"image\":\"$B\"}"
curl -s -o /dev/null -w "write wrong key: %{http_code}\n" -X POST "$R/v1/ocr" -H "Content-Type: application/json" -H "Authorization: Bearer not-the-key" -d "{\"image\":\"$B\"}"
curl -s -X POST "$R/v1/ocr" -H "Content-Type: application/json" -H "Authorization: Bearer $API_KEY" -d "{\"image\":\"$B\"}" \
| grep -o '"text":"[^"]*"' | head -1
A fresh deploy prints health: 200, write without key: 401, write wrong key: 401 and "text":"HELLO RAILWAY", the words in the test image. To caption a local file, locate a phrase and outline it:
curl -X POST "https://$RAILWAY_PUBLIC_DOMAIN/v1/caption" -H "Authorization: Bearer $API_KEY" -F file=@photo.jpg -F detail=detailed
curl -X POST "https://$RAILWAY_PUBLIC_DOMAIN/v1/detect" -H "Authorization: Bearer $API_KEY" -F file=@photo.jpg -F "phrase=the red car"
curl -X POST "https://$RAILWAY_PUBLIC_DOMAIN/v1/segment" -H "Authorization: Bearer $API_KEY" -F file=@photo.jpg -F "phrase=the red car"
A bad image answers 422 with the reason, an unknown model answers 400, and a request body over 20 MB answers 413. The API never fetches a URL for you: it only reads the bytes you send, so it cannot be pointed at an address inside your network.
Why Deploy Florence-2 Vision API on Railway?
Railway is a singular platform to deploy your infrastructure stack. Railway will host your infrastructure so you don't have to deal with configuration, while allowing you to vertically and horizontally scale it.
By deploying Florence-2 on Railway, you are one step closer to supporting a complete full-stack application with minimal burden. Host your servers, databases, AI agents, and more on Railway.
- Four vision jobs, one small model — captioning, OCR, detection and segmentation without separate services for each.
- Your images stay in your container — photos, scans and screenshots are read by your own service, not sent to a hosted vision API.
- No GPU bill and no per-call fee — a caption takes about three seconds on 8 vCPU and you pay for the container, not the call.
- Open weights, commercial use — the Florence-2 weights are MIT licensed.
- Private by default — a generated bearer key guards every route but the health check, and the container refuses to start without one.
Common Use Cases
- Alt text and image search — caption product photos and screenshots for accessibility and search indexes.
- Document and screenshot OCR — pull the text, with line positions, out of a scan, a receipt or a UI screenshot.
- Vision tool for AI agents — give an LLM agent one authenticated request that describes an image, finds an object in it or reads its text.
Dependencies for Florence-2 Hosting
- A service with 8 GB of RAM (peak resident memory measured at 5.5 GB after requests on both models; the default large model alone loads at 4.7 GB). Free and Trial plans cannot run it.
- No volume: the service keeps no state and the models are in the image.
Deployment Dependencies
- microsoft/Florence-2-large and Florence-2-base — the models, converted for
transformersas florence-community/Florence-2-large and florence-community/Florence-2-base - huggingface/transformers — the inference library, Apache-2.0, version 5.23.0
- bon5co/florence-railway — the Railway wrapper image this template deploys, Apache-2.0
- Weights licence: the Hugging Face model card of
florence-community/Florence-2-largedeclareslicense: mit, and so does the card offlorence-community/Florence-2-base. MIT allows commercial use and requires keeping the copyright and licence notice. Attribution: Florence-2, copyright Microsoft, licensed MIT.
Template Content

