Deploy Speaker Diarization | (Just Updated) Who Spoke When API, Nemotron on CPU, API Key Pre-Set

Nemotron speaker diarization API: who spoke when, up to 8 speakers

Deploy Speaker Diarization | (Just Updated) Who Spoke When API, Nemotron on CPU, API Key Pre-Set

Just deployed

Deploy and Host Speaker Diarization API (Nemotron-3-Diarization) on Railway

Speaker diarization answers "who spoke when": give it a recording and it returns time-stamped segments labelled speaker_0, speaker_1 and so on, for up to eight voices. This template serves NVIDIA Nemotron-3-Diarization, a 100M-parameter open model, behind one authenticated JSON API on CPU. The weights are already inside the image, an API key is generated for you at deploy, and no GPU is needed.

About Hosting Speaker Diarization

This template runs nvidia/Nemotron-3-Diarization with transformers 5.24.0 and PyTorch on CPU behind a small FastAPI server. The native transformers classes are used, so no remote model code and no NeMo toolkit. POST /v1/diarize takes audio as a multipart file or as a base64 audio field in a JSON body, in any format ffmpeg reads (wav, mp3, m4a, ogg, flac, opus, mp4), and returns segments with speaker, speaker_id, start and end in seconds, plus num_speakers and duration_seconds. Speakers are numbered by first appearance. Segments of different speakers may overlap, because the model detects overlapped speech. Add rttm=true to also get an RTTM string for scoring tools, and merge_gap (seconds) to join one speaker's neighbouring segments. mode defaults to offline, which gives the whole recording to the model; low_latency, very_low_latency and ultra_low_latency run the step-by-step streaming path of the same model, which is far slower on CPU. GET /v1/models describes the model.

The weights are baked into the image, so the first request is answered from disk and no model download hides inside a request that Railway's edge would abandon after five minutes. The service sizes its PyTorch thread pool from the container's own CPU quota instead of the host's core count, handles one request at a time so memory stays bounded, and refuses a recording longer than 1800 seconds or a file above 100 MB.

Measured on Railway, 8 vCPU: a 97.6 s conversation was diarized in 1.2 s (81x real time); a 586 s recording in 10.2 s (58x); a 1758 s recording in 28.5 s (62x), the whole request answered in 28.9 s. The default offline mode is the one to use. The streaming low_latency mode took 25.4 s on the 97.6 s clip, 3.9x real time, about 20 times slower for the same audio. The model loads in about 5 s after the container starts, and the process peak resident memory read 0.8 GB after boot, 1.3 GB after the 586 s file and 2.3 GB after the 1758 s file (peak_rss_mb on /health), so memory grows with recording length. Accuracy was not benchmarked here: the clip gave six speakers where the number actually present was not checked, so treat num_speakers as the model's estimate. Short back-channel utterances and heavy overlap are the usual weak spots of any diarizer.

Agent contract

  • Public URL: https://RAILWAY_PUBLIC_DOMAIN (the generated domain of the diarization service)
  • Private URL: http://diarization.railway.internal:8080
  • Credentials: the API_KEY variable of the diarization service
  • Auth: send Authorization: Bearer $API_KEY on every route except GET /health; a call without the key or with a wrong one answers 401
  • Health: GET /health returns 200 once the model is loaded (503 while it loads)
# needs: RAILWAY_PUBLIC_DOMAIN, API_KEY (a variable of the diarization service); also curl.
# signup: none - the API has no user accounts, only the API_KEY bearer
R="https://$RAILWAY_PUBLIC_DOMAIN"
n=0; until [ "$(curl -s -o /dev/null -w '%{http_code}' "$R/health")" = 200 ]; do n=$((n+1)); [ "$n" -le 60 ] || break; sleep 5; done
curl -s -o /dev/null -w "health: %{http_code}\n" "$R/health"
# a 98 s recording of a conversation (public test clip, Hugging Face hf-internal-testing)
curl -sL -o /tmp/meeting.mp3 https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/diarization_example.mp3
curl -s -o /dev/null -w "write without key: %{http_code}\n" -X POST "$R/v1/diarize" -F file=@/tmp/meeting.mp3
curl -s -o /dev/null -w "write wrong key: %{http_code}\n" -X POST "$R/v1/diarize" -H "Authorization: Bearer not-the-key" -F file=@/tmp/meeting.mp3
curl -s -X POST "$R/v1/diarize" -H "Authorization: Bearer $API_KEY" -F file=@/tmp/meeting.mp3 \
  | grep -o '"num_speakers":[0-9]*' | head -1

A fresh deploy prints health: 200, write without key: 401, write wrong key: 401 and a "num_speakers" count for the test clip. To diarize your own file and get an RTTM:

curl -X POST "https://$RAILWAY_PUBLIC_DOMAIN/v1/diarize" -H "Authorization: Bearer $API_KEY" -F file=@call.wav -F rttm=true -F merge_gap=0.5

An unreadable file answers 422 with the reason, an unknown mode answers 422, and a recording over the limit answers 413. The API never fetches a URL for you: it only reads the bytes you send, so it cannot be pointed at an address inside your network.

Why Deploy Speaker Diarization API on Railway?

Railway is a singular platform to deploy your infrastructure stack. Railway will host your infrastructure so you don't have to deal with configuration, while allowing you to vertically and horizontally scale it.

By deploying Speaker Diarization on Railway, you are one step closer to supporting a complete full-stack application with minimal burden. Host your servers, databases, AI agents, and more on Railway.

  • Who spoke when, in one request — speaker labels with start and end times, ready to merge with a transcript.
  • Your recordings stay in your container — calls and meetings are read by your own service, not sent to a hosted diarization API.
  • No GPU bill and no per-minute fee — about an hour of audio per minute of compute on 8 vCPU, and you pay for the container.
  • Open weights, commercial use — the OpenMDW-1.1 licence grants use without restriction and puts no restriction on outputs.
  • Private by default — a generated bearer key guards every route but the health check, and the container refuses to start without one.

Common Use Cases

  • Meeting and call transcripts — pair the segments with a speech-to-text service to label each line with its speaker.
  • Podcast and interview editing — find each guest's talk time and cut by speaker.
  • Speaker tool for AI agents — give an LLM agent one authenticated request that says how many people talk in a recording and when.

Dependencies for Speaker Diarization Hosting

  • A service with at least 4 GB of RAM for recordings up to the 1800 s limit (peak resident memory measured at 2.3 GB after a 1758 s file).
  • No volume: the service keeps no state and the model is in the image.

Deployment Dependencies

  • nvidia/Nemotron-3-Diarization — the model, a streaming Sortformer with up to 8 speakers
  • huggingface/transformers — the inference library, Apache-2.0, version 5.24.0
  • bon5co/diarization-railway — the Railway wrapper image this template deploys, Apache-2.0
  • Weights licence: the Hugging Face model card of nvidia/Nemotron-3-Diarization declares license: openmdw-1.1 and says "Use of this model is governed by the OpenMDW License Agreement, version 1.1". The agreement grants "permission ... free of charge, to deal in the Model Materials without restriction", requires that "if you distribute any portion of the Model Materials, you shall retain in your distribution (1) a copy of this agreement, and (2) all copyright notices and other notices of origin", and states that it "does not impose any restrictions or obligations with respect to any use, modification, or sharing of any outputs generated by using the Model Materials". The image keeps the agreement next to the weights. Attribution: Nemotron-3-Diarization, copyright NVIDIA Corporation, OpenMDW-1.1.

Template Content

More templates in this category

View Template
Chat Chat
Chat Chat, your own unified chat and search to AI platform.

okisdev
117
View Template
Weaviate
Vector database with hybrid search and an API key-protected API

GitHub Locker
1
View Template
Möbius Personal AI OS
Self-hosted AI agent that builds and runs apps for you, with memory.

Tracklayer
1