OpenAI Realtime Console

The OpenAI Realtime Console is intended as an inspector and interactive API reference for the OpenAI Realtime API. It comes packaged with two utility libraries, openai/openai-realtime-api-beta that acts as a Reference Client (for browser and Node.js) and /src/lib/wavtools which allows for simple audio management in the browser.

Using the console

The console requires an OpenAI API key (user key or project key) that has access to the Realtime API. You'll be prompted on startup to enter it. It will be saved via localStorage and can be changed at any time from the UI.

To start a session you'll need to connect. This will require microphone access. You can then choose between manual (Push-to-talk) and vad (Voice Activity Detection) conversation modes, and switch between them at any time.

There are two functions enabled;

• get_weather: Ask for the weather anywhere and the model will do its best to pinpoint the location, show it on a map, and get the weather for that location. Note that it doesn't have location access, and coordinates are "guessed" from the model's training data so accuracy might not be perfect.
• set_memory: You can ask the model to remember information for you, and it will store it in a JSON blob on the left.

You can freely interrupt the model at any time in push-to-talk or VAD mode.

Acknowledgements and contact

Thanks for checking out the Realtime Console. We hope you have fun with the Realtime API. Special thanks to the whole Realtime API team for making this possible. Please feel free to reach out, ask questions, or give feedback by creating an issue on the repository. You can also reach out and let us know what you think directly!

• OpenAI Developers / @OpenAIDevs
• Jordan Sitkin / API / @dustmason
• Mark Hudnall / API / @landakram
• Peter Bakkum / API / @pbbakkum
• Atty Eleti / API / @athyuttamre
• Jason Clark / API / @onebitToo
• Karolis Kosas / Design / @karoliskosas
• Keith Horwood / API + DX / @keithwhor
• Romain Huet / DX / @romainhuet
• Katia Gil Guzman / DX / @kagigz
• Ilan Bigio / DX / @ilanbigio
• Kevin Whinnery / DX / @kevinwhinnery