Text analysis solutions · 10–15%

Speech solutions

~3 min read

Speech to text (STT) transcribes spoken audio into text. Text to speech (TTS) synthesizes spoken audio from text. Azure gives you two ways to do both.

Two routes to speech — pick correctly

How the GenAI route works

You call the model like any other OpenAI API. Two calls to know:

• Transcribe (STT): client.audio.transcriptions.create(model, file, response_format="text") — you upload an audio file, you get its text back.
• Synthesize (TTS): client.audio.speech.with_streaming_response.create(model, voice="alloy", input=..., instructions=...) — you send text plus a voice name, you get audio back. instructions sets the tone (“speak in an excited tone”).
• The -transcribe-diarize model additionally labels who spoke each line (speaker separation).

Azure Speech SDK pattern (memorize)

Every Speech SDK app follows the same five steps:

1. Create a SpeechConfig with your key and endpoint (or region). It holds the connection details.
2. Optionally create an AudioConfig to say where audio comes from or goes to. Default is microphone in, speaker out; you can point it at a file instead.
3. Build the worker object from those two configs: a SpeechRecognizer (audio in, text out) or a SpeechSynthesizer (text in, audio out).
4. Call its method — recognize_once_async() to transcribe, speak_text_async() to speak.
5. Check result.reason to confirm it worked.

• Voice: speech_config.speech_synthesis_voice_name = 'en-US-Brian:DragonHDLatestNeural'.
• Output format: set_speech_synthesis_output_format(...).

SSML

SSML (Speech Synthesis Markup Language) is an XML-based markup language that controls how synthesized speech sounds. Instead of plain text, you send markup that sets:

• speaking style (mstts:express-as style="cheerful")
• breaks/pauses (<break>)
• phonemes — phonetic pronunciation (say “SQL” as “sequel”)
• prosody — pitch, timbre, speaking rate
• say-as rules (read a string as a date, time, phone number)
• inserted audio and multiple voices in one document

Speech translation

Speech translation converts spoken input in one language into text (or speech) in other languages, in one call.

• Three steps: 1) create a SpeechTranslationConfig and call add_target_language() for each target language (several are fine); 2) call TranslationRecognizer.recognize_once_async(); 3) read result.translations, a dict keyed by language code.
• Two ways to get spoken output: manual synthesis — loop over the translations and speak each one with a SpeechSynthesizer (works for several languages at once); or event-based synthesis — set voice_name on the config and collect the audio bytes inside the synthesizing event (1:1 only, no multi-language).

Voice Live API

The Voice Live API is an Azure Speech API for building agents that hold real-time voice conversations. It replaces the manual chain of speech-to-text, then a language model, then text-to-speech with one low-latency, bidirectional connection.

• Runs speech-to-speech over a WebSocket connection.
• JSON events: client events (session.update…) vs server events.
• Built-in noise suppression + echo cancellation; avatar streaming.
• Auth: Entra keyless (Bearer token) or API key. Endpoints: project wss://….services.ai.azure.com/voice-live/realtime vs model wss://….cognitiveservices.azure.com/voice-live/realtime.