---
title: "Speech: text to audio, and subtitle files"
description: "The SpeechProvider port with its engines and voices, the SSE route that streams progress and stores the audio as an asset, the daily character budget, the client call, and the block action that exports a transcript as WebVTT or SubRip."
canonical_url: "https://skaile-ai.github.io/forge-editor/guides/speech"
---
# Speech: text to audio, and subtitle files

> The SpeechProvider port with its engines and voices, the SSE route that streams progress and stores the audio as an asset, the daily character budget, the client call, and the block action that exports a transcript as WebVTT or SubRip.

The `speech` capability turns text into audio. The host passes one `SpeechProvider`; the editor adds the routes, the daily budget and the client call. The audio is a file like any render result: it is stored as an asset of the Document.

<table>
<thead>
  <tr>
    <th>
      Piece
    </th>
    
    <th>
      What it does
    </th>
    
    <th>
      Port (capability)
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <code>
        SpeechProvider
      </code>
    </td>
    
    <td>
      lists engines with their voices, synthesizes text, optionally transcribes audio
    </td>
    
    <td>
      <code>
        speech
      </code>
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        GET /api/editor/speech/engines
      </code>
    </td>
    
    <td>
      the engines and voices of the user
    </td>
    
    <td>
      <code>
        speech
      </code>
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        POST /api/editor/speech
      </code>
    </td>
    
    <td>
      one synthesis, streamed as server-sent events; the audio is stored as an asset of the Document
    </td>
    
    <td>
      <code>
        speech
      </code>
      
       and <code>
        assets
      </code>
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        EditorBackend['speech']
      </code>
    </td>
    
    <td>
      <code>
        engines()
      </code>
      
       and <code>
        synthesize(request, { onProgress })
      </code>
    </td>
    
    <td>
      <code>
        speech
      </code>
    </td>
  </tr>
  
  <tr>
    <td>
      Block action <code>
        transcript-export
      </code>
    </td>
    
    <td>
      downloads the timed segments of a transcript as <code>
        .vtt
      </code>
      
       or <code>
        .srt
      </code>
    </td>
    
    <td>
      none
    </td>
  </tr>
</tbody>
</table>

A synthesis lives as long as its request. There is no job store: the server streams progress, stores the file and ends the stream. A client that disconnects aborts the engine through the `AbortSignal` that `SpeechProvider.synthesize` receives.

## Provide a speech port

```ts [docs/snippets/speech-provider.ts]
import { createMemorySpeech, createOpenAiSpeech } from '@skaile/forge-editor/adapters/speech'
import type { EditorPorts } from '@skaile/forge-editor/server'

// The speech port is one provider. `createOpenAiSpeech` calls an OpenAI-compatible `/audio/speech` endpoint from the
// server, so the key never reaches a page; `createMemorySpeech` answers a second of silence, for tests and demos.
export function speechPort(apiKey: string | undefined): NonNullable<EditorPorts['speech']> {
  return apiKey
    ? createOpenAiSpeech({ endpoint: 'https://api.openai.com/v1', model: 'gpt-4o-mini-tts', apiKey })
    : createMemorySpeech()
}
```

`createOpenAiSpeech({ endpoint, model, apiKey?, id?, label?, voices?, fetch? })` offers one engine (id `openai`) and posts `{ model, input, voice, response_format }` to `<endpoint>/audio/speech`. Its voices default to `OPENAI_VOICES`; pass `voices` for a compatible server with other voices. `createMemorySpeech({ engines? })` is one engine `memory` with the voice `silent`: every synthesis answers one second of silence as WAV (`silentWav(seconds?)` builds it), and `mp3` is refused with `422`.

A host with several engines, for example a second vendor, implements `SpeechProvider` itself and lists each engine in `engines`. A provider that can also transcribe adds `transcribe`; the routes do not use it yet.

## The route

`POST /api/editor/speech` takes `{ documentId, engine, voice, text, format }` with `format` `'mp3'` or `'wav'`. The server checks the request before the stream starts:

<table>
<thead>
  <tr>
    <th>
      Status
    </th>
    
    <th>
      Cause
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <code>
        400
      </code>
    </td>
    
    <td>
      malformed body, empty text, text longer than the daily budget, or a Document ID that is not valid
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        401
      </code>
    </td>
    
    <td>
      no user
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        403
      </code>
    </td>
    
    <td>
      the user may not read the Document
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        404
      </code>
    </td>
    
    <td>
      unknown engine, or a voice the engine does not have
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        429
      </code>
    </td>
    
    <td>
      the text does not fit the characters the user has left today; <code>
        Retry-After
      </code>
      
       is the seconds until UTC midnight
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        501
      </code>
    </td>
    
    <td>
      the host has no <code>
        speech
      </code>
      
       or no <code>
        assets
      </code>
      
       port
    </td>
  </tr>
</tbody>
</table>

After that the answer is a `text/event-stream` with the events `progress` (`{ phase, percent? }`), then `done` (`{ asset, fileName, mime }`) or `error` (`{ message }`). The audio is stored through `ports.assets.storeAsset` as an asset of `documentId`; `asset` is its name, read with the asset route like any upload. An engine that fails ends the stream with `error`, stores nothing and gives the characters back.

## Daily budget

Each user may synthesize `SPEECH_CHARS_PER_DAY` (50 000) characters per UTC day. The count is kept in memory per server process and starts over at UTC midnight, so a restart or a second process starts a fresh count. The key is the user's `id` (see `userKey`, the same key the AI rate limit uses). A call that would pass the limit is refused with `429` and counts nothing.

A test or a host with another limit builds its own budget and hands it to the handlers through `HandlerContext.speechBudget`:

```ts
import { createSpeechBudget } from '@skaile/forge-editor/server'

const speechBudget = createSpeechBudget({ limit: 1000, now: () => Date.now() })
```

## The client

`backend.speech` is present when the server serves the `speech` capability. `backend.speech.engines()` lists the engines. `backend.speech.synthesize(request, { onProgress, signal })` resolves with `{ asset, fileName, mime, url }` and rejects with a `BackendError` that carries the server's message and status (`429` over the budget). The loop over the stream is `postSse` (`core/sse.ts`), the same helper `render` uses.

## Subtitle files from a transcript

A transcript is a run of speaker segments with `start` and `end` seconds (see [Media, mentions and transcripts](https://skaile-ai.github.io/forge-editor/guides/media-mentions-transcript)). The transcript feature adds the block action **„Untertitel exportieren“** (`transcript-export`) to a Typed block that holds timed segments. It is also offered in a read-only editor, because it does not change the Document. Its menu downloads the segments as WebVTT (`.vtt`) or SubRip (`.srt`), named after the block's title.

The functions are pure and DOM-free, so a server can write the same files:

<table>
<thead>
  <tr>
    <th>
      Function
    </th>
    
    <th>
      Result
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <code>
        vttOf(cues)
      </code>
    </td>
    
    <td>
      <code>
        WEBVTT
      </code>
      
      , then one <code>
        hh:mm:ss.mmm --> hh:mm:ss.mmm
      </code>
      
       block per cue; the speaker is a <code>
        <v Name>
      </code>
      
       voice tag
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        srtOf(cues)
      </code>
    </td>
    
    <td>
      numbered <code>
        hh:mm:ss,mmm --> hh:mm:ss,mmm
      </code>
      
       blocks; the speaker is a <code>
        Name:
      </code>
      
       prefix
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        transcriptCuesOf(node)
      </code>
    </td>
    
    <td>
      the cues of the timed segments in a node, in order
    </td>
  </tr>
</tbody>
</table>

A cue is `{ start, end, text, speaker? }`. A cue without text is left out, an end before the start is raised to the start, and a segment without a start time has no cue. A segment without an end time ends where the next cue starts, the last one two seconds after its start. The functions come from `@skaile/forge-editor/transcript`. For the port types see the [Speech port reference](https://skaile-ai.github.io/forge-editor/reference/speech-port).


## Sitemap

See the full [sitemap](https://skaile-ai.github.io/forge-editor/sitemap.md) for all pages.