Speech: text to audio, and subtitle files

The speech capability turns text into audio. The host passes one SpeechProvider; the editor adds the routes, the daily budget and the client call. The audio is a file like any render result: it is stored as an asset of the Document.

A synthesis lives as long as its request. There is no job store: the server streams progress, stores the file and ends the stream. A client that disconnects aborts the engine through the AbortSignal that SpeechProvider.synthesize receives.

Provide a speech port

import { createMemorySpeech, createOpenAiSpeech } from '@skaile/forge-editor/adapters/speech'
import type { EditorPorts } from '@skaile/forge-editor/server'

// The speech port is one provider. `createOpenAiSpeech` calls an OpenAI-compatible `/audio/speech` endpoint from the
// server, so the key never reaches a page; `createMemorySpeech` answers a second of silence, for tests and demos.
export function speechPort(apiKey: string | undefined): NonNullable<EditorPorts['speech']> {
  return apiKey
    ? createOpenAiSpeech({ endpoint: 'https://api.openai.com/v1', model: 'gpt-4o-mini-tts', apiKey })
    : createMemorySpeech()
}

createOpenAiSpeech({ endpoint, model, apiKey?, id?, label?, voices?, fetch? }) offers one engine (id openai) and posts { model, input, voice, response_format } to <endpoint>/audio/speech. Its voices default to OPENAI_VOICES; pass voices for a compatible server with other voices. createMemorySpeech({ engines? }) is one engine memory with the voice silent: every synthesis answers one second of silence as WAV (silentWav(seconds?) builds it), and mp3 is refused with 422.

A host with several engines, for example a second vendor, implements SpeechProvider itself and lists each engine in engines. A provider that can also transcribe adds transcribe; the routes do not use it yet.

The route

POST /api/editor/speech takes { documentId, engine, voice, text, format } with format 'mp3' or 'wav'. The server checks the request before the stream starts:

After that the answer is a text/event-stream with the events progress ({ phase, percent? }), then done ({ asset, fileName, mime }) or error ({ message }). The audio is stored through ports.assets.storeAsset as an asset of documentId; asset is its name, read with the asset route like any upload. An engine that fails ends the stream with error, stores nothing and gives the characters back.

Daily budget

Each user may synthesize SPEECH_CHARS_PER_DAY (50 000) characters per UTC day. The count is kept in memory per server process and starts over at UTC midnight, so a restart or a second process starts a fresh count. The key is the user's id (see userKey, the same key the AI rate limit uses). A call that would pass the limit is refused with 429 and counts nothing.

A test or a host with another limit builds its own budget and hands it to the handlers through HandlerContext.speechBudget:

import { createSpeechBudget } from '@skaile/forge-editor/server'

const speechBudget = createSpeechBudget({ limit: 1000, now: () => Date.now() })

The client

backend.speech is present when the server serves the speech capability. backend.speech.engines() lists the engines. backend.speech.synthesize(request, { onProgress, signal }) resolves with { asset, fileName, mime, url } and rejects with a BackendError that carries the server's message and status (429 over the budget). The loop over the stream is postSse (core/sse.ts), the same helper render uses.

Subtitle files from a transcript

A transcript is a run of speaker segments with start and end seconds (see Media, mentions and transcripts). The transcript feature adds the block action „Untertitel exportieren“ (transcript-export) to a Typed block that holds timed segments. It is also offered in a read-only editor, because it does not change the Document. Its menu downloads the segments as WebVTT (.vtt) or SubRip (.srt), named after the block's title.

The functions are pure and DOM-free, so a server can write the same files:

A cue is { start, end, text, speaker? }. A cue without text is left out, an end before the start is raised to the start, and a segment without a start time has no cue. A segment without an end time ends where the next cue starts, the last one two seconds after its start. The functions come from @skaile/forge-editor/transcript. For the port types see the Speech port reference.