Reference
Speech port
The SpeechProvider, SpeechEngineInfo and SpeechRequest types, the speech routes and events, the daily budget, and the createOpenAiSpeech and createMemorySpeech adapters.
The server types come from packages/editor/src/server/ports.ts (@skaile/forge-editor/server), the wire types from packages/editor/src/core/backend-types.ts (@skaile/forge-editor/core), the adapters from @skaile/forge-editor/adapters/speech. For how to provide a port and what the editor shows, see Speech.
SpeechProvider
EditorPorts.speech (capability speech). The audio is stored through EditorPorts.assets; without it POST /api/editor/speech answers 501.
| Member | Type | Meaning |
|---|---|---|
engines | (user: object) => Promise<SpeechEngineInfo[]> | The engines this user may use, each with its voices. |
synthesize | (request: SpeechRequest, user: object, signal: AbortSignal) => Promise<{ data: Uint8Array, mime: string }> | Reads request.text with request.voice of request.engine. signal aborts when the client goes away. Throw to fail the synthesis: the message reaches the client. Throw HttpError 404 for an unknown engine or voice. |
transcribe | (audio: Uint8Array, mime: string, user: object) => Promise<{ segments: SpeechSegment[] }> | Optional. The timed text of audio. |
Types
| Type | Fields |
|---|---|
SpeechEngineInfo | id, label (already in the user's language), voices: SpeechVoice[], optional formats (the formats it makes; absent: mp3 and wav; createMemorySpeech lists wav) |
SpeechVoice | id, label, lang (BCP 47, und for any language) |
SpeechRequest | engine, voice, text, format: SpeechFormat |
SpeechFormat | 'mp3' | 'wav' |
SpeechBody | SpeechRequest plus documentId, the Document the audio is stored under |
SpeechSegment | start, end (seconds into the audio), text |
Routes
Route (API key) | Request | Answer |
|---|---|---|
GET /api/editor/speech/engines (speechEngines) | none | SpeechEngineInfo[] |
POST /api/editor/speech (speechSynthesize) | SpeechBody | SSE: progress { phase: string, percent?: number }, then done { asset: string, fileName: string, mime: string } or error { message: string } |
Both routes are served only when ports.speech is present. Refusals before the stream starts: 400, 401, 403, 404 (unknown engine or voice), 429 (daily budget, with Retry-After in seconds), 501 (no assets port).
Daily budget
Export (@skaile/forge-editor/server) | Meaning |
|---|---|
SPEECH_CHARS_PER_DAY | 50_000, the characters one user may synthesize per UTC day. |
createSpeechBudget({ limit?, now? }) | A SpeechBudget held in memory. charge(key, chars) returns 0 when the characters fit, else the seconds until UTC midnight (at least 1) and counts nothing. refund(key, chars) gives characters back after a failed synthesis. |
HandlerContext.speechBudget | The budget the speech handler uses. Without it one budget per ports object applies, with the default limit. |
EditorBackend['speech']
| Member | Type | Meaning |
|---|---|---|
engines | (signal?) => Promise<SpeechEngineInfo[]> | The engines and voices of the user. |
synthesize | (request: SpeechBody, { onProgress?, signal? }?) => Promise<RenderResult> | Streams one synthesis; RenderResult is RenderDone (asset, fileName, mime) plus the url of the stored asset. Rejects with the server's message when it fails. |
@skaile/forge-editor/adapters/speech
| Export | Meaning |
|---|---|
createOpenAiSpeech(options) | A SpeechProvider over POST <endpoint>/audio/speech. Options endpoint, model, apiKey?, id? (default openai), label?, voices?, fetch?. An upstream error rejects with its status and the first 300 characters of its body. |
OPENAI_VOICES | The voices of OpenAI's speech models. |
createMemorySpeech(options?) | A SpeechProvider that answers one second of silence as WAV; option engines. Default engine memory, voice silent. mp3 is refused with 422. |
silentWav(seconds?) | The silent mono 16-bit WAV (8 kHz) as Uint8Array. |
Subtitle export (@skaile/forge-editor/transcript)
| Export | Meaning |
|---|---|
vttOf(cues) | WebVTT text of TranscriptCue[]. |
srtOf(cues) | SubRip text of TranscriptCue[]. |
transcriptCuesOf(node) | The TranscriptCue[] of the timed transcriptSegments in a node. |
TranscriptCue | { start, end, text, speaker? }, seconds. A SpeechSegment fits. |