Audio: read aloud, mixdown and ElevenLabs

The add-on @skaile/forge-editor-audio builds on the library's speech port and render port. It adds three things:

Both derive methods make a Typed block of the Block type recording and record its Derivation with named inputs. That is what lets the library tell when an input changed.

Wire the server

import { createRenderer, markdownTarget } from '@skaile/forge-editor/adapters/render'
import { audioMixdownTarget, createElevenLabsSpeech } from '@skaile/forge-editor-audio/server'

// installEditorServer({ ports: { assets, speech, render } })
export const speech = createElevenLabsSpeech({ apiKey: process.env.ELEVENLABS_API_KEY! })
export const render = createRenderer([markdownTarget(), audioMixdownTarget({ assets })])

The server code is the /server entry (it starts ffmpeg through node:child_process); the root entry of the package is browser-safe and holds the Feature and the pure pieces of a Derivation, as in the other add-ons.

assets is the host's AssetStore: the target reads the takes' audio from it, and the render route stores the mixdown in it.

engines() lists the voices of the account (GET /v2/voices, every page up to a limit). A voice's lang is the locale of its first verified language, else that language, else its language label, else und. synthesize posts the text to /v1/text-to-speech/{voice_id} as MP3 (mp3_44100_128) or WAV (wav_24000) and answers with the audio and the MIME type ElevenLabs sent. An engine or voice the account does not have is refused with 404 before any call. An error answer of ElevenLabs reaches the client with its own words (ElevenLabs: 401 Invalid API key). transcribe is not offered. The daily character budget of the speech route applies as for every provider.

Wire the client

import { audioFeature } from '@skaile/forge-editor-audio'

// Add it to the preset after the composites feature (it requires it).
const audio = audioFeature({ voice: { engine: 'elevenlabs', voice: '<voice_id>' }, mixdownMode: 'concat' })

The add-on carries its own small German and English tables for its notices (audioTranslator(locale)): the library's catalog is a closed set of keys.

Read aloud: audio-tts

Offered on a Container block with a title whose body holds text, when the backend has a speech capability. The body is read as plain text, one paragraph per body block. The result is a recording block right after the text block, titled „Sprache:

Edit the text block and the recording reads veraltet. Edit the recording's body and it reads bearbeitet (outputHash is sealed).

Mixdown: audio-mixdown

Offered on a Container block with at least two takes: recording blocks among its direct children that hold stored audio or video, in document order. A recording that is itself a mixdown is no take, so a second mixdown in the same container does not mix the first one in again. The method renders the block through the target audio-mixdown and adds the result as the last recording child of the container, titled „Mixdown:

The Derivation names one input per take:

When the audio of a take is replaced, the hash of its media block changes and derivationState with an inputHashOf reports the mixdown as stale. The library's chrome, tree dot and row badge do this for every Derived block. When the first take is deleted the state is missing.

import { mixdownDerivation, takesOfNode } from '@skaile/forge-editor-audio'

const takes = takesOfNode(containerNode)
const derivation = mixdownDerivation({ takes, createdAt: new Date().toISOString(), activityId: crypto.randomUUID() })

The render target

audioMixdownTarget({ assets, label?, accepts?, ffmpeg? }) accepts document, section and folder by default. A render request holds the container's JSON, so the target finds the takes itself with the same rule as the client (takesOf). It refuses a take whose asset belongs to another Document (foreign-take), because a render may only read the assets of the Document the user is allowed to read. Request option options.mode is overlay or the default concat.

The block action „Erzeugen als …“ of the library lists the target on every Block type it accepts, so a mixdown can also be started from there; the derive method adds the Derivation.

Takes are user uploads, and ffmpeg recognises a format by its content, not its file name. Every input therefore gets -format_whitelist (INPUT_FORMATS: mp3, wav, ogg, flac, mov/mp4/m4a, matroska/webm, aac, aiff, ac3, amr, caf, asf) and -protocol_whitelist file: an HLS playlist, a concat script or a DASH manifest that names other files or URLs is refused with ffmpeg-failed instead of followed.

ffmpeg brings every input to 44.1 kHz stereo before it joins (concat) or sums (amix) them, and writes MP3 with libmp3lame. The inputs are written to a temporary directory that is removed again, also after a failure.