---
title: "Audio: read aloud, mixdown and ElevenLabs"
description: "The add-on @skaile/forge-editor-audio: an ElevenLabs SpeechProvider, the derive methods audio-tts and audio-mixdown that make recording blocks with named Derivation inputs, and the render target audio-mixdown that mixes takes through ffmpeg."
canonical_url: "https://skaile-ai.github.io/forge-editor/guides/audio"
---
# Audio: read aloud, mixdown and ElevenLabs

> The add-on @skaile/forge-editor-audio: an ElevenLabs SpeechProvider, the derive methods audio-tts and audio-mixdown that make recording blocks with named Derivation inputs, and the render target audio-mixdown that mixes takes through ffmpeg.

The add-on `@skaile/forge-editor-audio` builds on the library's [speech port](https://skaile-ai.github.io/forge-editor/guides/speech) and [render port](https://skaile-ai.github.io/forge-editor/guides/render). It adds three things:

<table>
<thead>
  <tr>
    <th>
      Piece
    </th>
    
    <th>
      What it does
    </th>
    
    <th>
      Needs
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <code>
        createElevenLabsSpeech
      </code>
    </td>
    
    <td>
      a <code>
        SpeechProvider
      </code>
      
       over ElevenLabs: engine <code>
        elevenlabs
      </code>
      
       with the voices of the account
    </td>
    
    <td>
      an ElevenLabs API key
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        audioMixdownTarget
      </code>
    </td>
    
    <td>
      the render target <code>
        audio-mixdown
      </code>
      
       (<code>
        audio/mpeg
      </code>
      
      )
    </td>
    
    <td>
      <code>
        ffmpeg
      </code>
      
       on the <code>
        PATH
      </code>
      
       of the server, the <code>
        assets
      </code>
      
       port
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        audioFeature
      </code>
    </td>
    
    <td>
      the <a href="https://skaile-ai.github.io/forge-editor/guides/derive-methods">
        derive methods
      </a>
      
       <strong>
        „Als Sprache vorlesen“
      </strong>
      
       (<code>
        audio-tts
      </code>
      
      ) and <strong>
        „Mixdown erzeugen“
      </strong>
      
       (<code>
        audio-mixdown
      </code>
      
      )
    </td>
    
    <td>
      the <code>
        speech
      </code>
      
       and the <code>
        render
      </code>
      
       capability of the backend
    </td>
  </tr>
</tbody>
</table>

Both derive methods make a Typed block of the Block type `recording` and record its [Derivation](https://skaile-ai.github.io/forge-editor/guides/derive-methods) with **named inputs**. That is what lets the library tell when an input changed.

## Wire the server

```ts
import { createRenderer, markdownTarget } from '@skaile/forge-editor/adapters/render'
import { audioMixdownTarget, createElevenLabsSpeech } from '@skaile/forge-editor-audio/server'

// installEditorServer({ ports: { assets, speech, render } })
export const speech = createElevenLabsSpeech({ apiKey: process.env.ELEVENLABS_API_KEY! })
export const render = createRenderer([markdownTarget(), audioMixdownTarget({ assets })])
```

The server code is the `/server` entry (it starts `ffmpeg` through `node:child_process`); the root entry of the package is browser-safe and holds the Feature and the pure pieces of a Derivation, as in the other add-ons.

`assets` is the host's `AssetStore`: the target reads the takes' audio from it, and the render route stores the mixdown in it.

<table>
<thead>
  <tr>
    <th>
      Option of <code>
        createElevenLabsSpeech
      </code>
    </th>
    
    <th>
      Meaning
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <code>
        apiKey
      </code>
    </td>
    
    <td>
      sent as <code>
        xi-api-key
      </code>
      
      ; the key stays on the server
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        endpoint
      </code>
    </td>
    
    <td>
      base URL, default <code>
        https://api.elevenlabs.io
      </code>
      
      ; a regional host such as <code>
        https://api.eu.residency.elevenlabs.io
      </code>
      
       goes here
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        modelId
      </code>
    </td>
    
    <td>
      the speech model, default <code>
        eleven_multilingual_v2
      </code>
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        voiceSettings
      </code>
    </td>
    
    <td>
      <code>
        voice_settings
      </code>
      
       of every synthesis (<code>
        stability
      </code>
      
      , <code>
        similarity_boost
      </code>
      
      , <code>
        style
      </code>
      
      , <code>
        speed
      </code>
      
      , <code>
        use_speaker_boost
      </code>
      
      )
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        id
      </code>
      
      , <code>
        label
      </code>
    </td>
    
    <td>
      the engine's id and label in <code>
        engines()
      </code>
      
       (default <code>
        elevenlabs
      </code>
      
      , <code>
        ElevenLabs
      </code>
      
      )
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        voiceCacheMs
      </code>
    </td>
    
    <td>
      how long the voice list is kept (default five minutes)
    </td>
  </tr>
</tbody>
</table>

`engines()` lists the voices of the account (`GET /v2/voices`, every page up to a limit). A voice's `lang` is the locale of its first verified language, else that language, else its `language` label, else `und`. `synthesize` posts the text to `/v1/text-to-speech/{voice_id}` as MP3 (`mp3_44100_128`) or WAV (`wav_24000`) and answers with the audio and the MIME type ElevenLabs sent. An engine or voice the account does not have is refused with `404` before any call. An error answer of ElevenLabs reaches the client with its own words (`ElevenLabs: 401 Invalid API key`). `transcribe` is not offered. The daily character budget of the speech route applies as for every provider.

## Wire the client

```ts
import { audioFeature } from '@skaile/forge-editor-audio'

// Add it to the preset after the composites feature (it requires it).
const audio = audioFeature({ voice: { engine: 'elevenlabs', voice: '<voice_id>' }, mixdownMode: 'concat' })
```

<table>
<thead>
  <tr>
    <th>
      Option
    </th>
    
    <th>
      Meaning
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <code>
        voice
      </code>
    </td>
    
    <td>
      the voice <code>
        audio-tts
      </code>
      
       reads with; absent, or not offered by the service: the first voice of the surface's language, else the first voice
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        mixdownMode
      </code>
    </td>
    
    <td>
      <code>
        concat
      </code>
      
       (default, one after the other) or <code>
        overlay
      </code>
      
       (all at once, as long as the longest)
    </td>
  </tr>
</tbody>
</table>

The add-on carries its own small German and English tables for its notices (`audioTranslator(locale)`): the library's catalog is a closed set of keys.

## Read aloud: `audio-tts`

Offered on a Container block with a title whose body holds text, when the backend has a `speech` capability. The body is read as plain text, one paragraph per body block. The result is a `recording` block right after the text block, titled „Sprache: <title>

“, that holds the audio as its source. Its Derivation:

</title>



<table>
<thead>
  <tr>
    <th>
      Field
    </th>
    
    <th>
      Value
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <code>
        method
      </code>
    </td>
    
    <td>
      <code>
        tts
      </code>
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        from
      </code>
    </td>
    
    <td>
      the text block
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        inputs.text
      </code>
    </td>
    
    <td>
      the text block, with the hash of its body Markdown at the time it was read
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        inputs.voice
      </code>
    </td>
    
    <td>
      <code>
        { asset: 'voice:<engine>/<voice>' }
      </code>
      
       with the hash of engine and voice; no block holds that asset, so the library reads the voice as unknown, never as changed
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        inputHash
      </code>
    </td>
    
    <td>
      <code>
        derivationFingerprint
      </code>
      
       of method and input hashes
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        activity
      </code>
    </td>
    
    <td>
      <code>
        { id, agent: <engine>, at }
      </code>
    </td>
  </tr>
</tbody>
</table>

Edit the text block and the recording reads **veraltet**. Edit the recording's body and it reads **bearbeitet** (`outputHash` is sealed).

## Mixdown: `audio-mixdown`

Offered on a Container block with at least two **takes**: `recording` blocks among its direct children that hold stored audio or video, in document order. A recording that is itself a mixdown is no take, so a second mixdown in the same container does not mix the first one in again. The method renders the block through the target `audio-mixdown` and adds the result as the last `recording` child of the container, titled „Mixdown: <title>

“.

</title>



The Derivation names one input per take:

<table>
<thead>
  <tr>
    <th>
      Field
    </th>
    
    <th>
      Value
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <code>
        method
      </code>
    </td>
    
    <td>
      <code>
        mixdown
      </code>
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        from
      </code>
    </td>
    
    <td>
      the audio asset of the first take
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        inputs
      </code>
    </td>
    
    <td>
      <code>
        take-1
      </code>
      
       … <code>
        take-n
      </code>
      
       in mixing order: <code>
        { from: { asset }, hash }
      </code>
      
      , <code>
        hash
      </code>
      
       being the <code>
        hash
      </code>
      
       the take's media block recorded (<code>
        ''
      </code>
      
       when it has none, which reads as unknown)
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        inputHash
      </code>
    </td>
    
    <td>
      <code>
        derivationFingerprint({ method: 'mixdown', inputs })
      </code>
    </td>
  </tr>
</tbody>
</table>

When the audio of a take is replaced, the hash of its media block changes and `derivationState` with an `inputHashOf` reports the mixdown as **stale**. The library's chrome, tree dot and row badge do this for every Derived block. When the first take is deleted the state is **missing**.

```ts
import { mixdownDerivation, takesOfNode } from '@skaile/forge-editor-audio'

const takes = takesOfNode(containerNode)
const derivation = mixdownDerivation({ takes, createdAt: new Date().toISOString(), activityId: crypto.randomUUID() })
```

### The render target

`audioMixdownTarget({ assets, label?, accepts?, ffmpeg? })` accepts `document`, `section` and `folder` by default. A render request holds the container's JSON, so the target finds the takes itself with the same rule as the client (`takesOf`). It refuses a take whose asset belongs to another Document (`foreign-take`), because a render may only read the assets of the Document the user is allowed to read. Request option `options.mode` is `overlay` or the default `concat`.

The block action **„Erzeugen als …“** of the library lists the target on every Block type it accepts, so a mixdown can also be started from there; the derive method adds the Derivation.

<table>
<thead>
  <tr>
    <th>
      Error code (<code>
        AudioError.code
      </code>
      
      )
    </th>
    
    <th>
      Cause
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <code>
        ffmpeg-missing
      </code>
    </td>
    
    <td>
      no <code>
        ffmpeg
      </code>
      
       on the <code>
        PATH
      </code>
      
       (or at <code>
        ffmpeg
      </code>
      
       of the options)
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        ffmpeg-failed
      </code>
    </td>
    
    <td>
      ffmpeg exited with an error, for example an input that is no audio
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        too-few-takes
      </code>
    </td>
    
    <td>
      fewer than two takes
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        take-missing
      </code>
    </td>
    
    <td>
      the audio of a take is not in the asset store
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        foreign-take
      </code>
    </td>
    
    <td>
      the audio of a take is stored with another Document
    </td>
  </tr>
</tbody>
</table>

Takes are user uploads, and ffmpeg recognises a format by its content, not its file name. Every input therefore gets `-format_whitelist` (`INPUT_FORMATS`: mp3, wav, ogg, flac, mov/mp4/m4a, matroska/webm, aac, aiff, ac3, amr, caf, asf) and `-protocol_whitelist file`: an HLS playlist, a concat script or a DASH manifest that names other files or URLs is refused with `ffmpeg-failed` instead of followed.

ffmpeg brings every input to 44.1 kHz stereo before it joins (`concat`) or sums (`amix`) them, and writes MP3 with `libmp3lame`. The inputs are written to a temporary directory that is removed again, also after a failure.


## Sitemap

See the full [sitemap](https://skaile-ai.github.io/forge-editor/sitemap.md) for all pages.