# Custom Pronunciation
Source: https://docs.fluidinference.com/asr/custom-pronunciation
Override TTS pronunciation with custom lexicon files.
## Overview
FluidAudio TTS supports custom pronunciation dictionaries that override how specific words are pronounced. Essential for domain-specific terminology, brand names, acronyms, and proper nouns.
### Priority Order
1. Per-word phonetic overrides — Inline markup like `[word](/phonemes/)`
2. Custom lexicon — Your `word=phonemes` file entries
3. Case-sensitive built-in lexicon
4. Standard built-in lexicon
5. Grapheme-to-phoneme (G2P) — eSpeak-NG fallback
## File Format
```text theme={null}
# This is a comment
kokoro=kəkˈɔɹO
NASDAQ=nˈæzdæk
UN=junˈaɪtᵻd nˈeɪʃənz
```
Phonemes are compact IPA strings. Use whitespace to separate words in multi-word expansions.
## Word Matching
Three-tier strategy:
1. **Exact match** — `NASDAQ` matches only `NASDAQ`
2. **Case-insensitive** — `nasdaq` matches `NASDAQ`, `Nasdaq`
3. **Normalized** — Strips to letters/digits/apostrophes, lowercased
## Usage
### CLI
```bash theme={null}
swift run fluidaudio tts "The NASDAQ index rose today" \
--lexicon custom.txt --output output.wav
```
### Swift API
```swift theme={null}
let lexicon = try TtsCustomLexicon.load(from: fileURL)
let manager = TtSManager(customLexicon: lexicon)
try await manager.initialize()
let audio = try await manager.synthesize(text: "Welcome to Kokoro TTS")
// Update at runtime
manager.setCustomLexicon(newLexicon)
```
### Merging Lexicons
```swift theme={null}
let combined = baseLexicon.merged(with: domainLexicon)
```
## Example Lexicon
```text theme={null}
# Finance
NASDAQ=nˈæzdæk
EBITDA=iːbˈɪtdɑː
# Technology
NVIDIA=ɛnvˈɪdiə
Kubernetes=kuːbɚnˈɛtiːz
# Product Names
Kokoro=kəkˈɔɹO
FluidAudio=flˈuːɪd ˈɔːdioʊ
```
# Custom Vocabulary
Source: https://docs.fluidinference.com/asr/custom-vocabulary
CTC-based vocabulary boosting for domain-specific terms without retraining.
Custom vocabulary boosting is **batch mode only** (Parakeet TDT). It is not supported with streaming ASR (Parakeet EOU).
## Overview
FluidAudio's CTC-based custom vocabulary boosting enables accurate recognition of domain-specific terms (company names, technical jargon, proper nouns) without retraining the ASR model.
Based on the NVIDIA NeMo paper: [CTC-based Word Spotter](https://arxiv.org/abs/2406.07096).
## Architecture
The system uses two encoders processing the same audio:
1. **TDT Encoder** (Parakeet 0.6B) — Primary high-quality transcription
2. **CTC Encoder** (Parakeet 110M) — Keyword spotting with per-frame log-probabilities
Both encoders produce frames at the same rate (\~40ms), enabling direct timestamp comparison.
## Quick Start
```swift theme={null}
let asrManager = try await AsrManager.shared
let ctcModels = try await CtcModels.downloadAndLoad()
let ctcSpotter = CtcKeywordSpotter(models: ctcModels)
let vocabulary = CustomVocabularyContext(terms: [
CustomVocabularyTerm(text: "NVIDIA"),
CustomVocabularyTerm(text: "TensorRT"),
])
let result = try await asrManager.transcribe(
audioSamples,
customVocabulary: vocabulary
)
// result.text: "NVIDIA announced TensorRT optimizations"
```
## Aliases
Handle common misspellings or phonetic variations:
```swift theme={null}
let vocabulary = CustomVocabularyContext(terms: [
CustomVocabularyTerm(
text: "Hagen-Dazs",
aliases: ["Haagen-Dazs", "Hagen-Das", "Hagen Daz"]
),
CustomVocabularyTerm(
text: "macOS",
aliases: ["Mac OS", "Mac O S", "Macos"]
),
])
```
When a match is found via canonical or alias, the **canonical form** is used in the output.
## Detection Thresholds
| Parameter | Default | Description |
| ------------------------- | ------- | ----------------------------------------- |
| `defaultMinSpotterScore` | -15.0 | Minimum CTC score for keyword spotting |
| `defaultMinVocabCtcScore` | -12.0 | Minimum CTC score for vocabulary matching |
| `defaultCbw` | 3.0 | Context-biasing weight boost |
| `defaultMinSimilarity` | 0.52 | Minimum string similarity |
## Vocabulary Size Guidelines
| Size | Performance | Notes |
| ------------- | ----------- | ------------------------------------ |
| 1-50 terms | Excellent | Typical use case |
| 50-100 terms | Good | No noticeable latency |
| 100-230 terms | Tested | Validated with domain-specific lists |
## Memory
| Configuration | Peak RAM |
| ------------------ | -------- |
| TDT encoder only | \~66 MB |
| TDT + CTC encoders | \~130 MB |
## Why Batch Only
Custom vocabulary requires the complete CTC log-probability matrix for accurate scoring. Streaming ASR processes audio in small chunks (160-320ms), which is too short for reliable keyword spotting and rescoring. Keywords spanning chunk boundaries would be missed, and the rescorer cannot look ahead to future frames for optimal alignment.
## Benchmarks
[Earnings22](https://huggingface.co/datasets/revdotcom/earnings22) (771 files, 3.2h audio — earnings call transcripts with domain-specific terms):
| Metric | Value |
| ------------------ | ----------------------- |
| Average WER | 15.0% |
| Vocab Precision | 99.3% (TP=1068, FP=8) |
| Vocab Recall | 85.2% (TP=1068, FN=185) |
| Vocab F-score | 91.7% |
| Dict Pass (Recall) | 99.3% (1299/1308) |
| RTFx | 63.4x |
Precision = "of words we output, how many were correct?" Recall = "of words that should appear, how many did we find?"
The 63x RTFx is slower than TDT-only (156x) because two encoders run on the same audio. Still well above real-time.
# ASR Getting Started
Source: https://docs.fluidinference.com/asr/getting-started
Batch and streaming transcription with Parakeet models.
## When to Use
* **Transcribing recordings or files** — Use batch ASR (this page). 210x real-time, 2.5% WER on English.
* **Live captions while user speaks** — Use [Streaming ASR](/asr/streaming) with Parakeet EOU.
* **Domain-specific terms keep getting wrong** — Add [Custom Vocabulary](/asr/custom-vocabulary) boosting (91.7% F-score on earnings calls).
## Models
| Model | Languages | Audio Length | Use Case |
| ------------------- | ------------ | ------------ | ---------------------- |
| **Parakeet TDT v3** | 25 European | \~15s chunks | Default — multilingual |
| **Parakeet TDT v2** | English only | \~15s chunks | Best English accuracy |
| **Parakeet EOU** | English | 320ms chunks | Real-time streaming |
## Batch Transcription
Real-time factor: \~120x on M4 Pro (1 minute of audio in \~0.5 seconds).
```swift theme={null}
import FluidAudio
Task {
let models = try await AsrModels.downloadAndLoad(version: .v3) // .v2 for English-only
let asrManager = AsrManager(config: .default)
try await asrManager.initialize(models: models)
let samples = try AudioConverter().resampleAudioFile(
path: "path/to/audio.wav"
)
let result = try await asrManager.transcribe(samples, source: .system)
print("Transcription: \(result.text)")
print("Confidence: \(result.confidence)")
}
```
### Transcribing from a File URL
```swift theme={null}
let audioURL = URL(fileURLWithPath: "/path/to/audio.wav")
let result = try await asrManager.transcribe(audioURL, source: .system)
print(result.text)
```
Do not parse WAV/PCM bytes by hand. Always convert with `AudioConverter` so differing bit depths, channel layouts, metadata chunks, or compressed formats get normalized to the 16 kHz mono Float32 tensors that Parakeet expects.
## Choosing a Model Version
* **v2** — English only. Tighter vocabulary, better recall on long-form English audio.
* **v3** — 25 European languages. English accuracy is still strong, but the broader vocab slightly trails v2 on rare words.
Both share the same API surface—set `AsrModelVersion` in code or pass `--model-version` in the CLI.
```swift theme={null}
let models = try await AsrModels.downloadAndLoad(version: .v2)
```
## Benchmarks
[LibriSpeech test-clean](https://huggingface.co/datasets/openslr/librispeech_asr) (2,620 files, 5.4h audio):
| Model | WER | RTFx |
| -------------------- | ---- | ---- |
| Parakeet TDT v3 | 2.5% | 156x |
| Parakeet TDT v2 | 2.1% | 146x |
| Parakeet EOU (320ms) | 4.9% | 12x |
[FLEURS](https://huggingface.co/datasets/google/fleurs) (14,085 files, 44.9h audio, 25 languages):
| Model | Avg WER | RTFx |
| --------------- | ------- | ---- |
| Parakeet TDT v3 | 14.7% | 210x |
See [full benchmarks](/reference/benchmarks) for per-language breakdown.
## CLI
```bash theme={null}
# Transcribe (multilingual)
swift run fluidaudio transcribe audio.wav
# English-only (better recall)
swift run fluidaudio transcribe audio.wav --model-version v2
# Multiple files in parallel
swift run fluidaudio multi-stream audio1.wav audio2.wav
# Benchmark on LibriSpeech
swift run fluidaudio asr-benchmark --subset test-clean --max-files 50
```
# Manual Model Loading
Source: https://docs.fluidinference.com/asr/manual-model-loading
Deploy ASR models offline without HuggingFace downloads.
## Required Assets
Each ASR release ships four CoreML bundles plus vocabulary:
* `Preprocessor.mlmodelc`
* `Encoder.mlmodelc`
* `Decoder.mlmodelc`
* `JointDecision.mlmodelc`
* `parakeet_vocab.json`
## Directory Layout
```
/opt/models
└── parakeet-tdt-0.6b-v3-coreml
├── Preprocessor.mlmodelc
├── Encoder.mlmodelc
├── Decoder.mlmodelc
├── JointDecision.mlmodelc
└── parakeet_vocab.json
```
## Download Options
1. **Git LFS clone:**
```bash theme={null}
git lfs install
git clone https://huggingface.co/FluidInference/parakeet-tdt-0.6b-v3-coreml
```
2. **HuggingFace web UI** — download `.tar` archives
3. **Copy from cache** — from a machine that already ran `downloadAndLoad` (macOS: `~/Library/Application Support/FluidAudio/Models/`)
## Loading Without Downloads
```swift theme={null}
import FluidAudio
let repoDirectory = URL(
fileURLWithPath: "/opt/models/parakeet-tdt-0.6b-v3-coreml",
isDirectory: true
)
let models = try await AsrModels.load(
from: repoDirectory,
configuration: AsrModels.defaultConfiguration(),
version: .v3
)
let asrManager = AsrManager()
try await asrManager.initialize(models: models)
```
## Switching Versions
```swift theme={null}
let englishRepo = URL(fileURLWithPath: "/opt/models/parakeet-tdt-0.6b-v2-coreml")
let englishModels = try await AsrModels.load(from: englishRepo, version: .v2)
```
## Troubleshooting
* Use `AsrModels.modelsExist(at:)` to verify all bundles are present
* `parakeet_vocab.json` must sit beside the model bundles
* If you see `AsrModelsError.modelNotFound`, check folder names and `coremldata.bin` files
# Streaming ASR
Source: https://docs.fluidinference.com/asr/streaming
Real-time streaming transcription with Parakeet EOU and end-of-utterance detection.
## Overview
`StreamingEouAsrManager` provides real-time streaming ASR with End-of-Utterance detection using the Parakeet EOU 120M model.
## Quick Start
```swift theme={null}
let manager = StreamingEouAsrManager(chunkSize: .ms160, eouDebounceMs: 1280)
try await manager.loadModels(modelDir: modelsURL)
// Process audio incrementally
_ = try await manager.process(audioBuffer: buffer1)
_ = try await manager.process(audioBuffer: buffer2)
// Get final transcript
let transcript = try await manager.finish()
// Reset for next utterance
await manager.reset()
```
## Configuration
```swift theme={null}
let manager = StreamingEouAsrManager(
chunkSize: .ms320, // .ms160, .ms320, or .ms1600
eouDebounceMs: 1280 // Minimum silence before EOU triggers
)
```
## EOU Callback
```swift theme={null}
manager.setEouCallback { transcript in
print("End of utterance: \(transcript)")
}
```
## API
| Method | Description |
| ----------------------- | ------------------------------------------------- |
| `loadModels(modelDir:)` | Load CoreML models from directory |
| `process(audioBuffer:)` | Process audio incrementally |
| `finish()` | Finalize and return transcript |
| `reset()` | Reset state for next utterance |
| `appendAudio(_:)` | Append audio without processing (VAD integration) |
## Benchmarks
[LibriSpeech test-clean](https://huggingface.co/datasets/openslr/librispeech_asr) (2,620 files, 5.4h audio):
| Chunk Size | Latency | WER | RTFx |
| ---------- | ------------------ | ---- | ----- |
| 160ms | Lowest | \~8% | \~5x |
| 320ms | Balanced | \~5% | \~12x |
| 1600ms | Highest throughput | — | — |
320ms is the recommended default — best accuracy/latency tradeoff.
## CLI
```bash theme={null}
# Transcribe a file
swift run fluidaudio parakeet-eou --input audio.wav --use-cache
# Benchmark
swift run fluidaudio parakeet-eou --benchmark --chunk-size 160 --max-files 100 --use-cache
```
# Configuration
Source: https://docs.fluidinference.com/configuration
Model registry, proxy settings, and environment configuration.
## Model Registry URL
Models auto-download from HuggingFace by default. You can override this to use a mirror, local server, or air-gapped environment.
### Programmatic Override (recommended for apps)
```swift theme={null}
import FluidAudio
ModelRegistry.baseURL = "https://your-mirror.example.com"
let diarizer = DiarizerManager()
```
### Environment Variables (recommended for CLI/testing)
```bash theme={null}
export REGISTRY_URL=https://your-mirror.example.com
swift run fluidaudio transcribe audio.wav
# Or use the alias
export MODEL_REGISTRY_URL=https://models.internal.corp
```
### Xcode Scheme
1. Edit Scheme > Run > Arguments
2. Go to **Environment Variables** tab
3. Add: `REGISTRY_URL` = `https://your-mirror.example.com`
**Priority order:** programmatic override > env vars > default (HuggingFace)
## Proxy Configuration
If you're behind a corporate firewall, set the `https_proxy` environment variable:
```bash theme={null}
export https_proxy=http://proxy.company.com:8080
# Or for authenticated proxies:
export https_proxy=http://user:password@proxy.company.com:8080
```
### When to Use Which
| Scenario | Solution |
| -------------------------------------- | --------------------- |
| Local mirror or internal model server | Registry URL override |
| Behind a corporate firewall with proxy | Proxy configuration |
**Registry URL** — App requests from `your-mirror.com` instead of `huggingface.co`.
**Proxy** — App still requests `huggingface.co`, but traffic routes through the proxy.
In most cases, you only need one.
# Diarization Getting Started
Source: https://docs.fluidinference.com/diarization/getting-started
Speaker diarization — identify who spoke when in audio.
## When to Use
* **Post-recording analysis** (meetings, interviews) — Use the [Offline pipeline](/diarization/offline-pipeline). 15% DER, 122x real-time.
* **Real-time "who's speaking now"** — Use [Streaming diarization](/diarization/streaming). 26% DER at 5s chunks. Only use when you critically need real-time labels — offline is more accurate and still very fast.
* **Simple 2-4 speaker conversations** — Consider [Sortformer](/diarization/sortformer). Single model, no clustering, 32% DER. Better in noisy environments but limited to 4 speakers max — does not work well with 5+ people or heavy crosstalk.
## Quick Start
```swift theme={null}
import FluidAudio
let models = try await DiarizerModels.downloadIfNeeded()
let diarizer = DiarizerManager()
diarizer.initialize(models: models)
let samples = try AudioConverter().resampleAudioFile(
URL(fileURLWithPath: "meeting.wav")
)
let result = try diarizer.performCompleteDiarization(samples)
for segment in result.segments {
print("Speaker \(segment.speakerId): \(segment.startTimeSeconds)s - \(segment.endTimeSeconds)s")
}
```
## Configuration
```swift theme={null}
let config = DiarizerConfig(
clusteringThreshold: 0.7, // Speaker separation (0.0-1.0)
minSpeechDuration: 1.0, // Minimum segment duration (seconds)
minSilenceGap: 0.5, // Minimum silence between speakers
minActiveFramesCount: 10.0, // Minimum active frames
debugMode: false
)
let diarizer = DiarizerManager(config: config)
```
## Known Speaker Recognition
Pre-load speaker profiles for identification:
```swift theme={null}
let aliceAudio = loadAudioFile("alice_sample.wav")
let aliceEmbedding = try diarizer.extractEmbedding(aliceAudio)
let alice = Speaker(id: "Alice", name: "Alice", currentEmbedding: aliceEmbedding)
let bob = Speaker(id: "Bob", name: "Bob", currentEmbedding: bobEmbedding)
diarizer.speakerManager.initializeKnownSpeakers([alice, bob])
// Will use "Alice" instead of "Speaker_1" when matched
let result = try diarizer.performCompleteDiarization(audioSamples)
```
## Manual Model Loading
Stage Core ML bundles for offline deployment:
```swift theme={null}
let basePath = "/opt/models/speaker-diarization-coreml"
let segmentation = URL(fileURLWithPath: basePath)
.appendingPathComponent("pyannote_segmentation.mlmodelc")
let embedding = URL(fileURLWithPath: basePath)
.appendingPathComponent("wespeaker_v2.mlmodelc")
let models = try await DiarizerModels.load(
localSegmentationModel: segmentation,
localEmbeddingModel: embedding
)
```
## Benchmarks
[VoxConverse](https://www.robots.ox.ac.uk/~vgg/data/voxconverse/) (232 clips, multi-speaker conversations):
| Pipeline | Audio Length | DER | RTFx |
| ---------------------- | ------------ | ----- | ---- |
| Offline (default) | 10s windows | 15.1% | 122x |
| Offline (max accuracy) | 10s windows | 13.9% | 65x |
| Streaming | 5s chunks | 26.2% | 223x |
| Sortformer | 30.4s chunks | 31.7% | 127x |
Device comparison (offline pipeline, default config):
| Device | RTFx |
| -------------- | ---- |
| M2 MacBook Air | 150x |
| M1 iPad Pro | 120x |
| iPhone 14 Pro | 80x |
## CLI
```bash theme={null}
swift run fluidaudio process meeting.wav --output results.json --threshold 0.6
swift run fluidaudio diarization-benchmark --auto-download
```
# Offline Pipeline
Source: https://docs.fluidinference.com/diarization/offline-pipeline
Full VBx batch diarization with pyannote-compatible pipeline.
## Overview
`OfflineDiarizerManager` provides the full pyannote CoreML pipeline (powerset segmentation + VBx clustering) for highest accuracy offline diarization. Based on [pyannote/speaker-diarization-community-1](https://huggingface.co/pyannote/speaker-diarization-community-1).
Requires macOS 14 / iOS 17 or later.
## Quick Start
```swift theme={null}
import FluidAudio
let config = OfflineDiarizerConfig()
let manager = OfflineDiarizerManager(config: config)
try await manager.prepareModels()
let samples = try AudioConverter().resampleAudioFile(path: "meeting.wav")
let result = try await manager.process(audio: samples)
for segment in result.segments {
print("\(segment.speakerId) \(segment.startTimeSeconds)s - \(segment.endTimeSeconds)s")
}
```
### File-Based API
For large files, use memory-mapped streaming:
```swift theme={null}
let url = URL(fileURLWithPath: "meeting.wav")
let result = try await manager.process(url)
```
## Pipeline Stages
1. **Segmentation** — 10s/160k sample chunks through Core ML segmentation (589 frame-level log probabilities)
2. **Binarization** — Log probabilities to soft VAD weights
3. **Weight Interpolation** — `scipy.ndimage.zoom`-compatible half-pixel mapping
4. **Embedding Extraction** — FBANK + embedding backend, L2-normalized 256-d embeddings
5. **VBx Clustering** — AHC warm start + PLDA + iterative VBx refinement
6. **Timeline Reconstruction** — Timestamps with minimum gap/duration constraints
## Configuration
`OfflineDiarizerConfig` groups knobs by pipeline stage:
* `segmentation` — Window length (10s), step ratio, min on/off durations
* `embedding` — Batch size, overlap handling
* `clustering` — VBx warm-start threshold, Fa/Fb priors
* `vbx` — Max iterations, convergence tolerance
* `postProcessing` — Minimum gap duration
* `export` — Optional `embeddingsPath` for JSON dump
## Benchmarks
[VoxConverse](https://www.robots.ox.ac.uk/~vgg/data/voxconverse/) (232 clips, multi-speaker conversations). Segmentation uses 10s windows:
| Config | Audio Length | DER | JER | RTFx |
| ---------------------------------------------- | ------------ | ----- | ----- | ---- |
| Step ratio 0.2, min duration 1.0s (default) | 10s windows | 15.1% | 39.4% | 122x |
| Step ratio 0.1, min duration 0s (max accuracy) | 10s windows | 13.9% | 42.8% | 65x |
Default is \~2x faster for only \~1.2% worse DER. Use step ratio 0.1 for critical accuracy.
Reference: pyannote community-1 on CPU is 1.5-2x RTFx, on MPS is 20-25x RTFx. FluidAudio on ANE is 65-122x RTFx.
## CLI
```bash theme={null}
# Process a single file
swift run fluidaudio process meeting.wav --mode offline --threshold 0.6
# Benchmark on AMI dataset
swift run fluidaudio diarization-benchmark --mode offline \
--dataset ami-sdm --threshold 0.6 --auto-download
# With ground-truth RTTM
swift run fluidaudio process meeting.wav --mode offline \
--rttm ground_truth.rttm
```
# Sortformer
Source: https://docs.fluidinference.com/diarization/sortformer
NVIDIA's end-to-end streaming speaker diarization model.
## Overview
Sortformer is NVIDIA's end-to-end streaming speaker diarization model, converted to CoreML. Unlike the pyannote pipeline (segmentation + clustering), Sortformer is a single neural network with 4 fixed speaker slots.
Model: [FluidInference/diar-streaming-sortformer-coreml](https://huggingface.co/FluidInference/diar-streaming-sortformer-coreml)
## Key Properties
* 4 fixed speaker slots with real-time inference
* No separate segmentation + clustering stages
* Streaming only (no offline mode)
* Best for scenarios with 4 or fewer speakers
## When to Use Sortformer vs Pyannote
Sortformer can beat pyannote on benchmarks with certain configs, but benchmark DER does not always reflect production performance. In practice:
| Scenario | Recommendation | Why |
| ----------------------------------- | -------------------- | ------------------------------------------------------- |
| Noisy / background noise | **Sortformer** | More robust to non-speech audio |
| 4 or fewer speakers | **Sortformer** | Designed for this — single model, no clustering |
| 5+ speakers | **Pyannote offline** | Sortformer only has 4 speaker slots, will miss speakers |
| Overlapping speech (5+ people) | **Pyannote offline** | Sortformer breaks down with heavy crosstalk beyond 4 |
| Best overall accuracy | **Pyannote offline** | 15% DER vs 32% — more consistent in production |
| Streaming required, simple meetings | **Sortformer** | Single model, no clustering overhead |
Benchmarks are not always consistent with production usage. Pyannote's offline pipeline with aggressive tuning can score lower DER on AMI, but those configs may not generalize. Sortformer's 32% DER is more representative of real-world performance on meetings with 4 or fewer speakers.
## Benchmarks
[AMI SDM](https://groups.inf.ed.ac.uk/ami/corpus/) (16 meetings, single distant microphone). Audio length: 30.4s chunks (NVIDIA high-latency config):
| Metric | Value |
| ------------ | ------ |
| Average DER | 31.7% |
| Average Miss | 21.5% |
| Average FA | 0.5% |
| Average SE | 9.7% |
| Average RTFx | 126.7x |
See [full benchmarks](/reference/benchmarks) for per-meeting breakdown.
## CLI
```bash theme={null}
swift run fluidaudio sortformer-benchmark \
--nvidia-high-latency --hf --auto-download
```
# SpeakerManager API
Source: https://docs.fluidinference.com/diarization/speaker-manager
Track and manage speaker identities across audio chunks.
## Overview
`SpeakerManager` maintains an in-memory database of speakers, tracks their voice embeddings, and assigns consistent IDs across audio chunks.
`SpeakerManager` is compatible with `DiarizerManager` (streaming pipeline) only. `OfflineDiarizerManager` uses VBx clustering.
## Configuration
```swift theme={null}
let speakerManager = SpeakerManager(
speakerThreshold: 0.65, // Max cosine distance for speaker match
embeddingThreshold: 0.45, // Max distance for embedding updates
minSpeechDuration: 1.0, // Min seconds to create new speaker
minEmbeddingUpdateDuration: 2.0 // Min seconds to update embeddings
)
```
## Speaker Assignment
```swift theme={null}
let speaker = speakerManager.assignSpeaker(
embedding,
speechDuration: 2.5,
confidence: 0.95
)
```
**Behavior:**
1. Finds closest speaker using cosine distance
2. If distance \< `speakerThreshold`: assigns to existing speaker
3. If no match and duration >= `minSpeechDuration`: creates new speaker
4. Returns `nil` if speech too short
## Known Speakers
```swift theme={null}
let alice = Speaker(id: "alice", name: "Alice", currentEmbedding: aliceEmbedding)
let bob = Speaker(id: "bob", name: "Bob", currentEmbedding: bobEmbedding)
speakerManager.initializeKnownSpeakers([alice, bob])
```
### Initialization Modes
| Mode | Behavior |
| ------------ | --------------------------------------- |
| `.reset` | Clear database, add new speakers |
| `.merge` | Merge with existing speakers by ID |
| `.overwrite` | Replace existing speakers with same IDs |
| `.skip` | Skip if ID already exists |
## Speaker Management
```swift theme={null}
// Upsert
speakerManager.upsertSpeaker(speaker)
// Merge speakers
speakerManager.mergeSpeaker("1", into: "alice", mergedName: "Alice")
// Remove
speakerManager.removeSpeaker("1")
// Remove inactive
speakerManager.removeSpeakersInactive(for: 10.0)
// Permanent speakers
speakerManager.makeSpeakerPermanent("alice")
speakerManager.revokePermanence(from: "alice")
```
## Speaker Lookup
```swift theme={null}
// Find closest match
let (id, distance) = speakerManager.findSpeaker(with: embedding)
// Find all matches
let matches = speakerManager.findMatchingSpeakers(with: embedding)
// Get speaker by ID
if let speaker = speakerManager.getSpeaker(for: "speaker_1") {
print("\(speaker.name): \(speaker.duration)s")
}
// Count and IDs
print("Active: \(speakerManager.speakerCount)")
let ids = speakerManager.speakerIds
```
## Cosine Distance Guide
| Distance | Interpretation |
| -------- | ------------------------------------ |
| \< 0.3 | Same speaker (very high confidence) |
| 0.3-0.5 | Same speaker (high confidence) |
| 0.5-0.7 | Same speaker (medium confidence) |
| 0.7-0.9 | Different speakers |
| > 0.9 | Different speakers (high confidence) |
## Speaker Data Model
```swift theme={null}
public final class Speaker: Identifiable, Codable {
public let id: String
public var name: String
public var currentEmbedding: [Float] // 256-dim L2-normalized
public var duration: Float // Total speech (seconds)
public var createdAt: Date
public var updatedAt: Date
public var updateCount: Int
public var rawEmbeddings: [RawEmbedding] // Max 50 historical
public var isPermanent: Bool
}
```
## Thread Safety
`SpeakerManager` uses internal `DispatchQueue` with concurrent reads and barrier writes. All public methods are thread-safe.
# Streaming Diarization
Source: https://docs.fluidinference.com/diarization/streaming
Real-time speaker diarization for live audio streams.
## Overview
Process audio in chunks for real-time speaker labeling. Use this when you need speaker labels while transcription is happening. For most use cases, the [offline pipeline](/diarization/offline-pipeline) is more accurate.
## Quick Start
```swift theme={null}
let diarizer = DiarizerManager()
diarizer.initialize(models: models)
var stream = AudioStream(
chunkDuration: 5.0,
chunkSkip: 2.0,
streamStartTime: 0.0,
chunkingStrategy: .useMostRecent
)
stream.bind { chunk, time in
let results = try diarizer.performCompleteDiarization(chunk, atTime: time)
for segment in results.segments {
handleSpeakerSegment(segment)
}
}
for audioSamples in audioStream {
try stream.write(from: audioSamples)
}
```
## Chunk Size Considerations
| Chunk Size | Accuracy | Latency |
| ------------ | ---------------------- | ------- |
| \< 3 seconds | May fail or unreliable | Lowest |
| 3-5 seconds | Minimum viable | Low |
| 10 seconds | Optimal (recommended) | Medium |
| > 10 seconds | Good | Higher |
## Real-time Audio Capture
```swift theme={null}
class RealTimeDiarizer {
private let audioEngine = AVAudioEngine()
private let diarizer: DiarizerManager
private var audioStream: AudioStream
init() async throws {
let models = try await DiarizerModels.downloadIfNeeded()
diarizer = DiarizerManager()
diarizer.initialize(models: models)
audioStream = AudioStream(
chunkDuration: 5.0,
chunkSkip: 3.0,
streamStartTime: 0.0,
chunkingStrategy: .useFixedSkip
)
audioStream.bind { [weak self] chunk, _ in
Task {
let result = try self?.diarizer.performCompleteDiarization(chunk)
// Handle results
}
}
}
func startCapture() throws {
let inputNode = audioEngine.inputNode
let format = inputNode.outputFormat(forBus: 0)
inputNode.installTap(onBus: 0, bufferSize: 1024, format: format) {
[weak self] buffer, _ in
try? self?.audioStream.write(from: buffer)
}
audioEngine.prepare()
try audioEngine.start()
}
}
```
## Benchmarks
[AMI SDM](https://groups.inf.ed.ac.uk/ami/corpus/) (meeting recordings, single distant microphone):
| Audio Length | Overlap | Threshold | DER | RTFx | Best For |
| ------------ | ------- | --------- | ----- | ---- | --------------------------- |
| 5s chunks | 0s | 0.8 | 26.2% | 223x | Best accuracy/speed balance |
| 10s chunks | 0s | 0.7 | 33.3% | 392x | Higher throughput |
| 3s chunks | 1s | 0.85 | 49.7% | 51x | Lowest latency |
| 5s chunks | 2s | 0.8 | 43.0% | 69x | — |
Streaming diarization is 10-15% worse DER than offline. Only use streaming when you critically need real-time speaker labels. For most apps, offline is more than fast enough.
## Tips
* Keep one `DiarizerManager` per stream for consistent speaker IDs
* Always rebase per-chunk timestamps by `(chunkStartSample / sampleRate)`
* Provide 16 kHz mono Float32 samples
* Tune `speakerThreshold` and `embeddingThreshold` to trade off ID stability vs. sensitivity
# Audio Conversion
Source: https://docs.fluidinference.com/guides/audio-conversion
Convert any audio format to 16 kHz mono Float32 for FluidAudio pipelines.
## Overview
Most FluidAudio features expect 16 kHz mono Float32 samples. `AudioConverter` uses `AVAudioConverter` under the hood for sample-rate conversion, format conversion (e.g., Int16 → Float32), and channel mixing (stereo → mono).
## File Conversion
```swift theme={null}
import FluidAudio
let converter = AudioConverter()
let samples = try converter.resampleAudioFile(path: "path/to/audio.wav")
// samples: [Float] at 16 kHz mono
```
Supported inputs: WAV, M4A, MP3, FLAC — anything readable by `AVAudioFile`.
## Streaming Conversion
```swift theme={null}
let converter = AudioConverter()
func processChunk(_ pcmBuffer: AVAudioPCMBuffer) async throws {
let samples = try converter.resampleBuffer(pcmBuffer)
// Feed samples to ASR/VAD/diarization
}
```
Each conversion is stateless — reuse the same converter instance across formats.
# Manual Model Loading
Source: https://docs.fluidinference.com/guides/manual-model-loading
Deploy models offline without HuggingFace downloads.
## Overview
FluidAudio auto-downloads models from HuggingFace on first use. For offline or air-gapped environments, stage the CoreML bundles manually.
## Per-Module Guides
Each module has specific assets and loading APIs:
* **ASR**: [Manual Model Loading](/asr/manual-model-loading) — 4 CoreML bundles + vocabulary
* **Diarization**: [Getting Started](/diarization/getting-started#manual-model-loading) — segmentation + embedding models
* **VAD**: [Getting Started](/vad/getting-started#manual-model-loading) — single Silero VAD bundle
## General Pattern
1. **Download assets** via Git LFS, HuggingFace web UI, or copy from a machine that already ran the auto-downloader
2. **Stage in a directory** matching the expected layout
3. **Call the `load` API** with the staged directory URL
```swift theme={null}
// Example: ASR
let models = try await AsrModels.load(
from: URL(fileURLWithPath: "/opt/models/parakeet-tdt-0.6b-v3-coreml"),
version: .v3
)
// Example: Diarization
let models = try await DiarizerModels.load(
localSegmentationModel: segmentationURL,
localEmbeddingModel: embeddingURL
)
// Example: VAD
let vadModel = try MLModel(contentsOf: modelURL, configuration: config)
let manager = VadManager(config: .default, vadModel: vadModel)
```
## Cache Locations
Models are cached at `~/Library/Application Support/FluidAudio/Models/` on macOS after first download.
# Installation
Source: https://docs.fluidinference.com/installation
Add FluidAudio to your Swift project.
## Swift Package Manager
```swift theme={null}
dependencies: [
.package(url: "https://github.com/FluidInference/FluidAudio.git", from: "0.7.9"),
],
```
### Choosing a Product
FluidAudio provides two library products:
* **`FluidAudio`** (default) — Core functionality: ASR, diarization, VAD. Lightweight, no GPL dependencies.
* **`FluidAudioTTS`** — Text-to-Speech (Kokoro). Includes ESpeakNG framework (GPL-3.0). Only bundled if you explicitly add it.
**In Package.swift:**
```swift theme={null}
// Core features only (no GPL dependencies):
.product(name: "FluidAudio", package: "FluidAudio")
// Add TTS support (includes GPL ESpeakNG):
.product(name: "FluidAudioTTS", package: "FluidAudio")
```
**In Xcode:**
1. File > Add Package Dependencies
2. Enter the FluidAudio repository URL
3. Select your desired product (`FluidAudio` or `FluidAudioTTS`)
4. Add to your app target
## CocoaPods
We recommend using [cocoapods-spm](https://github.com/trinhngocthuyen/cocoapods-spm) for better SPM integration, but you can also use the podspec:
```ruby theme={null}
pod 'FluidAudio', '~> 0.7.8'
```
## Other Frameworks
| Platform | Package | Install |
| ----------------------- | ---------------------------------------------------------------------------------------------------- | ----------------------------------------------------- |
| **React Native / Expo** | [@fluidinference/react-native-fluidaudio](https://github.com/FluidInference/react-native-fluidaudio) | `npm install @fluidinference/react-native-fluidaudio` |
| **Rust / Tauri** | [fluidaudio-rs](https://github.com/FluidInference/fluidaudio-rs) | `cargo add fluidaudio-rs` |
The Kokoro TTS tooling currently ships arm64-only dependencies. See the [TTS docs](/tts/kokoro) if you hit linker errors targeting x86\_64.
# Introduction
Source: https://docs.fluidinference.com/introduction
Local audio AI for Apple devices — speech-to-text, speaker diarization, voice activity detection, and text-to-speech on the Neural Engine.
FluidAudio is a Swift SDK for fully local, low-latency audio AI on Apple devices. All inference runs on the Apple Neural Engine (ANE), keeping CPU and GPU free for your app.
## At a Glance
| Capability | Model | Speed | Accuracy | Languages |
| ------------------------- | ----------------- | ---------- | ---------------------------------- | ----------------- |
| **Transcription** | Parakeet TDT 0.6B | 210x RTFx | 2.5% WER (en), 14.7% avg (25 lang) | 25 European |
| **Streaming ASR** | Parakeet EOU 120M | 12x RTFx | 4.9% WER (en) | English |
| **Speaker Diarization** | Pyannote CoreML | 122x RTFx | 15% DER (offline) | Language-agnostic |
| **Streaming Diarization** | Sortformer | 127x RTFx | 31.7% DER | Language-agnostic |
| **Voice Activity** | Silero VAD v6 | 1230x RTFx | 96% accuracy | Language-agnostic |
| **Text-to-Speech** | Kokoro 82M | 23x RTFx | 48 voices | English |
| **Text-to-Speech** | PocketTTS 155M | Streaming | \~80ms first audio | English |
All benchmarks on M4 Pro. ASR on [LibriSpeech](https://huggingface.co/datasets/openslr/librispeech_asr) / [FLEURS](https://huggingface.co/datasets/google/fleurs), diarization on [VoxConverse](https://www.robots.ox.ac.uk/~vgg/data/voxconverse/) / [AMI](https://groups.inf.ed.ac.uk/ami/corpus/), VAD on [VOiCES](https://iqtlabs.github.io/voices/) / [MUSAN](https://www.openslr.org/17/). See [full benchmarks](/reference/benchmarks) for per-language breakdowns and device comparisons.
## When to Use Which
### Transcription
| Need | Use | Why |
| ------------------------------------- | --------------------------------- | ----------------------------------------------- |
| Transcribe recordings/files | **Parakeet TDT v3** | Fastest, 25 languages, 210x real-time |
| English-only, best accuracy | **Parakeet TDT v2** | 2.1% WER vs 2.5% on LibriSpeech |
| Live captions as user speaks | **Parakeet EOU** | 160ms chunks, end-of-utterance detection |
| Domain-specific terms (names, jargon) | **TDT + CTC vocabulary boosting** | 99.3% precision, 85.2% recall on earnings calls |
### Speaker Diarization
| Need | Use | Why |
| ------------------------------ | -------------------------- | ---------------------------------------------------- |
| Best accuracy (post-recording) | **Offline pipeline** (VBx) | 15% DER, full pyannote-compatible pipeline |
| Real-time "who's speaking now" | **Streaming pipeline** | 26% DER at 5s chunks, speaker tracking across chunks |
| Simple 2-4 speaker meetings | **Sortformer** | Single model, no clustering, 32% DER |
### Voice Activity Detection
| Need | Use | Why |
| -------------------------- | ------------------------ | --------------------------------------------- |
| Segment audio before ASR | **Offline segmentation** | Clean segments with min/max duration controls |
| Real-time speech detection | **Streaming VAD** | Per-chunk events with hysteresis |
### Text-to-Speech
| Need | Use | Why |
| ------------------------------------ | ------------- | ------------------------------------------- |
| Highest quality, full generation | **Kokoro** | 48 voices, SSML support, flow matching |
| Streaming audio (start playing fast) | **PocketTTS** | \~80ms to first audio, no espeak dependency |
## Platform Support
| Platform | Package |
| ----------------------- | ---------------------------------------------------------------------------------------------------- |
| **Swift (iOS / macOS)** | [FluidAudio](https://github.com/FluidInference/FluidAudio) |
| **React Native / Expo** | [@fluidinference/react-native-fluidaudio](https://github.com/FluidInference/react-native-fluidaudio) |
| **Rust / Tauri** | [fluidaudio-rs](https://github.com/FluidInference/fluidaudio-rs) |
## Showcase
40+ apps use FluidAudio for local speech recognition, speaker diarization, and text-to-speech.
| App | Description |
| ------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **[Voice Ink](https://tryvoiceink.com/)** | Local AI for instant, private transcription with near-perfect accuracy. Uses Parakeet ASR. |
| **[Spokenly](https://spokenly.app/)** | Mac dictation app for fast, accurate voice-to-text; supports real-time dictation and file transcription. Uses Parakeet ASR and speaker diarization. |
| **[Slipbox](https://slipbox.ai/)** | Privacy-first meeting assistant for real-time conversation intelligence. Uses Parakeet ASR (iOS) and speaker diarization across platforms. |
| **[Talat](https://talat.app)** | Privacy-focused AI meeting notes app. Featured in [TechCrunch](https://techcrunch.com/2026/03/24/talats-ai-meeting-notes-stay-on-your-machine-not-in-the-cloud/). Uses Parakeet ASR. |
| **[Paraspeech](https://paraspeech.com)** | AI powered voice to text. Fully offline. No subscriptions. |
| **[OpenOats](https://github.com/yazinsai/OpenOats)** | Open-source meeting note-taker that transcribes conversations in real time and surfaces relevant notes from your knowledge base. |
| **[Senko](https://github.com/narcotic-sh/senko)** | A very fast and accurate speaker diarization pipeline. A [good example](https://github.com/narcotic-sh/senko/commit/51dbd8bde764c3c6648dbbae57d6aff66c5ca15c) for Python integration. |
| **[macos-speech-server](https://github.com/dokterbob/macos-speech-server)** | OpenAI compatible STT/transcription and TTS/speech API server. |
| **[Whisper Mate](https://whisper.marksdo.com)** | Transcribes movies and audio locally; records and transcribes in real time from speakers or system apps. Uses speaker diarization. |
| **[BoltAI](https://boltai.com/)** | Write content 10x faster using parakeet models. |
| **[Voxeoflow](https://www.voxeoflow.app)** | Mac dictation app with real-time translation. Lightning-fast transcription in over 100 languages. |
| **[WhisKey](https://whiskey.asktobuild.app/)** | Privacy-first voice dictation keyboard for iOS and macOS. On-device transcription with 12+ languages, AI meeting summaries, and mindmap generation. |
| **[Summit AI Notes](https://summitnotes.app/)** | Local meeting transcription and summarization with speaker identification. Supports 100+ languages. |
| **[Snaply](https://snaply.ai)** | Free, Fast, 100% local AI dictation for Mac. |
| **[Enconvo](https://enconvo.com)** | AI Agent Launcher for macOS with voice input, live captions, and text-to-speech. |
| **[Speakmac](https://speakmac.app)** | Mac app that lets you type anywhere on your Mac using your voice. Fully local, private dictation built on FluidAudio. |
| **[Starling](https://github.com/Ryandonofrio3/Starling)** | Open Source, fully local voice-to-text transcription with auto-paste at your cursor. |
| **[Altic/Fluid Voice](https://github.com/altic-dev/Fluid-oss)** | Lightweight, fully free and Open Source Voice to Text dictation for macOS. |
| **[SamScribe](https://github.com/Steven-Weng/SamScribe)** | Open-source macOS app that captures and transcribes audio from your microphone and meeting apps in real-time. |
| **[Dictate Anywhere](https://github.com/hoomanaskari/mac-dictate-anywhere)** | Native macOS dictation app with global Fn key activation. Dictate into any app with 25 language support. |
| **[Hex](https://github.com/kitlangton/Hex)** | macOS app that lets you press-and-hold a hotkey to record your voice, transcribe it, and paste into any application. |
| **[Super Voice Assistant](https://github.com/ykdojo/super-voice-assistant)** | Open-source macOS voice assistant with local transcription. |
| **[VoiceTypr](https://github.com/moinulmoin/voicetypr)** | Open-source voice-to-text dictation for macOS and Windows. |
| **[Ora](https://futurelab.studio/ora)** | Local voice assistant for macOS with speech recognition and text-to-speech. |
| **[Flowstay](https://flowstay.app)** | Easy text-to-speech, local post-processing and Claude Code integration for macOS. Free forever. |
| **[Meeting Transcriber](https://github.com/pasrom/meeting-transcriber)** | macOS menu bar app that auto-detects, records, and transcribes meetings with dual-track speaker diarization. |
| **[Hitoku Draft](https://hitoku.me/draft)** | A local, private, voice writing assistant on your macOS menu bar. |
| **[Audite](https://github.com/zachatrocity/audite)** | macOS menu-bar app that records meetings and transcribes them locally into Markdown notes for Obsidian. |
| **[Muesli](https://github.com/pHequals7/muesli)** | Native macOS dictation and meeting transcription with \~0.13s latency. Automatic speaker diarization. |
| **[NanoVoice](https://apps.apple.com/kz/app/nanovoice/id6760539688)** | Free iOS voice keyboard for fast, private dictation in any app. |
| **[MiniWhisper](https://github.com/andyhtran/MiniWhisper)** | Open-source macOS menu bar for quick local voice-to-text with minimal setup. |
| **[Volocal](https://github.com/fikrikarim/volocal)** | Fully local voice AI on iOS. Uses streaming Parakeet EOU ASR and streaming PocketTTS. |
| **[VivaDicta](https://github.com/n0an/VivaDicta)** | Open-source iOS voice-to-text app with system-wide AI voice keyboard. 15+ AI providers, 40+ AI presets. |
| **[hongbomiao.com](https://github.com/hongbo-miao/hongbomiao.com)** | A personal R\&D lab that facilitates knowledge sharing. |
| **[mac-whisper-speedtest](https://github.com/anvanvan/mac-whisper-speedtest)** | Comparison of different local ASR, including one of the first versions of FluidAudio's ASR models. |
## Requirements
* macOS 14+ / iOS 17+
* Swift 5.10+
* Apple Silicon recommended
## Model Conversion
All FluidAudio models are converted through [möbius](https://github.com/FluidInference/mobius), our open-source model conversion framework. It handles export, numerical validation, and quantization for CoreML and other edge runtimes. See the [möbius docs](/mobius/getting-started) to convert your own models.
# Converting Models
Source: https://docs.fluidinference.com/mobius/converting-models
How to convert PyTorch models to CoreML using möbius — export, validate, quantize.
Each model conversion in möbius follows a three-step workflow: **export**, **validate**, **quantize**. The scripts and dependencies are self-contained per model directory.
## Workflow
```
PyTorch model
│
▼
convert-*.py Export to .mlpackage (CoreML)
│
▼
compare-*.py Validate numerical parity + measure latency
│
▼
quantize_coreml.py Sweep quantization variants (optional)
│
▼
.mlmodelc / .mlpackage Ready for FluidAudio or direct CoreML usage
```
## Step 1: Export
Each model directory contains a conversion script (e.g., `convert-parakeet.py`, `convert-coreml.py`). The script:
1. Loads the original PyTorch / NeMo / ONNX model
2. Traces or scripts the model with fixed input shapes
3. Converts to CoreML using `coremltools`
4. Saves `.mlpackage` files
```bash theme={null}
cd models/stt/parakeet-tdt-v3-0.6b/coreml
uv sync
uv run python convert-parakeet.py convert \
--nemo-path /path/to/model.nemo \
--output-dir ./build
```
### Fixed Input Shapes
CoreML requires static shapes at export time. Each model defines its input contract:
| Model | Input Shape | Duration |
| -------------------- | --------------- | -------------- |
| Parakeet TDT v3 | 240,000 samples | 15s at 16kHz |
| Parakeet EOU | 5,120 samples | 320ms at 16kHz |
| Silero VAD | 576 samples | 36ms at 16kHz |
| Silero VAD (256ms) | 4,160 samples | 256ms at 16kHz |
| Kokoro (5s variant) | Variable tokens | \~5s output |
| Kokoro (15s variant) | Variable tokens | \~15s output |
## Step 2: Validate
Parity scripts run the PyTorch and CoreML models side-by-side on identical inputs, comparing outputs numerically and measuring latency.
```bash theme={null}
uv run python compare-components.py compare \
--output-dir ./build \
--model-id nvidia/parakeet-tdt-0.6b-v3 \
--runs 10 --warmup 3
```
This produces:
* **Numerical diff** — max absolute error, max relative error, match/no-match per component
* **Latency comparison** — Torch CPU vs CoreML (CPU+ANE) with speedup ratios
* **Plots** — visual comparisons saved to `plots/` directory
* **metadata.json** — structured results for CI and reporting
### Example Parity Results (Parakeet TDT v3)
| Component | Max Abs Error | Match | Torch CPU | CoreML ANE | Speedup |
| ------------ | ------------- | ----- | --------- | ---------- | ------- |
| Encoder | 0.005 | Yes | 1030ms | 25ms | 40x |
| Preprocessor | 0.484 | Yes | 2.0ms | 1.2ms | 1.7x |
| Decoder | tolerance | Yes | 7.5ms | 4.3ms | 1.7x |
| Joint | 0.099 | Yes | 28ms | 23ms | 1.3x |
## Step 3: Quantize (Optional)
Quantization reduces model size and can improve latency on ANE. The sweep evaluates multiple strategies and reports the trade-offs.
```bash theme={null}
uv run python quantize_coreml.py \
--input-dir ./build \
--output-root ./build_quantized \
--compute-units ALL --runs 10
```
### Quantization Strategies
| Strategy | Size Reduction | Quality Impact | Best For |
| ------------------- | -------------- | -------------------------------- | ------------------------ |
| INT8 per-channel | \~2x smaller | Minimal loss | General deployment |
| INT8 per-tensor | \~2x smaller | Significant loss on large models | Small models only |
| 6-bit palettization | \~2.5x smaller | Varies by model | Size-constrained devices |
Results are saved to `quantization_summary.json` with per-component quality scores (1.0 = identical to baseline).
## Common CoreML Modifications
PyTorch models often need modifications for CoreML tracing. Common patterns:
| PyTorch Feature | CoreML Fix |
| ---------------------- | -------------------------------------- |
| `pack_padded_sequence` | Explicit LSTM states + masking |
| Dynamic shapes / loops | Fixed shapes, broadcasting |
| In-place operations | Pure functional transforms |
| Random generation | Deterministic inputs passed externally |
| Complex number ops | Real/imaginary split |
## Adding a New Model
1. Create the directory: `models/{class}/{name}/coreml/`
2. Add `pyproject.toml` with dependencies
3. Write `convert-*.py` — export script
4. Write `compare-*.py` — validation script (optional but recommended)
5. Add `README.md` documenting the conversion
6. Push converted weights to [Hugging Face](https://huggingface.co/FluidInference)
```bash theme={null}
mkdir -p models/stt/my-new-model/coreml
cd models/stt/my-new-model/coreml
# Initialize with uv
uv init
uv add coremltools torch
# Write your conversion script
# ... convert-my-model.py
```
## Deployment Targets
* **Minimum**: iOS 17 / macOS 14
* **Format**: MLProgram (`.mlpackage` for development, `.mlmodelc` for compiled)
* **Compute units**: Models traced with `CPU_ONLY` for determinism; runtime compute units set when loading (`.cpuAndNeuralEngine`, `.cpuAndGPU`, `.all`)
# Getting Started
Source: https://docs.fluidinference.com/mobius/getting-started
möbius — convert AI models for edge deployment on Apple Silicon, NPUs, and other accelerators.
[möbius](https://github.com/FluidInference/mobius) is a model conversion framework for running AI on edge devices. It converts models from PyTorch/ONNX to CoreML, ONNX Runtime, and OpenVINO — targeting Apple Neural Engine, NPUs, and embedded accelerators.
Every model that ships in FluidAudio was converted through möbius.
## Why möbius
Running AI on NVIDIA GPUs is straightforward. The edge is a different story — fragmented devices, different accelerators, format incompatibilities. möbius handles the conversion, validation, and quantization so you get production-ready models with a few commands.
Each conversion includes:
* **Parity validation** — numerical comparison between PyTorch and converted outputs
* **Latency benchmarks** — Torch CPU vs CoreML (ANE/GPU) on real inputs
* **Quantization sweeps** — size, speed, and quality trade-offs for int8, palettization, etc.
## Repository Structure
Models are organized by class, name, and target runtime. Each target directory is self-contained with its own `pyproject.toml` and dependencies managed by [uv](https://github.com/astral-sh/uv).
```
models/
├── emb/
│ └── cam++/coreml # Speaker embedding
├── segment-text/
│ └── coreml # Text segmentation
├── speaker-diarization/
│ ├── pyannote-community-1/coreml # Pyannote diarization
│ └── sortformer-streaming/ # Sortformer diarization
├── stt/
│ ├── canary-1b-v2/coreml # Canary ASR
│ ├── nemotron-speech-streaming-0.6b/coreml
│ ├── parakeet-realtime-eou-120m/coreml
│ ├── parakeet-tdt-ctc-110m/coreml
│ ├── parakeet-tdt-v2-0.6b/coreml
│ ├── parakeet-tdt-v3-0.6b/coreml # Current FluidAudio default
│ └── qwen3-asr-0.6b/coreml
├── tts/
│ ├── kokoro/coreml # Kokoro TTS
│ └── pocket_tts/coreml # PocketTTS
└── vad/
└── silero-vad/coreml # Silero VAD
```
## Converted Models
These models have been converted and published to [Hugging Face](https://huggingface.co/FluidInference):
| Class | Model | Source | CoreML |
| --------------- | -------------------- | --------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- |
| **STT** | Parakeet TDT v3 0.6B | [NVIDIA](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3) | [FluidInference](https://huggingface.co/FluidInference/parakeet-tdt-0.6b-v3-coreml) |
| **STT** | Parakeet TDT v2 0.6B | [NVIDIA](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v2) | [FluidInference](https://huggingface.co/FluidInference/parakeet-tdt-0.6b-v2-coreml) |
| **STT** | Parakeet EOU 120M | [NVIDIA](https://huggingface.co/nvidia/parakeet-tdt_ctc-110m) | [FluidInference](https://huggingface.co/FluidInference/parakeet-eou-120m-coreml) |
| **VAD** | Silero VAD v6 | [Silero](https://github.com/snakers4/silero-vad) | [FluidInference](https://huggingface.co/FluidInference/silero-vad-coreml) |
| **Diarization** | Pyannote Community 1 | [Pyannote](https://huggingface.co/pyannote/speaker-diarization-community-1) | [FluidInference](https://huggingface.co/FluidInference/speaker-diarization-coreml) |
| **TTS** | Kokoro 82M | [Hexgrad](https://huggingface.co/hexgrad/Kokoro-82M) | [FluidInference](https://huggingface.co/FluidInference/kokoro-82m-coreml) |
| **TTS** | PocketTTS 155M | [Kyutai](https://huggingface.co/kyutai/pocket-tts) | [FluidInference](https://huggingface.co/FluidInference/pocket-tts-coreml) |
| **Embedding** | CAM++ | [3D-Speaker](https://github.com/alibaba-damo-academy/3D-Speaker) | [FluidInference](https://huggingface.co/FluidInference/cam-plusplus-coreml) |
## Quick Start
```bash theme={null}
# Clone
git clone https://github.com/FluidInference/mobius.git
cd mobius
# Pick a model
cd models/stt/parakeet-tdt-v3-0.6b/coreml
# Set up environment
uv sync
# Convert
uv run python convert-parakeet.py convert \
--nemo-path /path/to/parakeet-tdt-0.6b-v3.nemo \
--output-dir parakeet_coreml
# Validate parity
uv run python compare-components.py compare \
--output-dir parakeet_coreml \
--runs 10 --warmup 3
```
Each model directory has its own README with specific conversion steps.
## Conversion Guidelines
* **Trace with `.CpuOnly`** — ensures deterministic tracing without ANE/GPU side effects
* **Target iOS 17+ / macOS 14+** — minimum deployment target for all CoreML exports
* **Use `uv`** — each model has isolated dependencies via its own `pyproject.toml`
* **Validate numerically** — always compare converted outputs against PyTorch reference
## License
Apache 2.0. See individual model directories for upstream model licenses.
# Quickstart
Source: https://docs.fluidinference.com/quickstart
Get up and running with FluidAudio in minutes.
## Install the Package
Add FluidAudio to your project using Swift Package Manager:
```swift theme={null}
dependencies: [
.package(url: "https://github.com/FluidInference/FluidAudio.git", from: "0.7.9"),
],
```
## Transcribe Audio
```swift theme={null}
import FluidAudio
Task {
let models = try await AsrModels.downloadAndLoad(version: .v3)
let asrManager = AsrManager(config: .default)
try await asrManager.initialize(models: models)
let audioURL = URL(fileURLWithPath: "/path/to/audio.wav")
let result = try await asrManager.transcribe(audioURL, source: .system)
print("Transcription: \(result.text)")
}
```
## Diarize Speakers
```swift theme={null}
import FluidAudio
Task {
let models = try await DiarizerModels.downloadIfNeeded()
let diarizer = DiarizerManager()
diarizer.initialize(models: models)
let samples = try AudioConverter().resampleAudioFile(
URL(fileURLWithPath: "meeting.wav")
)
let result = try diarizer.performCompleteDiarization(samples)
for segment in result.segments {
print("Speaker \(segment.speakerId): \(segment.startTimeSeconds)s - \(segment.endTimeSeconds)s")
}
}
```
## Detect Voice Activity
```swift theme={null}
import FluidAudio
Task {
let manager = try await VadManager(
config: VadConfig(defaultThreshold: 0.75)
)
let samples = try AudioConverter().resampleAudioFile(
URL(fileURLWithPath: "audio.wav")
)
var segmentation = VadSegmentationConfig.default
segmentation.minSpeechDuration = 0.25
let segments = try await manager.segmentSpeech(samples, config: segmentation)
for segment in segments {
print(String(format: "Speech %.2f–%.2fs", segment.startTime, segment.endTime))
}
}
```
## Synthesize Speech
```swift theme={null}
import FluidAudioTTS
Task {
let manager = TtSManager()
try await manager.initialize()
let audio = try await manager.synthesize(text: "Hello from FluidAudio!")
try audio.write(to: URL(fileURLWithPath: "/tmp/demo.wav"))
}
```
## CLI
```bash theme={null}
# Transcribe
swift run fluidaudio transcribe audio.wav
# Diarize
swift run fluidaudio process meeting.wav --mode offline --threshold 0.6
# TTS
swift run fluidaudio tts "Hello from FluidAudio" --output out.wav
```
# API Reference
Source: https://docs.fluidinference.com/reference/api
Complete API reference for FluidAudio.
## Common Patterns
### Audio Format
All pipelines expect **16 kHz mono Float32** samples. Use `AudioConverter` to normalize input:
```swift theme={null}
let converter = AudioConverter()
let samples = try converter.resampleAudioFile(path: "audio.wav")
```
### Model Registry
Override the default HuggingFace URL:
```swift theme={null}
ModelRegistry.baseURL = "https://your-mirror.example.com"
```
Or via environment: `REGISTRY_URL=...`
## Diarization
### DiarizerManager
| Method | Description |
| ------------------------------------------- | --------------------------------- |
| `initialize(models:)` | Initialize with Core ML models |
| `performCompleteDiarization(_:sampleRate:)` | Process audio and return segments |
| `cleanup()` | Release resources |
### OfflineDiarizerManager
| Method | Description |
| ----------------- | ------------------------------------- |
| `prepareModels()` | Download + compile Core ML bundles |
| `process(audio:)` | Process Float32 samples |
| `process(_:)` | Process from file URL (memory-mapped) |
## Voice Activity Detection
### VadManager
| Method | Description |
| ---------------------------------------- | ---------------------------------------- |
| `process(_:)` | Chunk-level probabilities for full audio |
| `segmentSpeech(_:config:)` | Speech segments with timestamps |
| `segmentSpeechAudio(_:config:)` | Speech segments with audio buffers |
| `processStreamingChunk(_:state:config:)` | Single-chunk streaming |
| `makeStreamState()` | Fresh streaming state |
## ASR
### AsrManager
| Method | Description |
| ----------------------- | -------------------------- |
| `initialize(models:)` | Load ASR models |
| `transcribe(_:source:)` | Transcribe Float32 samples |
| `transcribe(_:source:)` | Transcribe from file URL |
### AsrModels
| Method | Description |
| --------------------------- | ---------------------------- |
| `downloadAndLoad(version:)` | Download and compile models |
| `load(from:version:)` | Load from staged directory |
| `modelsExist(at:)` | Check if bundles are present |
## TTS
### TtSManager (Kokoro)
| Method | Description |
| --------------------------- | ------------------------------- |
| `initialize()` | Download and load Kokoro models |
| `synthesize(text:voice:)` | Generate audio Data |
| `synthesizeDetailed(text:)` | Generate with chunk metadata |
### PocketTtsManager
| Method | Description |
| ----------------------------------- | ---------------------------------- |
| `initialize()` | Download and load PocketTTS models |
| `synthesize(text:)` | Generate audio Data |
| `synthesizeToFile(text:outputURL:)` | Generate directly to file |
# Benchmarks
Source: https://docs.fluidinference.com/reference/benchmarks
Performance benchmarks across all FluidAudio capabilities on Apple Silicon.
Hardware: 2024 MacBook Pro, M4 Pro, 48GB RAM, macOS Tahoe 26.0 (unless noted).
## Transcription (Parakeet TDT v3)
25 European languages on [FLEURS](https://huggingface.co/datasets/google/fleurs):
| Language | WER% | CER% | RTFx | Files |
| ------------ | -------- | ------- | --------- | ---------- |
| Italian | 4.0 | 1.3 | 236.7 | 350 |
| Spanish | 4.5 | 2.2 | 221.7 | 350 |
| English (US) | 5.4 | 2.5 | 207.4 | 350 |
| French | 5.9 | 2.2 | 199.9 | 350 |
| German | 5.9 | 1.9 | 220.9 | 350 |
| Russian | 7.2 | 2.2 | 209.7 | 350 |
| Ukrainian | 7.2 | 2.5 | 201.9 | 350 |
| Dutch | 7.8 | 2.6 | 191.7 | 350 |
| Polish | 8.6 | 2.8 | 190.2 | 350 |
| Czech | 12.0 | 3.8 | 214.2 | 350 |
| Slovak | 12.6 | 4.4 | 227.6 | 350 |
| Bulgarian | 12.8 | 4.1 | 195.2 | 350 |
| Croatian | 14.0 | 4.3 | 204.9 | 350 |
| Romanian | 14.4 | 4.7 | 200.4 | 883 |
| Finnish | 14.8 | 3.1 | 222.0 | 918 |
| Swedish | 16.8 | 5.0 | 219.5 | 759 |
| Hungarian | 17.6 | 5.2 | 213.6 | 905 |
| Danish | 20.2 | 7.4 | 214.4 | 930 |
| Estonian | 20.1 | 4.2 | 225.3 | 893 |
| Maltese | 25.2 | 9.3 | 217.4 | 926 |
| Lithuanian | 25.0 | 6.8 | 202.8 | 986 |
| Latvian | 27.1 | 7.5 | 217.8 | 851 |
| Slovenian | 27.4 | 9.2 | 197.1 | 834 |
| Greek | 36.9 | 13.7 | 183.0 | 650 |
| **Average** | **14.7** | **4.7** | **209.8** | **14,085** |
### LibriSpeech (English)
| Model | Dataset | WER% | CER% | RTFx | Files |
| ------ | ---------- | ---- | ---- | ------ | ----- |
| TDT v3 | test-clean | 2.5% | 1.0% | 155.6x | 2,620 |
| TDT v2 | test-clean | 2.1% | 0.7% | 145.8x | 2,620 |
v2 has lower English WER — use it if you only need English.
### Model Compilation Times
First-load CoreML compile times (ANE compilation):
| Model | iPhone 16 Pro Max (cold) | iPhone 16 Pro Max (warm) | iPhone 13 (cold) |
| ------------- | -----------------------: | -----------------------: | ---------------: |
| Preprocessor | 9ms | — | 633ms |
| Encoder | 3,361ms | 162ms | 4,396ms |
| Decoder | 88ms | 8ms | 146ms |
| JointDecision | 48ms | 8ms | 72ms |
Cold start = first load after install. Warm = subsequent loads from ANE cache.
## Custom Vocabulary Boosting
Earnings22 benchmark (771 files, earnings call transcripts with domain-specific terms):
| Metric | Value |
| ------------------ | ----------------------- |
| Average WER | 15.0% |
| Vocab Precision | 99.3% (TP=1068, FP=8) |
| Vocab Recall | 85.2% (TP=1068, FN=185) |
| Vocab F-score | 91.7% |
| Dict Pass (Recall) | 99.3% (1299/1308) |
| RTFx | 63.4x |
| Total audio | 11,565s |
## Streaming ASR (Parakeet EOU)
Hardware: Apple M2, 2022, macOS 26.
LibriSpeech test-clean (2,620 files, 5.4h audio):
| Chunk Size | WER (Avg) | RTFx | Total Time |
| ---------- | --------- | ------ | ---------- |
| 320ms | 4.87% | 12.48x | 26min |
| 160ms | 8.29% | 4.78x | 68min |
320ms is the recommended default — best accuracy/latency tradeoff.
## Voice Activity Detection (Silero VAD v6)
### VOiCES Dataset (25 files, clean speech)
| Metric | Value |
| --------- | -------- |
| Accuracy | 96.0% |
| Precision | 100.0% |
| Recall | 95.8% |
| F1-Score | 97.9% |
| RTFx | 1,230.6x |
### MUSAN Full (2,016 files, mixed noise/music/speech)
| Metric | Value |
| --------- | -------- |
| Accuracy | 94.2% |
| Precision | 92.6% |
| Recall | 78.9% |
| F1-Score | 85.2% |
| RTFx | 1,220.7x |
## Speaker Diarization
### Offline Pipeline (VBx)
VoxConverse dataset (232 clips):
| Config | DER% | JER% | RTFx |
| ---------------------------------------------- | ----- | ----- | ---- |
| Step ratio 0.2, min duration 1.0s (default) | 15.1% | 39.4% | 122x |
| Step ratio 0.1, min duration 0s (max accuracy) | 13.9% | 42.8% | 65x |
The default is \~2x faster for only \~1.2% worse DER. Use step ratio 0.1 for critical accuracy.
Reference: pyannote community-1 on CPU is 1.5-2x RTFx, on MPS is 20-25x RTFx. FluidAudio on ANE is 65-122x RTFx.
### Streaming Pipeline (AMI SDM)
| Chunk | Overlap | Threshold | DER% | RTFx | Best For |
| ----- | ------- | --------- | ----- | ---- | --------------------------- |
| 5s | 0s | 0.8 | 26.2% | 223x | Best accuracy/speed balance |
| 10s | 0s | 0.7 | 33.3% | 392x | Higher throughput |
| 3s | 1s | 0.85 | 49.7% | 51x | Lowest latency |
| 5s | 2s | 0.8 | 43.0% | 69x | — |
5s chunks with 0.8 threshold is the recommended starting point for streaming.
Streaming diarization is 10-15% worse DER than offline. Only use streaming when you critically need real-time speaker labels. For most apps, offline is more than fast enough.
### Sortformer (End-to-End Streaming)
Hardware: Apple M2, 2022, macOS 26.1.
AMI SDM dataset, NVIDIA high-latency config (30.4s chunks):
| Meeting | DER% | Miss% | FA% | SE% | RTFx |
| ----------- | -------- | -------- | ------- | ------- | --------- |
| IS1009b | 16.4 | 10.6 | 0.6 | 5.3 | 127.0 |
| ES2004c | 23.8 | 17.8 | 0.3 | 5.7 | 126.5 |
| ES2004b | 23.9 | 18.7 | 0.2 | 5.0 | 123.9 |
| IS1009a | 26.5 | 16.0 | 1.4 | 9.1 | 134.4 |
| ES2004d | 28.3 | 19.7 | 0.3 | 8.3 | 123.5 |
| IS1009d | 29.1 | 16.5 | 1.0 | 11.6 | 127.9 |
| TS3003b | 31.1 | 27.1 | 0.6 | 3.4 | 125.5 |
| EN2002c | 31.8 | 20.1 | 0.2 | 11.5 | 126.0 |
| ES2004a | 33.7 | 24.6 | 0.1 | 9.0 | 127.2 |
| EN2002b | 34.0 | 20.2 | 0.6 | 13.3 | 127.7 |
| TS3003c | 34.4 | 31.1 | 0.3 | 3.1 | 126.6 |
| EN2002a | 35.6 | 20.0 | 0.4 | 15.2 | 125.4 |
| EN2002d | 37.1 | 20.1 | 0.5 | 16.5 | 125.5 |
| IS1009c | 38.1 | 12.8 | 0.9 | 24.4 | 129.2 |
| TS3003d | 41.0 | 32.0 | 0.1 | 8.8 | 125.6 |
| TS3003a | 41.8 | 36.8 | 0.7 | 4.3 | 125.7 |
| **Average** | **31.7** | **21.5** | **0.5** | **9.7** | **126.7** |
## Text-to-Speech
Comparison across frameworks generating the same text samples (1s to \~300s of output audio):
### Kokoro 82M
| Framework | Total RTFx | Peak RAM | Notes |
| ---------------- | ---------- | ----------- | ----------------------- |
| PyTorch CPU | 17.0x | 4.85 GB | Known memory leak |
| PyTorch MPS | 10.0x | 1.54 GB | Crashes on long strings |
| MLX | 23.8x | 3.37 GB | — |
| **Swift CoreML** | **23.2x** | **1.50 GB** | Lowest memory |
CoreML matches MLX speed with 55% less peak RAM. First run takes \~15s for ANE compilation, subsequent loads \~2s.
## Running Benchmarks
```bash theme={null}
# Transcription (all languages)
swift run -c release fluidaudio fleurs-benchmark --languages all --samples all
# Transcription (English, LibriSpeech)
swift run -c release fluidaudio asr-benchmark --max-files all
# Custom vocabulary
swift run -c release fluidaudio ctc-earnings-benchmark --auto-download
# Streaming ASR
swift run -c release fluidaudio parakeet-eou --benchmark --chunk-size 320 --use-cache
# VAD
swift run -c release fluidaudio vad-benchmark --dataset voices-subset --all-files --threshold 0.85
# Diarization (offline)
swift run -c release fluidaudio diarization-benchmark --mode offline --auto-download
# Diarization (streaming)
swift run -c release fluidaudio diarization-benchmark --mode streaming \
--dataset ami-sdm --threshold 0.8 --chunk-seconds 5.0 --overlap-seconds 0.0
# Sortformer
swift run -c release fluidaudio sortformer-benchmark --nvidia-high-latency --hf --auto-download
# TTS
swift run -c release fluidaudio tts --benchmark
```
# CLI Reference
Source: https://docs.fluidinference.com/reference/cli
FluidAudio command-line interface for testing and benchmarking.
## Installation
```bash theme={null}
cd FluidAudio
swift build -c release
# Binary at .build/release/fluidaudio
```
Or run directly:
```bash theme={null}
swift run fluidaudio
```
## Commands
### Transcription
```bash theme={null}
# Batch transcription
fluidaudio transcribe audio.wav
fluidaudio transcribe audio.wav --model-version v2 # English-only
# Multi-stream parallel transcription
fluidaudio multi-stream audio1.wav audio2.wav
# Streaming transcription (Parakeet EOU)
fluidaudio parakeet-eou --input audio.wav --use-cache
```
### Text-to-Speech
```bash theme={null}
fluidaudio tts "Hello from FluidAudio" --output demo.wav --voice af_heart
fluidaudio tts "Custom pronunciation" --lexicon custom.txt --output out.wav
```
### Diarization
```bash theme={null}
# Process a file
fluidaudio process meeting.wav --output results.json --threshold 0.6
# Offline mode
fluidaudio process meeting.wav --mode offline --threshold 0.6
# With ground-truth
fluidaudio process meeting.wav --rttm ground_truth.rttm
```
### Voice Activity Detection
```bash theme={null}
# Offline segmentation
fluidaudio vad-analyze audio.wav
# Streaming
fluidaudio vad-analyze audio.wav --streaming --min-silence-ms 300
# Both modes
fluidaudio vad-analyze audio.wav --mode both
```
### Benchmarks
```bash theme={null}
# ASR benchmark
fluidaudio asr-benchmark --subset test-clean --max-files 100
# FLEURS multilingual
fluidaudio fleurs-benchmark --languages en_us,fr_fr --samples 10
# Diarization benchmark
fluidaudio diarization-benchmark --auto-download
fluidaudio diarization-benchmark --single-file ES2004a --threshold 0.7
# VAD benchmark
fluidaudio vad-benchmark --num-files 40 --threshold 0.5
```
### Dataset Management
```bash theme={null}
fluidaudio download --dataset ami-sdm
fluidaudio download --dataset librispeech-test-clean
fluidaudio download --dataset librispeech-test-other
fluidaudio download --dataset vad
```
# Models
Source: https://docs.fluidinference.com/reference/models
CoreML model catalog and HuggingFace sources.
## ASR Models
### Batch Transcription
| Model | Description |
| ------------------- | ------------------------------------------------------ |
| **Parakeet TDT v3** | 25 European languages, 0.6B params. Default ASR model. |
| **Parakeet TDT v2** | English only, 0.6B params. Better English recall. |
TDT models process audio in chunks (\~15s with overlap). Fast enough for dictation-style workflows.
### Streaming Transcription
| Model | Description |
| ---------------- | -------------------------------------------------------------------------------------- |
| **Parakeet EOU** | 120M params. 160ms/320ms frames for real-time results with end-of-utterance detection. |
### Custom Vocabulary
| Model | Description |
| --------------------- | ----------------------------------------- |
| **Parakeet CTC 110M** | CTC-based keyword spotting alongside TDT. |
| **Parakeet CTC 0.6B** | Larger CTC variant. |
## VAD Models
| Model | Description |
| ----------------- | ------------------------------------------ |
| **Silero VAD v6** | Voice activity detection on 256ms windows. |
## Diarization Models
| Model | Description |
| ---------------------------- | ------------------------------------------------------------------------- |
| **Pyannote CoreML Pipeline** | Segmentation + WeSpeaker embeddings. Online and offline modes. |
| **Sortformer** | End-to-end streaming diarization. Single neural network, 4 speaker slots. |
## TTS Models
| Model | Description |
| -------------- | ---------------------------------------------------------------------- |
| **Kokoro TTS** | 82M params, 48 voices. Flow matching + Vocos vocoder. Requires espeak. |
| **PocketTTS** | 155M params. Autoregressive, no espeak dependency. |
## HuggingFace Sources
| Model | Repository |
| ---------------------- | --------------------------------------------------------------------------------------------------------------------------- |
| Parakeet TDT v3 | [FluidInference/parakeet-tdt-0.6b-v3-coreml](https://huggingface.co/FluidInference/parakeet-tdt-0.6b-v3-coreml) |
| Parakeet TDT v2 | [FluidInference/parakeet-tdt-0.6b-v2-coreml](https://huggingface.co/FluidInference/parakeet-tdt-0.6b-v2-coreml) |
| Parakeet CTC 110M | [FluidInference/parakeet-ctc-110m-coreml](https://huggingface.co/FluidInference/parakeet-ctc-110m-coreml) |
| Parakeet CTC 0.6B | [FluidInference/parakeet-ctc-0.6b-coreml](https://huggingface.co/FluidInference/parakeet-ctc-0.6b-coreml) |
| Parakeet EOU | [FluidInference/parakeet-realtime-eou-120m-coreml](https://huggingface.co/FluidInference/parakeet-realtime-eou-120m-coreml) |
| Silero VAD | [FluidInference/silero-vad-coreml](https://huggingface.co/FluidInference/silero-vad-coreml) |
| Diarization (Pyannote) | [FluidInference/speaker-diarization-coreml](https://huggingface.co/FluidInference/speaker-diarization-coreml) |
| Sortformer | [FluidInference/diar-streaming-sortformer-coreml](https://huggingface.co/FluidInference/diar-streaming-sortformer-coreml) |
| Kokoro TTS | [FluidInference/kokoro-82m-coreml](https://huggingface.co/FluidInference/kokoro-82m-coreml) |
| PocketTTS | [FluidInference/pocket-tts-coreml](https://huggingface.co/FluidInference/pocket-tts-coreml) |
# TTS Custom Pronunciation
Source: https://docs.fluidinference.com/tts/custom-pronunciation
Override TTS pronunciation with custom lexicon dictionaries.
See the [Custom Pronunciation guide](/asr/custom-pronunciation) for the full documentation on lexicon files, word matching, and Swift API usage.
This page exists as a navigation convenience — the custom pronunciation system is shared across TTS backends (Kokoro and PocketTTS).
# Kokoro TTS
Source: https://docs.fluidinference.com/tts/kokoro
High-quality text-to-speech synthesis with 48 voices.
## When to Use
* **Best quality, full generation** — Kokoro generates all frames at once. Use when you can wait for complete audio before playback.
* **Need streaming/immediate playback** — Use [PocketTTS](/tts/pocket-tts) instead (\~80ms to first audio).
## Specs
| Metric | Value |
| ------------- | ----------------------------- |
| Parameters | 82M |
| Voices | 48 |
| Speed | 23x RTFx |
| Peak RAM | 1.5 GB |
| Architecture | Flow matching + Vocos vocoder |
| Phonemization | eSpeak-NG (GPL-3.0) |
Model: [FluidInference/kokoro-82m-coreml](https://huggingface.co/FluidInference/kokoro-82m-coreml)
## Quick Start
### CLI
```bash theme={null}
swift run fluidaudio tts "Welcome to FluidAudio text to speech" \
--output ~/Desktop/demo.wav \
--voice af_heart
```
### Swift
```swift theme={null}
import FluidAudioTTS
let manager = TtSManager()
try await manager.initialize()
let audioData = try await manager.synthesize(text: "Hello from FluidAudio!")
try audioData.write(to: URL(fileURLWithPath: "/tmp/demo.wav"))
```
## Chunk Metadata
```swift theme={null}
let detailed = try await manager.synthesizeDetailed(
text: "FluidAudio can report chunk splits for you.",
variantPreference: .fifteenSecond
)
for chunk in detailed.chunks {
print("Chunk #\(chunk.index) -> variant: \(chunk.variant), tokens: \(chunk.tokenCount)")
print(" text: \(chunk.text)")
}
```
## Pipeline
```
text → espeak G2P → IPA phonemes → Kokoro model → audio
↑ ↑
custom lexicon SSML
overrides here overrides here
```
Because espeak runs **outside** the model as a preprocessing step, you can intercept and edit phonemes before they reach the neural network. This is what enables SSML, custom lexicon, and markdown pronunciation control.
## Pronunciation Control
Kokoro supports three ways to override pronunciation:
* **SSML tags** — ``, ``, ``. See [SSML documentation](/tts/ssml).
* **Custom lexicon** — word → IPA mapping files loaded via `setCustomLexicon()`. See [Custom Pronunciation](/tts/custom-pronunciation).
* **Markdown syntax** — inline `[word](/ipa/)` overrides in the input text.
## Kokoro vs PocketTTS
| | Kokoro | PocketTTS |
| ---------------------- | -------------------------------------- | --------------------------------- |
| Pipeline | text → espeak G2P → IPA → model | text → SentencePiece → model |
| Voice conditioning | Style embedding vector | 125 audio prompt tokens |
| Generation | All frames at once | Frame-by-frame autoregressive |
| Latency to first audio | Must wait for full generation | \~80ms after prefill |
| SSML support | Yes (``, ``, ``) | No |
| Custom lexicon | Yes (word → IPA) | No |
| Pronunciation control | Full (phoneme-level) | None (model decides internally) |
| Text preprocessing | Full (numbers, dates, currencies) | Minimal (whitespace, punctuation) |
## Benchmarks
Same text samples generating 1s to \~300s of output audio, M4 Pro:
| Framework | RTFx | Peak RAM | Notes |
| ---------------- | --------- | ----------- | ----------------------- |
| **Swift CoreML** | **23.2x** | **1.50 GB** | Lowest memory |
| MLX | 23.8x | 3.37 GB | — |
| PyTorch CPU | 17.0x | 4.85 GB | Known memory leak |
| PyTorch MPS | 10.0x | 1.54 GB | Crashes on long strings |
CoreML matches MLX speed with 55% less peak RAM. PocketTTS benchmarks coming soon.
## Enable in Your Project
### Package.swift
```swift theme={null}
dependencies: [
.package(url: "https://github.com/FluidInference/FluidAudio.git", from: "0.7.7"),
],
targets: [
.target(
name: "YourTarget",
dependencies: [
.product(name: "FluidAudioWithTTS", package: "FluidAudio")
]
)
]
```
### Import
```swift theme={null}
import FluidAudio // Core (ASR, diarization, VAD)
import FluidAudioTTS // TTS features
```
# PocketTTS
Source: https://docs.fluidinference.com/tts/pocket-tts
Autoregressive TTS with dynamic audio chunking and streaming output.
## Overview
PocketTTS (\~155M params) is an autoregressive TTS backend that generates audio frame-by-frame. No espeak dependency — uses SentencePiece tokenization directly. Audio starts streaming \~80ms after prefill.
Model: [FluidInference/pocket-tts-coreml](https://huggingface.co/FluidInference/pocket-tts-coreml)
## Quick Start
```swift theme={null}
import FluidAudioTTS
let manager = PocketTtsManager()
try await manager.initialize()
let audioData = try await manager.synthesize(text: "Hello, world!")
try await manager.synthesizeToFile(
text: "Hello, world!",
outputURL: URL(fileURLWithPath: "/tmp/output.wav")
)
```
## Architecture
```
PocketTtsManager.synthesize(text:)
→ chunkText() — split into max 50 token chunks
→ loadMimiInitialState() — 23 streaming state tensors
→ FOR EACH CHUNK:
→ tokenizer.encode() — SentencePiece
→ embedTokens() — table lookup
→ prefillKVCache() — 125 voice + N text tokens
→ GENERATE LOOP:
→ runFlowLMStep() — transformer_out + eos_logit
→ flowDecode() — 8 Euler steps → 32-dim latent
→ denormalize() → quantize() → runMimiDecoder()
→ 1920 audio samples per frame
→ concatenate + postprocess
→ WAV output (24kHz mono)
```
## Key State
### KV Cache
* 6 cache tensors `[2, 1, 512, 16, 64]` + 6 position counters
* Reset per chunk
### Mimi State
* 23 tensors for convolution history, attention caches, overlap-add buffers
* Continuous across chunks — keeps audio seamless
## Text Chunking
Long text splits at 50 tokens or fewer:
1. Sentence boundaries (`.!?`)
2. Clause boundaries (`,;:`)
3. Word boundaries (fallback)
## Pipeline
```
text → SentencePiece tokenizer → subword tokens → PocketTTS model → audio
↑
pronunciation decisions
happen inside model weights
(no external control)
```
Unlike [Kokoro](/tts/kokoro) which uses espeak to convert text to IPA phonemes **before** the model, PocketTTS feeds raw text tokens directly into the neural network. The model learned text→pronunciation mappings during training — there is no phoneme stage to intercept.
## Pronunciation Control
| Feature | Supported | Why |
| ----------------------------------- | ----------- | ---------------------------------------------- |
| SSML `` | No | No IPA layer — model has no phoneme vocabulary |
| Custom lexicon (word → IPA) | No | No phoneme stage to apply mappings |
| Markdown `[word](/ipa/)` | No | Same — no phoneme input |
| SSML `` (text substitution) | **Planned** | Text-level, can run before tokenizer |
| Text preprocessing (numbers, dates) | **Planned** | Text-level, can run before tokenizer |
**What can be added** — anything that operates on text before the SentencePiece tokenizer: number/date/currency expansion, text substitution, abbreviation expansion.
**What cannot be added without retraining** — anything that requires phoneme-level control. The model decides pronunciation from text tokens alone. See [Kokoro](/tts/kokoro) if you need pronunciation control.
## CoreML Details
* All 4 models loaded with `.cpuAndGPU` (ANE float16 causes artifacts in Mimi state)
* Compiled from `.mlpackage` → `.mlmodelc` on first load, cached on disk
* Thread-safe via actor pattern
## Benchmarks
Benchmarks in progress. Methodology follows [Kyutai's evaluation](https://kyutai.org/pocket-tts-technical-report) and their [tts\_longeval](https://github.com/kyutai-labs/tts_longeval) toolkit.
### Upstream (Kyutai, CPU)
[LibriSpeech test-clean](https://huggingface.co/datasets/openslr/librispeech_asr), WER via Whisper large-v3:
| Metric | PocketTTS (100M) | F5-TTS | DSM (313M) |
| ------------------------ | ------------------ | ------ | ---------- |
| WER | 1.84% | 2.21% | 1.84% |
| Audio Quality (ELO) | 2016 | — | — |
| Speaker Similarity (ELO) | 1898 | — | — |
| Runs on CPU | Yes (6x real-time) | No | No |
ELO from human pairwise evaluation (50 raters, 50 samples). Tested on Apple M3 and Intel Core Ultra 7.
### FluidAudio CoreML (planned)
We will benchmark the CoreML port against the upstream PyTorch CPU baseline using the same methodology:
| Metric | How | Dataset |
| ----------------------- | ------------------------------------------------------------------ | --------------------------------------------------------------------------------- |
| **WER** | Transcribe TTS output with Whisper large-v3, compare to input text | [LibriSpeech test-clean](https://huggingface.co/datasets/openslr/librispeech_asr) |
| **Speaker Similarity** | WavLM cosine similarity between prompt audio and generated audio | LibriSpeech test-clean |
| **RTFx** | Wall-clock generation time / audio duration | Variable length (1s to 300s) |
| **Time to First Audio** | Time from `synthesize()` call to first audio frame | Single sentence |
| **Peak RAM** | Instruments / `os_proc_memory` during generation | Variable length |
Additional datasets from [tts\_longeval](https://github.com/kyutai-labs/tts_longeval):
* **NTREX** — monologue sentences from news translation corpus
* **Synthetic Dialogs** — daily life, technical, and number-heavy scripts
* **SEED English** — adapted from ByteDance's SEED TTS Eval
Key comparisons: CoreML ANE vs PyTorch CPU (upstream), CoreML vs Kokoro CoreML (FluidAudio internal).
## License
CC-BY-4.0, inherited from [kyutai/pocket-tts](https://huggingface.co/kyutai/pocket-tts).
# SSML Support
Source: https://docs.fluidinference.com/tts/ssml
Control pronunciation with Speech Synthesis Markup Language tags.
## Supported Tags
### `` — Custom Pronunciation
```xml theme={null}
Kokoro
```
### `` — Text Substitution
```xml theme={null}
WWW
```
### `` — Content Type Interpretation
```xml theme={null}
123
```
## Say-As Types
| Type | Input | Output |
| -------------------------- | ------------ | ------------------------------------------ |
| `characters` / `spell-out` | `ABC` | "A B C" |
| `cardinal` / `number` | `123` | "one hundred twenty-three" |
| `ordinal` | `1` | "first" |
| `digits` | `123` | "one two three" |
| `date` | `12/25/2024` | "December twenty-fifth twenty twenty-four" |
| `time` | `2:30` | "two thirty" |
| `telephone` | `555-1234` | "five five five one two three four" |
| `fraction` | `3/4` | "three quarters" |
### Date Formats
| Format | Description | Example |
| ------ | ------------------------ | ------------ |
| `mdy` | Month-Day-Year (default) | `12/25/2024` |
| `dmy` | Day-Month-Year | `25/12/2024` |
| `ymd` | Year-Month-Day | `2024-01-15` |
| `md` | Month-Day | `12/25` |
| `dm` | Day-Month | `25/12` |
| `y` | Year only | `2024` |
| `m` | Month only | `12` |
| `d` | Day only | `25` |
## Usage
```swift theme={null}
import FluidAudioTTS
let ttsManager = TtSManager()
try await ttsManager.initialize()
let text = """
The price is 42 dollars.
Call us at 555-1234.
"""
let audio = try await ttsManager.synthesize(text: text, voice: .afHeart)
```
## Coexistence with Markdown
Both syntaxes work together:
```swift theme={null}
let text = """
Kokoro and [Misaki](/mɪˈsɑːki/)
"""
```
## Edge Cases
* **No SSML tags:** Text passes through unchanged (fast path)
* **Malformed tags:** Invalid SSML passes through as literal text
* **Unknown interpret-as:** Content returned unchanged
# VAD Getting Started
Source: https://docs.fluidinference.com/vad/getting-started
Voice activity detection with Silero VAD v6 on CoreML.
## When to Use
* **Pre-process audio before ASR** — Segment files into speech regions, skip silence. Reduces ASR processing by 30-50%.
* **Real-time speech detection** — Trigger recording or UI when user starts/stops speaking.
* **Improve diarization quality** — Filter noise before speaker embedding extraction. Reduces false speakers by 20-40%.
## Specs
| Metric | Value |
| ----------- | --------------------- |
| Model | Silero VAD v6 |
| Window size | 256ms |
| Memory | Minimal (runs on CPU) |
Model: [FluidInference/silero-vad-coreml](https://huggingface.co/FluidInference/silero-vad-coreml)
## Offline Segmentation
```swift theme={null}
import FluidAudio
let manager = try await VadManager(
config: VadConfig(defaultThreshold: 0.75)
)
let samples = try AudioConverter().resampleAudioFile(
URL(fileURLWithPath: "audio.wav")
)
var segmentation = VadSegmentationConfig.default
segmentation.minSpeechDuration = 0.25
segmentation.minSilenceDuration = 0.4
segmentation.speechPadding = 0.12
let segments = try await manager.segmentSpeech(samples, config: segmentation)
for (index, segment) in segments.enumerated() {
print(String(format: "Segment %02d: %.2f-%.2fs", index + 1, segment.startTime, segment.endTime))
}
```
### Get Audio Clips
```swift theme={null}
let clips = try await manager.segmentSpeechAudio(samples, config: segmentation)
print("Extracted \(clips.count) buffered segments ready for ASR")
```
### Chunk-Level Probabilities
```swift theme={null}
let results = try await manager.process(samples)
for (index, chunk) in results.enumerated() {
print(String(format: "Chunk %02d: prob=%.3f", index, chunk.probability))
}
```
## Manual Model Loading
Stage the Core ML bundle for offline environments:
```swift theme={null}
let modelURL = URL(
fileURLWithPath: "/opt/models/silero-vad-coreml/silero-vad-unified-256ms-v6.0.0.mlmodelc",
isDirectory: true
)
var configuration = MLModelConfiguration()
configuration.computeUnits = .cpuOnly
let vadModel = try MLModel(contentsOf: modelURL, configuration: configuration)
let manager = VadManager(config: .default, vadModel: vadModel)
```
## Benchmarks
[VOiCES](https://iqtlabs.github.io/voices/) (25 files, clean speech):
| Metric | Value |
| --------- | ------ |
| Accuracy | 96.0% |
| Precision | 100.0% |
| Recall | 95.8% |
| F1-Score | 97.9% |
| RTFx | 1,230x |
[MUSAN](https://www.openslr.org/17/) (2,016 files, mixed noise/music/speech):
| Metric | Value |
| --------- | ------ |
| Accuracy | 94.2% |
| Precision | 92.6% |
| Recall | 78.9% |
| F1-Score | 85.2% |
| RTFx | 1,221x |
## CLI
```bash theme={null}
# Offline segmentation
swift run fluidaudio vad-analyze audio.wav
# Streaming mode
swift run fluidaudio vad-analyze audio.wav --streaming --min-silence-ms 300
# Both modes
swift run fluidaudio vad-analyze audio.wav --mode both
# Benchmark
swift run fluidaudio vad-benchmark --num-files 50 --threshold 0.3
```
# Segmentation Config
Source: https://docs.fluidinference.com/vad/segmentation-config
Tune VAD segmentation for your use case.
## VadSegmentationConfig
```swift theme={null}
public struct VadSegmentationConfig {
var minSpeechDuration: TimeInterval // Default: 0.15s
var minSilenceDuration: TimeInterval // Default: 0.75s
var maxSpeechDuration: TimeInterval // Default: 14s
var speechPadding: TimeInterval // Default: 0.1s
var silenceThresholdForSplit: Float // Default: 0.3
var negativeThreshold: Float? // Default: nil (auto)
var negativeThresholdOffset: Float // Default: 0.15
var minSilenceAtMaxSpeech: TimeInterval // Default: 0.098s
var useMaxPossibleSilenceAtMaxSpeech: Bool // Default: true
}
```
## Parameters
| Parameter | Default | Description |
| ---------------------------------- | ------- | ------------------------------------------------------------------------------------------------------ |
| `minSpeechDuration` | 0.15s | Minimum speech to keep. Prevents clicks/coughs from being treated as speech. |
| `minSilenceDuration` | 0.75s | Silence required to end a segment. Prevents early cut-offs during brief pauses. |
| `maxSpeechDuration` | 14s | Force-split long segments to match ASR model limits. |
| `speechPadding` | 0.1s | Context padding on both sides of each segment. |
| `silenceThresholdForSplit` | 0.3 | Probability below which audio is treated as silence for splitting. |
| `negativeThreshold` | nil | Override for exit hysteresis threshold. If nil, computed as `baseThreshold - negativeThresholdOffset`. |
| `negativeThresholdOffset` | 0.15 | Gap between entry and exit thresholds. Creates a "sticky zone" to prevent rapid flipping. |
| `minSilenceAtMaxSpeech` | 0.098s | Minimum silence at forced split points. Ensures splits don't land mid-phoneme. |
| `useMaxPossibleSilenceAtMaxSpeech` | true | Split at the longest silence near max duration for cleaner boundaries. |
## Hysteresis
The entry/exit threshold system prevents rapid state toggling:
* **Enter speech** when probability > `baseThreshold`
* **Exit speech** when probability \< `negativeThreshold`
* **Stay in current state** when probability is between the two
The entry threshold defaults to `VadConfig.defaultThreshold` set when constructing `VadManager`.
# Streaming VAD
Source: https://docs.fluidinference.com/vad/streaming
Real-time voice activity detection with event callbacks.
## Overview
For streaming workloads, maintain a `VadStreamState` and process chunks individually. Each call emits at most one `VadStreamEvent` describing a speech start or end boundary.
## Quick Start
```swift theme={null}
import FluidAudio
let manager = try await VadManager()
var state = await manager.makeStreamState()
for chunk in microphoneChunks {
let result = try await manager.processStreamingChunk(
chunk,
state: state,
config: .default,
returnSeconds: true,
timeResolution: 2
)
state = result.state
print(String(format: "Probability: %.3f", result.probability))
if let event = result.event {
switch event.kind {
case .speechStart:
print("Speech began at \(event.time ?? 0) s")
case .speechEnd:
print("Speech ended at \(event.time ?? 0) s")
}
}
}
```
## VadStreamResult
| Property | Type | Description |
| ------------- | ----------------- | ------------------------------------- |
| `state` | `VadStreamState` | Updated state for next chunk |
| `event` | `VadStreamEvent?` | Speech start/end (only at boundaries) |
| `probability` | `Float` | Raw VAD probability (0.0-1.0) |
## Notes
* Chunks don't need to be exactly 4096 samples
* Call `makeStreamState()` to reset (equivalent to Silero's `reset_states`)
* Use `probability` for custom thresholding alongside the built-in hysteresis