PeerTube/packages/transcription-devtools/README.md

Transcription DevTools

Includes:

• JiWER CLI NodeJS wrapper
• Benchmark tool to test multiple transcription engines
• TypeScript classes to evaluate word-error-rate of files generated by the transcription

Build

npm run build

Benchmark

A benchmark of available transcribers might be run with:

npm run benchmark

┌────────────────────────┬───────────────────────┬───────────────────────┬──────────┬────────┬───────────────────────┐
│        (index)         │          WER          │          CER          │ duration │ model  │        engine         │
├────────────────────────┼───────────────────────┼───────────────────────┼──────────┼────────┼───────────────────────┤
│ 5yZGBYqojXe7nuhq1TuHvz │ '28.39506172839506%'  │  '9.62457337883959%'  │  '41s'   │ 'tiny' │   'openai-whisper'    │
│ x6qREJ2AkTU4e5YmvfivQN │ '29.75206611570248%'  │ '10.46195652173913%'  │  '15s'   │ 'tiny' │ 'whisper-ctranslate2' │
└────────────────────────┴───────────────────────┴───────────────────────┴──────────┴────────┴───────────────────────┘

The benchmark may be run with multiple model builtin sizes:

MODELS=tiny,small,large npm run benchmark

Jiwer

JiWER is a python tool for computing the word-error-rate of ASR systems. https://jitsi.github.io/jiwer/cli/

JiWER serves as a reference implementation to calculate errors rates between 2 text files:

• WER (Word Error Rate)
• CER (Character Error Rate)

Usage

const jiwerCLI = new JiwerClI('./reference.txt', './hypothesis.txt')

// WER as a percentage, ex: 0.03 -> 3%
console.log(await jiwerCLI.wer())

// CER as a percentage: 0.01 -> 1%
console.log(await jiwerCLI.cer())

// Detailed comparison report
console.log(await jiwerCLI.alignment())

Resources

• https://jitsi.github.io/jiwer/
• https://github.com/rapidfuzz/RapidFuzz