39

Does anyone know of an application that can convert audio to text?

2
  • I assume it is spoken text. Which language is that text in? Commented Jul 9, 2012 at 11:33
  • The speech text is in simple english.
    – Kopano
    Commented Jul 9, 2012 at 14:33

5 Answers 5

34

The software you can use is Vosk-api, a modern speech recognition toolkit based on neural networks. It supports 7+ languages and works on variety of platforms including RPi and mobile.

First you convert the file to the required format and then you recognize it:

ffmpeg -i file.mp3 -ar 16000 -ac 1 file.wav

Then install vosk-api with pip:

pip3 install vosk

Then use these steps:

git clone https://github.com/alphacep/vosk-api
cd vosk-api/python/example
wget https://alphacephei.com/kaldi/models/vosk-model-small-en-us-0.3.zip
unzip vosk-model-small-en-us-0.3.zip
mv vosk-model-small-en-us-0.3 model
python3 ./test_simple.py test.wav > result.json

The result will be stored in json format.

The same directory also contains an srt subtitle output example, which is easier to evaluate and can be directly useful to some users:

python3 -m pip install srt
python3 ./test_srt.py test.wav

The example given in the repository says in perfect American English accent and perfect sound quality three sentences which I transcribe as:

one zero zero zero one
nine oh two one oh
zero one eight zero three

The "nine oh two one oh" is said very fast, but still clear. The "z" of the before last "zero" sounds a bit like an "s".

The SRT generated above reads:

1
00:00:00,870 --> 00:00:02,610
what zero zero zero one

2
00:00:03,930 --> 00:00:04,950
no no to uno

3
00:00:06,240 --> 00:00:08,010
cyril one eight zero three

so we can see that several mistakes were made, presumably in part because we have the understanding that all words are numbers to help us.

Next I also tried with the vosk-model-en-us-aspire-0.2 which was a 1.4GB download compared to 36MB of vosk-model-small-en-us-0.3 and is listed at https://alphacephei.com/vosk/models:

mv model model.vosk-model-small-en-us-0.3
wget https://alphacephei.com/vosk/models/vosk-model-en-us-aspire-0.2.zip
unzip vosk-model-en-us-aspire-0.2.zip
mv vosk-model-en-us-aspire-0.2 model

and the result was:

1
00:00:00,840 --> 00:00:02,610
one zero zero zero one

2
00:00:04,026 --> 00:00:04,980
i know what you window

3
00:00:06,270 --> 00:00:07,980
serial one eight zero three

which got one more word correct.

Tested on vosk-api 7af3e9a334fbb9557f2a41b97ba77b9745e120b3.

9
  • also, as an addition to this answer, there's a cool demo of both speech recognition and voice command tools here: youtube.com/…
    – Daithí
    Commented Jan 8, 2015 at 10:22
  • How do you add an acoustic model to the system?
    – jarno
    Commented Feb 8, 2015 at 13:38
  • You just download it and unpack, there is no such thing as "add to the system" Commented Feb 8, 2015 at 13:56
  • 4
    Well, I installed packages pocketsphinx-utils, pocketsphinx-hmm-en-hub4wsj and pocketsphinx-lm-en-hub4 in universe repository of Ubuntu 14.04. Then pocketsphinx_continuous -infile file.wav -hmm en_US/hub4wsj_sc_8k -lm en_US/hub4.5000.DMP 2> pocketsphinx.log worked. Maybe they are not optimal packages, but they were best matches I could find in the repositories.
    – jarno
    Commented Feb 8, 2015 at 15:05
  • 1
    They are VERY unoptimal packages, it is not recommended to use them unless you want to complain about bad accuracy. It is better to install pocketsphinx from github, ubuntu version is very much outdated. Commented Feb 8, 2015 at 17:59
15

I know this is old, but to expand on Nikolay's answer and hopefully save someone some time in the future, in order to get an up-to-date version of pocketsphinx working you need to compile it from the github or sourceforge repository (not sure which is kept more up to date). Note the -j8 means run 8 separate jobs in parallel if possible; if you have more CPU cores you can increase the number.

git clone https://github.com/cmusphinx/sphinxbase.git
cd sphinxbase
./autogen.sh
./configure
make -j8
make -j8 check
sudo make install
cd ..
git clone https://github.com/cmusphinx/pocketsphinx.git
cd pocketsphinx
./autogen.sh
./configure
make -j8
make -j8 check
sudo make install
cd ..

Then, from: https://sourceforge.net/projects/cmusphinx/files/Acoustic%20and%20Language%20Models/US%20English/ download the newest versions of cmusphinx-en-us-....tar.gz and en-70k-....lm.gz

tar -xzf cmusphinx-en-us-....tar.gz
gunzip en-70k-....lm.gz

Then you can finally proceed with the steps from Nikolay's answer:

ffmpeg -i book.mp3 -ar 16000 -ac 1 book.wav
pocketsphinx_continuous -infile book.wav \
    -hmm cmusphinx-en-us-8khz-5.2 -lm en-70k-0.2.lm \
    2>pocketsphinx.log >book.txt

Sphinx works alright. I wouldn't rely on it to make a readable version of the text, but it's good enough that you can search it if you're looking for a particular quote. That works especially well if you use a search algorithm like Xapian (http://www.lesbonscomptes.com/recoll/) which accepts wildcards and doesn't require exact search expressions.

Hope this helps.

2
  • 4
    every thing works like a charm but in my case i had to run following command to fix pocketsphinx_continuous: error while loading shared libraries: libpocketsphinx.so.3: cannot open shared object file: No such file or directory -------> export LD_LIBRARY_PATH=/usr/local/lib -------> export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig Commented Sep 19, 2017 at 11:30
  • This is also recommended at cmusphinx.github.io/wiki/tutorialpocketsphinx/…
    – andrybak
    Commented Sep 19, 2019 at 21:58
12

I you are looking to convert speech to text you could try installing the julius package:

sudo apt install julius

Description:

"Julius" is a high-performance, two-pass large vocabulary continuous speech recognition (LVCSR) decoder software for speech-related researchers and developers.

Or another option that isn't in Ubuntu's repositories or in the Snap Store is Simon:

... is an open-source speech recognition program and replaces the mouse and keyboard.

Reference Links:

Julius:

Simon:

2
  • 2
    Could you add to this answer an example of how to run Julius? It is phenomenally unclear from the documentation. Commented Mar 7, 2020 at 0:17
  • In the future, it would be better to post one answer for the software Julius and one answer for the software Simon, so that each answer can be voted on and commented on separately.
    – Flimm
    Commented Jul 31, 2023 at 7:22
2

You can use Mozilla DeepSpeech is an opensource speech-to-text tool. But you will need to train the application or download Mozilla's pre-trained model. For my project, the accuracy was still not sufficient, as audio files were not good quality, and used Transcribear instead, a web based editor with speech-to-text capabilities, but you will need to be connected online to upload recordings to the Transcribear server.

3
  • I am considering transcribing about 70 hours by one speaker (accented, clear non-native en). Can DeepSpeech be trained using the existing mp3 files.
    – LenB
    Commented Apr 26, 2020 at 19:18
  • In the future, it would be better to post one answer for Mozilla DeepSpeech, and one answer for Transcribear, so that each answer can be voted on and commented on separately.
    – Flimm
    Commented Jul 31, 2023 at 7:22
  • Mozilla DeepSpeech hasn't seen any development ever since Mozilla fired the DeepSpeech team. See this issue for more details: github.com/mozilla/DeepSpeech/issues/3693
    – Flimm
    Commented Aug 23, 2023 at 13:24
2

vosk-transcriber official CLI from Vosk

I was randomly tab completing after installing Vosk today, previously mentioned at: https://askubuntu.com/a/423849/52975 when I saw they had added a nice CLI wrapper at last, so now tested on Ubuntu 23.10, you can install with the English model as:

pipx install vosk
mkdir -p ~/var/lib/vosk
cd ~/var/lib/vosk
wget https://alphacephei.com/vosk/models/vosk-model-en-us-0.22.zip
unzip vosk-model-en-us-0.22.zip
cd -

and then use as:

wget -O think.ogg https://upload.wikimedia.org/wikipedia/commons/4/49/Think_Thomas_J_Watson_Sr.ogg
vosk-transcriber -m ~/var/lib/vosk/vosk-model-en-us-0.22 -i think.ogg -o think.srt -t srt

-i will eat pretty much anything including compressed audio files like .ogg or even video files like .ogv, presumably FFmpeg at work.

Nice! Now they only need a vosk-transcriber --download-model en option and have a default -m directory to finally make things fully clean, but this is already a huge improvement of life.

I played with a few examples to informally evaluate accuracy at: https://unix.stackexchange.com/questions/256138/is-there-any-decent-speech-recognition-software-for-linux/613392#613392

OpenAI Whisper

https://github.com/openai/whisper

This answer is slightly adapted from: https://unix.stackexchange.com/questions/256138/is-there-any-decent-speech-recognition-software-for-linux/718354#718354 by Franck Dernoncourt.

OpenAI's Whisper (MIT license, Python 3.9, CLI) yields some highly accurate transcription. To use it (tested on Ubuntu 20.04 x64 LTS):

conda create -y --name whisperpy39 python==3.9
conda activate whisperpy39
pip install git+https://github.com/openai/whisper.git 
sudo apt update && sudo apt install ffmpeg
whisper recording.mp3
whisper recording.mp3 --model large

If using an Nvidia 3090 GPU, add the following after conda activate whisperpy39

pip install -f https://download.pytorch.org/whl/torch_stable.html
conda install pytorch==1.10.1 torchvision torchaudio cudatoolkit=11.0 -c pytorch

Performance benchmarks from OpenAI

Model inference time:

Size Parameters English-only model Multilingual model Required VRAM Relative speed
tiny 39 M tiny.en tiny ~1 GB ~32x
base 74 M base.en base ~1 GB ~16x
small 244 M small.en small ~2 GB ~6x
medium 769 M medium.en medium ~5 GB ~2x
large 1550 M N/A large ~10 GB 1x

WER on several corpus from https://cdn.openai.com/papers/whisper.pdf:

enter image description here

WER on several languages from https://github.com/openai/whisper/blob/main/language-breakdown.svg:

enter image description here

Related questions:

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .