22

I have a video that I want to create subtitles for. Is there a program that can perform rudimentary speech-to-text in order to

  1. Set the correct start/stop of each individual subtitle
  2. Create rudimentary text subtitles (using some sort of speech-to-text)

I know about gnome-subtitles. However, it requires extensive effort to create those subtitles manually. You need to select yourself the start and stop for each sentence.

YouTube has the above features (creates rudimentary text subtitles at the correct timings, using speech-to-text). However, I would rather not upload the videos to YouTube just to get my subtitles. Is it possible to do the subtitles efficiently on Ubuntu?

Update: I plan to use the .srt subtitles only, and do not need to hard code them on the videos. My biggest requirement is to have the program automatically find the start/stop for each sentence, so that I write the text in it.

Update #2: There is Speech-to-Text software for Linux, with the CMU Sphinx package. It is possible to use CMU Sphinx with a subtitle program according to this post. In addition, one subtitle tool is aware of this CMU Sphinx feature (web based tool), however there is no reference in the latest source code that they added CMU Sphinx. The quest continues to find a program that uses CMU Sphinx for rudimentary speech to text (which would set the correct timings as well), as YouTube already does.

1
  • there's an application called magpie that does something like this I think.
    – RolandiXor
    Jan 31, 2011 at 4:11

9 Answers 9

10

You have several alternatives:

YouTube

For the ones who do accept having to temporarily upload the video to YouTube (is mandatory to select video language) to get its subtitle (close caption, lyrics): Is possible to extract/download it with youtube-dl or yt-dlp:

yt-dlp --write-auto-sub \  # Write automatically generated subtitle file (YouTube only)
  --write-sub \                # Write subtitle file
  --sub-lang en,de,es \        # Languages of the subtitles to download (optional) separated by commas, use --list- subs for available language tags
  --convert-subs srt \         # Convert the subtitles to other format (currently supported: srt|ass|vtt|lrc)
  -o "~/%(uploader)s/%(playlist)s/%(playlist_index)s - %(title)s.%(ext)s" \  # OUTPUT TEMPLATE
  --skip-download \            # Do not download the video
  --ignore-errors vidURLorID   # Continue on download errors, for example to skip unavailable videos in a playlist

In one line and simplified:

yt-dlp --write-auto-sub --write-sub --sub-lang en --convert-subs srt --skip-download vidURLorID

If the conversion didn't work, convert it with FFmpeg:

ffmpeg -i myTitle.en.vtt output.srt

To convert from srt to txt:

sed -r -e 's/^\xef\xbb\xbf//' -e 's/\r//' -e 's/^[0-9]*$//' -e '/^[0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3} --> [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3}$/d' -e 's/^\s*$//' -e '/^$/d;s/<[^>]*>//g' output.srt | uniq > output.txt

Related answer.

Whisper (OpenAI)

Whisper is a general-purpose speech recognition model. It is trained on a large dataset of diverse audio and is also a multi-task model that can perform multilingual speech recognition as well as speech translation and language identification.

You can get some tools based on Whisper from this awesome list.

Live Captions

Live Captions is an application that provides live captioning for the Linux desktop.

Only the English language is supported currently. Other languages may produce gibberish or a bad phonetic translation.

On Flathub.

Kdenlive

Kdenlive have an Automatic Subtitling/Speech to text feature (optionally using Whisper).

3
  • yt-dlp didn't work. youtube.com/watch?v=fEsPOt8MG7E indicates that transcripts are disabled for that video. Could that be why?
    – YQ002lc2
    Sep 14, 2023 at 23:37
  • @YQ002lc2 The trick is to upload your video, let YouTube's algorithm generate the transcript/subtitles and download it with the above command. Doesn't work with any video without them. Sep 15, 2023 at 12:35
  • Oh, I see. That makes since now. But in this case, it's not my video, so trying to download and reupload it just to get subtitles generated would result in it being deleted and/or possibly flagged before any such automatic subtitle creation could occur...
    – YQ002lc2
    Sep 18, 2023 at 17:25
4

UPDATE:

autosub is no longer mantained. Another fork with GUI called pyTranscriber can be used.


You can use this Command-line utility

Autosub is a utility for automatic speech recognition and subtitle generation. It takes a video or an audio file as input, performs voice activity detection to find speech regions, makes parallel requests to Google Web Speech API to generate transcriptions for those regions, (optionally) translates them to a different language, and finally saves the resulting subtitles to disk.

https://github.com/agermanidis/autosub/

Python3 users, do this:

pip install git+https://github.com/BingLingGroup/autosub.git@alpha

Make sure you have ffmpeg installed.

3

I did not find a way to get the subtitle program to automatically add rudimentary subtitles, by analysing the voices in the video.

Therefore, the alternative that I use is

  1. Upload the video to Youtube (for example, privately) and use the in-build facility to create automatically rudimentary subtitles.

Then,

  1. Add the video to http://www.universalsubtitles.org/ and create manually the timeframes for each sentence, if the automated way in Youtube did not work, or sentences are mising.
  2. Use GNOME Subtitles (found in the Software Center) in order to clean up the subtitles and fix any timings.
2
  • 1
    This answer seems to be the most relevant to the question of automatically generating the subtitle files.
    – Garrett
    Mar 28, 2014 at 0:26
  • It's surprising that YouTube can automatically generate (rough) subtitles, but there's apparently no program which can.
    – Garrett
    Mar 28, 2014 at 1:21
3

I personally like Gnome Subtitles it is available in the repositories.

sudo apt-get install gnome-subtitles
0
3

I used Aegisub on Windows some years ago, and was really happy with it. Apparently it is available for Linux. It is pretty self explaining.

Aegisub only creates the subtitles file, e.g an .srt file. To combine the video and the subtitle to create a hard-coded subtitle you still need to use a second program.
On Windows I used VirtualDub, but it is not available for Linux. You can use VLC to do this on Linux:

Create your subs in Aegisub, saving it as usual as a .ass file.

Use VLC to add that subtitle track to your video. Subtitle -> Add subtitle file...

Configure the subtitle display style and settings so they display to your liking. Tools -> Preferences -> Subtitles/OSD

You can now watch the video to make sure the subs are displaying as you intended. For example I can check certain subs that I've specified in Aegisub to be displayed at the top of the screen rather than the bottom.

The output will be identical to how it looks now, so make sure all is good.

  1. Go to Media -> Convert/Save... (Ctrl + R).

  2. Under File Selection, add your video file. Tick "Use a subtitle file" and browse to your .ass sub file.

  3. Click the down arrow on the Convert/Save button and click Convert...(Alt + O).

  4. Under Settings, ensure the Convert option is ticked. Tick the Display the output option. Subs aren't added for some reason unless you tick this.

  5. Edit the profile so the video and audio settings are what you want. Under the subtitle tab, tick the Subtitles box, and use DVB subtitle codec. Make sure you tick 'Overlay subtitles on the video'. Press save.

  6. Enter a destination folder and filename in the Destination box.

  7. Press start.

Wait for it to be done, and that's it. The caveat with this method is that the encoding will happen in real-time with the video, so if you have a 2 hour video, it will take 2 hours. This is due to ticking the 'Display the output' box. But for some reason it only works when you tick this.

There are also other subtitle-editors.

Update:
I don't remember Aegisub having a functionality to automatically set beginning and end of a spoken sentence in the subtitles file. And I don't see a mention of such a function anywhere on the site. It is however with (key-combinations) pretty easy to set those times manually.

Is there even any program which has such a function (in any OS)?

6
  • I too used Aegsub on Windows, I didn't realize it was available for Linux.. thanks Pit :) ... Aegsub is a very competent subtitler... its default format is ASS (a evolution from SSA .. Sub-StationAlpha format) .. it handles Unicode and has special tools for preparing Karaoke text....
    – Peter.O
    Jan 31, 2011 at 11:56
  • 2
    Thanks for Aegisub. I am trying to figure out the workflow for this program. Can it scan the whole video and create the timings for each subtitle sentence? It does not appear to have a speech-to-text feature.
    – user4124
    Jan 31, 2011 at 12:01
  • You might want to read docs.aegisub.org/manual/….
    – Pit
    Jan 31, 2011 at 12:14
  • Aegisub is not generating subtitles automatically. We have to write subtitles using it. So probably this is not the solution for this question. Aug 3, 2018 at 17:03
  • The questions was edited 3 years(!) after my answer. Original "What program to use to create subtitles for a video" did not mention anything from automatic or text-to-speech.
    – Pit
    Aug 5, 2018 at 9:38
1

Inside Kdenlive video editor, in the top bar > project > subtitles > "Speech recognition" . You must first download the language pack from https://alphacephei.com/vosk/models , in kdenlive go to Settings > configure kdenlive > "Speech To Text".

0

Ok, found some tool which looks nice and similar to subtitle workshop - subtitle editor (apt-get install subtitleeditor).

Tried to compare it to Gnome Subtitles, subtitle editor looks more advance tool.

0

For KDE, a good subtitle editor is subtitlecomposer. Install it with the command:

sudo apt-get install subtitlecomposer
0

Speech note is worth mentioning here as it is free and can transcribe a vast number of languages. It comes as a flatpack so it is trivial to install. At the moment, you would have to use a tool like mpeg to extract the audio and feed it, but you can be the first to contribute to the completely open source code in github.... ;)

TLDR; soon this will hopefully be an option, but we are not completely there yet.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .