A good friend and his wife recently founded an AI startup in the lifestyle niche that uses machine learning to discover specific real-world patterns from videos.
For their business system, they need a pipeline that takes a video file, converts it to audio, and transcribes the audio to standard text that is then used for further processing. I couldn’t help but work on a basic solution to help fix their business problem.
Project Overview
I finished the project in three steps:
- First, install the necessary libraries.
- Second, convert the video to an audio file (
.mp4to.wav) - Third, convert the audio file to a speech file (
.wavto.txt). We first break the large audio file into smaller chunks and convert each of them separately due to the size restrictions of the used API.
Let’s get started!
Step 1: Install Libraries
We need the following import statements in our code:
# Import libraries
import speech_recognition as sr
import os
from pydub import AudioSegment
from pydub.silence import split_on_silence
import moviepy.editor as mp
Consequently, you need to pip install the following three libraries in your shell — assuming you run Python version 3.9:
pip3.9 install pydub
pip3.9 install SpeechRecognition
pip3.9 install moviepy
The os module is already preinstalled as a Python Standard Library.
If you need an additional guide on how to install Python libraries, check out this tutorial:
Recommended: Python Install Library Guide
Step 2: Video to Audio
Before you can do speech recognition on the video, we need to extract the audio as a .wav file using the moviepy.editor.VideoFileClip().audio.write_audiofile() method.
Here’s the code:
def video_to_audio(in_path, out_path): """Convert video file to audio file""" video = mp.VideoFileClip(in_path) video.audio.write_audiofile(out_path)
Recommended: Python Video to Audio
Step 3: Audio to Text
After extracting the audio file, we can start transcribing the speech from the .wav file using Google’s powerful speech recognition library on chunks of the potentially large audio file.
Using chunks instead of passing the whole audio file avoids an error for large audio files — Google has some restrictions on the audio file size.
However, you can play around with the splitting thresholds of 700ms silence—it can be more or less, depending on your concrete file.
Here’s the audio to text code function that worked for me:
def large_audio_to_text(path): """Split audio into chunks and apply speech recognition""" # Open audio file with pydub sound = AudioSegment.from_wav(path) # Split audio where silence is 700ms or greater and get chunks chunks = split_on_silence(sound, min_silence_len=700, silence_thresh=sound.dBFS-14, keep_silence=700) # Create folder to store audio chunks folder_name = "audio-chunks" if not os.path.isdir(folder_name): os.mkdir(folder_name) whole_text = "" # Process each chunk for i, audio_chunk in enumerate(chunks, start=1): # Export chunk and save in folder chunk_filename = os.path.join(folder_name, f"chunk{i}.wav") audio_chunk.export(chunk_filename, format="wav") # Recognize chunk with sr.AudioFile(chunk_filename) as source: audio_listened = r.record(source) # Convert to text try: text = r.recognize_google(audio_listened) except sr.UnknownValueError as e: print("Error:", str(e)) else: text = f"{text.capitalize()}. " print(chunk_filename, ":", text) whole_text += text # Return text for all chunks return whole_text
Need more info? Check out the following deep dive:
Recommended: Large Audio to Text? Here’s My Speech Recognition Solution in Python
Step 4: Putting It Together
Finally, we can combine our functions. First, we extract the audio from the video. Second, we chunk the audio into smaller files and recognize speech independently on each chunk using Google’s speech recognition module.
I added comments to annotate the most important parts of this code:
# Import libraries
import speech_recognition as sr
import os
from pydub import AudioSegment
from pydub.silence import split_on_silence
import moviepy.editor as mp def video_to_audio(in_path, out_path): """Convert video file to audio file""" video = mp.VideoFileClip(in_path) video.audio.write_audiofile(out_path) def large_audio_to_text(path): """Split audio into chunks and apply speech recognition""" # Open audio file with pydub sound = AudioSegment.from_wav(path) # Split audio where silence is 700ms or greater and get chunks chunks = split_on_silence(sound, min_silence_len=700, silence_thresh=sound.dBFS-14, keep_silence=700) # Create folder to store audio chunks folder_name = "audio-chunks" if not os.path.isdir(folder_name): os.mkdir(folder_name) whole_text = "" # Process each chunk for i, audio_chunk in enumerate(chunks, start=1): # Export chunk and save in folder chunk_filename = os.path.join(folder_name, f"chunk{i}.wav") audio_chunk.export(chunk_filename, format="wav") # Recognize chunk with sr.AudioFile(chunk_filename) as source: audio_listened = r.record(source) # Convert to text try: text = r.recognize_google(audio_listened) except sr.UnknownValueError as e: print("Error:", str(e)) else: text = f"{text.capitalize()}. " print(chunk_filename, ":", text) whole_text += text # Return text for all chunks return whole_text # Create a speech recognition object
r = sr.Recognizer() # Video to audio to text
video_to_audio('sample_video.mp4', 'sample_audio.wav')
result = large_audio_to_text('sample_audio.wav') # Print to shell and file
print(result)
print(result, file=open('result.txt', 'w'))
Store this code in a folder next to your video file 'sample_video.mp4' and run it. It will create an audio file 'sample_audio.wav' and chunk the audio and print the result to the shell, as well as to a file called 'result.txt'. This contains the transcription of the video file.
https://www.sickgaming.net/blog/2023/01/...cognition/