Encoding media files in Ruby using ffmpeg/mencoder with progress tracking

Posted by Dmytro Shteflyuk on under Ruby & Rails

In my current project I need to encode media files from any format to several predefined. Furthermore I need to track progress and show it for the customer. I don’t want to describe wich media formats I need and what troubles with converting (maybe it will be my future posts, if anybody interested), instead I will describe common idea how to implement encoder scripts and how to track progress.

First, I need to decide how will be progress shown. I can’t find anything simpler then plain text output to standard output PROGRESS: 56, where 56 is progress in percent, and ERROR on error. Every output format will be handled by separate script (for example, to produce video for iPod it will be ipod.rb).

There are two encoding software I need to use – mencoder and ffmpeg (I don’t count additional tools like flvtool2 or something else, because they took much lower process time). It means that all I need is to find common method of executing these programs and interpret theirs output (which already contains information enough to calculate progress).

Let’s get started. First we will run mencoder:

1
mencoder input.avi -o output.avi -oac lavc -ovc lavc -lavcopts vcodec=xvid:acodec=mp3 > output.txt

This is very simple command line and I hope you will not use it in real projects :-) As you can see, output.txt contains lines ended with r:

1
2
3
4
5
--skipped--
Pos:   0.1s      3f ( 1%)  0.00fps Trem:   0min   0mb  A-V:0.008 [0:0]
Pos:   0.2s      4f ( 2%)  0.00fps Trem:   0min   0mb  A-V:0.012 [0:0]
Pos:   0.2s      5f ( 3%)  0.00fps Trem:   0min   0mb  A-V:0.016 [0:0]
--skipped--

I will use IO.popen Ruby’s method to parse:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
class MediaFormatException < StandardError
end

def execute_mencoder(command)
  progress = nil
  IO.popen(command) do |pipe|
    pipe.each("r") do |line|
      if line =~ /Pos:[^(]*(s*(d+)%)/
        p = $1.to_i
        p = 100 if p > 100
        if progress != p
          progress = p
          print "PROGRESS: #{progress}n"
          $defout.flush
        end
      end
    end
  end
  raise MediaFormatException if $?.exitstatus != 0
end

First I defined class MediaFormatException which will be used to raise exceptions to inform caller about errors (we will talk about it later). Then you can see method execute_mencoder, which accepts command line. Progress information will be show on standard output, and no progress status will be shown twice.

Let’s continue with ffmpeg:

1
ffmpeg -y -i input.avi -vcodec xvid -acodec mp3 -ab 96 output.avi > output.txt

Weird! It produses output information to standard error!

1
ffmpeg -y -i input.avi -vcodec xvid -acodec mp3 -ab 96 output.avi &> output.txt

As we can see, it shows current frame and time, but not percents. But in the beginning of output it produces Duration: 00:00:24.9, therefor we can calculate progress ourself:

1
2
3
4
5
6
7
--skipped--
Duration: 00:00:24.9, start: 0.000000, bitrate: 331 kb/s
--skipped--
frame=   41 q=7.0 size=     116kB time=1.6 bitrate= 579.7kbits/s
frame=   78 q=12.0 size=     189kB time=3.1 bitrate= 497.2kbits/s
frame=  115 q=13.0 size=     254kB time=4.6 bitrate= 452.3kbits/s
--skipped--

Let’s do it!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
def execute_ffmpeg(command)
  progress = nil
  IO.popen(command) do |pipe|
    pipe.each("r") do |line|
      if line =~ /Duration: (d{2}):(d{2}):(d{2}).(d{1})/
        duration = (($1.to_i * 60 + $2.to_i) * 60 + $3.to_i) * 10 + $4.to_i
      end
      if line =~ /time=(d+).(d+)/
        if not duration.nil? and duration != 0
          p = ($1.to_i * 10 + $2.to_i) * 100 / duration
        else
          p = 0
        end
        p = 100 if p > 100
        if progress != p
          progress = p
          print "PROGRESS: #{progress}n"
          $defout.flush
        end
      end
    end
  end
  raise MediaFormatException if $?.exitstatus != 0
end

Here we are using the same exception class MediaFormatException. We have methods now and ready to test them.

1
2
3
4
5
6
7
8
9
10
command_mencoder = "mencoder input.avi -o output.avi -oac lavc -ovc lavc -lavcopts vcodec=xvid:acodec=mp3"
command_ffmpeg = "ffmpeg -y -i input.avi -vcodec xvid -acodec mp3 -ab 96 output.avi 2>&1"

begin
  execute_mencoder(command_mencoder)
  execute_ffmpeg(command_ffmpeg)
rescue
  print "ERRORn"
  exit 1
end

Please note, that we are redirected standard error output to standard output to handle progress of ffmpeg.

Looks not so good because we need to handle exception in every script. Let’s create method, which will do this instead of us:

1
2
3
4
5
6
def safe_execute
  yield
rescue
  print "ERRORn"
  exit 1
end

And here is example of using:

1
2
3
4
5
6
7
command_mencoder = "mencoder input.avi -o output.avi -oac lavc -ovc lavc -lavcopts vcodec=xvid:acodec=mp3"
command_ffmpeg = "ffmpeg -y -i input.avi -vcodec xvid -acodec mp3 -ab 96 output.avi 2>&1"

safe_execute do
  execute_mencoder(command_mencoder)
  execute_ffmpeg(command_ffmpeg)
end

It’s simple, right? Now we have plain common progress statistics on standard output and don’t need to handle different outputs from different encoders. Now we can use this script as a part of largest process, which can catch progress status and show it to user.

23 Responses to this entry

Subscribe to comments with RSS

said on October 11th, 2006 at 12:22 · Permalink

Интересная статья, добавляем в закладки. Спасибо!

erka
said on October 12th, 2006 at 10:31 · Permalink

Поддерживаю. Спасибо за статью. Хорошо, что ты нашел на нее время, которого у тебя щас нет. Тем не менее я жду еще одну про проблемы с кодированием.

said on January 3rd, 2007 at 09:11 · Permalink

to get the duration, change all the “d”s in the regex to “\d”s

said on January 3rd, 2007 at 16:55 · Permalink

Oh, thanks! Of course, you right. This is my syntax highlighter killed \ :-)
I have updated post.

Thanks again

said on January 23rd, 2007 at 07:52 · Permalink

I am using your method to process ffmpeg commands with images as my input, eg “ffmpeg -i %03d.jpg movie.mp4” and I keep receiving errors that “%03d.jpg: I/O error occured
Usually that means that input file is truncated and/or corrupted.”

I know the images are good, I can issue the same exact command via bash (using the same images) and it builds the movie with no probs.

Of course, my first assumption was that the % is messing something up so I tried doing “ffmpeg -i ‘%03d.jpg’ movie.mp4” but it did not change the out come any and I got the same error.

Any suggestions?

said on January 23rd, 2007 at 08:07 · Permalink

I just solved my previous problem. It was simply an issue of incorrect paths. Since my code is in a model it was trying to load images from my app/models/ directory. I just gave it a full path, eg

1
2
3
datadir = RAILS_ROOT + "/tmp/images/"
videodir = RAILS_ROOT + "/public/videos/"
"ffmpeg -i #{datadir}%03d.jpg #{videodir}mymovie.mp4"

For anyone else who may have this problem… it’s a silly little time consumer…

said on January 23rd, 2007 at 13:16 · Permalink

Yeah, you right. I’m using absolute path for input and output files too. Thanks for the notice.

Alex
said on May 15th, 2007 at 20:47 · Permalink

Хорошая статья, только у меня вопрос, а как заставить ffmpeg работать с xvid-ом?
у меня FreeBSD и на команду вроде

1
ffmpeg -y -i input.avi -vcodec xvid -acodec mp3 -ab 96 output.avi

говорит:

1
Unknown video codec 'xvid'

хотя ‘xvid’ установлен и mencoder с ним прекрасно работает.
Что делать?

said on May 15th, 2007 at 21:31 · Permalink

Наиболее вероятное объяснение — ffmpeg собран без поддержки xvid (ключ --enable-xvid). Скорее всего придется его пересобрать.

Посмотреть список форматов, поддерживаемых ffmpeg:

1
ffmpeg -formats

При запуске ffmpeg выводятся ключи, с которыми он собран.

Alex
said on May 16th, 2007 at 10:40 · Permalink

Спасибо за ответ, но не получается!
При установке из портов:
make config
No options to configure

если в ручную устанавливать:
./configure –help
особо выбора не предоставляет, не говоря уже об опции –enable-xvid :(
версия ffmpeg-0.4.9-pre1.tar.gz

в связи с этим другой вопрос, я решил использовать ffmpeg только потому что он делает нормальную копию звука (-acodec copy), в отличие от mencoder-а, который при опции -oac copy из дорожки “Dolby AC3 48000Hz 6ch 448Kbps” делает “DTS 48000Hz stereo 768Kbps” (так говорит Media Player Classic, приблизительно тоже и VirtualDub).
Может подскажешь как решить эту проблему?

said on May 16th, 2007 at 14:29 · Permalink

Вот тут уж сорри, действительно не знаю, чем помочь. Никогда не работал с ffmpeg из портов фри. Как-то привычнее скачать официальные исходники и поставить куда-нить в /opt, чтобы не размазывались по системе.

С проблемой AC3 -> DTS тоже не сталкивался. У меня обычно задача из видео нормального качества сделать говно типа flv или mp4 с минимальными параметрами…

said on June 19th, 2007 at 12:20 · Permalink

Hi, i tried to implement your Script, but it will not work for me. I get an Error: undefined method `execute_mencoder’ for main:Object (NoMethodError)

Why? :)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
class MediaFormatException < StandardError
end


command_mencoder = "/usr/bin/mencoder /home/download/medien_trailer.mov -o /home/robin/download/output.avi -oac lavc -ovc lavc -lavcopts vcodec=xvid:acodec=mp3"
command_ffmpeg = "/usr/bin/ffmpeg -y -i /home/download/medien_trailer.mov -vcodec xvid -acodec mp3 -ab 96 /home/robin/download/output.avi 2>&1"

begin
  execute_mencoder(command_mencoder)
  execute_ffmpeg(command_ffmpeg)
rescue
  print "ERROR\n"
  exit 1
end

def execute_mencoder(command)
  progress = nil
  Open3.popen3(command) do |pipe|
    pipe.each("\r") do |line|
      if line =~ /Pos:[^(]*(\s*(\d+)%)/
        p = $1.to_i
        p = 100 if p > 100
        if progress != p
          progress = p
          print "PROGRESS: #{progress}\n"
          $defout.flush
        end
      end
    end
  end
  raise MediaFormatException if $?.exitstatus != 0
end

def execute_ffmpeg(command)
  progress = nil
  Open3.popen3(command) do |pipe|
    pipe.each("\r") do |line|
      if line =~ /Duration: (\d{2}):(\d{2}):(\d{2}).(\d{1})/
        duration = (($1.to_i * 60 + $2.to_i) * 60 + $3.to_i) * 10 + $4.to_i
      end
      if line =~ /time=(\d+).(\d+)/
        if not duration.nil? and duration != 0
          p = ($1.to_i * 10 + $2.to_i) * 100 / duration
        else
          p = 0
        end
        p = 100 if p > 100
        if progress != p
          progress = p
          print "PROGRESS: #{progress}\n"
          $defout.flush
        end
      end
    end
  end
  raise MediaFormatException if $?.exitstatus != 0
end
said on June 19th, 2007 at 12:28 · Permalink

Baerz, you should define methods execute_mencoder and execute_ffmpeg before executing them, for example, right after definition of the class MediaFormatException.

said on June 19th, 2007 at 12:41 · Permalink

Hey, works :) Thanks…

said on December 14th, 2007 at 11:59 · Permalink

Помогите решить вот такую проблему:
конвертирую flv в mov
на входе 8 Мб
на выходе 51 Мб, т.е. в 6 раз больше.

команда такая:
ffmpeg -i sqlintro1.flv -sameq 01sql.mov

Подскажите, если не затруднит, какими ключами можно уменьшить размер выходного файла, чтобы он был примерно таким же как и входной.

Adam
said on January 23rd, 2008 at 00:20 · Permalink

Hi , Thanks for this tutorial.
I am on leopard FFmpeg version SVN-r9102, ruby 1.8.6.
The above code works fine for me but the progress wont update sequentially as expected.

Any idea what this could be?? Is it something to do with pipe.each(“\r”) ??

Chris
said on June 22nd, 2008 at 18:35 · Permalink

The ffmpeg progress counter will never update with Ruby 1.8.6 because duration is defined out of scope (the first if clause) when processing each line from ffmpeg’s output (the second if clause) — to fix it, change the line: “progress = nil” to “duration = progress = nil” and it will work fine.

Lele
said on June 23rd, 2008 at 15:25 · Permalink

Hi,
I’m trying to do something similar to what you did.
One thing I’m trying to add now is a timeout, what I would like to do is to check if I get some output from mencoder, if I don’t get any output on the stdour or error for more that 30 sec I want to quit the process.
Is that possible?
thanks very much in advance

Comments are closed

Comments for this entry are closed for a while. If you have anything to say – use a contact form. Thank you for your patience.