Using Coldfusion's CFFILE tag to monitor a progress log from FFMpeg

Question

I want to learn how to use the CFFILE tag from ColdFusion to read the contents of a text file. In my case, that text file is a progress log that is generated by FFMpeg while it transcodes a media file. I want to write a ColdFusion script that will poll the progress log periodically, until the log indicates that FFMpeg has finished its transcoding operation. On the client side I can then use Ajax to hit that ColdFusion script and show the user a "percentage completed" while FFMpeg does its work.

I got FFMpeg to generate the log file by using a new "progress" flag that recent versions of FFMpeg now support. Below I'll show you the way to use this flag, and also the generated output within the log file.

Here's the FFMpeg command:

ffmpeg -i c:\my_original_file.ogg c:\my_converted_file.mp3 -progress c:\my_progress.txt

The above command will cause FFMpeg to generate a log file called my_progress.txt.

Here's what it generates in the log file:

total_size=206150
out_time_ms=51410044
out_time=00:00:51.410044
dup_frames=0
drop_frames=0
progress=continue

The above 6 lines are generated repeatedly in the log file, with increasing values.

total_size=206150
out_time_ms=51410044
out_time=00:00:51.410044
dup_frames=0
drop_frames=0
progress=continue
total_size=412413
out_time_ms=102975756
out_time=00:01:42.975756
dup_frames=0
drop_frames=0
progress=continue
total_size=618363
out_time_ms=154463111
out_time=00:02:34.463111
dup_frames=0
drop_frames=0
progress=continue
total_size=824939
out_time_ms=206107189
out_time=00:03:26.107189
dup_frames=0
drop_frames=0
progress=continue

Finally, when the job completes, the final block of 6 lines are the last ones in the log file. Notice the "progress=end" on the last line:

total_size=9725902
out_time_ms=2431348011
out_time=00:40:31.348011
dup_frames=0
drop_frames=0
progress=end

I want to write a Coldfusion script using the CFFILE tag to read only the last 6 lines of the file (no matter how large the file has become), and to do this each time the script is called, by the browser, via Ajax. Finally I need to parse the values on these lines into variables so I can return some data to the caller.

I've researched progress bars for FFMpeg but they're in PHP which is hard for me, and besides, they parse the older formatted versions of FFMpeg's log files, and I would like to use the above newer formatting. Can anyone please help?

Answer 1

To get the last six lines tail will be faster, since it works backwards and only loads what you ask for (rather than reading in the whole file then looping through it). It'll obviously also use less memory, no matter the file size.

<cfexecute
    name      = "tail"
    arguments = "--lines=6 #Filename#"
    timeout   = 30
    variable  = "LastSixLines"
    />

<cfset Data = {} />
<cfloop index="CurLine" array=#LastSixLines.trim().split('\n')# >
    <cfset Data[ListFirst(CurLine,'=')] = ListRest(CurLine,'=') />
</cfloop>

As you appear to be on Windows, you likely need to install tail (it's pre-installed for Linux and MacOS). The simplest option to get it would be MSYS, which you may already have depending on what other software you use - for example, Git for Windows uses MSYS, and has tail.exe in its bin folder.

In that case, the second line above changes to something like:

    name      = "C:/Program Files/Git/bin/tail"

If you need the code to work on multiple systems, you can make that part a variable, (or put the appropriate directory onto the system PATH so it can be called from anywhere).

Answer 2

This will read your file and create a structure with the last 6 lines.

<cffile action="read" file="myfile.txt" variable="myfile">

<cfoutput>
    <cfset linecount = listlen(myfile,chr(10))>
    #linecount# lines

    <cfset count = 0>
    <cfset stData = {}>
    <cfloop list="#myfile#" index="i" delimiters="#chr(10)#">
        <cfset count++>
        <cfif count GT linecount - 6><!--- we only care about the last 6 lines --->
            #i#<Br>
            <cfset stData[listfirst(i,'=')] = listlast(i,'=')><!--- add data to stData --->
        </cfif>
    </cfloop>
</cfoutput>
<cfdump var="#stData#">

The code above will output the raw data from the text file and create a structure based on the = being a delimiter

You could also store your file into an array and pull out the last 6 rows like so. This option is about 2x as fast with smaller files (<200k lines), but as the file size grows the two options are nearly identical and then the first option becomes faster at around 1M lines

<cffile action="read" file="c:\ColdFusion10\cfusion\wwwroot\myfile.txt" variable="myfile">

<cfset filearray = listToArray(myfile,chr(10))>
<cfset linecount = arrayLen(filearray)>
<cfset stData = {}>
<cfloop from="0" to="5" index="i">
   <cfset stData[listfirst(filearray[linecount - i],'=')] = listlast(filearray[linecount - i],'=')>
</cfloop>