Comskip 0.79 build 32: Some bugs solved and Beyond TV added

[erik]

After some serious work on the MPEG decoder I think I have solved many of the audio/video synchronisation and timeline problems.
Some serious MPEG Transport Stream and DVR-MS testing could further improve Comskip so keep sending me your feedback.

From the change log

Changes in 0.79.32
- Added the BTV Smartskip output format, set output_btv=1 in the comskip.ini file to enable
- Bug solved: AC3 sound volume is too high causing missing of black frames
- Bug solved: Recording with 3/2 pull down has wrong frame numbers leading to incorrect cut points
- Bug solved: The volume_slip default value is too low
- Added parameter in ini file: ms_audio_delay=5 set to the number of frames audio lags video in dvr-ms files. Default value of 5 should be OK for most recordings.

Download from http://mk.kaashoek.com/comskip

[erik]

made an error during building.
If your comskip.exe does not do anything but exit, download build 32 again.

[geoffstewart]

Hey Erik,

Just tried out the 0.79.32 version. Works great but I don't think I was clear on the filename. It has to be the full filename including extension plus ".chapters.xml". For example, it must be "Law & Order-(Positive)-2006-04-05-0.mpg.chapters.xml". The current version of comskip does not have the ".mpg." part of that file name.

I can workaround this with my batch file at the moment, but if you have time, this would put the icing on the cake.

Thanks for your efforts on this.

Geoff

[erik]

Mmmm, Why did I expect this.....

Consider it done in the next build, will be available end of today.

[cmcquistion]

I did some testing of the new version (0.79.33) with output_btv=1 in the comskip.ini

It appears to create an mpg.chapters.xml file (YAY!!!), but the format looks slightly different from the BTV version of this file. I'm not sure whether it still works. I tried it, via BTV Link and it crashed Link, but this may have been for a different reason.

Here is a sample of a BTV chapter file's contents:

<cutlist>
<Region>
<start comment="00:02:21.2410998">1412410998</start>
<end comment="00:03:22.6691324">2026691324</end>
</Region>
<Region>
<start comment="00:11:56.4490662">7164490662</start>
<end comment="00:15:07.3397658">9073397658</end>
</Region>
</cutlist>



Now, here is a sample of the content of the chapter file created by Comskip:

<cutlist>
<Region><start>333667</start><end>1683348333</end></Region>
<Region><start>5014676333</start><end>5718379333</end></Region>
<Region><start>9794785000</start><end>10499155333</end></Region>
<Region><start>14948266667</start><end>15515500000</end></Region>
<Region><start>17934583333</start><end>17961610333</end></Region>
</cutlist>



Like I said, I'm not sure if it makes a functional difference or not. It could be that Snapstream included the "comments" in the file to show easy to read numbers for where the cuts are (minutes and seconds, as well as frames.)


*EDIT* I checked and the file DOES play back, properly, and the smartchapters are working, correctly.

[geoffstewart]

Ya, the comment attributes are in Snapstream's version for readability, I assume.

Geoff

[cmcquistion]

I have the new version of Comskip working perfectly and integrated with BTV's CustomPostProcessing schedule.

Here is the contents of the batch file to run (I named mine comskip.bat and it is in the same directory as the comskip.exe and comskip.ini):

"M:\Files_Backup\Snapstream\Comskip\comskip.ex e" %*
:: clean up log and txt files
del "%~pdn1".log
del "%~pdn1".logo.txt
del "%~pdn1".mpg.index.dat
del "%~pdn1".tp.index.dat
del "%~pdn1".txt

Here is the contents of my CustomPostProcessing.xml:

<CustomTasks>
<Task>
<GUID>531E0175-87F4-4d6f-B464-E603BC641DAE</GUID>
<Priority>DeferredImmediate</Priority>
<PropertyCollection>
<Property>
<Name>Command</Name>
<Value>M:\Files_Backup\Snapstream\Comskip\comskip. bat</Value>
</Property>
</PropertyCollection>
</Task>
</CustomTasks>

I have set Smartskip generation to DISABLED in Beyond TV's configuration (using BTV Web) and I turned off time constraints on Showsqueeze, so it runs immediately. I have a dedicated server for BTV and I watch programs from a Link machine, so I don't mind it running any time and popping up a DOS box. (If you wanted to run this during off-peak hours, instead of running immediately, you could set time constraints in the Showsqueeze options, and change the CustomPostProcessing step to run as "Deferred" instead of "DeferredImmediate" as I did, above.)

Here is the contents of my comskip.ini file. I took these entries directly from leecole, in another post, and modified a few things.

detect_method=43
validate_silence=1
validate_uniform=1
validate_scenechange=1
verbose=0
max_brightness=60
test_brightness=40
max_avg_brightness=25
max_commercialbreak=360
min_commercialbreak=25
max_commercial_size=125
min_commercial_size=4
min_show_segment_length=59
non_uniformity=500
max_volume=500
min_silence=20
ticker_tape=0
logo_at_bottom=0
punish=0
punish_threshold=1.3
punish_modifier=2
intelligent_brightness=1
logo_percentile=0.92
logo_threshold=0.70
punish_no_logo=1
aggressive_logo_rejection=1
connect_blocks_with_logo=1
cut_on_ar_change=1
delete_show_after_last_commercial=0
delete_show_before_or_after_current=0
remove_before=0
remove_after=0
shrink_logo=5
after_logo=0
padding=0
ms_audio_delay=5
max_repair_size=40
delete_logo_file=1
output_framearray=0
output_videoredo=0
output_womble=0
output_mls=0
output_cuttermaran=0
output_mpeg2schnitt=0
output_mpgtx=0
output_dvrcut=0
output_zoomplayer_chapter=0
output_zoomplayer_cutlist=0
output_edl=0
output_edlx=0
output_vcf=0
output_btv=1
output_projectx=0
output_demux=0
sage_framenumber_bug=0
sage_minute_bug=0
live_tv=0
live_tv_retries=1
cuttermaran_options="cut=\"true\" unattended=\"true\" muxResult=\"false\" snapToCutPoints=\"true\" closeApp=\"true\""
mpeg2schnitt_options="mpeg2schnitt.exe /S /E /R25 /Z %2 %1"
dvrcut_options="dvrcut \"%s.dvr-ms\" \"%s_clean.dvr-ms\" "

*EDIT* I just went in and removed all the comments in the comskip.ini to remove the line wrapping problem.

*EDIT 2* Changed detect_method=255 to detect_method=43, upon erik's suggestion. The commercial detection on HD recordings worked much better, after this change.

*EDIT 3* Updated my files, above, since some new releases have come out with some new options.

[erik]

Setting
detect_method=255
enable also silence based cutting which in the current version is only ment for transmissions without sufficient black frames.
It is suggested to use
detect_method=43 or
detect_method=47 when you want somewhat more agressive cutting.

The next release will automatically take care of enabling and disabling detection methods.

[cmcquistion]

I'll try that.

Thanks!

[cmcquistion]

I made that change (detect_method=43) and it working MUCH better at HD recordings, then it was working, before.

Thanks, again.

[PaulF]

Originally Posted by geoffstewart

Ya, the comment attributes are in Snapstream's version for readability, I assume.
Geoff

I have a script that converts the BTV xml into a Matroska chapter xml and then makes a Matroska video with mkvmerge. I used the comments because Matroska wants time not frame counts.

The player I use on my PalmT|X lets me skip chapters, then.

Is there any output file format that has the time of the cuts?

mkvmerge chapter xml

[geoffstewart]

No, there wasn't a file output format from comskip that has the time of the cuts in a easily readable format. That is why in this thread I had created a small utility to take a VideoRedo cut file and convert it to a snapstream smart chapter file. But the values in the the VideoRedo and smartchapter files is a time code, just a weird one that needs a touch of processing. I mention it here:

How do I get Comskip to work with v4.1?

In short, the last (right-most) seven digits of the value are fractions of a second. Any digit to the left of the first 7 are the seconds. A simple conversion from seconds to minutes and hours and you have your readable time code. That is what I did in my conversion utility. I can send you the source if that's of interest to you.

Hope this helps.

Geoff

[PaulF]

Originally Posted by geoffstewart

In short, the last (right-most) seven digits of the value are fractions of a second. Any digit to the left of the first 7 are the seconds. A simple conversion from seconds to minutes and hours and you have your readable time code.

Thanks! I thought the numbers were frame counts and I would have to divide by FPS to get time. I added the conversion you suggested to my Perl script. Now I can tweak comskip. It looks like my main program always has the logo so I may try logo only.

[erik]

Don't use only logo. You do need the others too.
Comskip should use as many clues as possible.
The newest version automatically disables detection methods when found not reliable.

[MickP]

<edit>An updated version of comproc.cmd can be found here</edit>

YAY! Comskip works beautifully.

Attached is a batch file (two actually) that I've made to process all files without a smartchapter file in a directory using comskip. It can be scheduled either on the local btv server or a remote machine to process files outside regular recording hours and will workk with unc paths. I use igCutter to manually process some files so comproc.cmd will also skip any files that end in _cut.mpg

Comskip is run at low priority so even if you're up playing games on the processing machine at 3:00 am it shouldn't cause too much of a problem.

To make it work:

copy comproc.cmd and comprocAction.cmd to the directory that contains the comskip files.

comproc has one parrameter which is the path to the directory containing the files to be processed. EG: comproc \\foo\q$\shows

comprocAction.cmd is called by comproc.cmd to process individual files and generally shouldn't be called directly unless it's for testing.

Use windows scheduled tasks to schedule comproc.cmd to run at a convenient time.

When scheduled as a task the command windows executing the processing will not be visible as they run in the background. A log file comProcLog.txt is created in the comskip directory and can be monitored to check the progress of the operation. I use "Tail for Win32" from SourceForge to keep an eye on this. Below is an example of the output;

------------------------------------------------------------------------
-- Mon 24/04/2006 3:00:00.70 - Begin processing files in:
-- \\hal\f$\shows
------------------------------------------------------------------------
Mon 24/04/2006 3:00:05.90 - Begin processing \\hal\f$\shows\Compass-(Who Wrote The Bible __ Pt 1)-2006-04-09-0.mpg
Mon 24/04/2006 3:20:01.59 - Completed processing \\hal\f$\shows\Compass-(Who Wrote The Bible __ Pt 1)-2006-04-09-0.mpg

Mon 24/04/2006 3:20:01.96 - Begin processing \\hal\f$\shows\Cutting Edge-(Israel and Bosnia)-2006-04-11-0.mpg
Mon 24/04/2006 3:42:36.46 - Completed processing \\hal\f$\shows\Cutting Edge-(Israel and Bosnia)-2006-04-11-0.mpg

Mon 24/04/2006 3:42:36.93 - Begin processing \\hal\f$\shows\Foreign Correspondent-(Another Sun_ Pt 1)-2006-04-11-0.mpg
Mon 24/04/2006 4:06:26.58 - Completed processing \\hal\f$\shows\Foreign Correspondent-(Another Sun_ Pt 1)-2006-04-11-0.mpg

Mon 24/04/2006 4:06:27.33 - Begin processing \\hal\f$\shows\Dateline-(Women of the Gaza Strip)-2006-04-13-0.mpg
Mon 24/04/2006 4:28:25.25 - Completed processing \\hal\f$\shows\Dateline-(Women of the Gaza Strip)-2006-04-13-0.mpg

Mon 24/04/2006 4:28:26.11 - Begin processing \\hal\f$\shows\Compass-(Who Wrote The Bible_ - Part 2)-2006-04-16-0.mpg
Mon 24/04/2006 4:48:05.36 - Completed processing \\hal\f$\shows\Compass-(Who Wrote The Bible_ - Part 2)-2006-04-16-0.mpg

Mon 24/04/2006 4:48:05.91 - Begin processing \\hal\f$\shows\Cutting Edge-(A War on Science)-2006-04-18-0.mpg
Mon 24/04/2006 5:13:03.16 - Completed processing \\hal\f$\shows\Cutting Edge-(A War on Science)-2006-04-18-0.mpg

Mon 24/04/2006 5:13:03.55 - Begin processing \\hal\f$\shows\Foreign Correspondent-(Another Sun_ Pt 2)-2006-04-18-0.mpg
Mon 24/04/2006 5:27:47.74 - Completed processing \\hal\f$\shows\Foreign Correspondent-(Another Sun_ Pt 2)-2006-04-18-0.mpg

Mon 24/04/2006 5:27:48.37 - Begin processing \\hal\f$\shows\The 7.30 Report-2006-04-19-0.mpg
Mon 24/04/2006 5:35:27.86 - Completed processing \\hal\f$\shows\The 7.30 Report-2006-04-19-0.mpg

Mon 24/04/2006 5:35:28.34 - Begin processing \\hal\f$\shows\Dateline-(Chester Wilmot)-2006-04-19-0.mpg
Mon 24/04/2006 6:05:50.43 - Completed processing \\hal\f$\shows\Dateline-(Chester Wilmot)-2006-04-19-0.mpg

Mon 24/04/2006 6:05:50.88 - Begin processing \\hal\f$\shows\The Glass House-(Akmal, David O'Doherty, Jason Byrne
Mon 24/04/2006 6:05:50.98 - Completed processing \\hal\f$\shows\The Glass House-(Akmal, David O'Doherty, Jason Byrne

Mon 24/04/2006 6:05:51.35 - Begin processing \\hal\f$\shows\The 7.30 Report-2006-04-20-0.mpg
Mon 24/04/2006 6:17:25.23 - Completed processing \\hal\f$\shows\The 7.30 Report-2006-04-20-0.mpg

Mon 24/04/2006 6:17:26.06 - Begin processing \\hal\f$\shows\Lost-(Revelation)-2006-04-20-0.mpg
Mon 24/04/2006 6:38:52.72 - Completed processing \\hal\f$\shows\Lost-(Revelation)-2006-04-20-0.mpg

Mon 24/04/2006 6:38:53.12 - Begin processing \\hal\f$\shows\Lateline-2006-04-20-0.mpg
Mon 24/04/2006 6:53:53.75 - Completed processing \\hal\f$\shows\Lateline-2006-04-20-0.mpg

Mon 24/04/2006 6:53:54.36 - Begin processing \\hal\f$\shows\Lost Worlds-(Michelangelo Superstar)-2006-04-23-0.mpg
Mon 24/04/2006 7:16:12.50 - Completed processing \\hal\f$\shows\Lost Worlds-(Michelangelo Superstar)-2006-04-23-0.mpg

Mon 24/04/2006 7:16:13.00 - Begin processing \\hal\f$\shows\Compass-(Gallipoli Pilgrimage)-2006-04-23-0.mpg
Mon 24/04/2006 7:36:20.50 - Completed processing \\hal\f$\shows\Compass-(Gallipoli Pilgrimage)-2006-04-23-0.mpg

------------------------------------------------------------------------
-- Mon 24/04/2006 7:36:20.55 - Completed processing files in:
-- \\hal\f$\shows
------------------------------------------------------------------------


Here's the content of the two cmd files:

comproc.cmd

 
@echo off
rem batch file by mick to process all files in a show folder
set folderpath=%1
echo.
echo Folder path is %folderpath%
echo.
echo ------------------------------------------------------------------------ >> comProcLog.txt
echo -- %date% %time% - Begin processing files in: >> comProcLog.txt
echo -- %folderpath% >> comProcLog.txt 
echo ------------------------------------------------------------------------ >> comProcLog.txt
dir %folderpath%\*.mpg /b /od > showlist.tmp
for /F "tokens=* delims=" %%i in (showlist.tmp) do start /wait /min cmd /c comprocAction.cmd %folderpath%\%%i
echo ------------------------------------------------------------------------ >> comProcLog.txt
echo -- %date% %time% - Completed processing files in: >> comProcLog.txt
echo -- %folderpath% >> comProcLog.txt 
echo ------------------------------------------------------------------------ >> comProcLog.txt
echo. >> comProcLog.txt

comprocAction.cmd

 
@echo off
rem this file should be called by comproc.cmd 
set showfile=%*
echo Show file is %showfile%
rem Check to see if file has been edited with igCutter
echo %showfile%|find "_cut.mpg" /i
if %errorlevel%==0 goto cutfile
echo Checking for %showfile%.chapters.xml
if exist "%showfile%.chapters.xml" goto alreadyProcessed
:needProcessing
echo Need to process %showfile%
echo.
echo %date% %time% - Begin processing %showfile% >> comProcLog.txt
start /wait /min /low cmd /c comskip.exe "%showfile%"
echo %date% %time% - Completed processing %showfile% >> comProcLog.txt
echo. >> comProcLog.txt
goto end
:alreadyProcessed
echo Already processed %showfile%
echo.
goto end
:cutfile
echo.
echo Skipping file %showfile% 
echo Already processed with igCutter.
echo. 
:end

Mick.