HD PVR reencoding script

[bforejt]

I'm not sure how long it will be until SS fully supports the HD PVR. Until then, I figured I would mess around with some free tools to see if I could make the .tp files a little less of a pain.

After a whole lot of experimentation, I now have a script that results in a HD Xvid file, and it looks the same as the original to me. I can manually run Smartskip, etc, on the new files, and they are also a bit smaller. The original is left behind should something go wrong. UPDATE: There can be A/V sync problems, and I'm still working on them.

Rencoding was not my first choice, but some of the issues are due to missing stuff in the h.264 video stream, which ruled out simple remuxing. (I tried quite a few variations on that theme.) If someone else can figure out how to repair the stream instead of reencoding it, I'm sure that would go much faster and be a better result.

I've included the current script below for reference. USE THE SCRIPT OR TOOLS AT YOUR OWN RISK!!!

Tools required:
ffmpeg -- Search on www.videohelp.com
metadataextractor -- SS Forums
now -- MS Win2k3 Resource Kit (optional, just used for logging)
(You could also use something like "cmdow" to hide the window if you ran this as a custom postprocessing task.)

USAGE (example):
CleanTP "c:\recordings\My Video File.tp"

Cheers,

-Brian

***************Begin CleanTP.cmd ***************
@echo off

Set LOG=<your log file>
Set TOOLS=<your tools directory>
Set BTV=<server>:<port>

if "%~x1"==".tp" goto :tpfile
%TOOLS%\Now %1 -- Not a TP file, will not process. >>%LOG%
goto :end

:tpfile
%TOOLS%\Now %1 -- Process starting. >>%LOG%

rem Start ffmpeg in low priority, in this window, and hang until finished
rem Video quality may be adjusted by using the qscale. (Suggested range 3-10, lower is better.)
start /B /low /wait %TOOLS%\ffmpeg -i %1 -f avi -vcodec libxvid -qscale 4 -acodec libmp3lame -ab 192k "%~dpn1.avi"

%TOOLS%\Now %1 -- Migrating metadata. >>%LOG%
%TOOLS%\metadataextractor --mode=extract --server=%BTV% %1
%TOOLS%\metadataextractor --mode=import --server=%BTV% "%~dpn1.avi"

rem Clean up the metadataextractor XML file...
del "%~dpn1.xml" >>%LOG%

:end
%TOOLS%\Now %1 -- Process Complete. >>%LOG%

[tscales]

OK, just tried it and appear to be clueless.

My 8+ GB file turned into a 63 MB avi file that BTV says is corrupt.

Also, the Meta data extractor and reinserter got error messages about an unauthenticated session.

Suggestions welcome. I REALLY want this to work.

[bforejt]

If you use a username and password for BTV, you'll need to add it to the metadataextractor command. (Run "metadataextractor" in a command prompt and it will show you the syntax. Should be --user=username and --password=password.) That said, when I run the script I see the same message about unauthenticated user, but it still works.

You changed the stuff at the top of the script to match your system, right?

I'm using ffmpeg version 12665, downloaded here: http://ffdshow.faireal.net/mirror/ffmpeg/

I suggest that you get that version, and run it manually to watch it work. All the script does is make it happen neatly, but you can do each step manually, and it might help wth troubleshooting.

So in a command prompt, run this:

ffmpeg -i "your input file.tp" -f avi -vcodec libxvid -qscale 4 -acodec libmp3lame -ab 192k "your output file.avi"

You should be able to watch ffmpeg run and see what it tells you. That would be a start. You can play back the file in Windows Media Player if it is ok.

Also, if you don't already have the Divx or Xvid full codec installed, it might be needed. This file will have the xvid identifier on the video stream, and I'm not sure if the divx codec included with BTV "knows" it can decode that. (I've always installed one or the other, so I never really tried.)

Finally, please note this isn't perfect yet. I'm still getting some audio/video sync problems with certain files, and I have a way to sorta "fix" it, but it isn't predictable yet. (I'm playing with the "-itsoffset" flag.) I think it is because of the weird way the HDPVR starts the video stream. (Judging from the errors that ffmpeg sees, it doesn't look like it starts with a key frame, which seems to cause issues.)

When it does work, though... it really makes it easier to deal with these files, and they look just as good to me.

Good luck,

-Brian

[tscales]

I'll try that version of ffmpeg and see if that helps. I do NOT have a userid/password on BTV though, so I'm a bit at a loss there.

In watching it on the first file, it got tons of errors at the beginning. Not sure it that was my problem.

I'll keep experimenting. Thanks!

Tom

[bforejt]

The errors at the beginning of the encode to be expected, I've seen a screen filled with them every time. What matters is that it locks on to the audio and video stream and outputs a clean file. (If you look above the errors, it will tell you what it sees in an "input" section. Should be h264 video 59.97 fps and 130kbps aac audio being recoded to xvid and mp3. At least... that's what it is on my system.)

The running display will show all sorts of useful information, like bitrate, expected file size, how many seconds it has processed, and frames per second it is processing. Should give you a really good handle on what is going on.

Good luck,

-Brian

[bforejt]

After two weeks of experimenting and watching the results, I think it is safe to say that reencoding HDPVR video this way works well (for now, until SS develops support within BTV). The only issue is A/V sync on certain files, and I have a solution to that now.

Most of the time, the command that will encode the video perfectly is this one:

ffmpeg -i "original file.tp" -f avi -vcodec libxvid -qscale 6 -acodec libmp3lame -ab 192k "output file.avi"

If you get A/V sync problems, then this one will fix it:

ffmpeg -itsoffset 1.5 -i "original file.tp" -f avi -vcodec libxvid -qscale 6 -acodec libmp3lame -ab 192k "output file.avi"

You don't have to wait for the entire encode to complete to figure out which one will work. If you do 5 minutes or so and cancel, you can usually see whether the lips match the words.

So if any of you are having trouble playing these HDPVR files back on slower machines, you might want to take a shot at this. The xvid/divx codec is much less taxing, and will play on lower end machines much better than h264.

Good luck,

-Brian

[derfel]

How long it takes to reencode on your computer (let me know your specs).

Thanks!

[tscales]

I have got to be doing something wrong. I compressed with the new command line and it compressed 2.3GB to 33MB. BTV says it is corrupt.

Any suggestions? Perhaps I'm using the wrong version of FFMpeg? Can you provide a link to the proper version? I had real trouble finding it, even with the link to the main page.

Thanks.

[bforejt]

The version of ffmpeg I use is here: http://ffdshow.faireal.net/mirror/ffmpeg/ Version 12665

And to answer the other question, 1 hour of video takes about 3 hours to process on my 2.7 GHz dual core AMD.

I run this command in the script now: ffmpeg -i %1 -f avi -v 2 -vcodec libxvid -qscale 6 -acodec libmp3lame -ab 192k "%~dpn1.avi" The "-v 2" adds some additional logging, and it will show you dropped frames. The files with the A/V sync issue always have a load of dropped frames at the beginning, so I know right away if the encode isn't going to be good. If I see the dropped frames I cancel right away and run a script that has this statement instead: ffmpeg -itsoffset 1.3 -i %1 -f avi -v 2 -vcodec libxvid -qscale 6 -acodec libmp3lame -ab 192k "%~dpn1.avi"

My next task is to see if I can detect the dropped frames to make the whole process automatic.

I'm sorry you are having trouble with running ffmpeg. From the sounds of things, I'd wager it is crashing early in the process, which is why the files are small and corrupt. Please manualy run it in a command prompt (without using the script at first) to make sure it is working ok. Start by running "ffmpeg -version" to make sure you are actually executing the "correct" one. After that, run this:

ffmpeg -i <input file name.tp> -f avi -v 2 -vcodec libxvid -qscale 6 -acodec libmp3lame -ab 192k <output file name.avi>


Remember to replace the input and output file names with the actual names of the files you want to process and create.

Good luck,

-Brian

[tscales]

With the exception of the V 2, that is EXACTLY what I've run, including the ffmpeg version. I've been running it from the command line and it appears to be working (runs all the way through).

I can't figure out what is different about my environment.

Weird.

[tscales]

Do I need to install the mp3Lame libraries? Would it have come in the package? Could that be my problem?

I'm also running on a quad core. Doubt if that matters.

[bforejt]

The thing that makes both ffmpeg and mencoder so cool is that the libraries they need are all compiled in. So there's nothing else to install.

Maybe try using the mpeg4 codec instead of the xvid codec. Replace the word "libxvid" with "mpeg4 -vtag xvid" to try it.

You could also try x264 by replacing "libxvid" with "libx264".

Not seeing what you are seeing, it's a bit of a mystery to me why it seems to work but comes up with such odd results. It's not like ffmpeg is an untried tool (it seems to be almost a standard), but maybe there's a bug. You could also try some older versions of ffmpeg to see if it results in something different.

-Brian

[bforejt]

Success!

The only issue (for me) has been the AV sync, and as it turns out, mencoder does an equal job without any AV sync issues. Plus, I know that at least one person had trouble with ffmpeg, so maybe this tool will work better in general.

I got my mencoder binary here: http://oss.netfarm.it/mplayer-win32.php

The ffmpeg line in my script has been replaced by this:
start /B /low /wait %TOOLS%\mencoder %* -ffourcc xvid -ovc lavc -lavcopts vqscale=3.3:autoaspect:mpeg_quant:threads=2:mbd=2: v4mv:trell -oac mp3lame -lameopts cbr:br=192 -o "%~dpn1.avi"

Quality is very good with these parameters, speed is ok (~20fps), and it compresses a good bit over the original. If you don't use "mbd=2:v4mv:trell" it will go twice as fast, and still look decent. (See the mencoder docs for more info.)

So that's it.

Enjoy!

-Brian

[bits]

I must be missing something here! Why for the love of video would you even want to re-encode h264 to xvid or any other video format? Why spend the money on the HD PVR only to re-encode? There must be a rational explanation, otherwise it is pure lunacy!

[bforejt]

Not the first time I've been called a lunatic.

However, I'm not sure that it is deserving in this case. I love the visual quality of the "real" HD recordings, which is why I'm an early adopter of the HD PVR. That said, the reasons I have for reencoding are many...

#1. I can smartskip the new files. (HD PVR native files fail, and that's annoying.)

#2. I can burn DVDs from them. (Same deal, HD PVR files fail.)

#3. I can showsqueeze again to other formats as necessary (to watch on my ipod, for example)

#4. Watching them is far less annoying. (i.e. no green blocks, no lost audio on skipping around. And... you do a lot of skipping around because smartskip fails on the HD PVR originals, so this issue is compounded.)

#5. They are more watchable on lower end machines in my home.

#6. Since BTV doesn't actually support the HD PVR, my recordings are a default 8 mbps. That's such overkill for h264 that a less advanced codec (Xvid) actually produces smaller files at similar quality.

#7. And it's fun to mess with the encoders. (For me, at least.)

There's no doubt that leaving the videos alone would be far better, but that isn't possible since they are so painful to deal with that I actually shut the HDPVR down for a bit while I worked this out. I just figured that some people might want a way to make the video a little less annoying while we wait for SS to finish developing support.

I did try to simply fix the container format, btw. The watchability issues seem to be related to the video stream, so a new container did no good.

-Brian