New HTPC

[Fickey]

Regarding fragmentation on NTFS drives, I recall reading three things that cause problems. First, NTFS intentionally places gaps between files for expansion. Second, I think it begins writing in the first available cluster, which may be those tiny expansion gaps, or deleted *.dat files. Third, almost all defraggers also leave these gaps.

Now, take all this with a grain of salt, as I'm too lazy to document it and some or all may no longer apply to XP or Vista. Snapstream may also be doing something proprietary to work around this, since 4.3 introduced a "New disk allocation method to reduce disk fragmentation."

[HomeyFour]

Originally Posted by Jakesty

Homey, go to Intels website to download those specific drivers for the G41 chipset. Sounds like you have an HP or DELL and often they don't provide support to downgrade. I had to do this with an HP we purchased at work. This probably still won't "MAKE" the video work with BluRay, but in my experience I have had success using PowerDVD 7.

Well I downloaded the XP drivers from Intel. It's a home-built system, and PowerDVD 9 which came with my BluRay burner, won't play BluRays with that driver and XP. When I get home tonight, I'll boot back into my XP image and see if I can convince XP to load the Vista drivers somehow and see if I can fool Windows somehow!

[Rich A]

Almost there. Tonight was a very busy recording night. And I must say I'm pretty pleased with my unusual set-up. Tonight I recorded 6 Digital HD, and 2 analog in three hours of time.

I just checked (1/2 hour after the last show finished) My quest to keep fragmentation down seems to work well. After all that recording here are the stats.

Op. System Drive C: 2 % fragmentation
Scratch drive (swap file temp etc) Drive D: 0 %
Recording Drive E: 0 %
Storage drive F: 0 %
Storage drive G: 48 %

All the shows recorded just happened to all be moved to their proper folders on the G: drive. The system is lightning fast. The comment about SS changing something recently to reduce fragmentation was something I also remember seeing. But I have to say that I've tested this and doing a one hour recording with BTV to a brand new formatted empty drive results in the drive being heavily fragmented. But then I'm also seeing a lot of fragmentation when I use the BTV "copy" function of showsqueeze to move the file from one drive to another. More testing needed there.

Now the last problem. Getting SnapStream's Smart chapter to work AFTER the files have been moved. Unfortunately the "custom Postprocessing" occurs first. The precedence is:

1. Custom Post processing
2. Smart Chapter Generation
3. Show squeeze.

Since I'm using Show Squeeze to sort and distribute (copy / delete) the files to their ultimate storage drives .. I need to have that done first. The "custom post processing" that I am doing is deleting all the *.dat files, which after the shows are moved keeps the recording drive empty and fragment free. The "copy" parameter used in ShowSqueeze is VERY useful and I think a powerful tool that is only limited by the way SS has set the order of post processing, Smart chapter gen. and Showsqueeze. It does allow you to take any recording and set it to always be moved to it's own different hard drive and folder. And that frees up the recording drive when it becomes time to watch the shows while the recording drive is busy.

I have to keep Smart Chapters turned off because it naturally occurs before showsqueeze, which then keeps the load on the recording drive.

The system would be "perfect" if I could change the order like this:

1. ShowSqueeze
2. Custom post processing
3. Smart Chapter gen.

I wonder if there might be something in an xml file or other parameter I might be able to use to change the order?

[Rich A]

One question I have is why does the BTV showsqueeze "copy" function result in a severely fragmented file?

I ran some tests yesterday trying to figure this out. I did a test record of a one hour digital show (about 5 GB) I had a completely blank freshly formatted hard drive (250 GB) and I set BTV' ShowSqueeze to "copy" the file to that new hard drive. The result was the file as written to the new drive was very badly fragmented.

Huh? So I re-formatted the storage drive ... nice clean and empty. Then did a simple DOS file copy of the recorded BTV drive from it's original hard drive to the new freshly formatted 2nd drive. The result .. NO fragmentation.

The old fashion manual file "copy" command just writes the file to the empty drive in one long uninterrupted sequential writing. Trying to do the same thing with the BTV "copy" puts the file all over the disk. What's up with that? I think the ShowSqueeze file "copy" is not doing a system based "copy" per se. I could understand a file being written and getting fragmented if the drive had multiple other files and the free space was somewhat fragmented to begin with. But why does the BTV process not write sequentially when there was no fragmentation or files at all on the destination drive?

[blaze818]

hey rich,
i know this is a little off topic but i wanted to know about your separate drive you use as a swap file. i have been looking around on the web and found people saying a whole bunch of different things. some people say that you should have 2.5X the amount of ram as your page file and others say thats a bad idea. some people also say it should be on a usb stick or CF drive....i just wanted to know what you know about this, since you sure do look (and sound) wise!

[Jakesty]

Originally Posted by HomeyFour

Well I downloaded the XP drivers from Intel. It's a home-built system, and PowerDVD 9 which came with my BluRay burner, won't play BluRays with that driver and XP. When I get home tonight, I'll boot back into my XP image and see if I can convince XP to load the Vista drivers somehow and see if I can fool Windows somehow!

Homey, Looks like POWERDVD should be ok for you as long as you have a decent CPU.

Requirements: Here

Rich, time to open a ticket with SS.

Blaze, the page file is generally important if you're low on memory. Otherwise there's not much of a reason to have a large Page file. It used to be 1.5 times your memory, but again unless you're running out of memory much at all, going larger is just a waste.

[Fickey]

Originally Posted by Rich A

...The old fashion manual file "copy" command just writes the file to the empty drive in one long uninterrupted sequential writing. Trying to do the same thing with the BTV "copy" puts the file all over the disk. What's up with that?...

Does your defragger have a drive map showing where the data is located? I know at least the O&O free defragger does, and even lists the contents of each cluster. Does allowing BTV to copy just one file to a fresh & clean drive result in fragmentation? If not, maybe it's those "expansion gaps" between files getting filled in that are resulting in fragmentation?

[Bigbird999]

+1 for O&O for great visual picture of the data on the drive.

In Windoze, the best way to defrag a file is to use explorer to copy it to another location that has contiguous free space. No matter how many fragments there are, windows will read them in the correct order and write them sequentially to the new location. Even if you select multiple files at the same time, windoze moves them one at a time so the result is the same as if you move them one at a time.

Rich is recording several shows at once and moving them as soon as they are finished. One would expect that BTV would move the shows one at a time like windows does, and produce fragment free copies on his storage drive. But maybe BTV somehow moves all of the shows at the same time resulting in fragmented files just like when recording multiple shows.

IIRC, showsqueeze can utilize multiple cores (ie do 2, 3 or 4, simultaneously - depending on # of cores) so using showsqueeze in copy mode probably utilizes all the cores and copies them simultaneously rather than sequentially.

The other thing that is an issue is "How fragmented is fragmented". There is file fragmentation, free space fragmentation, folder fragmentation and disk fragmentation and getting a handle on what each means is difficult. All of the various disk software gives you a number like % fragmented, but what does it really mean? I'm pretty sure that O&O defines a file broken into 2 or more fragments as fragmented. I recently copied a system image (15 GB) to a 0% (according to 0&0) fragmented drive. The result was that the image was split into 3 chunks and when I ran a report, O&O showed the drive as 40% fragmented because a single large file was split into 3 pieces. Even if it were in 3000 pieces, each piece would be 5 MB (the size of an MP3 file). So What? Even if BTV HD recordings files have 10,000 fragments there willnot be a performance issue.

BB

Edit attach screen shot

[Rich A]

Originally Posted by Fickey

Does your defragger have a drive map showing where the data is located? I know at least the O&O free defragger does, and even lists the contents of each cluster. Does allowing BTV to copy just one file to a fresh & clean drive result in fragmentation? If not, maybe it's those "expansion gaps" between files getting filled in that are resulting in fragmentation?

I have several "defraggers" at my disposal. I prefer Piriform's "Defraggler". It's quick, very low system impact, has lots of information, can be auto-scheduled and of course is free. I also have "Diskkeeper", "O & O" etc. I ONLY use this defragger on the main system drive. NONE of the other drives are ever defragged and at the start of every recording "day" they are usually at ZERO fragmentation. Here is a screen shot at what my drives look like "every" day. The main one .. E: is the drive where ALL shows are recorded to. And by doing what I'm doing, that drive is 0 percent fragmented all the time. Sometimes when I'm recording several shows at once. that drive becomes fragmented. But within a minute of the end of any show recording, the recorded show is moved (takes a few seconds) to it's permanent storage drive. I use ShowSqueeze "copy" to move the shows and then at that time also delete all the *.dat and other temporary files BTV writes during the recording process. The result is the drive dedicated for recording new shows is almost always at zero fragmentation. Yes I can get a lot of information about the files in question. Generally the recorded BTV files "as recorded" consist of a 5 GB file that is in 4 to 6 fragments.

Using a windows "copy" or "move" results in a continuously intact file at the destination. Using BTV's Showsqueeze "copy" feature results in each 5 GB file being fragmented into 4 to 6 parts. And that is even when the file is being copied to a freshly formatted drive with NO fragmentation.

I don't have a clue about those "expansion gaps" Never heard of that. Besides, why would those "expansion gaps" only affect BTV files and not a normal windows "copy or move" command?

The drive of interest is the "Unsorted Shows E" drive. That drive is almost always zero percent fragmented any time I check it. Yes I can (and have) examined the fragmented files and drive sectors etc. in detail. The below drive info was as of 4:15 PM The second drive info was as of 11:15 PM on the same day after a few shows had finished at 11:00 PM.

[Rich A]

Originally Posted by Bigbird999

+1 for O&O for great visual picture of the data on the drive.

In Windoze, the best way to defrag a file is to use explorer to copy it to another location that has contiguous free space. No matter how many fragments there are, windows will read them in the correct order and write them sequentially to the new location. Even if you select multiple files at the same time, windoze moves them one at a time so the result is the same as if you move them one at a time.

Rich is recording several shows at once and moving them as soon as they are finished. One would expect that BTV would move the shows one at a time like windows does, and produce fragment free copies on his storage drive. But maybe BTV somehow moves all of the shows at the same time resulting in fragmented files just like when recording multiple shows.

IIRC, showsqueeze can utilize multiple cores (ie do 2, 3 or 4, simultaneously - depending on # of cores) so using showsqueeze in copy mode probably utilizes all the cores and copies them simultaneously rather than sequentially.

The other thing that is an issue is "How fragmented is fragmented". There is file fragmentation, free space fragmentation, folder fragmentation and disk fragmentation and getting a handle on what each means is difficult. All of the various disk software gives you a number like % fragmented, but what does it really mean? I'm pretty sure that O&O defines a file broken into 2 or more fragments as fragmented. I recently copied a system image (15 GB) to a 0% (according to 0&0) fragmented drive. The result was that the image was split into 3 chunks and when I ran a report, O&O showed the drive as 40% fragmented because a single large file was split into 3 pieces. Even if it were in 3000 pieces, each piece would be 5 MB (the size of an MP3 file). So What? Even if BTV HD recordings files have 10,000 fragments there willnot be a performance issue.

BB

Edit attach screen shot

Windows copy. Yup .. BTV showsqueeze "copy" .. nope.

The fragmented files in my case are large. 5 GB on average. So if just one is fragmented into 6 fragments that over-all space being used would be a higher percentage as opposed to say a dozen 1 mb fragmented files.

As far as Smartchapter and ShowSqueeze are concerned both seem to be able to use multiple processors. I only have a dual core, but during my testing, I sat down and watched a one hour HD show. During that time two other shows finished recording and within a few seconds they were copied to their final destination. And because smartchapters are disabled (otherwize they run before the show is moved which defeats all the advantages for moving the shows out of the recording drive) I paused my viewing and started a smart chapter generation on two 1080 digital hd shows. Normally a one hour digital show takes my new pc about 50 minutes to fully generate the chapter points. Using dual cores at the same time smartchapter finished both in 70 minutes. Now I'm curious to see what the quad core will do.

I'm now positive that anyone can improve performance by using one drive dedicated for capture only and then moving the shows to another physical drive when done. Last night while watching a recorded show, smart skip was doing two other shows that were already moved. Oh and I had 2 other shows being recorded too. I just can't tell you how much better playback is.

The recording drive is always empty
Nothing other than recording goes on the recording drive
When all recordings are finished they are immediately moved off the recording drive.

There are only two issues.

When BTV is writing a show to the disc, it is always fragmented in 4 to 6 parts.

I have to do smartchapters manually after the shows were moved. I cannot change the preferred order of the post process stuff so that smartchapters is done last.

Other than that, just the way it is now .. it's working wonderfully.

Oh and the old HTPC was averaging 240 watts .. that would be at the UPS AC input which ran the PC, external drive box, and various other cable amps etc. The new setup is now at 180 watts. Yay, I've already convinced my wife that with all the money I'm saving I should buy a better projector! Or maybe more hard drives?

[Zogg]

Lots of great info here, thanks, Rich. I've enabled 1 smartchapter process in the past to run while other recordings were going on, and we always get some skipping on the Link system while watching something else. I think I'll try your method and see how it works.

Try using comskip instead of smartchapters. You can add it to your batch file where you want it, and it works very well. I get better results with it than I do with smartchapters. It's very easy to test manually if you want to. The settings take just a couple of tweaks for BTV, then drag the recording onto the .exe and it will run.

[Rich A]

How does it work? Does it scan the folder(s) for new shows to work on?

[Zogg]

No, comskip is a command line program that scans the file you tell it to and produces the chapter file. There is a gui interface but in this context it's best to run it from your batch file. There's a huge thread on Commercial Detection with a lot of batch file examples but here is my simple one that simply deletes unneeded files and runs comskip. This is the post processing batch file.

@ECHO OFF
::
::
:: Comskip script to commercial detection on Videos, using Comskip
:: (Last Edited 03-22-07)
::
::
SET COMSKIPFOLDER=C:\scripts\comskip
::
::
IF EXIST "%~pdn1*log*" DEL "%~pdn1*log*"
IF EXIST "%~pdn1*txt" DEL "%~pdn1*txt"
IF EXIST "%~pdn1*index.dat" DEL "%~pdn1*index.dat"
IF EXIST "%~pdnx1.chapters.xml" DEL "%~pdnx1.chapters.xml"
IF EXIST "%~pdn1.chp" DEL "%~pdn1.chp"
IF EXIST "%~pdn1.vprj" DEL "%~pdn1.vprj"
ECHO .
ECHO Running Commercial Detection, using Comskip
ECHO .
"%COMSKIPFOLDER%\comskip.exe" "%~pdnx1"
::
:: clean up log and txt files
IF EXIST "%~pdn1*log*" DEL "%~pdn1*log*"
IF EXIST "%~pdn1*txt" DEL "%~pdn1*txt"
IF EXIST "%~pdn1*index.dat" DEL "%~pdn1*index.dat"

You could put your file copy in there too, although you would have to do some trickery to figure out which folder to put it in.

One downside is that a command window will pop up on the server while this is running, but there are a couple of ways to fix that. I downloaded a free batch file compiler and compiled this into a .exe file which solved the popup problem.

Edit: This post has the changes you need to make to the comskip settings file. I use these for all my recordings and it works well.

[Rich A]

Thanks Z ... But I don't think this will work in my setup. If I understand the script, you are chapterizing (my new word ha ha) the BTV show in a specific folder.

And my shows are spread over multiple drives and folders. I think it's the same problem as you alluded to about moving the files with a script. There is no way to determine which file gets moved to which drive and folder. The BTV ShowSqueeze Copy Option does this just fine.

Remember my setup results in a clean zero fragmented recording drive. I don't want to do any thing to the recorded drive except move the files to their proper end destination.

If I have Comskip do it's thing while the show resides on the main recording drive, then I'm kinda negating the whole purpose of moving the show. And BTV's SmartChapter always worked very well for me, so I might as well use that .. which ends up with the whole thing working like it used to.

Besides, the BTV SmartChapter generation is "multi-core" aware. And IF I could make it work (after moving the file) I could run multiple SC processes at the same time.

[Zogg]

Yeah, that's why I said you'd have to do some batch file trickery to get the recording into your category folder.

But you've mentioned 2 problems, ShowSqueeze copy not giving you a defragmented file, and then having to manually do the SmartChaptering, so to fully automate this then you might have to resort to a complex batch file. If so then comskip is a good bet because it's free, works well, and can be called from a batch file. Just something to think about.

For me, I don't categorize the recordings, we just pull them up by series name in the viewscape. So I think I can record to a temporary drive, then when done have the custom post processing script move the file to the storage drive and then kick off comskip. The main reason I'm using comskip is that it works well on the HDPVR recordings, unlike SmartChapters, but I think it's more accurate on .TP and .MPG files as well.

Whatever you do, I'm glad you got the new box working well. Cheers!