[plugin] XMLTV Importer

[StephaneM]

Originally Posted by Ivo-G

The latest version has the log conflicts to Eventlog feature.../...Is there any way you could do an optional Log to XML format feature?

Why not, but for what purpose? To be able for you to read this log and incorporate this into another process?
Regards,
Stéphane.

[Ivo-G]

Yep, I want to use the conflict data to fix the conflicts and re-run the import.
The problem I have at the moment is that the eventlog isn't exportable in a format that keeps the data together. The extra linefeeds in the data field break up the log file. If you could output it as XML, I could parse the XML data and use it to fix the problems.

Even better would be to have the 2 conflicting shows included in their own object, with their original tags included. For instance:

Warning 5-4-2006 9:25:09 PM
The show 'Friends'
on channel 0008, from 2006-04-11 23:55 to 00:30
is conflicting with 'Law & order: Criminal intent'
from 2006-04-11 23:00 to 00:00

The last show was removed.

Could be represented as:

<conflict detected="20060405212500" losershow="1">
 <conflicting show="1" >
  <programme start="20060411230000" stop="20060412000000" channel="37">
   <title>Law &amp; order: Criminal intent</title>
  </programme>
 </conflicting>
 <conflicting show="2" >
  <programme start="20060411235500" stop="20060412003000" channel="37">
   <title>Friends</title>
  </programme>
 </conflicting>
</conflict>

Where the bold text is a straight copy from the input XMLTV file

[Ivo-G]

I know it would be better to get this fixed in the grabber, but I have no idea how to go about this. I have a working grabber now, but the website is screwing things up IMO.

Funny thing is that their webpages do show the correct info, they're probably parsing the wrong info through a XSLT file first before showing it, whereas the grabber gets the raw data.

[nstocke]

Since installing and starting to use version 4.5 Imports seem to take much longer than they did in previous versions. Is this expected behavior or might I have something wrong that I need to look into?

Normally my update batch file would start at approximately 4:00 a.m. and be done before 5:30 a.m. I just checked and it was still running at 7:15 a.m. It did complete prior to 7:30 a.m.


I am processing zap2it.com data from two sources local broadcast programing and 4DTV shows. The file is about 34 -35 meg is size. It was processing a file about 42 meg is size but I trimmed it down yesterday by deleting channels I no longer can get or I'm no longer interested in on the Zap2it site and then ran a full update last night so it would only be processing one more day's worth of data today.

Thanks
Neal

[StephaneM]

Originally Posted by nstocke

Since installing and starting to use version 4.5 Imports seem to take much longer than they did in previous versions. Is this expected behavior or might I have something wrong that I need to look into?

Yes this is expected behavior as your are using a Zap2it XMLTV.

In version 4.5 the Importer use the EPGID furnished by Zap2it this cause extra lookups in the lookup table and searching the lookup table takes a long time (especially when using the importer since a long time as the lookup table grows up).

Regards,
Stéphane.

[nstocke]

Originally Posted by StephaneM

Yes this is expected behavior as your are using a Zap2it XMLTV.

In version 4.5 the Importer use the EPGID furnished by Zap2it this cause extra lookups in the lookup table and searching the lookup table takes a long time (especially when using the importer since a long time as the lookup table grows up).

Regards,
Stéphane.

Thanks for the quick reply. Is there anything I can do that would reduce the time? I currently process my XMLTV file with a utility that marks episodes as new. I"m not sure what criteria it uses. Do you think it would be better to use the Importer to determine new episodes or disable that function? Do you know if Zap2it is the only sources for XMLTV data for North America?

Thanks in advance for any info you can throw my way. I can always just start the process earlier so I'm just fine. But I'm always looking for ways to improve how my system works.

Neal

[StephaneM]

Originally Posted by nstocke

Thanks for the quick reply. Is there anything I can do that would reduce the time? I currently process my XMLTV file with a utility that marks episodes as new. I"m not sure what criteria it uses. Do you think it would be better to use the Importer to determine new episodes or disable that function?

You can continue to use this utility. This not what lengthen the whole process. Before version 4.5 the Importer was creating EPGID : that is the unique reference for each show. For coherence and speed all EPGID are kept into a lookup table : for each show the Importer compute three hashes and search for them in the lookup table and the result of that search give the EPGID. This search is fact because lookup table is indexed on the hashes.

With version 4.5, the Importer is given the EPGID by Zap2it, so the Importer will check that the Zap2it EPGID and the computed EPGID are the same, if they are not, the Importer will correct its lookup table. As such, it will do a search through the lookup table by EPGID and it is taking some time.

However the more you'll use the Importer, the less "new" shows it will encounter and the less the Importer will be forced to correct the lookup table. So normally (If I have done things right the Import time should be the same as before as time goes by).

Do you know if Zap2it is the only sources for XMLTV data for North America?

No

Thanks in advance for any info you can throw my way. I can always just start the process earlier so I'm just fine. But I'm always looking for ways to improve how my system works.

Well if you want to speed the process the only thing that can be done is : only download data for "Today" and "Last Imported Day + 1".

For instance on monday you have just imported 7 (Mon, Tue, Wen, Thu, Fri, Sat, Sun) days of program guide. The next day only download Zap2it data for Tuesday and next monday, and so on (On Wednesday, only import for Wed and next Tue). This will cut down both download and import times.

Regards,
Stéphane.

[Maddog]

I don't seam to get this XMLTV thing, I need serious help I have read most of this topic, but no light yet

I have BeyondTV 3.50, and have tried the guide for installing XMLTV: http://www.lmgestion.net/@en-us/4/22/20/21/article.asp

From StephaneM's five point, I will ask my questions:

1) Install BTV 3.5 --this is done
2) Grab XMLTV data --I downloadet two files: xmltv and xmltv_prereq . But I dont know where to put them...
3) Install XMLTV Importer and import your XMLTV data into BTV --tried this, but I dont know where my XMLTV file are located...
4) Configure BTV to use the new lineups imported --not there yet..
5) Install the Web guide (after installing IIS through Add/Remove Programs - Add/Remove Windows Components in Control Panel). --IIS installed..

So basicly I am stuck with what to do with the grabber files. If I execute the xlmtv.exe, there will only pop up a dos window a brief moment... AND where is this xmltv file located???

Hoping for a helping hand here.

[StephaneM]

Hello,

Originally Posted by Maddog

If I execute the xlmtv.exe, there will only pop up a dos window a brief moment... AND where is this xmltv file located???

The XMLTV package is a command line utility. So you have to open a Windows command line to run it (Start -> Run -> type "cmd" (without the quotes).

I will suppose you uncompressed the XMLTV package to c:\XMLTV (that is xmltv.exe is in C:\XMLTV\xmltv.exe)

First type in the commande line: cd c:\xmltv
this will change current directory to the directory where xmltv.exe is located, then type: xmltv.exe tv_grab_xx --configure
this will enter the configuration stage for the grabber, you'll have to replace xx by the country code, for instance tv_grab_fr (France), tv_grab_na_dd (North America) etc..

When the configuration stage is done, you'll grab data by typing:
xmltv.exe tv_grab_xx --output epg.xml (this is also the command line that you can put in the Importer itself so as it will download subsequent tv listings itself)

This will download program guide and put the XMLTV file (epg.xml) in C:\XMLTV\epg.xml

You can review xmltv.exe documentation in the c:\xmltv\doc\man directory.

Regards,
Stéphane.

[Maddog]

Thanks that clearing things for me, StephaneM. I will try that out after I have reinstalled windows.

Thanks again for your fast replay on my problem!!

[fragmental]

I did everything according to the instructions, but when I check web admin the lineup is not available. No where to be found.]

When I check the log it says

[2006/9/11 12:24:03] Guide Update started.
[2006/9/11 12:24:07] Guide Updater Service: Initializing...
[2006/9/11 12:24:07] Guide Updater Service: Getting data from Third Party Source...
[2006/9/11 12:24:07] Guide Updater Service: Processing new data...
[2006/9/11 12:24:08] Guide Updater Service: Setting base channel lineups...
[2006/9/11 12:24:09] Guide Updater Service: Processing data complete. Loading new data...
[2006/9/11 12:24:12] Guide Updater Service: New guide data loaded. Clean update complete.
[2006/9/11 12:24:12] The source "Antenna (Analog) - Mercury TV Card WDM Video Capture" is associated with lineup "2a2ac77e-c0fc-48b9-8db6-060eec20512c", but that lineup could not be found.
[2006/9/11 12:24:12] Defaulting the source "Antenna (Analog) - Mercury TV Card WDM Video Capture" to the first available lineup.

[StephaneM]

Hello,

Originally Posted by fragmental

I did everything according to the instructions, but when I check web admin the lineup is not available. No where to be found.

You'll see this behavior sometimes when deleting a lineup or when changing a lineup.

This is because the Importer is setting the lineup for BTV (to avoid you to use the WebAdmin and circumvent a BTV bug with some video capture cards).

Usually this is not a serious error and you can ignore it or go to the WebAdmin to reset the lineup for the video source (change it to another, apply and change it back).

Regards,
Stéphane.

[fragmental]

BTV only has 5 days worth of listings. I don't that the btv xmltv import program is running the xmltv.exe application to update the programs. There doesn't seem to be a lot of information about the grabber arguments section, so maybe I have that wrong? I wondered if maybe it had something to do with the the xmltv import program not finding the config file for xmltv.exe or something?

I'm trying to check to see if I can download more listing manually from the command line, right now, but I seem to be having problems. If I can get it to work, I will let you.

Edit: yeah that worked.

[StephaneM]

Originally Posted by fragmental

I don't that the btv xmltv import program is running the xmltv.exe application to update the programs. There doesn't seem to be a lot of information about the grabber arguments section, so maybe I have that wrong? I wondered if maybe it had something to do with the the xmltv import program not finding the config file for xmltv.exe or something?

You're right, there is no informations about the grabber arguments because it is grabber specific.

Anyway here is a few tips. Usually you have to launch the grabber with a command line, for instance : xmltv.exe xmltv_grab_fr --output epg.xml

The command arguments here are : "xmltv_grab_fr --output epg.xml" (without the quotes).

Also note that the grabber is launched from it's own directory (the "Start in" directory) so it will find the config file for xmltv.exe and all other files relative to the xmltv.exe directory. For instance "--output epg.xml" tells the grabber to save the XMLTV data into epg.xml, this file will be saved in the xmltv.exe directory (because the Importer start the grabber from it's directory).


Regards,
Stéphane.

[Maddog]

Originally Posted by StephaneM

Hello,

The XMLTV package is a command line utility. So you have to open a Windows command line to run it (Start -> Run -> type "cmd" (without the quotes).

I will suppose you uncompressed the XMLTV package to c:\XMLTV (that is xmltv.exe is in C:\XMLTV\xmltv.exe)

First type in the commande line: cd c:\xmltv
this will change current directory to the directory where xmltv.exe is located, then type: xmltv.exe tv_grab_xx --configure
this will enter the configuration stage for the grabber, you'll have to replace xx by the country code, for instance tv_grab_fr (France), tv_grab_na_dd (North America) etc..

When the configuration stage is done, you'll grab data by typing:
xmltv.exe tv_grab_xx --output epg.xml (this is also the command line that you can put in the Importer itself so as it will download subsequent tv listings itself)

This will download program guide and put the XMLTV file (epg.xml) in C:\XMLTV\epg.xml

You can review xmltv.exe documentation in the c:\xmltv\doc\man directory.

Regards,
Stéphane.

Thanks Stephane, i've got it

But I have still an issue... There is no picture or sound, only channel id and EPG is displayed in BeyondTV.

In order to get picture and sound again I have to open Beyond TV Web Admin (http://localhost:8129 ), select in "Configure", "edit a video source" my "ATI AVStream Analog Capture Lineup - 1" instead of "Snapstream.Net"

I'am a little confused here
Appreciate further assistance