[plugin] XMLTV Importer

[FoxBat]

Originally Posted by avidav

I always get the following error message: "Unable to start Beyond TV update: Unable to connect to BTV Web Admin (Server was unable to process request)".
...
Heeeeelp!

I am sorry to hear this - please post the .reg files you was using. I still have mine, it is possible that we can locate the error to this file.

[avidav]

OK, I have it sorted out now.
1) The error message regarding not being able to connect to the Web Admin: I uninstalled the whole BTV application, manually removed all remains from the registry, deleted the installation folder and just did a clean fresh install from scratch, after which all worked again.
2) Not having the Program Guide in the BTV main menu: I did some reading on this and other threads and then I realized that I was trying to import data which was all in the past; in other words I did not do a new xmltv grab but used the file already there and which was all in the past. Hence no data was imported into the BTV database, hence no Program Guide button.
After doing a fresh grab and import, the button showed up in all its glory, and there was actual data inside.
And the channels were ordered as I wanted them :-)
Stephane - just a thought: maybe you could, in one of the future versions of the importer, issue a warning if there is no data imported to BTV.
And, by the way, your program is superb and your willingness to help is even more so, so I am on my way to make a donation. Good work!

[Ivo-G]

Stephane, I'm having the following issue:

Certain shows which are clearly in the XMLTV file do not get imported into BTV's EPG. When I check 3rdPartyGuideData\Programs0.xml they just do not appear in there at all. In the EPG in BTV the empty timeslots show as Unavailable. Shows before and after are fine. Ths is happening with different channels, shows and times.

I even tried to delete (after backing them up) all of the files in 3rdPartyGuideData, the Lookup.xml in the BTVXMLTVImport directory, and SS_BTV_DB.db, basically starting from scratch. I just can't get the shows in. What can I do? can you help me?

Ivo

[StephaneM]

Hello Ivo,

Originally Posted by Ivo-G

Certain shows which are clearly in the XMLTV file do not get imported into BTV's EPG.../...Shows before and after are fine. Ths is happening with different channels, shows and times.

This could happen when there is an overlap. I will issue a new version of the Importer (4.2) within few days, it may adress this issue.

However I don't think that you are severly affected. In the worst case you can have a whole day missing. Basically, if your XMLTV grabber is cumulative, that is when your run your grabbber, it will add or replace programs but not necessarily delete old programs, you can have many programs that are in conflicts between each other.

When the Importer detect a conflict, it will delete data already in program guide so as there is no overlap at all.

Depending on the order conflicting shows are imported, it can completely erase a time slot. For instance you have three consecutive shows in your XMLTV file :

A : 20:00-22:30
B : 22:30-22:35
C : 22-35-23:30

and

A1 : 20:00-22:35
B1 : 22:35-22:40
C1 : 22:40-23:35

A and A1 / B and B1 / C and C1 are the same show, but A,B,C where grabbed on week1 and A1,B1,C1 grabbed on week2. The diff between week1 and week2 is a 5min sliding.

The shows normally are imported sorte by start time, so it gives.

A is imported
A1 is imported, deleting A
B is imported, deleting A1
B1 is imported
C is imported, deleting B1
C1 is imported, deleting C

So you will only have the show B and C1.

There is nothing much that can be done to adress this problem in the Importer itself.

Regards,
Stéphane.

[jbartlett777]

Originally Posted by StephaneM

A is imported
A1 is imported, deleting A
B is imported, deleting A1
B1 is imported
C is imported, deleting B1
C1 is imported, deleting C

So you will only have the show B and C1.

There is nothing much that can be done to adress this problem in the Importer itself.

Does the XMLTV feed have a unique id for each show? Instead of referencing by the time slot, you can reference by it instead. This should resolve this overlap conflict since you would only get "A".

[StephaneM]

Originally Posted by jbartlett777

Does the XMLTV feed have a unique id for each show?

No.

Even though, this not the Importer job to decide which one between A or A1 that is the good show to Import, this is something that only the grabber can decide (because it knows in that case that A1 is the new schedule for A).

And the Importer must remove all conflicts during the import process. Otherwise the program guide is a little funny.

Regards,
Stéphane.

[Ivo-G]

Problem is that I get this behaviour even when I start from scratch, with a blank 3rdPartyGuideData directory, no SS_BTV_DB.db, no lookup.xml. My XMLTV only has one reference to each show, and I even tried to import a single day XMLTV versus a 7 day version.

The shows are missing from the files in 3rdPartyGuideData, aren't the files in this directory produced by BTVXMLTV Importer before importing into BTV?

And yes I'm severly affected, as some of my wife's (and mine) favourite shows are being deleted at the moment, which could have an adverse effect on my wellbeing

[StephaneM]

Originally Posted by Ivo-G

Problem is that I get this behaviour even when I start from scratch, with a blank 3rdPartyGuideData directory, no SS_BTV_DB.db, no lookup.xml.

I was referring to conflicts inside the XMLTV file (wether it is the same show or not). So deleting previous imported data has no effect, the Importer is designed to handle conflicts between imports (as opposed to conflicts within the XMLTV data itself).

The shows are missing from the files in 3rdPartyGuideData, aren't the files in this directory produced by BTVXMLTV Importer before importing into BTV?

Yes.

However conflicts within the XMLTV file is one of the possibility, it may be something else. Try the new version of the Importer and see if it resolve the issue. If not, send me your XMLTV file to lmgestion@hotmail.com and tell me what show is not imported (Title and air date, don't forget to tell me the time offset you are using).

Regards,
Stéphane.

[Ivo-G]

Thanks for your offline help Stephane. You've made me realize that the problem is indeed with the XMLTV file, as the source I'm getting it from thinks a show is starting before the one before has ended.

But couldn't this detection be solved in BTVXMLTV Importer? You're already detecting when this happens, couldn't you collate the conflicts and show them in a screen where the overlap can be resolved manually, or by setting a preference for which show should win?
If the overlap is say 5 minutes, why not cut the difference down the middle, or allow the user to say "Well, give the 2nd show the 5 minutes"?

Think about it.

Thanks again for a great tool, I think I'm going to donate some more!

[StephaneM]

Originally Posted by Ivo-G

But couldn't this detection be solved in BTVXMLTV Importer? You're already detecting when this happens, couldn't you collate the conflicts and show them in a screen where the overlap can be resolved manually, or by setting a preference for which show should win?

I don't think so: usually the import is unattended, so asking the user to choose is a bad idea (wether to choose how to handle the conflicts or assigning a preference etc...) this is just too confusing and time consuming.

If the overlap is say 5 minutes, why not cut the difference down the middle, or allow the user to say "Well, give the 2nd show the 5 minutes"?

This trick could work for the precise situation you describe, but what to do when a show conflicts with two, three, four shows...

I personally already had conflicts such as this one, and always fixed XMLTV grabber to resolve it (It's easy for me because I have my own grabber), but this is the way to do it, and not confusing the "genres":

* Grabber is responsible for the data : accuracy and formatting (to XMLTV, compliant with the DTD)

* Importer is responsible for the transformation of the data for BTV

* BTV is responsible of handling the data (loading and presenting)

I can eventually think about additionnal software dedicated for this task, but not included in the Importer.

Regards,
Stéphane.

[Ivo-G]

Originally Posted by StephaneM

This trick could work for the precise situation you describe, but what to do when a show conflicts with two, three, four shows...

Any method that still kept the show in would be preferable IMO to tossing out the whole show and leaving an Unavailable slot. I can't program in an Unavailable recording, whereas if I still see the show I could work with the early start / late stop recording control to still get it recorded properly. Shows recorded through a Record All (New) job would still hit my radar, and I could deal with the problem.

Originally Posted by StephaneM

I can eventually think about additional software dedicated for this task, but not included in the Importer.

That would also work indeed. I my particular case I found out that the web site I'm using had the wrong times, even on the webfront, so the XML file grabbed also had the mistake.

[StephaneM]

Originally Posted by Ivo-G

Any method that still kept the show in would be preferable.../...I can't program in an Unavailable recording

I'm aware of the consequences, but this doesn't change the root of the issue and how it can be handled.

It's all a matter of context. In the Importer context: shows are not related to each other by any rules, specifically regarding conflict. Because at the moment the Importer detects a conflict it cannot know if the conflict is with two or more newly grabbed shows (so from the same XMLTV file) or with show that comes from more than one XMLTV file.

And that alone make a great difference in term of handling the situation. But there are other factors: are the conflicting shows following each other, how are there interleaving etc...

For instance for resolving the conflict you had, the best method would be to do prior to the import a linear scan of programs and compare the start of program n+1 and the stop of program n, if they don't match, then change stop of program n+1 to be start of program n (or the opposite). This simple scan, would make the import three times longer than it is actually and will only deals with one type of conflicts.

I my particular case I found out that the web site I'm using had the wrong times, even on the webfront, so the XML file grabbed also had the mistake.

This is exactly why I say that the grabber should take care of these issues. Because that problem is an easy one. The grabber works sequentially : one program after another, channel by channel. So in the grabber context, it easy to notice that show n+1 start before show n ends, and correct the error without even slowing the grabbing process. Moreover by doing this the grabber cannot make mistakes, because in his context this is always true.

My grabber do this and even more because I grab more that one web site and so there is in the grabber a very strong algorithm to hande conflict resolution : by time and show content. And all of this is done without impacting the speed of the grabbing process. And even if I wanted to apply the same conflict resolution in the Importer I would not be able to because the original grabbing context is lost.

I hope that now your understand better the situation. But I've plans to alert you in case of conflicts or error and you'll then be able to handle the situation with another tool.

Regards,
Stéphane.

[Ivo-G]

Have you released your grabber for public use? Have any plans to do so if you haven't?

[StephaneM]

Originally Posted by Ivo-G

Have you released your grabber for public use? Have any plans to do so if you haven't?

No it is not released, neither it will be. It is not a all purpose/web site grabber, it is only a script so it really can't be.

[Ivo-G]

Stéphane,

The latest version has the log conflicts to Eventlog feature. Thanks for that.
Is there any way you could do an optional Log to XML format feature, and allow the user to choose which logging method (if any) to use?

Greetings, Ivo