Quantcast
Channel: Kodi Community Forum - Scrapers
Viewing all 171 articles
Browse latest View live
↧

need help scaping metadata text files

November 22, 2015, 8:53 am
$
0
0
I am switching from using my tired Tivo series 3 to using Kodi. I have been struggling to create a scrapper that will read metadata stored in text files for each of the TV shows I have archived from my the Tivo. For some backgroung, the text file is generated by a program called KMTTG that downloads from the Tivo and uses a third program called pyTivo to generate the metadata file with the same filename but ending in txt. Unfortuanely, althought the data in it has a lot in common with an NFO file, it is not in the NFO format, but rather is a list of name/value pairs separated by a colon (see below).

title : Sample Title
seriesTitle : Sample Title
description : As something happens deep down in the Midnight Zone, something else happens somewhere else..
time : 2015-02-18T00:02:00Z
mpaaRating : G1
isEpisode : true
iso_duration : PT16M
episodeTitle : The Episode title goes here.
isEpisodic : true
showingBits : 1
tvRating : x3
displayMajorNumber : 22
callsign : ABC2/ABC4
seriesId : SH0000262601
programId : EP0000262601-0004454490
vProgramGenre : Animation
vSeriesGenre : Animation
vSeriesGenre : Interests

I have been reading the scrappers wiki at "http://kodi.wiki/view/HOW-TO_Write_Media_Info_Scrapers" and other posts on this forum, such as from PKO66 (http://forum.kodi.tv/showthread.php?tid=36422). It looks to me like it shouldn't be too difficult to put together a scrapper to parse the metadata text file and populate the library with at least the Title and plot. Maybe it isn't too hard, but I can't get it to work. The logic I worked on was that it was much like scrapping an NFO file. This is what I have tried to do:
1. Change the extension of the text file to .NFO
2. Make a scrapper XML with an NFO section that simply sent everything to the getdetails section.
<NfoUrl dest="3">
<RegExp input="$$1" output="\1" dest="3">
<expression noclean="1"/>
</RegExp>
</NfoUrl>
3. In the getdtails section make the regex convert the name:value pairs to generate the listing of detailed information in the correct format as detailed in section 1.3 on HOW-TO_Write_Media_Info_Scrapers (wiki)

I have used ScraperXMLEditor3_6_9_5 to test the regex and feeding in the appropriate test data it looks like I am getting a good output from the GetDetails section... so I think my regex there is ok. What I suspect i going wrong is that the NFO section of the scraper does not like that the content of the file is not in the correct NFO format and is not parsing it to the GetDetails section.

This looked like it would be an elegant solution, but it doesn't appear to be working.

Any thoughts or recommendations?

Regards,
David
↧
Search

Adding custom property

November 27, 2015, 6:53 pm
$
0
0
Hi,

I am not sure if this is the right section of the forum. Basically what I want to do is adding some kind of custom property to my *.nfo or movie.xml file and it should show up on the Kodi UI. How would I accomplish this? At the moment I am using Media Center Master to sort my library and using only local movie info in kodi. Adding the property to the file is not the propblem, but I guess I would have to alter the scraper which is paring the *.nfo or/and movie.xml file and the skin to actually show me what I want, am I right? Is there any documentation on either of these two things?


Sorry for my bad English, hopefully you understand, what I am trying to do and can point me in the right direction. Thanks.
↧
↧

Custom Function in built-in scraper?

November 28, 2015, 10:39 am
$
0
0
Hi everyone,

I'm trying to adapt the built-in tmdb scraper to better handle my files.
So far it seems to work when I test it with ScraperEdit (except for that it terminates with an empty warning. But it does that with the untampered one as well). But modifying the existing one seems kind of dirty, I would very much like to just have my own scraper which calls the tmdb scraper.
Or if that isn't possible at least keep the functions separate, so that I only need to include them in the tmdb scraper.

So far I tried the latter. I have added an import statement to the addon.xml of the tmdb scraper
Code:
<requires>
    <import addon="xbmc.metadata" version="2.1.0"/>
    <import addon="metadata.common.imdb.com" version="2.7.8"/>
    <import addon="metadata.common.themoviedb.org" version="2.13.1"/>
    <import addon="plugin.video.youtube" version="4.4.10" optional="true"/>
    <import addon="metadata.myextract" version="1.0"/>
</requires>

and added the following RegExp to it's CreateSearchUrl

Code:
<CreateSearchUrl dest="3">
    <RegExp input="$$1" output="&lt;url&gt;http://api.tmdb.org/3/search/movie?api_key=57983e31fb435df4df77afb854740ea9&amp;amp;query=\1&amp;amp;year=$$4&amp;amp;language=$INFO[language]&lt;/url&gt;" dest="3">
        <RegExp input="$$2" output="\1" dest="4">
            <expression clear="yes">(.+)</expression>
        </RegExp>
<!-- added by me ------->
        <RegExp input="$$1" output="&lt;chain function=&quot;GetTitleFromFilename&quot;&gt;$$1&lt;/chain&gt;" dest="1" >
            <expression encode = "1" />
        </RegExp>
<!---------------------->
        <expression noclean="1" />
    </RegExp>
</CreateSearchUrl>
I pretty much copied that from the other function calls in the file. What I don't understand yet, is what the buffer after the function="..." part is for?
Also I'm not quite sure how the buffers transfer from inside the function call to outside. As far as I understand they are separate and no matter what destination I set in the function definition it will be written into the destination set in the function call?

I have tested this with ScraperEdit. It just seems to put the whole function call into the search url, which of course won't work.
Below is my "library", again pretty much copied existing ones. Seemed straight forward, but maybe I still made a mistake?

Any help is appreciated!

The library is defined in %AppData%\Roaming\Kodi\addons\metadata.myextract\addon.xml :
Code:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<addon id="metadata.myextract"
       name="My title extractor"
       version="1.0"
       provider-name="kaeptnjoker">
  <requires>
    
  </requires>
  <extension point="xbmc.metadata.scraper.library"
             language="en"
             library="myextract.xml"/>
  <extension point="xbmc.addon.metadata">
    <summary lang="en">My title extractor</summary>
    <description lang="en">Extracts movie title from filename.</description>
    <platform>all</platform>
    <license>GPL v2.0</license>
    <forum></forum>
    <website></website>
    <email></email>
    <source></source>
  </extension>
</addon>


The function itself is in %AppData%\Roaming\Kodi\addons\metadata.myextract\myextract.xml :
Code:
<?xml version="1.0" encoding="UTF-8">
<scraperfunctions>

    <GetTitleFromFilename dest="5">
        <RegExp dest="5" output="\3+\1" input="$$1">
            <RegExp dest="1" output="\1 " input="$$1">
                <RegExp dest="1" output="\1" input="$$1">
                    <expression>^([^_]+)_(xvid|divx|h264|mpg1|mpg2|spark)(_\d){0,1}((_(de|en|fr|ja|zh|fi|es|hu|da|it|r​o|sv|no|pl|hs|ma|zz)\.[0-6])+)(_sub)*\.(mkv|avi|mpg|mp4)$</expression>
                </RegExp>
                <expression repeat="yes">([^.]+(\.\.\.)?(\.(?=\.))?)</expression>
            </RegExp>
            <expression encode="1">^([^,]*)(,(.*))? $</expression>
        </RegExp>
    </GetTitleFromFilename>
    
</scraperfunctions>
↧

Scraper JustUseTheFileName stopped working since V15.x

December 11, 2015, 10:46 pm
$
0
0
Hi Everyone,

I have been using the "JustUseTheFileName" scraper to scrape my various mixed downloaded content consisting of Movies & TV Shows.

It worked perfectly in Kodi V14.2 and earlier by simply building a list of content which I could sort by descending date added order under the Movies menu

Ever since I upgraded to v15.2, the scraper stopped working. It just scans the library for new content and appear to be doing its job but the Movies menu is gone and no content is available.

The problem persist in v16.0 beta3

Does anyone know why or how I can fix this?

Thank you
↧

Revised: parse selected result in $$2?

December 18, 2015, 5:51 pm
$
0
0
My last question was worded very badly. Let me try this.

Upon entering GetDetails I want to parse the selected <entity>(url) chosen from the list returned by GetSearchResults. Using ScraperEdit I see the selected <entity>(url) is in $$2.

Using ScraperEdit all works well. $$2 is the input, run the regex on it, and it moves what I need to the buffer I designated.

Using Kodi not so well. $$2 as the input, use (.+) as regex, returns the full list of all <entity> found in GetSearchResults function. I used (.+) as a test so I could see what was actually being input/parsed from $$2 in the Kodi debug log.

It does not appear that Kodi puts the selected <entity> in $$2.

Is there a way to get the selected <entity> and parse it?

I have tried many things and can't get it to work. I test with only one regex in GetDetails that would attempt to make this parse.

The reason I need this is so GetDetails runs it function based on the selected <entity>. The url in the selected <entity> contains the ID I need to parse/use in following CustomFunctions.

I've read thru the forum and can't find what I am asking.

Thanks!
↧
↧

url function question

December 22, 2015, 8:41 pm
$
0
0
Why is it that if a url function precedes a custom function, the details are returned for the function preceding that url function until another custom function is run? i.e.: details for GetAlbumInfobyREID are returned in GetMBWorkID and GetFanartvThumbbyMBrgid. This displays <studio> multiple times for example.

In the scraper I wrote here if I move GetAlbumThumbCAAbyRGID to the bottom of the tree, the following url functions add details from GetAlbumInfobyREID twice, once for GetMBWorkID and again for GetFanartvThumbbyMBrgid.

It seems that a url function is not completed unless it is at the end of the function list.

Hope this makes sense.

I could produce a debug to show this if needed. But it is more than probable that I just don't understand the logic and I need to end the url function somehow. Smile
↧

which xml file set the content of an media source

January 14, 2016, 5:11 pm
$
0
0
Hallo,

im looking for the xml File where you can find the configuration for each media source an his content for the right scraper.
Can find it

any ideas

lantoeter
↧

IMDB Scraper - TV Information

January 28, 2016, 8:13 am
$
0
0
I love Universal Movie Scraper because it pulls all the info from IMDB which is accurate. I couldn't find any TV Scraper that gets info from IMDB, and numerous TV Shows have wrong episode numbering on tvdb and themoviedb. I even made an account over there to fix the info but most of it is locked and the mods are no help too. For example, most of the kids TV shows that I have for my kids, mostly Disney Junior shows, have two stories in one episode. On IMDB they are correctly labeled as per official websites and DisneyWiKi etc. However, on tvdb and themoviedb each story is labeled as separate episode, which makes a season with 25 episodes on IMDB to be 50 episodes on tvdb. Shows like Little Charmers, Doc Mcstuffin etc. So my question is: Is there any Add-on / Scraper for TV Info from IMDB?
↧
Search

TVDB Scraper 1.8.3 gives incorrect information

January 31, 2016, 1:45 pm
$
0
0
Hi,

When scanning my TV series TVDB Scraper 1.8.3 gives incorrect information.
When I revert back to version 1.8.3, the scraper works correctly.
Is this a known issue?
↧
↧

Modifying TVDB scraper to get Episode Thumbs from The Movie Database (TMDB)

March 1, 2016, 1:10 pm
$
0
0
Hi Guys,

I'm trying to modify the TVDB scraper from the Kodi repo to scrape the episode thumbnail from The Movie Database (TMDB) as a substitute, with all other data to remain being scraped using the TVDB service as normal.

I've been reading up on the very helpful Wiki guide: HOW-TO:Write_media_scrapers (wiki) but unfortunately, I can't seem to make any sense of the part explaining how to get information from elsewhere (Chapter 3)

Would anyone be able to tell me how I can achieve this or, even better, post a modified version to achieve this?

In the TVDB xml, line 304 is the relevant part for episode thumbs.

I would really appreciate if anyone knowledgeable in scraper development is willing to donate a few mins of their time to advise me on exactly how to achieve this, bearing in mind I've no experience of scraper writing and base all my knowledge on this topic from reading the Wiki page.

Many thanks! WinkBig Grin


EDIT: I'll be using my own API key, so no need to worry about that part.
↧

parse json

March 3, 2016, 11:46 pm
$
0
0
Hi all,
json code: from link https://inet-publicapi.mw.elion.ee/dtv-a...minutv-web

Code:
[{"version":"2.0","data":{"context_objects":{"playables":{"channel_1":{"id":"channel_1","commercial_free":false,"consumed":null,"min_pos":0,"max_pos":null,"object_id":1,"object_type":"channel","start_pos":0,"streams":[{"audio_tracks":[{"type":"audio","language":"est","format":null,"pid":null,"source":null},{"type":"audio","language":"nld","format":null,"pid":811,"source":null}],"bitrate":null,"duration":null,"sources":["http://hls-streamer-1-2.minutv.ee/7edb718421e48c9d7104b5080c80970a_1457087825web/input/ch01/index.m3u8","http://hls-streamer-1-1.minutv.ee/7edb718421e48c9d7104b5080c80970a_1457087825web/input/ch01/index.m3u8"],"text_tracks":[],"type":"hls","valid_until":"2016-03-04T12:37:05+02:00","video_tracks":[{"resolution_type":"sd"}],"uiName":null}],"stream_type":"continuous"}},"channels":{"1":{"features":{"net_recordable":true,"catchup":true},"icons":[{"type":"h40pxw90px","url":"h40pxw90px/1.png","md5":"3ea047acbaca857d9bef3af7e2280d80"},{"type":"90x40","url":"90x40/1.png","md5":"3ea047acbaca857d9bef3af7e2280d80"},{"type":"72x49","url":"72x49/1.jpg","md5":"31fd5c0b0f7021fefb3ce19d972047c6"},{"type":"87x46","url":"87x46/1.png","md5":"bf920bb81e8123b513be561c3e10222c"},{"type":"90x40_white","url":"90x40_white/1.png","md5":"fd8e7c0cbdc7c98329e505922e702551"}],"id":1,"packet_ids":[1,12,13,14,15,30,34,35,36,38,44,46,129,130,162,163,166,167,181,182,190,191,224,2​25,233],"streams":[{"audio_tracks":[{"type":"audio","language":"est","format":null,"pid":null,"source":null},{"type":"audio","language":"nld","format":null,"pid":811,"source":null}],"bitrate":null,"duration":null,"sources":[],"text_tracks":[{"type":"subtitles","language":"est"},{"type":"subtitles","language":"rus"},{"type":"subtitles","language":"ned"}],"type":"multicast","valid_until":null,"video_tracks":[{"resolution_type":"hd"}],"uiName":null},{"audio_tracks":[{"type":"audio","language":"est","format":null,"pid":null,"source":null},{"type":"audio","language":"nld","format":null,"pid":811,"source":null}],"bitrate":null,"duration":null,"sources":[],"text_tracks":[{"type":"subtitles","language":"est"},{"type":"subtitles","language":"rus"},{"type":"subtitles","language":"ned"}],"type":"multicast","valid_until":null,"video_tracks":[{"resolution_type":"sd"}],"uiName":null},{"audio_tracks":[{"type":"audio","language":"est","format":null,"pid":null,"source":null},{"type":"audio","language":"nld","format":null,"pid":811,"source":null}],"bitrate":null,"duration":null,"sources":["http://hls-streamer-1-2.minutv.ee/7edb718421e48c9d7104b5080c80970a_1457087825web/input/ch01/index.m3u8","http://hls-streamer-1-1.minutv.ee/7edb718421e48c9d7104b5080c80970a_1457087825web/input/ch01/index.m3u8"],"text_tracks":[],"type":"hls","valid_until":"2016-03-04T12:37:05+02:00","video_tracks":[{"resolution_type":"sd"}],"uiName":null},{"audio_tracks":[{"type":"audio","language":"est","format":null,"pid":null,"source":null},{"type":"audio","language":"nld","format":null,"pid":811,"source":null}],"bitrate":null,"duration":null,"sources":[],"text_tracks":[],"type":"multicast","valid_until":null,"video_tracks":[{"resolution_type":"pip"}],"uiName":null}],"title":"ETV"}}},"playable_id":"channel_1","channel_id":1},"error_code":null,"handler":"channel.start_playing","message":null,"message_type":null,"success":true}]

pls help me to parse json to kodi items = list()
where should i start to get "title" and "source" key to list?

Br,
Andris
↧

Issue with movie scraper, and returning title

March 15, 2016, 12:55 pm
$
0
0
Hi..
I have had this issue for some time now, but haven't gotten to look into it before now, cause I am about to start from scratch again after a house rebuild.

I have added my movies, and it have startet to scan them, and everything goes fine.

Then suddenly it finds a title from 1977, and it just resubmits it to the movielist over and over and over, so I think I have over 100 entries of this movie title that I do not even got..

Some others probably have had the same issue, but didn't know what to search for.

Hope that someone can help me solve my issue

Thanks in advance
↧

HTTP headers ignored

March 20, 2016, 12:39 pm
$
0
0
Hello I am trying to set header information in a url using that information: HTTP (wiki) . I looked up on debug log information and there is no message about http headers.

Here is an example about the url that I am trying to use:
Code:
http://s15.vimple.ru:8081/vv68/730508.mp4?v=3296a293-7b29-41b5-803a-204e5e0ef691&t=635941098541617500&d=6116&sig=ef145d9f558ee3e57dcde6b668c2c769|DNT=1&connection=keep-alive&Accept-Language=en-US,en;q=0.5&accept-encoding=gzip,deflate&Cookie=UniversalUserID=dfcc0ee95aca4db5b416626f4d0dcc88&Referer=http://player.vimple.ru/iframe/3296a2937b2941b5803a204e5e0ef691&Accept=video/webm,video/ogg,video/*;q=0.9,application/ogg;q=0.7,audio/*;q=0.6,*/*;q=0.5&user-agent=Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:44.0) Gecko/20100101 Firefox/44.0

I tested this url on pc it works. I spent a week looking the reason.
EDIT: How do you pass cookies to kodi?
Any help would be appreciated. Thank you.
↧
↧

local db scraper

March 21, 2016, 11:14 am
$
0
0
Hello,

I'm trying to create a scraper to read from a local db, and I'm having some issues.

Here's a bit of background first... I want to use anidb to look up anime shows and movies, but I immediately get banned when I use the provided anidb scraper. So instead I wrote a program to look up show data for given files and store it locally, then wrote another program which serves this data in a kodi-friendly way. The final step is writing a kodi scraper to read from this local db.

However, I'm having absolutely no luck getting this to work.

For instance, if I have a file called "ninja scroll (bluray - 1080p).mkv", I would look up the data using "http://linux-media:8080/animelookup/anime?filename=ninja scroll (bluray - 1080p)", where linux-media is a computer on my lan specified in /etc/hosts. This returns:

Code:
<?xml version="1.0"?><details><title>Juubee Ninpuuchou</title><year>1993</year><rating>6.2</rating><thumb>http://img7.anidb.net/pics/anime/42966.jpg</thumb><poster>http://img7.anidb.net/pics/anime/42966.jpg</poster><plot>Feudal Japan ? a time of danger, intrigue and deception. Kibagami Juubee is a masterless ninja who travels the land alone, hiring his services to those with gold... or a worthy cause. His fearsome abilities have served him well, but a hideous plot to overthrow the government threatens to end his wandering ways and possibly his life. When a small village succumbs to a terrible plague, a team of ninjas are sent to investigate, and realize that all is not as it seems. Ambushed, they are wiped out by a terrifying man-monster with incredible powers, leaving only one alive ? Kagero, a beautiful female ninja, whose touch can bring instant death. Juubee saves her from a fate worse than death, and unwittingly becomes drawn into the web of treachery. He is soon faced with his greatest challenge, an enemy for whom death holds no fear...</plot><genre>violence</genre><genre>samurai</genre><genre>past</genre><genre>ninja</genre><genre>martial arts</genre><genre>action</genre><genre>Japan</genre><genre>Asia</genre><genre>nudity</genre><genre>horror</genre></details>

My scraper looks like this:

Code:
<?xml version="1.0" encoding="UTF-8"?>
<scraper framework="1.1" date="2013-04-04">
    <NfoUrl dest="3">
        <RegExp input="$$1" output="\1" dest="3">
            <expression />
        </RegExp>
    </NfoUrl>

    <CreateSearchUrl dest="3">
        <RegExp input="$$1" output="&lt;url&gt;http://linux-media:8080/animelookup/anime?search=\1&lt;/url&gt;" dest="3">
            <expression/>
        </RegExp>
    </CreateSearchUrl>

    <GetSearchResults dest="8">
        <RegExp input="$$1" output="&lt;?xml version=&quot;1.0&quot; encoding=&quot;iso-8859-1&quot; standalone=&quot;yes&quot;?&gt;&lt;results&gt;&lt;entity&gt;&lt;title&gt;\1&lt;/title&gt;&lt;url&gt;http://linux-media:8080/animelookup/anime?filename=\1&lt;/url&gt;&lt;/entity&gt;&lt;/results&gt;" dest="8">
            <expression>.*&lt;body&gt;(.*)&lt;/body&gt;.*</expression>
        </RegExp>
    </GetSearchResults>

    <GetDetails dest="3">
        <RegExp input="$$1" output="\1" dest="3">
            <expression>.*&lt;body&gt;(.*)&lt;/body&gt;.*</expression>
        </RegExp>
    </GetDetails>
</scraper>

I've edited advancedsettings.xml to add this line to the tvshowmatching tag (prepend):

Code:
<regexp>([^\/]+)$</regexp>

This isn't working and I'm not sure why. Kodi's logfile says there was an error processing the xml. I'm sure it's something really noobish. I can change the output of my server if that's complicating things.

Does anyone have any suggestions for me? Any help would be appreciated. Thanks.
↧

Scaper issue

March 23, 2016, 12:37 pm
$
0
0
I have the Mini Series Hatfield & McCoys starting Kevin Costner, Not listed from the MovieDb Scraper.It only list the Original 1975 and Christian Slater 2012 Verison, But when I go online to there site Its there. Any Idea Thanks
↧
Search

Site scrape

March 24, 2016, 1:08 am
$
0
0
I want to add this and this websites to an add-on, who can help me with scrape these websites?
↧

scrapers - vagalume

March 29, 2016, 4:58 pm
↧
↧

Scrapping youtube - python 2.7.x

May 3, 2016, 1:05 pm
$
0
0
Hi guys,

I wish to create an addon that everytime you lunch it it will search for a specific string on youtube (that is in order to pull all of the new videos for that string).
I saw I can use the youtube python official API but I have no idea how to install it on KODI.
Also using my google developer key limits me to an X amount of searches, so if I will publish this addon with my developer key it will exceeding the search quota within 10 minutes.

How can I use the youtube API in python to search for a string and extract the youtube video ID, description and jpg?

Regards,
un4get.
↧

Keep Original Local Title TMDB

May 13, 2016, 7:22 pm
$
0
0
GDay,
I asked over on Feature request for a Switch so while scraping new content, Kodi would keep my title names, and somebody suggested that it would possibly be an easy thing to add into a scraper.
If someone here could Mod the TMDB scraper for me to do this, I would be forever greatful and quote their name to gods on high.

There are a couple of problems i keep having with the way I sort my movies and TV Shows, and, if i ever lose my movie database, like i did this week, it takes me weeks to get it all back together.

This mod to the scraper would solve everything for me in 1 go.

So Please Please Please and Thanks in Advance and Help all in one.

Cheers
↧

Apply scraper to video list inside of an addon

May 16, 2016, 9:09 am
$
0
0
Hello,

as I am developing a video addon for Kodi I would like to add scraper information to the list of videos (e.g. IMDB rating, trailer).

As far as I know scrapers could only be applied to media sources of kodi (e.g. folders, smb shares, nfs shares etc.) and could not be applied inside of an addon.

In general my understanding is that a scaper needs only the "file names" as input. In my case it would be the movie titles of the video listing.

Is there any way (e.g. function call) to apply scapers to a video list inside of an addon?

- Raspispass
↧
Viewing all 171 articles
Browse latest View live
Search