Webscrape Youtube with Yahoo Pipes

This is the first part of a multi step tutorial on how to webscape Youtube using Yahoo pipes.
By the end of this series you will have an RSS feed of different videos all webscraped from different Youtube queries.
I will then show you how to filter out videos below a certain number of views or below a certain ranking.
In the final stage of this series I will show how to add this RSS feed to wordpress to create a constantly updating automated blog full of quality content.

A quick example
Dubstep is a genre of music. I want to search Youtube for Dubstep music and all its sub genres.

Lets begin

1. Select the “URL Builder” module from the URL section of modules.

2. URL Builder module
- “BASE” you want to copy and paste the following line in it.

http://gdata.youtube.com/feeds/base/videos

-Query parametres (you need 4 boxes)
Box 1 – “q” – “Dubstep”    This defines the search we are doing on youtube
Box 2 – “client” – ytapi-youtube-search
Box 3 – “alt” – “rss”
Box 4 – “v” – “2″

Check out the screenshot below

3. You can repeat this process adding more “URL builder modules” and changing the “q” box.  In this example I have two queries for youtube.  ”dubstep” and “heavy dubstep”.

4. For each “URL builder” module you need to attach a “fetch data” module
In “path to item list” put in “channel.item”
I don’t know why but this seems to create a readable RSS format that you should be familiar with

5.  Choose the “Union” module and attach all your “fetch data” modules to it. Remember if you have many many queries you can keep adding “union modules”.

6. I would recommend using the “sort” module and sort by item.y:published.
Sorting RSS feeds by date using Yahoo Pipes

7. Connect up your pipe to the output and run the pipe. You should have a list from Youtube with one or more queries all combined into one.

In the next part of this series I will show you how to filter out the low ranking or low viewed videos which should help increase the relevance of your live RSS feed.

Dynamic page queries with YQL in YahooPipes

This tutorial will show you how to query multiple pages with YQL. For example query the webpage of every news item in your RSS feed to gain extra information.

My example is going to query the tek-9 RSS feed to find out how many comments were made on each item.

1. Select the fetch feed module and enter the feed of your choice.

http://www.tek-9.org/rss/news/cs/

2. Select the “loop operator” and place the “string builder” into it

3. In the first box type the following
select * from html where url=’

3. In the second box
item.link

4. In the third box
‘ and xpath=’/html/body/div[5]/div[2]/div/h2′

Note (make sure the Xpath matches the query you are looking for). Please refer back to the earlier YQL tutorials if you get lost here

5. Assign results to item.field
(in my case, item.comments)



6. Add another Loop module

7. Place the YQL module into the loop module

8. In the body of the loop module type in the item created. (in my case “item.comments”)

9. Assign “all” to item.comments

10. This will run through every item on your RSS feed. Enjoy

Sorting YQL into RSS feed format using YahooPipes

This tutorial will show how to sort the jargon YQL produces in YahooPipes into a familiar RSS format.



1. The YQL query here is
select * from html where url=’http://www.cadred.org’ and xpath=’/html/body/div/div/div[3]/div[5]/div[3]/div/div’
The output is not useful for an RSS feed and needs sorting out before it can be used

2. Select the Loop module from operators

3. Place the “item builder” module into the “loop” module

4. This process is very difficult and requires some trial error.
On the left hand side of the “item builder” input the field you want to create.
Then from the drop down try to select the path that is the closest match to the jargon from the YQL debugger.
Sometimes you will get lucky and you will find it works first time! However sometimes the dropdown path is incorrect and you will need to try and figure out the correct path and manually enter it



5. Repeat this process for every field you want.
Use the “rename module” to create virtual fields hilst you figure out the exact path of the item you want. Then add the exact path to the item builder. This will save time having to refresh through the loop module which can be sensitive to incorrect paths.

Using YQL to select specific information in Yahoo Pipes

YQL can be used instead of the fetch page module to select specific areas of a website. It is useful for selecting very specific elements such as the comments on a topic within a forum. It helps cut back on having to use complex regex which can be a real time saver.

To follow the steps in this tutorial you must first install firebug. Click here for help.

1. Select the YQL module from the sources area of YahooPipes

2. Paste the follow YQL query into the YQL module. Replacing the website address with your own, and the xpath with your own.

select * from html where url=’http://www.tek-9.org/news/announcing_the_powercup_on_esports_heaven-2095.html’ and xpath=’/html/body/div[5]/div[2]/div/h2′

4. Make sure there are no spaces in this query.

3. Depending on the accuracy of your YQL query you made find you have several nodes (01,2,3,4) in the debugger. You can either try to make a more specific xpath by going back to firebug (this is not always possible) or move onto the next tutorial to learn how to use the “item builder”

Using firebug to find the Xpath on a webpage (needed for YQL)

1. Google “firebug” and install the addon

2. Navigate to the website wish to find the YQL for and select the blue arrow in the upper right hand side of firebug

3. Using the arrow select the area of the site you need. This should highlight an area of HTML in the debugger

4. Right click on the highlighted zone and select “copy xpath”

5. Note down/copy this “xpath” for later use!

Increasing RSS description detail by using fetch page with YahooPipes

This tutorial will demonstrate how to add more content to a RSS feeds description. Most RSS feeds lack detail to encourage you to click through to the site. This method gets around the problem!

1. Select the fetch feed module and enter in your RSS feed

2. Under operators select the “loop” module

3. Under operators select the “fetch page” module and place it in the “loop” module

4. In the “fetch page : URL ” type in “item.link”

5. Follow the instruction from here to cut down a page for the description

6. Make sure the “Loop” module is set to assign “all” to the item.description

7. Link the “loop” module to the “pipeout” module

Using the fetch page module to extract information using YahooPipes

1. Under operators select the fetch page module

2. Copy and paste the URL of the website you want into the fetch page

3. Highlight the fetchpage module and select item.description

4. In the Item.description click on the ‘Source’

5. Search for the HTML you wish to start your page from

6. Enter the HTML code in the ‘Cut content from’ box of the fetch page

7. Search For the HTML for the end of the page and put that in the “to” box

8. Check the item.description for the cut down version of the web page