veyepar: Web scraping: Reliably and efficiently pull data from pages that don't expect it

Hi user

Admin Login:

Name: Web scraping: Reliably and efficiently pull data from pages that don't expect it
--client psf --show pycon_2012 --room H3 672 --force
Prev: 6 Introduction to Django
Next: 6 Introduction to Game Development

show more...

Author(s):	Asheesh Laroia
Location	H3-Tutorials VIII
Date	mar Wed 07	Days Raw Files
Start	13:20	First Raw Start	13:20
Duration	03:20:00	Offset	0:00:00
End	16:40	Last Raw End	16:15
Chapters	00:00 1:18:07 2:26:26 2:30:42 2:39:38
Total cuts_time	159 min.

http://us.pycon.org/2012/schedule/presentation/317/
raw-playlist raw-mp4-playlist encoded-files-playlist host public mp4 svg png

final audio

assets release.pdf Web_scraping_Reliably_and_efficiently_pull_data_from_pages_that_dont_expect_it.json
logs
Admin: episode episode list cut list raw files day marks day marks day image_files

State:
Locked:	clear this to unlock
Locked by:	user/process that locked.
Start:	initially scheduled time from master, adjusted to match reality
Duration:	length in hh:mm:ss
Name:	Video Title (shows in video search results)
Emails:	email(s) of the presenter(s)
Released:	has someone authorised pubication
Normalise:
Channelcopy:	m=mono, 01=copy left to right, 10=right to left, 00=ignore.
Thumbnail:	filename.png
Description:	Exciting information is trapped in web pages and behind HTML forms. In this tutorial, you'll learn how to parse those pages and when to apply advanced techniques that make scraping faster and more stable. We'll cover parallel downloading with Twisted, gevent, and others; analyzing sites behind SSL; driving JavaScript-y sites with Selenium; and evading common anti-scraping techniques. markdown
Comment:	<?xml version='1.0' encoding='UTF-8'?> <ns0:entry xmlns:ns0="http://www.w3.org/2005/Atom" xmlns:ns1="http://schemas.google.com/g/2005" xmlns:ns2="http://search.yahoo.com/mrss/" xmlns:ns3="http://gdata.youtube.com/schemas/2007" xmlns:ns4="http://purl.org/atom/app#"><ns0:category scheme="http://schemas.google.com/g/2005#kind" term="http://gdata.youtube.com/schemas/2007#video" /><ns0:category label="Education" scheme="http://gdata.youtube.com/schemas/2007/categories.cat" term="Education" /><ns0:category scheme="http://gdata.youtube.com/schemas/2007/keywords.cat" term="psf" /><ns0:category scheme="http://gdata.youtube.com/schemas/2007/keywords.cat" term="python pycon pycon2012" /><ns0:category scheme="http://gdata.youtube.com/schemas/2007/keywords.cat" term="pycon_2012" /><ns0:category scheme="http://gdata.youtube.com/schemas/2007/keywords.cat" term="AsheeshLaroia" /><ns0:id>http://gdata.youtube.com/feeds/api/videos/52wxGESwQSA</ns0:id><ns0:author><ns0:name>NextDayVideo</ns0:name><ns0:uri>https://gdata.youtube.com/feeds/api/users/NextDayVideo</ns0:uri></ns0:author><ns0:content type="text">Asheesh Laroia Exciting information is trapped in web pages and behind HTML forms. In this tutorial, you'll learn how to parse those pages and when to apply advanced techniques that make scraping faster and more stable. We'll cover parallel downlo</ns0:content><ns0:updated>2012-03-09T01:10:16.000Z</ns0:updated><ns0:published>2012-03-09T01:10:16.000Z</ns0:published><ns1:comments><ns1:feedLink countHint="0" href="https://gdata.youtube.com/feeds/api/videos/52wxGESwQSA/comments?client=ndv" rel="http://gdata.youtube.com/schemas/2007#comments" /></ns1:comments><ns2:group><ns2:keywords>psf, python pycon pycon2012, pycon_2012, AsheeshLaroia</ns2:keywords><ns2:description type="plain">Asheesh Laroia Exciting information is trapped in web pages and behind HTML forms. In this tutorial, you'll learn how to parse those pages and when to apply advanced techniques that make scraping faster and more stable. We'll cover parallel downlo</ns2:description><ns2:title type="plain">Web scraping: Reliably and efficiently pull data from pages that don't expect it</ns2:title><ns3:duration seconds="0" /><ns2:content duration="0" expression="full" isDefault="true" medium="video" type="application/x-shockwave-flash" url="https://www.youtube.com/v/52wxGESwQSA?version=3&f=user_uploads&c=ndv&d=Aarb2r5skm2_yNTuKEinXdAO88HsQjpE1a8d1GxQnGDm&app=youtube_gdata" ns3:format="5" /><ns2:thumbnail height="360" time="00:00:00" url="http://i.ytimg.com/vi/52wxGESwQSA/0.jpg" width="480" /><ns2:thumbnail height="90" time="00:00:00" url="http://i.ytimg.com/vi/52wxGESwQSA/1.jpg" width="120" /><ns2:thumbnail height="90" time="00:00:00" url="http://i.ytimg.com/vi/52wxGESwQSA/2.jpg" width="120" /><ns2:thumbnail height="90" time="00:00:00" url="http://i.ytimg.com/vi/52wxGESwQSA/3.jpg" width="120" /><ns2:category label="Education" scheme="http://gdata.youtube.com/schemas/2007/categories.cat">Education</ns2:category><ns2:category scheme="http://gdata.youtube.com/schemas/2007/developertags.cat">psf</ns2:category><ns2:category scheme="http://gdata.youtube.com/schemas/2007/developertags.cat">python pycon pycon2012</ns2:category><ns2:category scheme="http://gdata.youtube.com/schemas/2007/developertags.cat">pycon_2012</ns2:category><ns2:category scheme="http://gdata.youtube.com/schemas/2007/developertags.cat">AsheeshLaroia</ns2:category><ns2:player url="https://www.youtube.com/watch?v=52wxGESwQSA&feature=youtube_gdata_player" /></ns2:group><ns0:title type="text">Web scraping: Reliably and efficiently pull data from pages that don't expect it</ns0:title><ns4:control><ns4:draft>yes</ns4:draft><ns3:state name="processing" /></ns4:control><ns0:link href="https://www.youtube.com/watch?v=52wxGESwQSA&feature=youtube_gdata" rel="alternate" type="text/html" /><ns0:link href="https://gdata.youtube.com/feeds/api/videos/52wxGESwQSA/responses?client=ndv" rel="http://gdata.youtube.com/schemas/2007#video.responses" type="application/atom+xml" /><ns0:link href="https://gdata.youtube.com/feeds/api/videos/52wxGESwQSA/ratings?client=ndv" rel="http://gdata.youtube.com/schemas/2007#video.ratings" type="application/atom+xml" /><ns0:link href="https://gdata.youtube.com/feeds/api/videos/52wxGESwQSA/complaints?client=ndv" rel="http://gdata.youtube.com/schemas/2007#video.complaints" type="application/atom+xml" /><ns0:link href="https://gdata.youtube.com/feeds/api/videos/52wxGESwQSA/related?client=ndv" rel="http://gdata.youtube.com/schemas/2007#video.related" type="application/atom+xml" /><ns0:link href="https://gdata.youtube.com/feeds/api/users/nextdayvideo/uploads/52wxGESwQSA?client=ndv" rel="self" type="application/atom+xml" /><ns0:link href="https://gdata.youtube.com/feeds/api/users/nextdayvideo/uploads/52wxGESwQSA?client=ndv" rel="edit" type="application/atom+xml" /></ns0:entry> production notes

2012-03-07/13:20:00.dv Apply: 13:20:03 - 14:38:10 ( 01:18:07 ) S: 13:20:00 - E: 14:38:12 D: 01:18:12 (Start: 3) (End: 01:18:10) show more...

vlc ~/Videos/veyepar/psf/pycon_2012/dv/H3/2012-03-07/13:20:00.dv :start-time=03.0 --audio-desync=0

	Raw File	Cut List
	13:20:00		seconds: 3.0 Wall: 13:20:03
Duration	01:18:12
	14:38:12		seconds: 4690.0 Wall: 14:38:10
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

audio viz

2012-03-07/14:38:14.dv Apply: 14:38:14 - 14:49:45 ( 00:11:31 ) S: 14:38:14 - E: 14:49:45 D: 00:11:31 show more...

vlc ~/Videos/veyepar/psf/pycon_2012/dv/H3/2012-03-07/14:38:14.dv :start-time=00.0 --audio-desync=0

	Raw File	Cut List
	14:38:14		seconds: 0.0 Wall: 14:38:14
Duration	00:11:31
	14:49:45		seconds: 0.0 Wall: 14:38:14
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

audio viz

2012-03-07/14:49:45.dv Apply: 14:49:45 - 14:52:10 ( 00:02:25 ) S: 14:49:45 - E: 14:52:10 D: 00:02:25 show more...

vlc ~/Videos/veyepar/psf/pycon_2012/dv/H3/2012-03-07/14:49:45.dv :start-time=00.0 --audio-desync=0

	Raw File	Cut List
	14:49:45		seconds: 0.0 Wall: 14:49:45
Duration	00:02:25
	14:52:10		seconds: 0.0 Wall: 14:49:45
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

audio viz

2012-03-07/14:52:10.dv Apply: 14:52:10 - 16:00:29 ( 01:08:19 ) S: 14:52:10 - E: 16:00:29 D: 01:08:19 show more...

vlc ~/Videos/veyepar/psf/pycon_2012/dv/H3/2012-03-07/14:52:10.dv :start-time=00.0 --audio-desync=0

	Raw File	Cut List
	14:52:10		seconds: 0.0 Wall: 14:52:10
Duration	01:08:19
	16:00:29		seconds: 0.0 Wall: 14:52:10
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

audio viz

2012-03-07/16:00:32.dv Apply: 16:00:32 - 16:04:48 ( 00:04:16 ) S: 16:00:32 - E: 16:04:48 D: 00:04:16 show more...

vlc ~/Videos/veyepar/psf/pycon_2012/dv/H3/2012-03-07/16:00:32.dv :start-time=00.0 --audio-desync=0

	Raw File	Cut List
	16:00:32		seconds: 0.0 Wall: 16:00:32
Duration	00:04:16
	16:04:48		seconds: 0.0 Wall: 16:00:32
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

audio viz

2012-03-07/16:04:48.dv Apply: 16:04:48 - 16:13:44 ( 00:08:56 ) S: 16:04:48 - E: 16:13:44 D: 00:08:56 show more...

vlc ~/Videos/veyepar/psf/pycon_2012/dv/H3/2012-03-07/16:04:48.dv :start-time=00.0 --audio-desync=0

	Raw File	Cut List
	16:04:48		seconds: 0.0 Wall: 16:04:48
Duration	00:08:56
	16:13:44		seconds: 0.0 Wall: 16:04:48
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

audio viz

2012-03-07/16:13:45.dv Apply: 16:13:45 - 16:13:50 ( 00:00:05 ) S: 16:13:45 - E: 16:13:50 D: 00:00:05 show more...

vlc ~/Videos/veyepar/psf/pycon_2012/dv/H3/2012-03-07/16:13:45.dv :start-time=00.0 --audio-desync=0

	Raw File	Cut List
	16:13:45		seconds: 0.0 Wall: 16:13:45
Duration	00:00:05
	16:13:50		seconds: 0.0 Wall: 16:13:45
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

audio viz

2012-03-07/16:13:50.dv Apply: 16:13:50 - 16:15:00 ( 00:01:10 ) S: 16:13:50 - E: 16:15:00 D: 00:01:10 show more...

vlc ~/Videos/veyepar/psf/pycon_2012/dv/H3/2012-03-07/16:13:50.dv :start-time=00.0 --audio-desync=0

	Raw File	Cut List
	16:13:50		seconds: 0.0 Wall: 16:13:50
Duration	00:01:10
	16:15:00		seconds: 0.0 Wall: 16:13:50
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

audio viz

Rf filename:	root is .../show/dv/location/, example: 2013-03-13/13:13:30.dv
Sequence:
get this:	check and save to add this

2012-03-07/13:20:00.dv
audio viz

audio viz

2012-03-07/14:38:14.dv
audio viz

audio viz

2012-03-07/14:49:45.dv
audio viz

audio viz

2012-03-07/14:52:10.dv
audio viz

audio viz

2012-03-07/16:00:32.dv
audio viz

audio viz

2012-03-07/16:04:48.dv
audio viz

audio viz

2012-03-07/16:13:45.dv
audio viz

audio viz

2012-03-07/16:13:50.dv
audio viz

audio viz

Veyepar Video Eyeball Processor and Review