Hi
user
Admin Login:
Username:
Password:
Name:
Web scraping: Reliably and efficiently pull data from pages that don't expect it
--client
psf
--show
pycon_2012
--room H3 672 --force
Next: 6 Introduction to Game Development
show more...
Marks
Author(s):
Asheesh Laroia
Location
H3-Tutorials VIII
Date
mar Wed 07
Days Raw Files
Start
13:20
First Raw Start
13:20
Duration
03:20:00
Offset
0:00:00
End
16:40
Last Raw End
16:15
Chapters
00:00
1:18:07
2:26:26
2:30:42
2:39:38
Total cuts_time
159 min.
http://us.pycon.org/2012/schedule/presentation/317/
raw-playlist
raw-mp4-playlist
encoded-files-playlist
host
public
mp4
svg
png
assets
release.pdf
Web_scraping_Reliably_and_efficiently_pull_data_from_pages_that_dont_expect_it.json
logs
Admin:
episode
episode list
cut list
raw files day
marks day
marks day
image_files
State:
---------
borked
edit
encode
push to queue
post
richard
review 1
email
review 2
make public
tweet
to-miror
conf
done
Locked:
clear this to unlock
Locked by:
user/process that locked.
Start:
initially scheduled time from master, adjusted to match reality
Duration:
length in hh:mm:ss
Name:
Video Title (shows in video search results)
Emails:
email(s) of the presenter(s)
Released:
has someone authorised pubication
Unknown
Yes
No
Normalise:
Channelcopy:
m=mono, 01=copy left to right, 10=right to left, 00=ignore.
Thumbnail:
filename.png
Description:
markdown
Exciting information is trapped in web pages and behind HTML forms. In this tutorial, you'll learn how to parse those pages and when to apply advanced techniques that make scraping faster and more stable. We'll cover parallel downloading with Twisted, gevent, and others; analyzing sites behind SSL; driving JavaScript-y sites with Selenium; and evading common anti-scraping techniques.
Comment:
production notes
<?xml version='1.0' encoding='UTF-8'?> <ns0:entry xmlns:ns0="http://www.w3.org/2005/Atom" xmlns:ns1="http://schemas.google.com/g/2005" xmlns:ns2="http://search.yahoo.com/mrss/" xmlns:ns3="http://gdata.youtube.com/schemas/2007" xmlns:ns4="http://purl.org/atom/app#"><ns0:category scheme="http://schemas.google.com/g/2005#kind" term="http://gdata.youtube.com/schemas/2007#video" /><ns0:category label="Education" scheme="http://gdata.youtube.com/schemas/2007/categories.cat" term="Education" /><ns0:category scheme="http://gdata.youtube.com/schemas/2007/keywords.cat" term="psf" /><ns0:category scheme="http://gdata.youtube.com/schemas/2007/keywords.cat" term="python pycon pycon2012" /><ns0:category scheme="http://gdata.youtube.com/schemas/2007/keywords.cat" term="pycon_2012" /><ns0:category scheme="http://gdata.youtube.com/schemas/2007/keywords.cat" term="AsheeshLaroia" /><ns0:id>http://gdata.youtube.com/feeds/api/videos/52wxGESwQSA</ns0:id><ns0:author><ns0:name>NextDayVideo</ns0:name><ns0:uri>https://gdata.youtube.com/feeds/api/users/NextDayVideo</ns0:uri></ns0:author><ns0:content type="text">Asheesh Laroia Exciting information is trapped in web pages and behind HTML forms. In this tutorial, you'll learn how to parse those pages and when to apply advanced techniques that make scraping faster and more stable. We'll cover parallel downlo</ns0:content><ns0:updated>2012-03-09T01:10:16.000Z</ns0:updated><ns0:published>2012-03-09T01:10:16.000Z</ns0:published><ns1:comments><ns1:feedLink countHint="0" href="https://gdata.youtube.com/feeds/api/videos/52wxGESwQSA/comments?client=ndv" rel="http://gdata.youtube.com/schemas/2007#comments" /></ns1:comments><ns2:group><ns2:keywords>psf, python pycon pycon2012, pycon_2012, AsheeshLaroia</ns2:keywords><ns2:description type="plain">Asheesh Laroia Exciting information is trapped in web pages and behind HTML forms. In this tutorial, you'll learn how to parse those pages and when to apply advanced techniques that make scraping faster and more stable. We'll cover parallel downlo</ns2:description><ns2:title type="plain">Web scraping: Reliably and efficiently pull data from pages that don't expect it</ns2:title><ns3:duration seconds="0" /><ns2:content duration="0" expression="full" isDefault="true" medium="video" type="application/x-shockwave-flash" url="https://www.youtube.com/v/52wxGESwQSA?version=3&f=user_uploads&c=ndv&d=Aarb2r5skm2_yNTuKEinXdAO88HsQjpE1a8d1GxQnGDm&app=youtube_gdata" ns3:format="5" /><ns2:thumbnail height="360" time="00:00:00" url="http://i.ytimg.com/vi/52wxGESwQSA/0.jpg" width="480" /><ns2:thumbnail height="90" time="00:00:00" url="http://i.ytimg.com/vi/52wxGESwQSA/1.jpg" width="120" /><ns2:thumbnail height="90" time="00:00:00" url="http://i.ytimg.com/vi/52wxGESwQSA/2.jpg" width="120" /><ns2:thumbnail height="90" time="00:00:00" url="http://i.ytimg.com/vi/52wxGESwQSA/3.jpg" width="120" /><ns2:category label="Education" scheme="http://gdata.youtube.com/schemas/2007/categories.cat">Education</ns2:category><ns2:category scheme="http://gdata.youtube.com/schemas/2007/developertags.cat">psf</ns2:category><ns2:category scheme="http://gdata.youtube.com/schemas/2007/developertags.cat">python pycon pycon2012</ns2:category><ns2:category scheme="http://gdata.youtube.com/schemas/2007/developertags.cat">pycon_2012</ns2:category><ns2:category scheme="http://gdata.youtube.com/schemas/2007/developertags.cat">AsheeshLaroia</ns2:category><ns2:player url="https://www.youtube.com/watch?v=52wxGESwQSA&feature=youtube_gdata_player" /></ns2:group><ns0:title type="text">Web scraping: Reliably and efficiently pull data from pages that don't expect it</ns0:title><ns4:control><ns4:draft>yes</ns4:draft><ns3:state name="processing" /></ns4:control><ns0:link href="https://www.youtube.com/watch?v=52wxGESwQSA&feature=youtube_gdata" rel="alternate" type="text/html" /><ns0:link href="https://gdata.youtube.com/feeds/api/videos/52wxGESwQSA/responses?client=ndv" rel="http://gdata.youtube.com/schemas/2007#video.responses" type="application/atom+xml" /><ns0:link href="https://gdata.youtube.com/feeds/api/videos/52wxGESwQSA/ratings?client=ndv" rel="http://gdata.youtube.com/schemas/2007#video.ratings" type="application/atom+xml" /><ns0:link href="https://gdata.youtube.com/feeds/api/videos/52wxGESwQSA/complaints?client=ndv" rel="http://gdata.youtube.com/schemas/2007#video.complaints" type="application/atom+xml" /><ns0:link href="https://gdata.youtube.com/feeds/api/videos/52wxGESwQSA/related?client=ndv" rel="http://gdata.youtube.com/schemas/2007#video.related" type="application/atom+xml" /><ns0:link href="https://gdata.youtube.com/feeds/api/users/nextdayvideo/uploads/52wxGESwQSA?client=ndv" rel="self" type="application/atom+xml" /><ns0:link href="https://gdata.youtube.com/feeds/api/users/nextdayvideo/uploads/52wxGESwQSA?client=ndv" rel="edit" type="application/atom+xml" /></ns0:entry>
2012-03-07/13:20:00.dv
Apply:
13:20:03 - 14:38:10 ( 01:18:07 )
S:
13:20:00 -
E:
14:38:12
D:
01:18:12
(
Start:
3) (
End:
01:18:10)
show more...
vlc ~/Videos/veyepar/psf/pycon_2012/dv/H3/2012-03-07/13:20:00.dv :start-time=03.0 --audio-desync=0
Raw File
Cut List
13:20:00
seconds: 3.0
Wall: 13:20:03
Duration
01:18:12
14:38:12
seconds: 4690.0
Wall: 14:38:10
Comments:
mp4
mp4.m3u
dv.m3u
Split:
Sequence:
:
delete
2012-03-07/14:38:14.dv
Apply:
14:38:14 - 14:49:45 ( 00:11:31 )
S:
14:38:14 -
E:
14:49:45
D:
00:11:31
show more...
vlc ~/Videos/veyepar/psf/pycon_2012/dv/H3/2012-03-07/14:38:14.dv :start-time=00.0 --audio-desync=0
Raw File
Cut List
14:38:14
seconds: 0.0
Wall: 14:38:14
Duration
00:11:31
14:49:45
seconds: 0.0
Wall: 14:38:14
Comments:
mp4
mp4.m3u
dv.m3u
Split:
Sequence:
:
delete
2012-03-07/14:49:45.dv
Apply:
14:49:45 - 14:52:10 ( 00:02:25 )
S:
14:49:45 -
E:
14:52:10
D:
00:02:25
show more...
vlc ~/Videos/veyepar/psf/pycon_2012/dv/H3/2012-03-07/14:49:45.dv :start-time=00.0 --audio-desync=0
Raw File
Cut List
14:49:45
seconds: 0.0
Wall: 14:49:45
Duration
00:02:25
14:52:10
seconds: 0.0
Wall: 14:49:45
Comments:
mp4
mp4.m3u
dv.m3u
Split:
Sequence:
:
delete
2012-03-07/14:52:10.dv
Apply:
14:52:10 - 16:00:29 ( 01:08:19 )
S:
14:52:10 -
E:
16:00:29
D:
01:08:19
show more...
vlc ~/Videos/veyepar/psf/pycon_2012/dv/H3/2012-03-07/14:52:10.dv :start-time=00.0 --audio-desync=0
Raw File
Cut List
14:52:10
seconds: 0.0
Wall: 14:52:10
Duration
01:08:19
16:00:29
seconds: 0.0
Wall: 14:52:10
Comments:
mp4
mp4.m3u
dv.m3u
Split:
Sequence:
:
delete
2012-03-07/16:00:32.dv
Apply:
16:00:32 - 16:04:48 ( 00:04:16 )
S:
16:00:32 -
E:
16:04:48
D:
00:04:16
show more...
vlc ~/Videos/veyepar/psf/pycon_2012/dv/H3/2012-03-07/16:00:32.dv :start-time=00.0 --audio-desync=0
Raw File
Cut List
16:00:32
seconds: 0.0
Wall: 16:00:32
Duration
00:04:16
16:04:48
seconds: 0.0
Wall: 16:00:32
Comments:
mp4
mp4.m3u
dv.m3u
Split:
Sequence:
:
delete
2012-03-07/16:04:48.dv
Apply:
16:04:48 - 16:13:44 ( 00:08:56 )
S:
16:04:48 -
E:
16:13:44
D:
00:08:56
show more...
vlc ~/Videos/veyepar/psf/pycon_2012/dv/H3/2012-03-07/16:04:48.dv :start-time=00.0 --audio-desync=0
Raw File
Cut List
16:04:48
seconds: 0.0
Wall: 16:04:48
Duration
00:08:56
16:13:44
seconds: 0.0
Wall: 16:04:48
Comments:
mp4
mp4.m3u
dv.m3u
Split:
Sequence:
:
delete
2012-03-07/16:13:45.dv
Apply:
16:13:45 - 16:13:50 ( 00:00:05 )
S:
16:13:45 -
E:
16:13:50
D:
00:00:05
show more...
vlc ~/Videos/veyepar/psf/pycon_2012/dv/H3/2012-03-07/16:13:45.dv :start-time=00.0 --audio-desync=0
Raw File
Cut List
16:13:45
seconds: 0.0
Wall: 16:13:45
Duration
00:00:05
16:13:50
seconds: 0.0
Wall: 16:13:45
Comments:
mp4
mp4.m3u
dv.m3u
Split:
Sequence:
:
delete
2012-03-07/16:13:50.dv
Apply:
16:13:50 - 16:15:00 ( 00:01:10 )
S:
16:13:50 -
E:
16:15:00
D:
00:01:10
show more...
vlc ~/Videos/veyepar/psf/pycon_2012/dv/H3/2012-03-07/16:13:50.dv :start-time=00.0 --audio-desync=0
Raw File
Cut List
16:13:50
seconds: 0.0
Wall: 16:13:50
Duration
00:01:10
16:15:00
seconds: 0.0
Wall: 16:13:50
Comments:
mp4
mp4.m3u
dv.m3u
Split:
Sequence:
:
delete
Rf filename:
root is .../show/dv/location/, example: 2013-03-13/13:13:30.dv
Sequence:
get this:
check and save to add this
2012-03-07/13:20:00.dv
2012-03-07/14:38:14.dv
2012-03-07/14:49:45.dv
2012-03-07/14:52:10.dv
2012-03-07/16:00:32.dv
2012-03-07/16:04:48.dv
2012-03-07/16:13:45.dv
2012-03-07/16:13:50.dv
Veyepar
Video Eyeball Processor and Review
