Hi
user
Admin Login:
Username:
Password:
Name:
SciPy + MapReduce with Disco
--client
enthought
--show
scipy_2012
--room 106 1180 --force
Next: 11 Bringing High Performance to Python/Numpy Without Changing a Single Line of Code.
show more...
Marks
Author(s):
Al Barrentine
Location
106
Date
jul Wed 18
Days Raw Files
Start
14:20
First Raw Start
14:23
Duration
20:00
Offset
0:03:09
End
14:40
Last Raw End
14:47
Chapters
00:00
Total cuts_time
23 min.
raw-playlist
raw-mp4-playlist
encoded-files-playlist
host
archive
public
tweet
mp4
svg
png
assets
release.pdf
SciPy_MapReduce_with_Disco.json
logs
Admin:
episode
episode list
cut list
raw files day
marks day
marks day
image_files
State:
---------
borked
edit
encode
push to queue
post
richard
review 1
email
review 2
make public
tweet
to-miror
conf
done
Locked:
clear this to unlock
Locked by:
user/process that locked.
Start:
initially scheduled time from master, adjusted to match reality
Duration:
length in hh:mm:ss
Name:
Video Title (shows in video search results)
Emails:
email(s) of the presenter(s)
Released:
has someone authorised pubication
Unknown
Yes
No
Normalise:
Channelcopy:
m=mono, 01=copy left to right, 10=right to left, 00=ignore.
Thumbnail:
filename.png
Description:
markdown
MapReduce has become one of two dominant paradigms in distributed computing (along with MPI). Yet many times, implementing an algorithm as a MapReduce job - especially in Python - forces us to sacrifice efficiency (BLAS routines, etc.) in favor of data parallelism. In my work, which involves writing distributed learning algorithms for processing terabytes of Twitter data at SocialFlow, I've come to advocate a form of "vectorized MapReduce" which integrates efficient numerical libraries like numpy/scipy into the MapReduce setting, yielding both faster per-machine performance and reduced I/O, which is often a major bottleneck. I'll also highlight some features of Disco (a Python/Erlang MapReduce implementation from Nokia) which make it a very compelling choice for writing scientific MapReduce jobs in Python.
Comment:
production notes
<?xml version='1.0' encoding='UTF-8'?> <ns0:entry xmlns:ns0="http://www.w3.org/2005/Atom"><ns0:category scheme="http://schemas.google.com/g/2005#kind" term="http://gdata.youtube.com/schemas/2007#video" /><ns0:category label="Education" scheme="http://gdata.youtube.com/schemas/2007/categories.cat" term="Education" /><ns0:category scheme="http://gdata.youtube.com/schemas/2007/keywords.cat" term="enthought" /><ns0:category scheme="http://gdata.youtube.com/schemas/2007/keywords.cat" term="scipy_2012" /><ns0:category scheme="http://gdata.youtube.com/schemas/2007/keywords.cat" term="HPC" /><ns0:category scheme="http://gdata.youtube.com/schemas/2007/keywords.cat" term="AlBarrentine" /><ns0:id>http://gdata.youtube.com/feeds/api/videos/ofCddrXkw0U</ns0:id><ns0:author><ns0:name>NextDayVideo</ns0:name><ns0:uri>https://gdata.youtube.com/feeds/api/users/NextDayVideo</ns0:uri></ns0:author><ns0:content type="text">Al Barrentine</ns0:content><ns0:updated>2012-08-02T22:13:44.000Z</ns0:updated><ns0:published>2012-08-02T22:13:44.000Z</ns0:published><ns1:comments xmlns:ns1="http://schemas.google.com/g/2005"><ns1:feedLink countHint="0" href="https://gdata.youtube.com/feeds/api/videos/ofCddrXkw0U/comments?client=ndv" rel="http://gdata.youtube.com/schemas/2007#comments" /></ns1:comments><ns1:group xmlns:ns1="http://search.yahoo.com/mrss/"><ns1:keywords>enthought, scipy_2012, HPC, AlBarrentine</ns1:keywords><ns1:description type="plain">Al Barrentine</ns1:description><ns1:title type="plain">SciPy + MapReduce with Disco</ns1:title><ns2:duration seconds="0" xmlns:ns2="http://gdata.youtube.com/schemas/2007" /><ns1:content duration="0" expression="full" isDefault="true" medium="video" type="application/x-shockwave-flash" url="https://www.youtube.com/v/ofCddrXkw0U?version=3&f=user_uploads&c=ndv&d=Aarb2r5skm2_yNTuKEinXdAO88HsQjpE1a8d1GxQnGDm&app=youtube_gdata" ns2:format="5" xmlns:ns2="http://gdata.youtube.com/schemas/2007" /><ns1:thumbnail height="360" time="00:00:00" url="http://i.ytimg.com/vi/ofCddrXkw0U/0.jpg" width="480" /><ns1:thumbnail height="90" time="00:00:00" url="http://i.ytimg.com/vi/ofCddrXkw0U/1.jpg" width="120" /><ns1:thumbnail height="90" time="00:00:00" url="http://i.ytimg.com/vi/ofCddrXkw0U/2.jpg" width="120" /><ns1:thumbnail height="90" time="00:00:00" url="http://i.ytimg.com/vi/ofCddrXkw0U/3.jpg" width="120" /><ns1:category label="Education" scheme="http://gdata.youtube.com/schemas/2007/categories.cat">Education</ns1:category><ns1:category scheme="http://gdata.youtube.com/schemas/2007/developertags.cat">enthought</ns1:category><ns1:category scheme="http://gdata.youtube.com/schemas/2007/developertags.cat">scipy_2012</ns1:category><ns1:category scheme="http://gdata.youtube.com/schemas/2007/developertags.cat">HPC</ns1:category><ns1:category scheme="http://gdata.youtube.com/schemas/2007/developertags.cat">AlBarrentine</ns1:category><ns1:player url="https://www.youtube.com/watch?v=ofCddrXkw0U&feature=youtube_gdata_player" /></ns1:group><ns0:title type="text">SciPy + MapReduce with Disco</ns0:title><ns1:control xmlns:ns1="http://purl.org/atom/app#"><ns1:draft>yes</ns1:draft><ns2:state name="processing" xmlns:ns2="http://gdata.youtube.com/schemas/2007" /></ns1:control><ns0:link href="https://www.youtube.com/watch?v=ofCddrXkw0U&feature=youtube_gdata" rel="alternate" type="text/html" /><ns0:link href="https://gdata.youtube.com/feeds/api/videos/ofCddrXkw0U/responses?client=ndv" rel="http://gdata.youtube.com/schemas/2007#video.responses" type="application/atom+xml" /><ns0:link href="https://gdata.youtube.com/feeds/api/videos/ofCddrXkw0U/ratings?client=ndv" rel="http://gdata.youtube.com/schemas/2007#video.ratings" type="application/atom+xml" /><ns0:link href="https://gdata.youtube.com/feeds/api/videos/ofCddrXkw0U/complaints?client=ndv" rel="http://gdata.youtube.com/schemas/2007#video.complaints" type="application/atom+xml" /><ns0:link href="https://gdata.youtube.com/feeds/api/videos/ofCddrXkw0U/related?client=ndv" rel="http://gdata.youtube.com/schemas/2007#video.related" type="application/atom+xml" /><ns0:link href="https://gdata.youtube.com/feeds/api/users/nextdayvideo/uploads/ofCddrXkw0U?client=ndv" rel="self" type="application/atom+xml" /><ns0:link href="https://gdata.youtube.com/feeds/api/users/nextdayvideo/uploads/ofCddrXkw0U?client=ndv" rel="edit" type="application/atom+xml" /></ns0:entry>
2012-07-18/12_23_09.dv
Apply:
14:23:09 - 14:23:17 ( 00:00:08 )
S:
14:23:09 -
E:
14:23:17
D:
00:00:08
show more...
vlc ~/Videos/veyepar/enthought/scipy_2012/dv/106/2012-07-18/12_23_09.dv :start-time=00.0 --audio-desync=0
Raw File
Cut List
14:23:09
seconds: 0.0
Wall: 14:23:09
Duration
00:00:08
14:23:17
seconds: 0.0
Wall: 14:23:09
Comments:
mp4
mp4.m3u
dv.m3u
Split:
Sequence:
:
delete
2012-07-18/12_23_17.dv
Apply:
14:23:17 - 14:47:07 ( 00:23:50 )
S:
14:23:17 -
E:
14:47:07
D:
00:23:50
show more...
vlc ~/Videos/veyepar/enthought/scipy_2012/dv/106/2012-07-18/12_23_17.dv :start-time=00.0 --audio-desync=0
Raw File
Cut List
14:23:17
seconds: 0.0
Wall: 14:23:17
Duration
00:23:50
14:47:07
seconds: 0.0
Wall: 14:23:17
Comments:
mp4
mp4.m3u
dv.m3u
Split:
Sequence:
:
delete
Rf filename:
root is .../show/dv/location/, example: 2013-03-13/13:13:30.dv
Sequence:
get this:
check and save to add this
2012-07-18/12_23_09.dv
2012-07-18/12_23_17.dv
Veyepar
Video Eyeball Processor and Review
