veyepar: SciPy + MapReduce with Disco

Hi user

Admin Login:

Name: SciPy + MapReduce with Disco
--client enthought --show scipy_2012 --room 106 1180 --force
Prev: 11 OpenMG: A New Multigrid Implementation in Python
Next: 11 Bringing High Performance to Python/Numpy Without Changing a Single Line of Code.

show more...

Author(s):	Al Barrentine
Location	106
Date	jul Wed 18	Days Raw Files
Start	14:20	First Raw Start	14:23
Duration	20:00	Offset	0:03:09
End	14:40	Last Raw End	14:47
Chapters	00:00
Total cuts_time	23 min.

raw-playlist raw-mp4-playlist encoded-files-playlist host archive public tweet mp4 svg png

final audio

assets release.pdf SciPy_MapReduce_with_Disco.json
logs
Admin: episode episode list cut list raw files day marks day marks day image_files

State:
Locked:	clear this to unlock
Locked by:	user/process that locked.
Start:	initially scheduled time from master, adjusted to match reality
Duration:	length in hh:mm:ss
Name:	Video Title (shows in video search results)
Emails:	email(s) of the presenter(s)
Released:	has someone authorised pubication
Normalise:
Channelcopy:	m=mono, 01=copy left to right, 10=right to left, 00=ignore.
Thumbnail:	filename.png
Description:	MapReduce has become one of two dominant paradigms in distributed computing (along with MPI). Yet many times, implementing an algorithm as a MapReduce job - especially in Python - forces us to sacrifice efficiency (BLAS routines, etc.) in favor of data parallelism. In my work, which involves writing distributed learning algorithms for processing terabytes of Twitter data at SocialFlow, I've come to advocate a form of "vectorized MapReduce" which integrates efficient numerical libraries like numpy/scipy into the MapReduce setting, yielding both faster per-machine performance and reduced I/O, which is often a major bottleneck. I'll also highlight some features of Disco (a Python/Erlang MapReduce implementation from Nokia) which make it a very compelling choice for writing scientific MapReduce jobs in Python. markdown
Comment:	<?xml version='1.0' encoding='UTF-8'?> <ns0:entry xmlns:ns0="http://www.w3.org/2005/Atom"><ns0:category scheme="http://schemas.google.com/g/2005#kind" term="http://gdata.youtube.com/schemas/2007#video" /><ns0:category label="Education" scheme="http://gdata.youtube.com/schemas/2007/categories.cat" term="Education" /><ns0:category scheme="http://gdata.youtube.com/schemas/2007/keywords.cat" term="enthought" /><ns0:category scheme="http://gdata.youtube.com/schemas/2007/keywords.cat" term="scipy_2012" /><ns0:category scheme="http://gdata.youtube.com/schemas/2007/keywords.cat" term="HPC" /><ns0:category scheme="http://gdata.youtube.com/schemas/2007/keywords.cat" term="AlBarrentine" /><ns0:id>http://gdata.youtube.com/feeds/api/videos/ofCddrXkw0U</ns0:id><ns0:author><ns0:name>NextDayVideo</ns0:name><ns0:uri>https://gdata.youtube.com/feeds/api/users/NextDayVideo</ns0:uri></ns0:author><ns0:content type="text">Al Barrentine</ns0:content><ns0:updated>2012-08-02T22:13:44.000Z</ns0:updated><ns0:published>2012-08-02T22:13:44.000Z</ns0:published><ns1:comments xmlns:ns1="http://schemas.google.com/g/2005"><ns1:feedLink countHint="0" href="https://gdata.youtube.com/feeds/api/videos/ofCddrXkw0U/comments?client=ndv" rel="http://gdata.youtube.com/schemas/2007#comments" /></ns1:comments><ns1:group xmlns:ns1="http://search.yahoo.com/mrss/"><ns1:keywords>enthought, scipy_2012, HPC, AlBarrentine</ns1:keywords><ns1:description type="plain">Al Barrentine</ns1:description><ns1:title type="plain">SciPy + MapReduce with Disco</ns1:title><ns2:duration seconds="0" xmlns:ns2="http://gdata.youtube.com/schemas/2007" /><ns1:content duration="0" expression="full" isDefault="true" medium="video" type="application/x-shockwave-flash" url="https://www.youtube.com/v/ofCddrXkw0U?version=3&f=user_uploads&c=ndv&d=Aarb2r5skm2_yNTuKEinXdAO88HsQjpE1a8d1GxQnGDm&app=youtube_gdata" ns2:format="5" xmlns:ns2="http://gdata.youtube.com/schemas/2007" /><ns1:thumbnail height="360" time="00:00:00" url="http://i.ytimg.com/vi/ofCddrXkw0U/0.jpg" width="480" /><ns1:thumbnail height="90" time="00:00:00" url="http://i.ytimg.com/vi/ofCddrXkw0U/1.jpg" width="120" /><ns1:thumbnail height="90" time="00:00:00" url="http://i.ytimg.com/vi/ofCddrXkw0U/2.jpg" width="120" /><ns1:thumbnail height="90" time="00:00:00" url="http://i.ytimg.com/vi/ofCddrXkw0U/3.jpg" width="120" /><ns1:category label="Education" scheme="http://gdata.youtube.com/schemas/2007/categories.cat">Education</ns1:category><ns1:category scheme="http://gdata.youtube.com/schemas/2007/developertags.cat">enthought</ns1:category><ns1:category scheme="http://gdata.youtube.com/schemas/2007/developertags.cat">scipy_2012</ns1:category><ns1:category scheme="http://gdata.youtube.com/schemas/2007/developertags.cat">HPC</ns1:category><ns1:category scheme="http://gdata.youtube.com/schemas/2007/developertags.cat">AlBarrentine</ns1:category><ns1:player url="https://www.youtube.com/watch?v=ofCddrXkw0U&feature=youtube_gdata_player" /></ns1:group><ns0:title type="text">SciPy + MapReduce with Disco</ns0:title><ns1:control xmlns:ns1="http://purl.org/atom/app#"><ns1:draft>yes</ns1:draft><ns2:state name="processing" xmlns:ns2="http://gdata.youtube.com/schemas/2007" /></ns1:control><ns0:link href="https://www.youtube.com/watch?v=ofCddrXkw0U&feature=youtube_gdata" rel="alternate" type="text/html" /><ns0:link href="https://gdata.youtube.com/feeds/api/videos/ofCddrXkw0U/responses?client=ndv" rel="http://gdata.youtube.com/schemas/2007#video.responses" type="application/atom+xml" /><ns0:link href="https://gdata.youtube.com/feeds/api/videos/ofCddrXkw0U/ratings?client=ndv" rel="http://gdata.youtube.com/schemas/2007#video.ratings" type="application/atom+xml" /><ns0:link href="https://gdata.youtube.com/feeds/api/videos/ofCddrXkw0U/complaints?client=ndv" rel="http://gdata.youtube.com/schemas/2007#video.complaints" type="application/atom+xml" /><ns0:link href="https://gdata.youtube.com/feeds/api/videos/ofCddrXkw0U/related?client=ndv" rel="http://gdata.youtube.com/schemas/2007#video.related" type="application/atom+xml" /><ns0:link href="https://gdata.youtube.com/feeds/api/users/nextdayvideo/uploads/ofCddrXkw0U?client=ndv" rel="self" type="application/atom+xml" /><ns0:link href="https://gdata.youtube.com/feeds/api/users/nextdayvideo/uploads/ofCddrXkw0U?client=ndv" rel="edit" type="application/atom+xml" /></ns0:entry> production notes

2012-07-18/12_23_09.dv Apply: 14:23:09 - 14:23:17 ( 00:00:08 ) S: 14:23:09 - E: 14:23:17 D: 00:00:08 show more...

vlc ~/Videos/veyepar/enthought/scipy_2012/dv/106/2012-07-18/12_23_09.dv :start-time=00.0 --audio-desync=0

	Raw File	Cut List
	14:23:09		seconds: 0.0 Wall: 14:23:09
Duration	00:00:08
	14:23:17		seconds: 0.0 Wall: 14:23:09
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

audio viz

2012-07-18/12_23_17.dv Apply: 14:23:17 - 14:47:07 ( 00:23:50 ) S: 14:23:17 - E: 14:47:07 D: 00:23:50 show more...

vlc ~/Videos/veyepar/enthought/scipy_2012/dv/106/2012-07-18/12_23_17.dv :start-time=00.0 --audio-desync=0

	Raw File	Cut List
	14:23:17		seconds: 0.0 Wall: 14:23:17
Duration	00:23:50
	14:47:07		seconds: 0.0 Wall: 14:23:17
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

audio viz

Rf filename:	root is .../show/dv/location/, example: 2013-03-13/13:13:30.dv
Sequence:
get this:	check and save to add this

2012-07-18/12_23_09.dv
audio viz

audio viz

2012-07-18/12_23_17.dv
audio viz

audio viz

Veyepar Video Eyeball Processor and Review