veyepar: Solving the import problem: Scalable Dynamic Loading Network File Systems

Hi user

Admin Login:

Name: Solving the import problem: Scalable Dynamic Loading Network File Systems
--client enthought --show scipy_2012 --room 106 1172 --force
Prev: 11 Copperhead: Data Parallel Python
Next: 0 mystic: a simple model-independent inversion framework

show more...

Author(s):	Aron Ahmadia, Jed Brown, William Scullin
Location	106
Date	jul Wed 18	Days Raw Files
Start	11:05	First Raw Start	10:52
Duration	20:00	Offset	0:12:16
End	11:25	Last Raw End	11:31
Chapters	00:00
Total cuts_time	23 min.

raw-playlist raw-mp4-playlist encoded-files-playlist host archive public tweet mp4 svg png

final audio

assets release.pdf Solving_the_import_problem_Scalable_Dynamic_Loading_Network_File_Systems.json
logs
Admin: episode episode list cut list raw files day marks day marks day image_files

State:
Locked:	clear this to unlock
Locked by:	user/process that locked.
Start:	initially scheduled time from master, adjusted to match reality
Duration:	length in hh:mm:ss
Name:	Video Title (shows in video search results)
Emails:	email(s) of the presenter(s)
Released:	has someone authorised pubication
Normalise:
Channelcopy:	m=mono, 01=copy left to right, 10=right to left, 00=ignore.
Thumbnail:	filename.png
Description:	The most common programming paradigm for scientific computing, SPMD (Single Program Multiple Data), catastrophically interacts with the loading strategies of dynamically linked executables and network-attached file systems on even moderately sized high performance computing clusters. This difficulty is further exacerbated by "function-shipped" I/O on modern supercomputer compute nodes, preventing the deployment of simple solutions. In this talk, we introduce a two-component solution: collfs, a set of low-level MPI-collective file operations that can selectively shadow file system access in a library, and walla, a set of Python import hooks for seamlessly enabling parallel dynamic loading scalable to tens of thousands of cores. markdown
Comment:	<?xml version='1.0' encoding='UTF-8'?> <ns0:entry xmlns:ns0="http://www.w3.org/2005/Atom"><ns0:category scheme="http://schemas.google.com/g/2005#kind" term="http://gdata.youtube.com/schemas/2007#video" /><ns0:category label="Education" scheme="http://gdata.youtube.com/schemas/2007/categories.cat" term="Education" /><ns0:category scheme="http://gdata.youtube.com/schemas/2007/keywords.cat" term="enthought" /><ns0:category scheme="http://gdata.youtube.com/schemas/2007/keywords.cat" term="scipy_2012" /><ns0:category scheme="http://gdata.youtube.com/schemas/2007/keywords.cat" term="HPC" /><ns0:category scheme="http://gdata.youtube.com/schemas/2007/keywords.cat" term="AronAhmadia" /><ns0:category scheme="http://gdata.youtube.com/schemas/2007/keywords.cat" term="JedBrown" /><ns0:category scheme="http://gdata.youtube.com/schemas/2007/keywords.cat" term="WilliamScullin" /><ns0:id>http://gdata.youtube.com/feeds/api/videos/BpuykTOy4a0</ns0:id><ns0:author><ns0:name>NextDayVideo</ns0:name><ns0:uri>https://gdata.youtube.com/feeds/api/users/NextDayVideo</ns0:uri></ns0:author><ns0:content type="text">Aron Ahmadia, Jed Brown, William Scullin</ns0:content><ns0:updated>2012-08-02T22:06:31.000Z</ns0:updated><ns0:published>2012-08-02T22:06:31.000Z</ns0:published><ns1:comments xmlns:ns1="http://schemas.google.com/g/2005"><ns1:feedLink countHint="0" href="https://gdata.youtube.com/feeds/api/videos/BpuykTOy4a0/comments?client=ndv" rel="http://gdata.youtube.com/schemas/2007#comments" /></ns1:comments><ns1:group xmlns:ns1="http://search.yahoo.com/mrss/"><ns1:keywords>enthought, scipy_2012, HPC, AronAhmadia, JedBrown, WilliamScullin</ns1:keywords><ns1:description type="plain">Aron Ahmadia, Jed Brown, William Scullin</ns1:description><ns1:title type="plain">Solving the import problem: Scalable Dynamic Loading Network File Systems</ns1:title><ns2:duration seconds="0" xmlns:ns2="http://gdata.youtube.com/schemas/2007" /><ns1:content duration="0" expression="full" isDefault="true" medium="video" type="application/x-shockwave-flash" url="https://www.youtube.com/v/BpuykTOy4a0?version=3&f=user_uploads&c=ndv&d=Aarb2r5skm2_yNTuKEinXdAO88HsQjpE1a8d1GxQnGDm&app=youtube_gdata" ns2:format="5" xmlns:ns2="http://gdata.youtube.com/schemas/2007" /><ns1:thumbnail height="360" time="00:00:00" url="http://i.ytimg.com/vi/BpuykTOy4a0/0.jpg" width="480" /><ns1:thumbnail height="90" time="00:00:00" url="http://i.ytimg.com/vi/BpuykTOy4a0/1.jpg" width="120" /><ns1:thumbnail height="90" time="00:00:00" url="http://i.ytimg.com/vi/BpuykTOy4a0/2.jpg" width="120" /><ns1:thumbnail height="90" time="00:00:00" url="http://i.ytimg.com/vi/BpuykTOy4a0/3.jpg" width="120" /><ns1:category label="Education" scheme="http://gdata.youtube.com/schemas/2007/categories.cat">Education</ns1:category><ns1:category scheme="http://gdata.youtube.com/schemas/2007/developertags.cat">enthought</ns1:category><ns1:category scheme="http://gdata.youtube.com/schemas/2007/developertags.cat">scipy_2012</ns1:category><ns1:category scheme="http://gdata.youtube.com/schemas/2007/developertags.cat">HPC</ns1:category><ns1:category scheme="http://gdata.youtube.com/schemas/2007/developertags.cat">AronAhmadia</ns1:category><ns1:category scheme="http://gdata.youtube.com/schemas/2007/developertags.cat">JedBrown</ns1:category><ns1:category scheme="http://gdata.youtube.com/schemas/2007/developertags.cat">WilliamScullin</ns1:category><ns1:player url="https://www.youtube.com/watch?v=BpuykTOy4a0&feature=youtube_gdata_player" /></ns1:group><ns0:title type="text">Solving the import problem: Scalable Dynamic Loading Network File Systems</ns0:title><ns1:control xmlns:ns1="http://purl.org/atom/app#"><ns1:draft>yes</ns1:draft><ns2:state name="processing" xmlns:ns2="http://gdata.youtube.com/schemas/2007" /></ns1:control><ns0:link href="https://www.youtube.com/watch?v=BpuykTOy4a0&feature=youtube_gdata" rel="alternate" type="text/html" /><ns0:link href="https://gdata.youtube.com/feeds/api/videos/BpuykTOy4a0/responses?client=ndv" rel="http://gdata.youtube.com/schemas/2007#video.responses" type="application/atom+xml" /><ns0:link href="https://gdata.youtube.com/feeds/api/videos/BpuykTOy4a0/ratings?client=ndv" rel="http://gdata.youtube.com/schemas/2007#video.ratings" type="application/atom+xml" /><ns0:link href="https://gdata.youtube.com/feeds/api/videos/BpuykTOy4a0/complaints?client=ndv" rel="http://gdata.youtube.com/schemas/2007#video.complaints" type="application/atom+xml" /><ns0:link href="https://gdata.youtube.com/feeds/api/videos/BpuykTOy4a0/related?client=ndv" rel="http://gdata.youtube.com/schemas/2007#video.related" type="application/atom+xml" /><ns0:link href="https://gdata.youtube.com/feeds/api/users/nextdayvideo/uploads/BpuykTOy4a0?client=ndv" rel="self" type="application/atom+xml" /><ns0:link href="https://gdata.youtube.com/feeds/api/users/nextdayvideo/uploads/BpuykTOy4a0?client=ndv" rel="edit" type="application/atom+xml" /></ns0:entry> production notes

2012-07-18/08_52_44.dv Apply: 10:52:44 - 11:07:22 ( 00:14:38 ) S: 10:52:44 - E: 11:07:22 D: 00:14:38 show more...

vlc ~/Videos/veyepar/enthought/scipy_2012/dv/106/2012-07-18/08_52_44.dv :start-time=00.0 --audio-desync=0

	Raw File	Cut List
	10:52:44		seconds: 0.0 Wall: 10:52:44
Duration	00:14:38
	11:07:22		seconds: 0.0 Wall: 10:52:44
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

audio viz

2012-07-18/09_07_23.dv Apply: 11:07:23 - 11:07:28 ( 00:00:05 ) S: 11:07:23 - E: 11:07:28 D: 00:00:05 show more...

vlc ~/Videos/veyepar/enthought/scipy_2012/dv/106/2012-07-18/09_07_23.dv :start-time=00.0 --audio-desync=0

	Raw File	Cut List
	11:07:23		seconds: 0.0 Wall: 11:07:23
Duration	00:00:05
	11:07:28		seconds: 0.0 Wall: 11:07:23
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

audio viz

2012-07-18/09_07_29.dv Apply: 11:07:29 - 11:31:18 ( 00:23:49 ) S: 11:07:29 - E: 11:31:18 D: 00:23:49 show more...

vlc ~/Videos/veyepar/enthought/scipy_2012/dv/106/2012-07-18/09_07_29.dv :start-time=00.0 --audio-desync=0

	Raw File	Cut List
	11:07:29		seconds: 0.0 Wall: 11:07:29
Duration	00:23:49
	11:31:18		seconds: 0.0 Wall: 11:07:29
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

audio viz

Rf filename:	root is .../show/dv/location/, example: 2013-03-13/13:13:30.dv
Sequence:
get this:	check and save to add this

2012-07-18/08_52_44.dv
audio viz

audio viz

2012-07-18/09_07_23.dv
audio viz

audio viz

2012-07-18/09_07_29.dv
audio viz

audio viz

Veyepar Video Eyeball Processor and Review