veyepar: Parallel High Performance Statistical Bootstrapping in Python

Hi user

Admin Login:

Name: Parallel High Performance Statistical Bootstrapping in Python
--client enthought --show scipy_2012 --room Ballroom_CDE 1184 --force
Prev: 11 SymPy Stats - Uncertainty Modeling
Next: 0 Antonia

show more...

Author(s):	Aakash Prasad
Location	Ballroom (CDE)
Date	jul Wed 18	Days Raw Files
Start	15:30	First Raw Start	15:04
Duration	20:00	Offset	0:25:37
End	15:50	Last Raw End	15:55
Chapters	00:00 0:10:51
Total cuts_time	12 min.

raw-playlist raw-mp4-playlist encoded-files-playlist host archive public tweet mp4 svg png

final audio

assets release.pdf Parallel_High_Performance_Statistical_Bootstrapping_in_Python.json
logs
Admin: episode episode list cut list raw files day marks day marks day image_files

State:
Locked:	clear this to unlock
Locked by:	user/process that locked.
Start:	initially scheduled time from master, adjusted to match reality
Duration:	length in hh:mm:ss
Name:	Video Title (shows in video search results)
Emails:	email(s) of the presenter(s)
Released:	has someone authorised pubication
Normalise:
Channelcopy:	m=mono, 01=copy left to right, 10=right to left, 00=ignore.
Thumbnail:	filename.png
Description:	BLB ("Bag of Little Bootstraps") is a method to assess the quality of a statistical estimator based upon subsets of sample distributions. BLB is a variant of and solves the same class of problems as the general bootstrap. Unfortunately, the general bootstrap is a computationally demanding operation when given large data sets, and does not parallelize easily. BLB is an attractive alternative due to its its structural and computational properties which allow for much better parallelization. However, two obstacles exist to realizing this parallelism in practice. First, expressing the parallelism inherent in the algorithm requires quite different code depending on the platform (for example, multi-core with Cilk-aware compiler vs. GPU with CUDA or OpenCL vs. shared-nothing cloud computing with Spark or Hadoop). Second, even given the skeleton code for a particular platform, the specific estimator function being computed is different for each application, making it difficult to encapsulate the BLB pattern in a library. We apply the SEJITS technology (Selective Embedded Just-in-Time Specialization) to solve both problems: scientists can write applications in Python that make use of estimator functions also written in (a subset of) Python, and just-in-time code generation techniques are used to "lower" these functions to efficiency-level languages and inline them into an execution template optimized for each type of parallel platform; in this paper we focus on the multicore environment. The result is that Python applications that use BLB are source- and performance-portable, with performance comparable to hand-written code in low-level languages. We expect that code variants produced for the multicore environment can reasonably support data sets in the order of tens of gigabytes in size, and that variants for the cloud environment can support data sets in the order of terabytes in size. Preliminary results from a simple application of linear regression show a 13.6x speedup as a result of using 16 cores instead of 1 core, and this parallel performance was obtained by simply coding the linear estimator function in Python. Our runtime compiler (specializer) for BLB augments a growing family of such compilers that will ultimately allow Python applications to make use of generic computational patterns across different platforms. markdown
Comment:	<?xml version='1.0' encoding='UTF-8'?> <ns0:entry xmlns:ns0="http://www.w3.org/2005/Atom"><ns0:category scheme="http://schemas.google.com/g/2005#kind" term="http://gdata.youtube.com/schemas/2007#video" /><ns0:category label="Education" scheme="http://gdata.youtube.com/schemas/2007/categories.cat" term="Education" /><ns0:category scheme="http://gdata.youtube.com/schemas/2007/keywords.cat" term="enthought" /><ns0:category scheme="http://gdata.youtube.com/schemas/2007/keywords.cat" term="scipy_2012" /><ns0:category scheme="http://gdata.youtube.com/schemas/2007/keywords.cat" term="HPC" /><ns0:category scheme="http://gdata.youtube.com/schemas/2007/keywords.cat" term="AakashPrasad" /><ns0:category scheme="http://gdata.youtube.com/schemas/2007/keywords.cat" term="DavidHoward" /><ns0:category scheme="http://gdata.youtube.com/schemas/2007/keywords.cat" term="ShoaibKamil" /><ns0:category scheme="http://gdata.youtube.com/schemas/2007/keywords.cat" term="ArmandoFox" /><ns0:id>http://gdata.youtube.com/feeds/api/videos/pwx5lPBjI54</ns0:id><ns0:author><ns0:name>NextDayVideo</ns0:name><ns0:uri>https://gdata.youtube.com/feeds/api/users/NextDayVideo</ns0:uri></ns0:author><ns0:content type="text">Aakash Prasad, David Howard, Shoaib Kamil, Armando Fox</ns0:content><ns0:updated>2012-08-02T22:35:58.000Z</ns0:updated><ns0:published>2012-08-02T22:35:58.000Z</ns0:published><ns1:comments xmlns:ns1="http://schemas.google.com/g/2005"><ns1:feedLink countHint="0" href="https://gdata.youtube.com/feeds/api/videos/pwx5lPBjI54/comments?client=ndv" rel="http://gdata.youtube.com/schemas/2007#comments" /></ns1:comments><ns1:group xmlns:ns1="http://search.yahoo.com/mrss/"><ns1:keywords>enthought, scipy_2012, HPC, AakashPrasad, DavidHoward, ShoaibKamil, ArmandoFox</ns1:keywords><ns1:description type="plain">Aakash Prasad, David Howard, Shoaib Kamil, Armando Fox</ns1:description><ns1:title type="plain">Parallel High Performance Statistical Bootstrapping in Python</ns1:title><ns2:duration seconds="0" xmlns:ns2="http://gdata.youtube.com/schemas/2007" /><ns1:content duration="0" expression="full" isDefault="true" medium="video" type="application/x-shockwave-flash" url="https://www.youtube.com/v/pwx5lPBjI54?version=3&f=user_uploads&c=ndv&d=Aarb2r5skm2_yNTuKEinXdAO88HsQjpE1a8d1GxQnGDm&app=youtube_gdata" ns2:format="5" xmlns:ns2="http://gdata.youtube.com/schemas/2007" /><ns1:thumbnail height="360" time="00:00:00" url="http://i.ytimg.com/vi/pwx5lPBjI54/0.jpg" width="480" /><ns1:thumbnail height="90" time="00:00:00" url="http://i.ytimg.com/vi/pwx5lPBjI54/1.jpg" width="120" /><ns1:thumbnail height="90" time="00:00:00" url="http://i.ytimg.com/vi/pwx5lPBjI54/2.jpg" width="120" /><ns1:thumbnail height="90" time="00:00:00" url="http://i.ytimg.com/vi/pwx5lPBjI54/3.jpg" width="120" /><ns1:category label="Education" scheme="http://gdata.youtube.com/schemas/2007/categories.cat">Education</ns1:category><ns1:category scheme="http://gdata.youtube.com/schemas/2007/developertags.cat">enthought</ns1:category><ns1:category scheme="http://gdata.youtube.com/schemas/2007/developertags.cat">scipy_2012</ns1:category><ns1:category scheme="http://gdata.youtube.com/schemas/2007/developertags.cat">HPC</ns1:category><ns1:category scheme="http://gdata.youtube.com/schemas/2007/developertags.cat">AakashPrasad</ns1:category><ns1:category scheme="http://gdata.youtube.com/schemas/2007/developertags.cat">DavidHoward</ns1:category><ns1:category scheme="http://gdata.youtube.com/schemas/2007/developertags.cat">ShoaibKamil</ns1:category><ns1:category scheme="http://gdata.youtube.com/schemas/2007/developertags.cat">ArmandoFox</ns1:category><ns1:player url="https://www.youtube.com/watch?v=pwx5lPBjI54&feature=youtube_gdata_player" /></ns1:group><ns0:title type="text">Parallel High Performance Statistical Bootstrapping in Python</ns0:title><ns1:control xmlns:ns1="http://purl.org/atom/app#"><ns1:draft>yes</ns1:draft><ns2:state name="processing" xmlns:ns2="http://gdata.youtube.com/schemas/2007" /></ns1:control><ns0:link href="https://www.youtube.com/watch?v=pwx5lPBjI54&feature=youtube_gdata" rel="alternate" type="text/html" /><ns0:link href="https://gdata.youtube.com/feeds/api/videos/pwx5lPBjI54/responses?client=ndv" rel="http://gdata.youtube.com/schemas/2007#video.responses" type="application/atom+xml" /><ns0:link href="https://gdata.youtube.com/feeds/api/videos/pwx5lPBjI54/ratings?client=ndv" rel="http://gdata.youtube.com/schemas/2007#video.ratings" type="application/atom+xml" /><ns0:link href="https://gdata.youtube.com/feeds/api/videos/pwx5lPBjI54/complaints?client=ndv" rel="http://gdata.youtube.com/schemas/2007#video.complaints" type="application/atom+xml" /><ns0:link href="https://gdata.youtube.com/feeds/api/videos/pwx5lPBjI54/related?client=ndv" rel="http://gdata.youtube.com/schemas/2007#video.related" type="application/atom+xml" /><ns0:link href="https://gdata.youtube.com/feeds/api/users/nextdayvideo/uploads/pwx5lPBjI54?client=ndv" rel="self" type="application/atom+xml" /><ns0:link href="https://gdata.youtube.com/feeds/api/users/nextdayvideo/uploads/pwx5lPBjI54?client=ndv" rel="edit" type="application/atom+xml" /></ns0:entry> <?xml version='1.0' encoding='UTF-8'?> <ns0:entry xmlns:ns0="http://www.w3.org/2005/Atom" xmlns:ns1="http://schemas.google.com/g/2005" xmlns:ns2="http://search.yahoo.com/mrss/" xmlns:ns3="http://gdata.youtube.com/schemas/2007" xmlns:ns4="http://purl.org/atom/app#"><ns0:category scheme="http://schemas.google.com/g/2005#kind" term="http://gdata.youtube.com/schemas/2007#video" /><ns0:category label="Education" scheme="http://gdata.youtube.com/schemas/2007/categories.cat" term="Education" /><ns0:category scheme="http://gdata.youtube.com/schemas/2007/keywords.cat" term="enthought" /><ns0:category scheme="http://gdata.youtube.com/schemas/2007/keywords.cat" term="scipy_2012" /><ns0:category scheme="http://gdata.youtube.com/schemas/2007/keywords.cat" term="HPC" /><ns0:category scheme="http://gdata.youtube.com/schemas/2007/keywords.cat" term="AakashPrasad" /><ns0:id>http://gdata.youtube.com/feeds/api/videos/vxEOYn_wedQ</ns0:id><ns0:author><ns0:name>NextDayVideo</ns0:name><ns0:uri>https://gdata.youtube.com/feeds/api/users/NextDayVideo</ns0:uri></ns0:author><ns0:content type="text">Aakash Prasad BLB ("Bag of Little Bootstraps") is a method to assess the quality of a statistical estimator based upon subsets of sample distributions. BLB is a variant of and solves the same class of problems as the general bootstrap. Unfortunately,</ns0:content><ns0:updated>2012-09-04T06:22:51.000Z</ns0:updated><ns0:published>2012-09-04T06:22:51.000Z</ns0:published><ns1:comments><ns1:feedLink countHint="0" href="https://gdata.youtube.com/feeds/api/videos/vxEOYn_wedQ/comments?client=NextDayVideo" rel="http://gdata.youtube.com/schemas/2007#comments" /></ns1:comments><ns2:group><ns2:keywords>enthought, scipy_2012, HPC, AakashPrasad</ns2:keywords><ns2:description type="plain">Aakash Prasad BLB ("Bag of Little Bootstraps") is a method to assess the quality of a statistical estimator based upon subsets of sample distributions. BLB is a variant of and solves the same class of problems as the general bootstrap. Unfortunately,</ns2:description><ns2:title type="plain">Parallel High Performance Statistical Bootstrapping in Python</ns2:title><ns3:duration seconds="0" /><ns2:content duration="0" expression="full" isDefault="true" medium="video" type="application/x-shockwave-flash" url="https://www.youtube.com/v/vxEOYn_wedQ?version=3&f=user_uploads&c=NextDayVideo&d=Aarb2r5skm2_yNTuKEinXdAO88HsQjpE1a8d1GxQnGDm&app=youtube_gdata" ns3:format="5" /><ns2:thumbnail height="360" time="00:00:00" url="http://i.ytimg.com/vi/vxEOYn_wedQ/0.jpg" width="480" /><ns2:thumbnail height="90" time="00:00:00" url="http://i.ytimg.com/vi/vxEOYn_wedQ/1.jpg" width="120" /><ns2:thumbnail height="90" time="00:00:00" url="http://i.ytimg.com/vi/vxEOYn_wedQ/2.jpg" width="120" /><ns2:thumbnail height="90" time="00:00:00" url="http://i.ytimg.com/vi/vxEOYn_wedQ/3.jpg" width="120" /><ns2:category label="Education" scheme="http://gdata.youtube.com/schemas/2007/categories.cat">Education</ns2:category><ns2:category scheme="http://gdata.youtube.com/schemas/2007/developertags.cat">enthought</ns2:category><ns2:category scheme="http://gdata.youtube.com/schemas/2007/developertags.cat">scipy_2012</ns2:category><ns2:category scheme="http://gdata.youtube.com/schemas/2007/developertags.cat">HPC</ns2:category><ns2:category scheme="http://gdata.youtube.com/schemas/2007/developertags.cat">AakashPrasad</ns2:category><ns2:player url="https://www.youtube.com/watch?v=vxEOYn_wedQ&feature=youtube_gdata_player" /></ns2:group><ns0:title type="text">Parallel High Performance Statistical Bootstrapping in Python</ns0:title><ns4:control><ns4:draft>yes</ns4:draft><ns3:state name="processing" /></ns4:control><ns0:link href="https://www.youtube.com/watch?v=vxEOYn_wedQ&feature=youtube_gdata" rel="alternate" type="text/html" /><ns0:link href="https://gdata.youtube.com/feeds/api/videos/vxEOYn_wedQ/responses?client=NextDayVideo" rel="http://gdata.youtube.com/schemas/2007#video.responses" type="application/atom+xml" /><ns0:link href="https://gdata.youtube.com/feeds/api/videos/vxEOYn_wedQ/ratings?client=NextDayVideo" rel="http://gdata.youtube.com/schemas/2007#video.ratings" type="application/atom+xml" /><ns0:link href="https://gdata.youtube.com/feeds/api/videos/vxEOYn_wedQ/complaints?client=NextDayVideo" rel="http://gdata.youtube.com/schemas/2007#video.complaints" type="application/atom+xml" /><ns0:link href="https://gdata.youtube.com/feeds/api/videos/vxEOYn_wedQ/related?client=NextDayVideo" rel="http://gdata.youtube.com/schemas/2007#video.related" type="application/atom+xml" /><ns0:link href="https://gdata.youtube.com/feeds/api/users/nextdayvideo/uploads/vxEOYn_wedQ?client=NextDayVideo" rel="self" type="application/atom+xml" /><ns0:link href="https://gdata.youtube.com/feeds/api/users/nextdayvideo/uploads/vxEOYn_wedQ?client=NextDayVideo" rel="edit" type="application/atom+xml" /></ns0:entry> production notes

2012-07-18/13_04_23.dv Apply: 15:04:23 - 15:30:28 ( 00:26:05 ) S: 15:04:23 - E: 15:30:28 D: 00:26:05 show more...

vlc ~/Videos/veyepar/enthought/scipy_2012/dv/Ballroom_CDE/2012-07-18/13_04_23.dv :start-time=00.0 --audio-desync=0

	Raw File	Cut List
	15:04:23		seconds: 0.0 Wall: 15:04:23
Duration	00:26:05
	15:30:28		seconds: 0.0 Wall: 15:04:23
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

audio viz

2012-07-18/13_30_30.dv Apply: 15:30:30 - 15:41:21 ( 00:10:51 ) S: 15:30:30 - E: 15:41:21 D: 00:10:51 show more...

vlc ~/Videos/veyepar/enthought/scipy_2012/dv/Ballroom_CDE/2012-07-18/13_30_30.dv :start-time=00.0 --audio-desync=0

	Raw File	Cut List
	15:30:30		seconds: 0.0 Wall: 15:30:30
Duration	00:10:51
	15:41:21		seconds: 0.0 Wall: 15:30:30
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

audio viz

2012-07-18/13_41_21.dv Apply: 15:41:21 - 15:42:31 ( 00:01:10 ) S: 15:41:21 - E: 15:42:31 D: 00:01:10 show more...

vlc ~/Videos/veyepar/enthought/scipy_2012/dv/Ballroom_CDE/2012-07-18/13_41_21.dv :start-time=00.0 --audio-desync=0

	Raw File	Cut List
	15:41:21		seconds: 0.0 Wall: 15:41:21
Duration	00:01:10
	15:42:31		seconds: 0.0 Wall: 15:41:21
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

audio viz

2012-07-18/13_42_32.dv Apply: 15:42:32 - 15:43:51 ( 00:01:19 ) S: 15:42:32 - E: 15:43:51 D: 00:01:19 show more...

vlc ~/Videos/veyepar/enthought/scipy_2012/dv/Ballroom_CDE/2012-07-18/13_42_32.dv :start-time=00.0 --audio-desync=0

	Raw File	Cut List
	15:42:32		seconds: 0.0 Wall: 15:42:32
Duration	00:01:19
	15:43:51		seconds: 0.0 Wall: 15:42:32
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

audio viz

2012-07-18/13_44_21.dv Apply: 15:44:21 - 15:55:33 ( 00:11:12 ) S: 15:44:21 - E: 15:55:33 D: 00:11:12 show more...

vlc ~/Videos/veyepar/enthought/scipy_2012/dv/Ballroom_CDE/2012-07-18/13_44_21.dv :start-time=00.0 --audio-desync=0

	Raw File	Cut List
	15:44:21		seconds: 0.0 Wall: 15:44:21
Duration	00:11:12
	15:55:33		seconds: 0.0 Wall: 15:44:21
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

audio viz

Rf filename:	root is .../show/dv/location/, example: 2013-03-13/13:13:30.dv
Sequence:
get this:	check and save to add this

2012-07-18/13_04_23.dv
audio viz

audio viz

2012-07-18/13_30_30.dv
audio viz

audio viz

2012-07-18/13_41_21.dv
audio viz

audio viz

2012-07-18/13_42_32.dv
audio viz

audio viz

2012-07-18/13_44_21.dv
audio viz

audio viz

Veyepar Video Eyeball Processor and Review