Hi
user
Admin Login:
Username:
Password:
Name:
Total Recall: flmake and the Quest for Reproducibility
--client
enthought
--show
scipy_2012
--room Ballroom_CDE 1203 --force
Prev: 11 IPython: tools for the entire lifecycle of research computing
Next: 11 nD image segmentation using learned region agglomeration with the Ray Python library
show more...
Author(s):
Anthony Scopatz
Location
Ballroom (CDE)
Date
jul Thu 19
Days Raw Files
Start
10:20
First Raw Start
10:03
Duration
20:00
Offset
0:16:26
End
10:40
Last Raw End
10:51
Chapters
00:00
0:14:33
Total cuts_time
21 min.
raw-playlist
raw-mp4-playlist
encoded-files-playlist
host
archive
public
tweet
mp4
svg
png
assets
release.pdf
Total_Recall_flmake_and_the_Quest_for_Reproducibility.json
logs
Admin:
episode
episode list
cut list
raw files day
marks day
marks day
image_files
State:
---------
borked
edit
encode
push to queue
post
richard
review 1
email
review 2
make public
tweet
to-miror
conf
done
Locked:
clear this to unlock
Locked by:
user/process that locked.
Start:
initially scheduled time from master, adjusted to match reality
Duration:
length in hh:mm:ss
Name:
Video Title (shows in video search results)
Emails:
email(s) of the presenter(s)
Released:
Unknown
Yes
No
has someone authorised pubication
Normalise:
Channelcopy:
m=mono, 01=copy left to right, 10=right to left, 00=ignore.
Thumbnail:
filename.png
Description:
`FLASH`_ is a high-performance computing (HPC) multi-physics code which is used to perform astrophysical and high-energy density physics simulations. It runs on the full range of systems from laptops to workstations to 100,000 processor super computers - such as the Blue Gene/P at Argonne National Laboratory. Historically, FLASH was born from a collection of unconnected legacy codes written primarily in Fortran and merged into a single project. Over the past 13 years major sections have been rewritten in other languages. For instance, I/O is now implemented in C. However building, testing, and documentation are all performed in Python. FLASH has a unique architecture which compiles *simulation specific* executables for each new type of run. This is aided by an object-oriented-esque inheritance model that is implemented by inspecting the file system's directory hierarchy. This allows FLASH to compile to faster machine code than a compile-once strategy. However it also places a greater importance on the Python build system. To run a FLASH simulation, the user must go through three basic steps: setup, build, and execution. Canonically, each of these tasks are independently handled by the user. However, with the recent advent of `flmake`_ - a Python workflow management utility for FLASH - such tasks may now be performed in a repeatable way. Previous workflow management tools have been written for FLASH. (For example, the "Milad system" was implemented entirely in Makefiles.) However, none of the priorattempts have placed reproducibility as their primary concern. This is in part becausefully capturing the setup metadata requires alterations to the build system. The development of flmake started by rewriting the existing build systemto allow FLASH to be run outside of the main line subversion repository. It separates outproject and simulation directories independent of the FLASH source directory. Thesedirectories are typically under their own version control. Moreover for each of the important tasks (setup, build, run, etc), a sidecar metadata *description* file is either written or appended to. This is a simple dictionary-of-dictionaries JSON file which stores the environment of the system and the state of the code when each flmake command is run. This metadata includes the version information of both the FLASH main line and project repositories. However, it also may include *all* local modifications since the last commit. A patch is automatically generated using the Python standard library ``difflib`` module and stored directly in the description. Along with universally unique identifiers, logging, and Python run control files, the flmake utility may use the description files to fully reproduce a simulation by re-executing each command in its original environment and state. While ``flmake reproduce`` makes a useful debugging tool, it fundamentally increases the scientific merit of FLASH simulations. The methods described above may be used whenever source code itself is distributed. While this is true for FLASH (uncommon amongst compiledcodes), most Python packages also distribute their source. Therefore the same reproducibility strategy is applicable and highly recommended for Python simulation codes. Thus flmake shows that reproducibility - which is notably absent from most computational science projects - is easily attainable using only version control and standard library modules. .. _FLASH: http://flash.uchicago.edu/site/ .. _flmake: http://flash.uchicago.edu/site/flashcode/user_support/tools4b/usersguide/flmake/index.htm
markdown
Comment:
<?xml version='1.0' encoding='UTF-8'?> <ns0:entry xmlns:ns0="http://www.w3.org/2005/Atom"><ns0:category scheme="http://schemas.google.com/g/2005#kind" term="http://gdata.youtube.com/schemas/2007#video" /><ns0:category label="Education" scheme="http://gdata.youtube.com/schemas/2007/categories.cat" term="Education" /><ns0:category scheme="http://gdata.youtube.com/schemas/2007/keywords.cat" term="enthought" /><ns0:category scheme="http://gdata.youtube.com/schemas/2007/keywords.cat" term="scipy_2012" /><ns0:category scheme="http://gdata.youtube.com/schemas/2007/keywords.cat" term="General" /><ns0:category scheme="http://gdata.youtube.com/schemas/2007/keywords.cat" term="AnthonyScopatz" /><ns0:id>http://gdata.youtube.com/feeds/api/videos/ce4UU0ltbPA</ns0:id><ns0:author><ns0:name>NextDayVideo</ns0:name><ns0:uri>https://gdata.youtube.com/feeds/api/users/NextDayVideo</ns0:uri></ns0:author><ns0:content type="text">Anthony Scopatz</ns0:content><ns0:updated>2012-08-02T22:50:50.000Z</ns0:updated><ns0:published>2012-08-02T22:50:50.000Z</ns0:published><ns1:comments xmlns:ns1="http://schemas.google.com/g/2005"><ns1:feedLink countHint="0" href="https://gdata.youtube.com/feeds/api/videos/ce4UU0ltbPA/comments?client=ndv" rel="http://gdata.youtube.com/schemas/2007#comments" /></ns1:comments><ns1:group xmlns:ns1="http://search.yahoo.com/mrss/"><ns1:keywords>enthought, scipy_2012, General, AnthonyScopatz</ns1:keywords><ns1:description type="plain">Anthony Scopatz</ns1:description><ns1:title type="plain">Total Recall: flmake and the Quest for Reproducibility</ns1:title><ns2:duration seconds="0" xmlns:ns2="http://gdata.youtube.com/schemas/2007" /><ns1:content duration="0" expression="full" isDefault="true" medium="video" type="application/x-shockwave-flash" url="https://www.youtube.com/v/ce4UU0ltbPA?version=3&f=user_uploads&c=ndv&d=Aarb2r5skm2_yNTuKEinXdAO88HsQjpE1a8d1GxQnGDm&app=youtube_gdata" ns2:format="5" xmlns:ns2="http://gdata.youtube.com/schemas/2007" /><ns1:thumbnail height="360" time="00:00:00" url="http://i.ytimg.com/vi/ce4UU0ltbPA/0.jpg" width="480" /><ns1:thumbnail height="90" time="00:00:00" url="http://i.ytimg.com/vi/ce4UU0ltbPA/1.jpg" width="120" /><ns1:thumbnail height="90" time="00:00:00" url="http://i.ytimg.com/vi/ce4UU0ltbPA/2.jpg" width="120" /><ns1:thumbnail height="90" time="00:00:00" url="http://i.ytimg.com/vi/ce4UU0ltbPA/3.jpg" width="120" /><ns1:category label="Education" scheme="http://gdata.youtube.com/schemas/2007/categories.cat">Education</ns1:category><ns1:category scheme="http://gdata.youtube.com/schemas/2007/developertags.cat">enthought</ns1:category><ns1:category scheme="http://gdata.youtube.com/schemas/2007/developertags.cat">scipy_2012</ns1:category><ns1:category scheme="http://gdata.youtube.com/schemas/2007/developertags.cat">General</ns1:category><ns1:category scheme="http://gdata.youtube.com/schemas/2007/developertags.cat">AnthonyScopatz</ns1:category><ns1:player url="https://www.youtube.com/watch?v=ce4UU0ltbPA&feature=youtube_gdata_player" /></ns1:group><ns0:title type="text">Total Recall: flmake and the Quest for Reproducibility</ns0:title><ns1:control xmlns:ns1="http://purl.org/atom/app#"><ns1:draft>yes</ns1:draft><ns2:state name="processing" xmlns:ns2="http://gdata.youtube.com/schemas/2007" /></ns1:control><ns0:link href="https://www.youtube.com/watch?v=ce4UU0ltbPA&feature=youtube_gdata" rel="alternate" type="text/html" /><ns0:link href="https://gdata.youtube.com/feeds/api/videos/ce4UU0ltbPA/responses?client=ndv" rel="http://gdata.youtube.com/schemas/2007#video.responses" type="application/atom+xml" /><ns0:link href="https://gdata.youtube.com/feeds/api/videos/ce4UU0ltbPA/ratings?client=ndv" rel="http://gdata.youtube.com/schemas/2007#video.ratings" type="application/atom+xml" /><ns0:link href="https://gdata.youtube.com/feeds/api/videos/ce4UU0ltbPA/complaints?client=ndv" rel="http://gdata.youtube.com/schemas/2007#video.complaints" type="application/atom+xml" /><ns0:link href="https://gdata.youtube.com/feeds/api/videos/ce4UU0ltbPA/related?client=ndv" rel="http://gdata.youtube.com/schemas/2007#video.related" type="application/atom+xml" /><ns0:link href="https://gdata.youtube.com/feeds/api/users/nextdayvideo/uploads/ce4UU0ltbPA?client=ndv" rel="self" type="application/atom+xml" /><ns0:link href="https://gdata.youtube.com/feeds/api/users/nextdayvideo/uploads/ce4UU0ltbPA?client=ndv" rel="edit" type="application/atom+xml" /></ns0:entry>
production notes
2012-07-19/08_03_34.dv
Apply:
10:03:34 - 10:24:45 ( 00:21:11 )
S:
10:03:34 -
E:
10:24:45
D:
00:21:11
show more...
vlc ~/Videos/veyepar/enthought/scipy_2012/dv/Ballroom_CDE/2012-07-19/08_03_34.dv :start-time=00.0 --audio-desync=0
Raw File
Cut List
10:03:34
seconds: 0.0
Wall: 10:03:34
Duration
00:21:11
10:24:45
seconds: 0.0
Wall: 10:03:34
Comments:
mp4
mp4.m3u
dv.m3u
Split:
Sequence:
:
delete
2012-07-19/08_24_46.dv
Apply:
10:24:46 - 10:28:35 ( 00:03:49 )
S:
10:24:46 -
E:
10:28:35
D:
00:03:49
show more...
vlc ~/Videos/veyepar/enthought/scipy_2012/dv/Ballroom_CDE/2012-07-19/08_24_46.dv :start-time=00.0 --audio-desync=0
Raw File
Cut List
10:24:46
seconds: 0.0
Wall: 10:24:46
Duration
00:03:49
10:28:35
seconds: 0.0
Wall: 10:24:46
Comments:
mp4
mp4.m3u
dv.m3u
Split:
Sequence:
:
delete
2012-07-19/08_28_35.dv
Apply:
10:28:35 - 10:29:31 ( 00:00:56 )
S:
10:28:35 -
E:
10:29:31
D:
00:00:56
show more...
vlc ~/Videos/veyepar/enthought/scipy_2012/dv/Ballroom_CDE/2012-07-19/08_28_35.dv :start-time=00.0 --audio-desync=0
Raw File
Cut List
10:28:35
seconds: 0.0
Wall: 10:28:35
Duration
00:00:56
10:29:31
seconds: 0.0
Wall: 10:28:35
Comments:
mp4
mp4.m3u
dv.m3u
Split:
Sequence:
:
delete
2012-07-19/08_29_32.dv
Apply:
10:29:32 - 10:29:37 ( 00:00:05 )
S:
10:29:32 -
E:
10:29:37
D:
00:00:05
show more...
vlc ~/Videos/veyepar/enthought/scipy_2012/dv/Ballroom_CDE/2012-07-19/08_29_32.dv :start-time=00.0 --audio-desync=0
Raw File
Cut List
10:29:32
seconds: 0.0
Wall: 10:29:32
Duration
00:00:05
10:29:37
seconds: 0.0
Wall: 10:29:32
Comments:
mp4
mp4.m3u
dv.m3u
Split:
Sequence:
:
delete
2012-07-19/08_29_37.dv
Apply:
10:29:37 - 10:44:10 ( 00:14:33 )
S:
10:29:37 -
E:
10:44:10
D:
00:14:33
show more...
vlc ~/Videos/veyepar/enthought/scipy_2012/dv/Ballroom_CDE/2012-07-19/08_29_37.dv :start-time=00.0 --audio-desync=0
Raw File
Cut List
10:29:37
seconds: 0.0
Wall: 10:29:37
Duration
00:14:33
10:44:10
seconds: 0.0
Wall: 10:29:37
Comments:
mp4
mp4.m3u
dv.m3u
Split:
Sequence:
:
delete
2012-07-19/08_44_11.dv
Apply:
10:44:11 - 10:51:05 ( 00:06:54 )
S:
10:44:11 -
E:
10:51:05
D:
00:06:54
show more...
vlc ~/Videos/veyepar/enthought/scipy_2012/dv/Ballroom_CDE/2012-07-19/08_44_11.dv :start-time=00.0 --audio-desync=0
Raw File
Cut List
10:44:11
seconds: 0.0
Wall: 10:44:11
Duration
00:06:54
10:51:05
seconds: 0.0
Wall: 10:44:11
Comments:
mp4
mp4.m3u
dv.m3u
Split:
Sequence:
:
delete
Rf filename:
root is .../show/dv/location/, example: 2013-03-13/13:13:30.dv
Sequence:
get this:
check and save to add this
2012-07-19/08_03_34.dv
2012-07-19/08_24_46.dv
2012-07-19/08_28_35.dv
2012-07-19/08_29_32.dv
2012-07-19/08_29_37.dv
2012-07-19/08_44_11.dv
Veyepar
Video Eyeball Processor and Review
