veyepar: tardis - an interpreter for command-line parallel execution

Author(s):	Alan McCulloch
Location	Track 2
Date	sep Sat 13	Days Raw Files
Start	16:30	First Raw Start	error-in-template
Duration	00:30:00	Offset	None
End	17:00	Last Raw End
Chapters
Total cuts_time	None min.

https://kiwi.pycon.org/schedule/presentation/50/
raw-playlist raw-mp4-playlist encoded-files-playlist host public mp4 svg png

assets release.pdf tardis_an_interpreter_for_commandline_parallel_execution.json
logs
Admin: episode episode list cut list raw files day marks day marks day image_files

State:

Locked:

clear this to unlock

Locked by:

user/process that locked.

Start:

initially scheduled time from master, adjusted to match reality

Duration:

length in hh:mm:ss

Name:

Video Title (shows in video search results)

Emails:

email(s) of the presenter(s)

Released:

has someone authorised pubication

Normalise:

Channelcopy:

m=mono, 01=copy left to right, 10=right to left, 00=ignore.

Thumbnail:

filename.png

Description:

markdown

= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = 
Alan McCulloch:
tardis - an interpreter for command-line parallel execution
= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = 
@ Kiwi PyCon 2014 - Saturday, 13 Sep 2014 - Track 2
http://kiwi.pycon.org/

**Audience level**

Intermediate

**Description**

This talk is about simplifying the command line interface to local or cluster based parallel computing.

**Abstract**

Ideally the user of a command shell would be unaware whether their commands were executed as a single process on the local machine, or as many concurrent processes on either their local machine or a remote cluster, apart from the reduced time taken to complete the command if executed as concurrent processes. We have developed an approach which we call “command conditioning” in which the user marks up a command with hints to the interpreter which are used to transform the marked-up command into “(re)conditioned” native shell commands which the interpreter then launches concurrently and monitors, collects and collates output and termination status. We have implemented an initial python based command-conditioning interpreter called tardis. We describe tardis, give examples of the class of compute tasks for which it is suited, and briefly outline key compute-cluster design characteristics which support this approach. We also touch on future work such as potential integration with Galaxy, a popular python based workflow system.

**Slides**

https://speakerdeck.com/nzpug/alan-mcculloch-tardis-an-interpreter-for-command-line-parallel-execution

Comment:

production notes