veyepar: Script Layering for large-scale number crunching in Python

Author(s):	Catherine Moroney
Location	Mystic Theatre
Date	nov Sun 03	Days Raw Files
Start	11:25	First Raw Start	11:03
Duration	0:18:0	Offset	0:22:15
End	11:43	Last Raw End	12:03
Chapters	00:00 0:07:44
Total cuts_time	22 min.

https://2019.northbaypython.org/schedule/presentation/8/
raw-playlist raw-mp4-playlist encoded-files-playlist host archive tweet mp4 svg png

assets release.pdf Script_Layering_for_largescale_number_crunching_in_Python.json
logs
Admin: episode episode list cut list raw files day marks day marks day image_files

State:

Locked:

clear this to unlock

Locked by:

user/process that locked.

Start:

initially scheduled time from master, adjusted to match reality

Duration:

length in hh:mm:ss

Name:

Video Title (shows in video search results)

Emails:

email(s) of the presenter(s)

Released:

has someone authorised pubication

Normalise:

Channelcopy:

m=mono, 01=copy left to right, 10=right to left, 00=ignore.

Thumbnail:

filename.png

Description:

markdown

Python's combination of number-crunching and plotting prowess, combined with scripting makes it easy to submit a massive number of scientific jobs in a fully-automated fashion.  I often need to run hundreds or thousands of orbits through the processing software and by dividing the duty into multiple scripts, each calling the one below, I have a customizable and automated way of finding the input files, caching them to a local disk, creating the input files, submitting each orbit to our batch-processing system and monitoring their progress so the entire chain consisting of 10-15 executables can all be run in order with a single command.

Python's scripting capabilities, combined with its number-crunching utility make it very easy to gather up all the data, submit thousands of jobs and then monitor their progress, all with a single command.  One script submits a single orbit, another one calls the first to run a month or year worth of data, and then a third calls the second to run the entire processing chain.  Script layering!

Comment:

production notes

2019-11-03/11_03_14.ts Apply: 11:04:42 - 11:17:51 ( 00:13:09 ) S: 11:03:14 - E: 11:33:13 D: 00:29:59 (Start: 88.0) (End: 877.0) show more...

vlc ~/Videos/veyepar/nbpy/nbpy19/dv/mystic/2019-11-03/11_03_14.ts :start-time=088.0 --audio-desync=0

	Raw File	Cut List
	11:03:14		seconds: 88.0 Wall: 11:04:42
Duration	00:29:59
	11:33:13		seconds: 877.0 Wall: 11:17:51
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

2019-11-03/11_03_14.ts Apply: 11:17:51 - 11:25:30 ( 00:07:39 ) S: 11:03:14 - E: 11:33:13 D: 00:29:59 (Start: 877.0) (End: 1336.0) show more...

vlc ~/Videos/veyepar/nbpy/nbpy19/dv/mystic/2019-11-03/11_03_14.ts :start-time=0877.0 --audio-desync=0

	Raw File	Cut List
	11:03:14		seconds: 877.0 Wall: 11:17:51
Duration	00:29:59
	11:33:13		seconds: 1336.0 Wall: 11:25:30
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

2019-11-03/11_03_14.ts Apply: 11:25:29 - 11:33:13 ( 00:07:44 ) S: 11:03:14 - E: 11:33:13 D: 00:29:59 (Start: 1335) show more...

vlc ~/Videos/veyepar/nbpy/nbpy19/dv/mystic/2019-11-03/11_03_14.ts :start-time=01335.0 --audio-desync=0

	Raw File	Cut List
	11:03:14		seconds: 1335.0 Wall: 11:25:29
Duration	00:29:59
	11:33:13		seconds: 0.0 Wall: 11:03:14
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

2019-11-03/11_33_14.ts Apply: 11:48:23 - 12:03:13 ( 00:14:50 ) S: 11:33:14 - E: 12:03:13 D: 00:29:59 (Start: 909.0) show more...

vlc ~/Videos/veyepar/nbpy/nbpy19/dv/mystic/2019-11-03/11_33_14.ts :start-time=0909.0 --audio-desync=0

	Raw File	Cut List
	11:33:14		seconds: 909.0 Wall: 11:48:23
Duration	00:29:59
	12:03:13		seconds: 0.0 Wall: 11:33:14
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete