veyepar: Hassle Free ETL with PySpark

Author(s):	Rob Howley
Location	Room CR4
Date	jul Sat 16	Days Raw Files
Start	15:15	First Raw Start	15:12
Duration	00:25:00	Offset	0:02:30
End	15:40	Last Raw End	15:42
Chapters	00:00
Total cuts_time	23 min.

https://2016.pygotham.org/talks/275/hassle-free-etl-with-pyspark
raw-playlist raw-mp4-playlist encoded-files-playlist host archive mp4 svg png

assets release.pdf Hassle_Free_ETL_with_PySpark.json
logs
Admin: episode episode list cut list raw files day marks day marks day image_files

State:
Locked:	clear this to unlock
Locked by:	user/process that locked.
Start:	initially scheduled time from master, adjusted to match reality
Duration:	length in hh:mm:ss
Name:	Video Title (shows in video search results)
Emails:	email(s) of the presenter(s)
Released:	has someone authorised pubication
Normalise:
Channelcopy:	m=mono, 01=copy left to right, 10=right to left, 00=ignore.
Thumbnail:	filename.png
Description:	While the models of data science get all the press, the real work is in the maze of data preprocessing and pipelines. The goal of this talk is to get a glimpse into how you can use Python and the distributed power of Spark to simplify your (data) life, ditch the ETL boilerplate and get to the insights. We’ll intro PySpark and considerations in ETL jobs with respect to code structure and performance. markdown
Comment:	delay=1.0 production notes

2016-07-16/15_12_30.ts Apply: 15:12:30 - 15:15:18 ( 00:02:48 ) S: 15:12:30 - E: 15:42:29 D: 00:29:59 (End: 168.0) show more...

vlc ~/Videos/veyepar/big_apple_py/pygotham_2016/dv/Room_CR4/2016-07-16/15_12_30.ts :start-time=00.0 --audio-desync=0

	Raw File	Cut List
	15:12:30		seconds: 0.0 Wall: 15:12:30
Duration	00:29:59
	15:42:29		seconds: 168.0 Wall: 15:15:18
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

2016-07-16/15_12_30.ts Apply: 15:15:08 - 15:38:47 ( 00:23:38 ) S: 15:12:30 - E: 15:42:29 D: 00:29:59 (Start: 158.535797) (End: 1577.0) show more...

vlc ~/Videos/veyepar/big_apple_py/pygotham_2016/dv/Room_CR4/2016-07-16/15_12_30.ts :start-time=0158.535797 --audio-desync=0

	Raw File	Cut List
	15:12:30		seconds: 158.535797 Wall: 15:15:08
Duration	00:29:59
	15:42:29		seconds: 1577.0 Wall: 15:38:47
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

2016-07-16/15_12_30.ts Apply: 15:38:47 - 15:42:29 ( 00:03:42 ) S: 15:12:30 - E: 15:42:29 D: 00:29:59 (Start: 1577.0) show more...

vlc ~/Videos/veyepar/big_apple_py/pygotham_2016/dv/Room_CR4/2016-07-16/15_12_30.ts :start-time=01577.0 --audio-desync=0

	Raw File	Cut List
	15:12:30		seconds: 1577.0 Wall: 15:38:47
Duration	00:29:59
	15:42:29		seconds: 0.0 Wall: 15:12:30
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

Rf filename:	root is .../show/dv/location/, example: 2013-03-13/13:13:30.dv
Sequence:
get this:	check and save to add this