veyepar: Luigi - Batch Data Processing in Python

Hi user

Admin Login:

Name: Luigi - Batch Data Processing in Python
--client continuum --show pydata_sv_2013 --room a1 2245 --force
Prev: 5 Data Visualization With Nodebox
Next: 5 Big Data in Fashion

show more...

Author(s):	Elias Freider
Location	A1
Date	mar Wed 20	Days Raw Files
Start	10:00	First Raw Start	10:05
Duration	00:50:00	Offset	0:05:04
End	10:50	Last Raw End	11:27
Chapters	00:00 0:34:56
Total cuts_time	38 min.

raw-playlist raw-mp4-playlist encoded-files-playlist host archive mp4 svg png

final audio

assets release.pdf Luigi_Batch_Data_Processing_in_Python.json
logs
Admin: episode episode list cut list raw files day marks day marks day image_files

State:
Locked:	clear this to unlock
Locked by:	user/process that locked.
Start:	initially scheduled time from master, adjusted to match reality
Duration:	length in hh:mm:ss
Name:	Video Title (shows in video search results)
Emails:	email(s) of the presenter(s)
Released:	has someone authorised pubication
Normalise:
Channelcopy:	m=mono, 01=copy left to right, 10=right to left, 00=ignore.
Thumbnail:	filename.png
Description:	Luigi is Spotify's recently open sourced Python framework for batch data processing including dependency resolution and monitoring. We will demonstrate how Luigi can help you get started with data processing in Hadoop MapReduce as well as on your local workstation. Spotify has terabytes of data being logged by backend services every day for everything from debugging to reporting reasons. The logs are basically huge semi-structured text files that can be parsed using a few lines of Python. From this data aggregated reports need to be created, data needs to be pushed into SQL databases for internal dashboards, related artists need to be calculated using complex algorithms and a lot of other tasks need to be performed, many of which have to be run on an daily or even hourly basis. A lot of the initial processing steps are very similar for the many data products that are produced, and instead of re-doing a lot of work, intermediate results are stored and form dependencies for later tasks. The dependency graph forms a data pipeline. Luigi was created for managing task dependencies, monitoring the progress of the data pipeline and providing frameworks for common batch processing tasks. markdown
Comment:	production notes

2013-03-20/10:05:04.dv Apply: 10:05:04 - 10:09:58 ( 00:04:54 ) S: 10:05:04 - E: 10:09:58 D: 00:04:54 show more...

vlc ~/Videos/veyepar/continuum/pydata_sv_2013/dv/a1/2013-03-20/10:05:04.dv :start-time=00.0 --audio-desync=0

	Raw File	Cut List
	10:05:04		seconds: 0.0 Wall: 10:05:04
Duration	00:04:54
	10:09:58		seconds: 0.0 Wall: 10:05:04
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

audio viz

2013-03-20/10:09:58.dv Apply: 10:09:58 - 10:44:54 ( 00:34:56 ) S: 10:09:58 - E: 10:44:54 D: 00:34:56 show more...

vlc ~/Videos/veyepar/continuum/pydata_sv_2013/dv/a1/2013-03-20/10:09:58.dv :start-time=00.0 --audio-desync=0

	Raw File	Cut List
	10:09:58		seconds: 0.0 Wall: 10:09:58
Duration	00:34:56
	10:44:54		seconds: 0.0 Wall: 10:09:58
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

audio viz

2013-03-20/10:44:55.dv Apply: 10:44:55 - 10:48:50 ( 00:03:55 ) S: 10:44:55 - E: 10:48:50 D: 00:03:55 show more...

vlc ~/Videos/veyepar/continuum/pydata_sv_2013/dv/a1/2013-03-20/10:44:55.dv :start-time=00.0 --audio-desync=0

	Raw File	Cut List
	10:44:55		seconds: 0.0 Wall: 10:44:55
Duration	00:03:55
	10:48:50		seconds: 0.0 Wall: 10:44:55
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

audio viz

2013-03-20/10:48:50.dv Apply: 10:48:50 - 10:57:41 ( 00:08:51 ) S: 10:48:50 - E: 10:57:41 D: 00:08:51 show more...

vlc ~/Videos/veyepar/continuum/pydata_sv_2013/dv/a1/2013-03-20/10:48:50.dv :start-time=00.0 --audio-desync=0

	Raw File	Cut List
	10:48:50		seconds: 0.0 Wall: 10:48:50
Duration	00:08:51
	10:57:41		seconds: 0.0 Wall: 10:48:50
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

audio viz

2013-03-20/10:57:42.dv Apply: 10:57:42 - 11:16:26 ( 00:18:44 ) S: 10:57:42 - E: 11:16:26 D: 00:18:44 show more...

vlc ~/Videos/veyepar/continuum/pydata_sv_2013/dv/a1/2013-03-20/10:57:42.dv :start-time=00.0 --audio-desync=0

	Raw File	Cut List
	10:57:42		seconds: 0.0 Wall: 10:57:42
Duration	00:18:44
	11:16:26		seconds: 0.0 Wall: 10:57:42
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

audio viz

2013-03-20/11:16:26.dv Apply: 11:16:26 - 11:27:10 ( 00:10:44 ) S: 11:16:26 - E: 11:27:10 D: 00:10:44 show more...

vlc ~/Videos/veyepar/continuum/pydata_sv_2013/dv/a1/2013-03-20/11:16:26.dv :start-time=00.0 --audio-desync=0

	Raw File	Cut List
	11:16:26		seconds: 0.0 Wall: 11:16:26
Duration	00:10:44
	11:27:10		seconds: 0.0 Wall: 11:16:26
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

audio viz

Rf filename:	root is .../show/dv/location/, example: 2013-03-13/13:13:30.dv
Sequence:
get this:	check and save to add this

2013-03-20/10:05:04.dv
audio viz

audio viz

2013-03-20/10:09:58.dv
audio viz

audio viz

2013-03-20/10:44:55.dv
audio viz

audio viz

2013-03-20/10:48:50.dv
audio viz

audio viz

2013-03-20/10:57:42.dv
audio viz

audio viz

2013-03-20/11:16:26.dv
audio viz

audio viz

Veyepar Video Eyeball Processor and Review