veyepar: Luigi - Batch Data Processing in Python

Hi user

Admin Login:

Name: Luigi - Batch Data Processing in Python
--client continuum --show pydata_sv_2013 --room a1 2245 --force
Next: 5 Big Data in Fashion

show more... Marks

Author(s):	Elias Freider
Location	A1
Date	mar Wed 20	Days Raw Files
Start	10:00	First Raw Start	10:05
Duration	00:50:00	Offset	0:05:04
End	10:50	Last Raw End	11:27
Chapters	00:00 0:34:56
Total cuts_time	38 min.

raw-playlist raw-mp4-playlist encoded-files-playlist host archive mp4 svg png

final audio

assets release.pdf Luigi_Batch_Data_Processing_in_Python.json
logs
Admin: episode episode list cut list raw files day marks day marks day image_files

State:

Locked:

clear this to unlock

Locked by:

user/process that locked.

Start:

initially scheduled time from master, adjusted to match reality

Duration:

length in hh:mm:ss

Name:

Video Title (shows in video search results)

Emails:

email(s) of the presenter(s)

Released:

has someone authorised pubication

Normalise:

Channelcopy:

m=mono, 01=copy left to right, 10=right to left, 00=ignore.

Thumbnail:

filename.png

Description:

markdown

Luigi is Spotify's recently open sourced Python framework for batch data
processing including dependency resolution and monitoring. We will demonstrate
how Luigi can help you get started with data processing in Hadoop MapReduce as
well as on your local workstation.

Spotify has terabytes of data being logged by backend services every day for
everything from debugging to reporting reasons. The logs are basically huge
semi-structured text files that can be parsed using a few lines of Python.
From this data aggregated reports need to be created, data needs to be pushed
into SQL databases for internal dashboards, related artists need to be
calculated using complex algorithms and a lot of other tasks need to be
performed, many of which have to be run on an daily or even hourly basis.

A lot of the initial processing steps are very similar for the many data
products that are produced, and instead of re-doing a lot of work,
intermediate results are stored and form dependencies for later tasks. The
dependency graph forms a data pipeline.

Luigi was created for managing task dependencies, monitoring the progress of
the data pipeline and providing frameworks for common batch processing tasks.

Comment:

production notes

2013-03-20/10:05:04.dv Apply: 10:05:04 - 10:09:58 ( 00:04:54 ) S: 10:05:04 - E: 10:09:58 D: 00:04:54 show more...

vlc ~/Videos/veyepar/continuum/pydata_sv_2013/dv/a1/2013-03-20/10:05:04.dv :start-time=00.0 --audio-desync=0

	Raw File	Cut List
	10:05:04		seconds: 0.0 Wall: 10:05:04
Duration	00:04:54
	10:09:58		seconds: 0.0 Wall: 10:05:04
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

audio viz

2013-03-20/10:09:58.dv Apply: 10:09:58 - 10:44:54 ( 00:34:56 ) S: 10:09:58 - E: 10:44:54 D: 00:34:56 show more...

vlc ~/Videos/veyepar/continuum/pydata_sv_2013/dv/a1/2013-03-20/10:09:58.dv :start-time=00.0 --audio-desync=0

	Raw File	Cut List
	10:09:58		seconds: 0.0 Wall: 10:09:58
Duration	00:34:56
	10:44:54		seconds: 0.0 Wall: 10:09:58
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

audio viz

2013-03-20/10:44:55.dv Apply: 10:44:55 - 10:48:50 ( 00:03:55 ) S: 10:44:55 - E: 10:48:50 D: 00:03:55 show more...

vlc ~/Videos/veyepar/continuum/pydata_sv_2013/dv/a1/2013-03-20/10:44:55.dv :start-time=00.0 --audio-desync=0

	Raw File	Cut List
	10:44:55		seconds: 0.0 Wall: 10:44:55
Duration	00:03:55
	10:48:50		seconds: 0.0 Wall: 10:44:55
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

audio viz

2013-03-20/10:48:50.dv Apply: 10:48:50 - 10:57:41 ( 00:08:51 ) S: 10:48:50 - E: 10:57:41 D: 00:08:51 show more...

vlc ~/Videos/veyepar/continuum/pydata_sv_2013/dv/a1/2013-03-20/10:48:50.dv :start-time=00.0 --audio-desync=0

	Raw File	Cut List
	10:48:50		seconds: 0.0 Wall: 10:48:50
Duration	00:08:51
	10:57:41		seconds: 0.0 Wall: 10:48:50
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

audio viz

2013-03-20/10:57:42.dv Apply: 10:57:42 - 11:16:26 ( 00:18:44 ) S: 10:57:42 - E: 11:16:26 D: 00:18:44 show more...

vlc ~/Videos/veyepar/continuum/pydata_sv_2013/dv/a1/2013-03-20/10:57:42.dv :start-time=00.0 --audio-desync=0

	Raw File	Cut List
	10:57:42		seconds: 0.0 Wall: 10:57:42
Duration	00:18:44
	11:16:26		seconds: 0.0 Wall: 10:57:42
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

audio viz

2013-03-20/11:16:26.dv Apply: 11:16:26 - 11:27:10 ( 00:10:44 ) S: 11:16:26 - E: 11:27:10 D: 00:10:44 show more...

vlc ~/Videos/veyepar/continuum/pydata_sv_2013/dv/a1/2013-03-20/11:16:26.dv :start-time=00.0 --audio-desync=0

	Raw File	Cut List
	11:16:26		seconds: 0.0 Wall: 11:16:26
Duration	00:10:44
	11:27:10		seconds: 0.0 Wall: 11:16:26
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

audio viz

Rf filename:

root is .../show/dv/location/, example: 2013-03-13/13:13:30.dv

Sequence:

get this:

check and save to add this

2013-03-20/10:05:04.dv
audio viz

audio viz

2013-03-20/10:09:58.dv
audio viz

audio viz

2013-03-20/10:44:55.dv
audio viz

audio viz

2013-03-20/10:48:50.dv
audio viz

audio viz

2013-03-20/10:57:42.dv
audio viz

audio viz

2013-03-20/11:16:26.dv
audio viz

audio viz

Veyepar Video Eyeball Processor and Review