veyepar: The Split-Apply-Combine Pattern for Data Science in Python

Hi user

Admin Login:

Name: The Split-Apply-Combine Pattern for Data Science in Python
--client pyconza --show pyconza2015 --room room211 10322 --force
Prev: 11 A little scripting goes a long way: automating data processing in science
Next: 11 Community Security - ANPR (Automated Number Plate Recognition) on your Pi

show more...

Author(s):	Tobias Brandt
Location	Room 211
Date	oct Thu 01	Days Raw Files
Start	14:15	First Raw Start	14:14
Duration	00:45:00	Offset	0:00:37
End	15:00	Last Raw End	14:54
Chapters	00:00
Total cuts_time	39 min.

https://za.pycon.org/talks/12/
raw-playlist raw-mp4-playlist encoded-files-playlist host archive public tweet mp4 svg png

final audio

assets release.pdf The_SplitApplyCombine_Pattern_for_Data_Science_in_Python.json
logs
Admin: episode episode list cut list raw files day marks day marks day image_files

State:
Locked:	clear this to unlock
Locked by:	user/process that locked.
Start:	initially scheduled time from master, adjusted to match reality
Duration:	length in hh:mm:ss
Name:	Video Title (shows in video search results)
Emails:	email(s) of the presenter(s)
Released:	has someone authorised pubication
Normalise:
Channelcopy:	m=mono, 01=copy left to right, 10=right to left, 00=ignore.
Thumbnail:	filename.png
Description:	Many data science problems involve the application of a split-apply-combine pattern, where you break up a big dataset into independent pieces, operate on each piece in isolation and then put all the pieces back together. This crops up in all stages of a data analysis: * During data preparation, when performing group-wise ranking, standardisation, or normalisation. * During modelling, when fitting separate models to each group. * During communication, when creating summaries or visualisations for display or analysis. Python has many tools that make it easy to utilise this strategy when solving data science problems. These range from list and dictionary comprehensions in the language, the map and reduce functions and itertools and functools modules in the standard library to dedicated packages like Pandas, PyToolz, Blaze and Dask. Explicit recognition of the applicability of the pattern allows one to reuse standard components for the bookkeeping code that handles the splitting and combining of the independent pieces. This allows one to concentrate on the data analysis code that is unique to the problem at hand. Since implicit in the pattern is the independence of the pieces, its applicability immediately implies a strategy for parallelisation which allows one to easily scale one's solution from single core to out-of-core computation on multiple machines, often with only very few changes to the code required. This talk will introduce the pattern and how to recognise it by presenting some common code blocks. We will then look at some of the tools available, in particular Pandas and PyToolz, demonstrate their use, and discuss their strengths and weaknesses. Finally we'll show how to take a simple analysis and parallelise it to process a dataset that is too large to fit in memory. markdown
Comment:	production notes

2015-10-01/14_14_23.dv Apply: 14:14:23 - 14:53:54 ( 00:39:31 ) S: 14:14:23 - E: 14:53:54 D: 00:39:31 show more...

vlc ~/Videos/veyepar/pyconza/pyconza2015/dv/room211/2015-10-01/14_14_23.dv :start-time=00.0 --audio-desync=0

	Raw File	Cut List
	14:14:23		seconds: 0.0 Wall: 14:14:23
Duration	00:39:31
	14:53:54		seconds: 0.0 Wall: 14:14:23
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

audio viz

2015-10-01/14_53_54.dv Apply: 14:53:54 - 14:54:08 ( 00:00:14 ) S: 14:53:54 - E: 14:54:08 D: 00:00:14 show more...

vlc ~/Videos/veyepar/pyconza/pyconza2015/dv/room211/2015-10-01/14_53_54.dv :start-time=00.0 --audio-desync=0

	Raw File	Cut List
	14:53:54		seconds: 0.0 Wall: 14:53:54
Duration	00:00:14
	14:54:08		seconds: 0.0 Wall: 14:53:54
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

audio viz

Rf filename:	root is .../show/dv/location/, example: 2013-03-13/13:13:30.dv
Sequence:
get this:	check and save to add this

2015-10-01/14_14_23.dv
audio viz

audio viz

2015-10-01/14_53_54.dv
audio viz

audio viz

Veyepar Video Eyeball Processor and Review