veyepar: The Split-Apply-Combine Pattern for Data Science in Python

Author(s):	Tobias Brandt
Location	Room 211
Date	oct Thu 01	Days Raw Files
Start	14:15	First Raw Start	14:14
Duration	00:45:00	Offset	0:00:37
End	15:00	Last Raw End	14:54
Chapters	00:00
Total cuts_time	39 min.

https://za.pycon.org/talks/12/
raw-playlist raw-mp4-playlist encoded-files-playlist host archive public tweet mp4 svg png

assets release.pdf The_SplitApplyCombine_Pattern_for_Data_Science_in_Python.json
logs
Admin: episode episode list cut list raw files day marks day marks day image_files

State:

Locked:

clear this to unlock

Locked by:

user/process that locked.

Start:

initially scheduled time from master, adjusted to match reality

Duration:

length in hh:mm:ss

Name:

Video Title (shows in video search results)

Emails:

email(s) of the presenter(s)

Released:

has someone authorised pubication

Normalise:

Channelcopy:

m=mono, 01=copy left to right, 10=right to left, 00=ignore.

Thumbnail:

filename.png

Description:

markdown

Many data science problems involve the application of a split-apply-combine pattern, where you break up a big dataset into independent pieces, operate on each piece in isolation and then put all the pieces back together. This crops up in all stages of a data analysis:

* During data preparation, when performing group-wise ranking, standardisation, or normalisation.

* During modelling, when fitting separate models to each group.

* During communication, when creating summaries or visualisations for display or analysis.

Python has many tools that make it easy to utilise this strategy when solving data science problems. These range from list and dictionary comprehensions in the language, the *map* and *reduce* functions and *itertools* and *functools* modules in the standard library to dedicated packages like *Pandas*, *PyToolz*, *Blaze* and *Dask*.

Explicit recognition of the applicability of the pattern allows one to reuse standard components for the bookkeeping code that handles the splitting and combining of the independent pieces. This allows one to concentrate on the data analysis code that is unique to the problem at hand. Since implicit in the pattern is the independence of the pieces, its applicability immediately implies a strategy for parallelisation which allows one to easily scale one's solution from single core to out-of-core computation on multiple machines, often with only very few changes to the code required.

This talk will introduce the pattern and how to recognise it by presenting some common code blocks. We will then look at some of the tools available, in particular *Pandas* and *PyToolz*, demonstrate their use, and discuss their strengths and weaknesses. Finally we'll show how to take a simple analysis and parallelise it to process a dataset that is too large to fit in memory.

Comment:

production notes