veyepar: Functional programming in Python with Toolz and fn.py

Author(s):	Juan Nunez-Iglesias
Location	Wool Museum
Date	feb Tue 02	Days Raw Files
Start	11:50	First Raw Start	11:44
Duration	0:30:00	Offset	0:05:11
End	12:20	Last Raw End	13:21
Chapters	00:00
Total cuts_time	22 min.

https://linux.conf.au/schedule/30358/view_talk
raw-playlist raw-mp4-playlist encoded-files-playlist host archive tweet mp4 svg png

assets release.pdf Functional_programming_in_Python_with_Toolz_and_fnpy.json
logs
Admin: episode episode list cut list raw files day marks day marks day image_files

State:

Locked:

clear this to unlock

Locked by:

user/process that locked.

Start:

initially scheduled time from master, adjusted to match reality

Duration:

length in hh:mm:ss

Name:

Video Title (shows in video search results)

Emails:

email(s) of the presenter(s)

Released:

has someone authorised pubication

Normalise:

Channelcopy:

m=mono, 01=copy left to right, 10=right to left, 00=ignore.

Thumbnail:

filename.png

Description:

markdown

In my brief experience people rarely take this [streaming] route. They use single-threaded in-memory Python until it breaks, and then seek out Big Data Infrastructure like Hadoop/Spark at relatively high productivity overhead. ~ Matt Rocklin

That quote succinctly summarises my computational life, right up until recent months.

In “traditional” programming, you load a dataset into memory, process it in some way, and output the result. This is simple to understand. But in streaming programs, a function processes some of the data, yields the processed chunk, then downstream functions deal with that chunk, then the original function receives a bit more, and so on… All these things are going on at the same time! How can one keep them straight?

This talk will introduce Matt Rocklin’s Toolz library which makes functional programming easy in Python and provides a framework to write elegant, concise code to analyse bigger-than-memory data, and fn.py, which has even more FP constructs. I’ll present streaming data analysis using FP from the ground up, from a simple “hello-world” example to image illumination correction and streaming extensions to scikit-learn classifiers, and analysing a genome in a few minutes.

Comment:

production notes