veyepar: Handling Big Data with Python

Author(s):	Florian Wilhelm
Location	KOMED Saal
Date	oct Thu 17	Days Raw Files
Start	15:30	First Raw Start	15:29
Duration	00:30.0:00	Offset	0:00:23
End	16:00	Last Raw End	16:01
Chapters	00:00 0:20:48 0:20:56
Total cuts_time	28 min.

https://2013.de.pycon.org/schedule/sessions/2/
raw-playlist raw-mp4-playlist encoded-files-playlist host public tweet mp4 svg png

assets release.pdf Handling_Big_Data_with_Python.json
logs
Admin: episode episode list cut list raw files day marks day marks day image_files

State:

Locked:

clear this to unlock

Locked by:

user/process that locked.

Start:

initially scheduled time from master, adjusted to match reality

Duration:

length in hh:mm:ss

Name:

Video Title (shows in video search results)

Emails:

email(s) of the presenter(s)

Released:

has someone authorised pubication

Normalise:

Channelcopy:

m=mono, 01=copy left to right, 10=right to left, 00=ignore.

Thumbnail:

filename.png

Description:

markdown

The talk gives a small introduction of how Blue Yonder applies machine learning and Predictive Analytics in various fields as well as the challenges of Big Data. Using the example of Blue Yonder's machine learning software NeuroBayes, we show the made efforts and hit dead ends in order to provide a flexible and yet easy to use interface for NeuroBayes to Data Scientists. Since NeuroBayes is written in FORTRAN for performance reasons different interface approaches were tried which lead us eventually to a Python interface. In the talk we elaborate on the up- and downsides of the different approaches and the various reasons why Python won the race with an emphasize on the benefits of the Python ecosystem itself. 
Also, we discuss performance as well as scalability issues with Python and how we address them. In detail, we show the application of Cython to speed up calculations in the Python interface layer as well as distributed computing in a private cloud called Stratosphere. Scalability and efficiency is of utmost importance when data processing is time critical.  Our overall goal is to give the audience an overview how Python fits in the software ecosystem of a company handling Big Data.

Comment:

production notes