veyepar: Python in an Evolving Enterprise System

Author(s):	Angelica Pando, Dave Himrod, Steve Kannan
Location	A1
Date	mar Tue 19	Days Raw Files
Start	10:15	First Raw Start	10:26
Duration	00:50:00	Offset	0:11:15
End	11:05	Last Raw End	11:07
Chapters	00:00 0:00:11 0:00:12
Total cuts_time	36 min.

raw-playlist raw-mp4-playlist encoded-files-playlist host archive mp4 svg png

assets release.pdf Python_in_an_Evolving_Enterprise_System.json
logs
Admin: episode episode list cut list raw files day marks day marks day image_files

State:

Locked:

clear this to unlock

Locked by:

user/process that locked.

Start:

initially scheduled time from master, adjusted to match reality

Duration:

length in hh:mm:ss

Name:

Video Title (shows in video search results)

Emails:

email(s) of the presenter(s)

Released:

has someone authorised pubication

Normalise:

Channelcopy:

m=mono, 01=copy left to right, 10=right to left, 00=ignore.

Thumbnail:

filename.png

Description:

markdown

Our data pipeline is growing like crazy, processing more than 30 terabytes of
data every day and more than tripling in the last year alone. In 2011, we
moved our data pipeline to a Hadoop stack in order to enable horizontal
scalability for future growth. Our optimization tools used for data
exploration, aggregations, and general data hackery are critical for updating
budgets and optimization data. However, these tools are built in Python, and
integrating them with our Hadoop data pipeline has been an enormous challenge.
Our continued explosive growth demands increased efficiency, whether that's in
simplifying our infrastructure or building more shared services. Over the past
few months, we evaluated multiple solutions for integrating Python with Hadoop
including using Hadoop Streaming, PIG with Jython UDFs, writing MapReduce in
Jython, and of course, why not just do it in Java? In our talk, we'll explore
the different Python-Hadoop integration options, share our evaluation process
and best practices, and invite an interactive dialogue of lessons learned.

Comment:

production notes