veyepar: Python in an Evolving Enterprise System

Author(s):	Angelica Pando, Dave Himrod, Steve Kannan
Location	A1
Date	mar Tue 19	Days Raw Files
Start	10:15	First Raw Start	10:26
Duration	00:50:00	Offset	0:11:15
End	11:05	Last Raw End	11:07
Chapters	00:00 0:00:11 0:00:12
Total cuts_time	36 min.

raw-playlist raw-mp4-playlist encoded-files-playlist host archive mp4 svg png

assets release.pdf Python_in_an_Evolving_Enterprise_System.json
logs
Admin: episode episode list cut list raw files day marks day marks day image_files

State:
Locked:	clear this to unlock
Locked by:	user/process that locked.
Start:	initially scheduled time from master, adjusted to match reality
Duration:	length in hh:mm:ss
Name:	Video Title (shows in video search results)
Emails:	email(s) of the presenter(s)
Released:	has someone authorised pubication
Normalise:
Channelcopy:	m=mono, 01=copy left to right, 10=right to left, 00=ignore.
Thumbnail:	filename.png
Description:	Our data pipeline is growing like crazy, processing more than 30 terabytes of data every day and more than tripling in the last year alone. In 2011, we moved our data pipeline to a Hadoop stack in order to enable horizontal scalability for future growth. Our optimization tools used for data exploration, aggregations, and general data hackery are critical for updating budgets and optimization data. However, these tools are built in Python, and integrating them with our Hadoop data pipeline has been an enormous challenge. Our continued explosive growth demands increased efficiency, whether that's in simplifying our infrastructure or building more shared services. Over the past few months, we evaluated multiple solutions for integrating Python with Hadoop including using Hadoop Streaming, PIG with Jython UDFs, writing MapReduce in Jython, and of course, why not just do it in Java? In our talk, we'll explore the different Python-Hadoop integration options, share our evaluation process and best practices, and invite an interactive dialogue of lessons learned. markdown
Comment:	production notes