veyepar: Large-Scale Recommendation System with Python and Spark

Author(s):	Phil Anderson
Location	Cartoon 1
Date	jul Sat 28	Days Raw Files
Start	12:00	First Raw Start	11:43
Duration	0:30:0	Offset	0:16:09
End	12:30	Last Raw End	12:43
Chapters	00:00 0:14:12
Total cuts_time	25 min.

https://pyohio.org/2018/schedule/presentation/58/
raw-playlist raw-mp4-playlist encoded-files-playlist host archive tweet mp4 svg png

assets release.pdf LargeScale_Recommendation_System_with_Python_and_Spark.json
logs
Admin: episode episode list cut list raw files day marks day marks day image_files

State:

Locked:

clear this to unlock

Locked by:

user/process that locked.

Start:

initially scheduled time from master, adjusted to match reality

Duration:

length in hh:mm:ss

Name:

Video Title (shows in video search results)

Emails:

email(s) of the presenter(s)

Released:

has someone authorised pubication

Normalise:

Channelcopy:

m=mono, 01=copy left to right, 10=right to left, 00=ignore.

Thumbnail:

filename.png

Description:

markdown

# Abstract
We will briefly cover the Kroger Company and its digital properties, along with its current recommendation systems and need for a new one.  We will then move into a deep dive of the system we developed, covering the Python APIs for large-scale data processing tool Spark, and the underlying Hadoop Distributed File System (HDFS) - focusing on how we utilized each in our implementation.  We’ll also discuss process scheduling and coordination via Apache Airflow, along with its Python API and use of Python eggs.  Finally, we will show the recommendation system in action, and discuss plans for testing and improvement.

Talk will be organized as follows:

## Intro - Context Setting (5 min)

* What is Kroger?  
   * What is 84.51?  
      * What is Digital Personalization at 84.51?

* Landscape: Kroger’s digital properties 
   * Typically use Retention-focused Recommendation Systems 
      * These tend to work extremely well with grocery’s cyclic purchase cycles
   * Need for Acquisition-based Recommendation System

## Body - Technical Deep Dive (20 min)
### New Product Recommender - Ensemble Recommendation System

### Part 0:
* Hadoop & Spark, and their Python API

### Part 1: Collaborative Filtering 
* Overview 
   * Training (PySpark)
   * Implementation (PySpark)
      * Roadblocks

### Part 2: Regularized Regression
* Overview
   * Training 
   * Implementation (PySpark)

### Part 3: Process Scheduling
   * Overview of Airflow
      * Directed Acyclic Graphs
      * Python directive script layout
      * Python Eggs

### Part 4: Live view of system on kroger.com

## Conclusion (5 min)
   * Next Steps - Testing

Comment:

production notes