veyepar: Large-Scale Recommendation System with Python and Spark

Hi user

Admin Login:

Name: Large-Scale Recommendation System with Python and Spark
--client pyohio --show pyohio_2018 --room cartoon1 14182 --force
Prev: 11 Go with the Flow: Automating Your Workflows with Airflow
Next: 11 Software Engineering For Beginners: A Jr. Developer's Guide

show more...

Author(s):	Phil Anderson
Location	Cartoon 1
Date	jul Sat 28	Days Raw Files
Start	12:00	First Raw Start	11:43
Duration	0:30:0	Offset	0:16:09
End	12:30	Last Raw End	12:43
Chapters	00:00 0:14:12
Total cuts_time	25 min.

https://pyohio.org/2018/schedule/presentation/58/
raw-playlist raw-mp4-playlist encoded-files-playlist host archive tweet mp4 svg png

final audio

assets release.pdf LargeScale_Recommendation_System_with_Python_and_Spark.json
logs
Admin: episode episode list cut list raw files day marks day marks day image_files

State:
Locked:	clear this to unlock
Locked by:	user/process that locked.
Start:	initially scheduled time from master, adjusted to match reality
Duration:	length in hh:mm:ss
Name:	Video Title (shows in video search results)
Emails:	email(s) of the presenter(s)
Released:	has someone authorised pubication
Normalise:
Channelcopy:	m=mono, 01=copy left to right, 10=right to left, 00=ignore.
Thumbnail:	filename.png
Description:	# Abstract We will briefly cover the Kroger Company and its digital properties, along with its current recommendation systems and need for a new one. We will then move into a deep dive of the system we developed, covering the Python APIs for large-scale data processing tool Spark, and the underlying Hadoop Distributed File System (HDFS) - focusing on how we utilized each in our implementation. We’ll also discuss process scheduling and coordination via Apache Airflow, along with its Python API and use of Python eggs. Finally, we will show the recommendation system in action, and discuss plans for testing and improvement. Talk will be organized as follows: ## Intro - Context Setting (5 min) * What is Kroger? * What is 84.51? * What is Digital Personalization at 84.51? * Landscape: Kroger’s digital properties * Typically use Retention-focused Recommendation Systems * These tend to work extremely well with grocery’s cyclic purchase cycles * Need for Acquisition-based Recommendation System ## Body - Technical Deep Dive (20 min) ### New Product Recommender - Ensemble Recommendation System ### Part 0: * Hadoop & Spark, and their Python API ### Part 1: Collaborative Filtering * Overview * Training (PySpark) * Implementation (PySpark) * Roadblocks ### Part 2: Regularized Regression * Overview * Training * Implementation (PySpark) ### Part 3: Process Scheduling * Overview of Airflow * Directed Acyclic Graphs * Python directive script layout * Python Eggs ### Part 4: Live view of system on kroger.com ## Conclusion (5 min) * Next Steps - Testing markdown
Comment:	delay=1.0 production notes

2018-07-28/11_43_51.ts Apply: 11:43:51 - 11:59:34 ( 00:15:43 ) S: 11:43:51 - E: 12:13:50 D: 00:29:59 (End: 943.0) show more...

vlc ~/Videos/veyepar/pyohio/pyohio_2018/dv/cartoon1/2018-07-28/11_43_51.ts :start-time=00.0 --audio-desync=0

	Raw File	Cut List
	11:43:51		seconds: 0.0 Wall: 11:43:51
Duration	00:29:59
	12:13:50		seconds: 943.0 Wall: 11:59:34
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

audio viz

2018-07-28/11_43_51.ts Apply: 11:59:34 - 11:59:38 ( 00:00:04 ) S: 11:43:51 - E: 12:13:50 D: 00:29:59 (Start: 943.0) (End: 947.0) show more...

vlc ~/Videos/veyepar/pyohio/pyohio_2018/dv/cartoon1/2018-07-28/11_43_51.ts :start-time=0943.0 --audio-desync=0

	Raw File	Cut List
	11:43:51		seconds: 943.0 Wall: 11:59:34
Duration	00:29:59
	12:13:50		seconds: 947.0 Wall: 11:59:38
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

audio viz

2018-07-28/11_43_51.ts Apply: 11:59:38 - 12:13:50 ( 00:14:12 ) S: 11:43:51 - E: 12:13:50 D: 00:29:59 (Start: 947.0) show more...

vlc ~/Videos/veyepar/pyohio/pyohio_2018/dv/cartoon1/2018-07-28/11_43_51.ts :start-time=0947.0 --audio-desync=0

	Raw File	Cut List
	11:43:51		seconds: 947.0 Wall: 11:59:38
Duration	00:29:59
	12:13:50		seconds: 0.0 Wall: 11:43:51
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

audio viz

2018-07-28/12_13_51.ts Apply: 12:13:51 - 12:25:25 ( 00:11:34 ) S: 12:13:51 - E: 12:43:50 D: 00:29:59 (End: 694.0) show more...

vlc ~/Videos/veyepar/pyohio/pyohio_2018/dv/cartoon1/2018-07-28/12_13_51.ts :start-time=00.0 --audio-desync=0

	Raw File	Cut List
	12:13:51		seconds: 0.0 Wall: 12:13:51
Duration	00:29:59
	12:43:50		seconds: 694.0 Wall: 12:25:25
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

audio viz

2018-07-28/12_13_51.ts Apply: 12:25:25 - 12:43:50 ( 00:18:25 ) S: 12:13:51 - E: 12:43:50 D: 00:29:59 (Start: 694.0) show more...

vlc ~/Videos/veyepar/pyohio/pyohio_2018/dv/cartoon1/2018-07-28/12_13_51.ts :start-time=0694.0 --audio-desync=0

	Raw File	Cut List
	12:13:51		seconds: 694.0 Wall: 12:25:25
Duration	00:29:59
	12:43:50		seconds: 0.0 Wall: 12:13:51
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

audio viz

Rf filename:	root is .../show/dv/location/, example: 2013-03-13/13:13:30.dv
Sequence:
get this:	check and save to add this

2018-07-28/11_43_51.ts
audio viz

audio viz

2018-07-28/12_13_51.ts
audio viz

audio viz

Veyepar Video Eyeball Processor and Review