veyepar: Making a scalable course search engine with python

Author(s):	Robert King
Location	Track 2
Date	sep Sun 14	Days Raw Files
Start	15:40	First Raw Start	error-in-template
Duration	01:30:00	Offset	None
End	17:10	Last Raw End
Chapters
Total cuts_time	None min.

https://kiwi.pycon.org/schedule/presentation/62/
raw-playlist raw-mp4-playlist encoded-files-playlist host public mp4 svg png

assets release.pdf Making_a_scalable_course_search_engine_with_python.json
logs
Admin: episode episode list cut list raw files day marks day marks day image_files

State:

Locked:

clear this to unlock

Locked by:

user/process that locked.

Start:

initially scheduled time from master, adjusted to match reality

Duration:

length in hh:mm:ss

Name:

Video Title (shows in video search results)

Emails:

email(s) of the presenter(s)

Released:

has someone authorised pubication

Normalise:

Channelcopy:

m=mono, 01=copy left to right, 10=right to left, 00=ignore.

Thumbnail:

filename.png

Description:

markdown

= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = 
Robert King:
Making a scalable course search engine with Python
= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = 
@ Kiwi PyCon 2014 - Sunday, 14 Sep 2014 - Track 2
http://kiwi.pycon.org/

**Audience level**

Experienced

**Description**

Creating a custom search engine with python on google app engine. Serve large spikes in search request traffic. Allow students to find course reviews across multiple universities and countries.

**Abstract**

- introduction to the real world problem - Students need to be able to find university courses across multiple countries and universities.
 - first - explore how to solve the problem - collect course data & decide on a rough solution
 - second - create minimum viable product & see how people use it. Iteratively make it better.
 - second continued: Organise a big website launch event before you've created the website - then proceed to write 10K lines of code in the week before launch.
 - third - analyse the 50K most recent search terms & make a simple tree data structure to help improve search performance.
 - four - caching & cache invalidation
 - five - Maybe I'll do an online marketing campaign halfway through the talk and show graphs of the app responding in real time.

- Covers Data analysis with python (csv, matplotlib, networkx, collections.Counter, logfile parsing)

- Covers "Futures" - doing RPC calls in parallel.
 - Unit testing & simulating all things. - Being able to see how adjusting search functionality effects query times & quality of results.

- Some tasteful jokes to keep things entertaining ;)

**Slides**

https://speakerdeck.com/nzpug/robert-king-making-a-scalable-course-search-engine-with-python

Comment:

production notes