veyepar: Dimensionality reduction - squeezing out the good stuff with PCA

Author(s):	Aabir Abubaker Kar
Location	Baobab
Date	oct Fri 12	Days Raw Files
Start	10:00	First Raw Start	09:54
Duration	00:45:00	Offset	0:05:31
End	10:45	Last Raw End	10:48
Chapters	00:00 0:00:09 0:19:25 0:22:59 0:23:01
Total cuts_time	39 min.

https://za.pycon.org/talks/28-dimensionality-reduction-squeezing-out-the-good-stuff-with-pca/
raw-playlist raw-mp4-playlist encoded-files-playlist host archive tweet mp4 svg png

assets release.pdf Dimensionality_reduction_squeezing_out_the_good_stuff_with_PCA.json
logs
Admin: episode episode list cut list raw files day marks day marks day image_files

State:

Locked:

clear this to unlock

Locked by:

user/process that locked.

Start:

initially scheduled time from master, adjusted to match reality

Duration:

length in hh:mm:ss

Name:

Video Title (shows in video search results)

Emails:

email(s) of the presenter(s)

Released:

has someone authorised pubication

Normalise:

Channelcopy:

m=mono, 01=copy left to right, 10=right to left, 00=ignore.

Thumbnail:

filename.png

Description:

markdown

Data is often high-dimensional - millions of pixels, frequencies, categories. A lot of this detail is unnecessary for data analysis - but how much exactly? This talk will discuss the ideas and techniques of dimensionality reduction, provide useful mathematical intuition about how it's done, and show you how Netflix uses it to lead you from binge to binge.

In this session, we'll start by remembering what data really is and what it stands for. Data is a structured set of numbers, and these numbers typically (hopefully!) hold some information. This will lead us naturally to the concept of a *high-dimensional dataspace*, the mystical realm in which data lives. It turns out that data in this space displays an extremely useful 'selection bias' - ***a datapoint can be known by the company it keeps***. This is one of the basic ideas behind **k-means clustering**, which we will briefly discuss.

We'll then talk about the *informative-ness* of certain dimensions of the data space over others. This lays the mathematical foundation for the technique of **Principal Component Analysis** (PCA), which we will run on the Netflix movie dataset using *scikit-learn*.

We will also touch upon **tSNE**, another popular dimensionality-reduction algorithms.

I will be using *scikit-learn* for processing and *matplotlib* for visualization. The purpose of this session is to introduce dimensionality-reduction to those who do not know it, and to provide useful guiding intuitions to those who do. We'll also discuss some seminal use-cases, with tips and warnings for your own applications.

Comment:

production notes