veyepar: Better Web Scraping Through Reverse Engineering AJAX Calls

Author(s):	Diego Berrocal
Location	Mystic Theatre
Date	nov Sat 03	Days Raw Files
Start	16:00	First Raw Start	15:33
Duration	0:30:0	Offset	0:26:45
End	16:30	Last Raw End	16:33
Chapters	00:00 0:25:33
Total cuts_time	25 min.

https://2018.northbaypython.org/schedule/presentation/18/
raw-playlist raw-mp4-playlist encoded-files-playlist host archive tweet mp4 svg png

assets release.pdf Better_Web_Scraping_Through_Reverse_Engineering_AJAX_Calls.json
logs
Admin: episode episode list cut list raw files day marks day marks day image_files

State:

Locked:

clear this to unlock

Locked by:

user/process that locked.

Start:

initially scheduled time from master, adjusted to match reality

Duration:

length in hh:mm:ss

Name:

Video Title (shows in video search results)

Emails:

email(s) of the presenter(s)

Released:

has someone authorised pubication

Normalise:

Channelcopy:

m=mono, 01=copy left to right, 10=right to left, 00=ignore.

Thumbnail:

filename.png

Description:

markdown

Have you ever wished you could automate any task in a website? Be it ordering food from your favorite vendor with an online shopping cart, or find out the weekly items for sale in your local grocery store website. Due to the rise of libraries like React and the proliferation of client-side rendered webapps, chances are that you may be inclined to use dynamic scrapers like [Selenium](https://www.seleniumhq.org/) or [Puppeteer](https://developers.google.com/web/tools/puppeteer/). However, because these webapps need to initialize their components, they still make AJAX calls to services, exposing “internal” APIs we can use. This is not how its developers intended us to use the site, because it can change without notice, but that doesn’t mean it’s not useful!
After leaving this talk, you’ll be able to backwards-engineer these “internal” APIs and replicate any trivial flow as well as knowing how to use Puppeteer to directly interact with the page. You’ll also learn the criteria to choose when to use a dynamic scraper vs just using the “internal” API webapps use.

Comment:

production notes

2018-11-03/16_03_15.ts Apply: 16:04:56 - 16:05:19 ( 00:00:23 ) S: 16:03:15 - E: 16:33:15 D: 00:30:00 (Start: 101.0) (End: 124.0) show more...

vlc ~/Videos/veyepar/nbpy/nbpy18/dv/mystic/2018-11-03/16_03_15.ts :start-time=0101.0 --audio-desync=0

	Raw File	Cut List
	16:03:15		seconds: 101.0 Wall: 16:04:56
Duration	00:30:00
	16:33:15		seconds: 124.0 Wall: 16:05:19
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

2018-11-03/16_03_15.ts Apply: 16:05:19 - 16:30:52 ( 00:25:33 ) S: 16:03:15 - E: 16:33:15 D: 00:30:00 (Start: 124.0) (End: 1657.0) show more...

vlc ~/Videos/veyepar/nbpy/nbpy18/dv/mystic/2018-11-03/16_03_15.ts :start-time=0124.0 --audio-desync=0

	Raw File	Cut List
	16:03:15		seconds: 124.0 Wall: 16:05:19
Duration	00:30:00
	16:33:15		seconds: 1657.0 Wall: 16:30:52
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

2018-11-03/16_03_15.ts Apply: 16:30:52 - 16:31:11 ( 00:00:19 ) S: 16:03:15 - E: 16:33:15 D: 00:30:00 (Start: 1657.0) (End: 1676.0) show more...

vlc ~/Videos/veyepar/nbpy/nbpy18/dv/mystic/2018-11-03/16_03_15.ts :start-time=01657.0 --audio-desync=0

	Raw File	Cut List
	16:03:15		seconds: 1657.0 Wall: 16:30:52
Duration	00:30:00
	16:33:15		seconds: 1676.0 Wall: 16:31:11
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete