Hi
user
Admin Login:
Username:
Password:
Name:
Hassle Free ETL with PySpark
--client
big_apple_py
--show
pygotham_2016
--room Room_CR4 11073 --force
Next: 11 A tale of two cellphones: Python on Android and iOS
show more...
Marks
Author(s):
Rob Howley
Location
Room CR4
Date
jul Sat 16
Days Raw Files
Start
15:15
First Raw Start
15:12
Duration
00:25:00
Offset
0:02:30
End
15:40
Last Raw End
15:42
Chapters
00:00
Total cuts_time
23 min.
https://2016.pygotham.org/talks/275/hassle-free-etl-with-pyspark
raw-playlist
raw-mp4-playlist
encoded-files-playlist
host
archive
mp4
svg
png
assets
release.pdf
Hassle_Free_ETL_with_PySpark.json
logs
Admin:
episode
episode list
cut list
raw files day
marks day
marks day
image_files
State:
---------
borked
edit
encode
push to queue
post
richard
review 1
email
review 2
make public
tweet
to-miror
conf
done
Locked:
clear this to unlock
Locked by:
user/process that locked.
Start:
initially scheduled time from master, adjusted to match reality
Duration:
length in hh:mm:ss
Name:
Video Title (shows in video search results)
Emails:
email(s) of the presenter(s)
Released:
has someone authorised pubication
Unknown
Yes
No
Normalise:
Channelcopy:
m=mono, 01=copy left to right, 10=right to left, 00=ignore.
Thumbnail:
filename.png
Description:
markdown
While the models of data science get all the press, the real work is in the maze of data preprocessing and pipelines. The goal of this talk is to get a glimpse into how you can use Python and the distributed power of Spark to simplify your (data) life, ditch the ETL boilerplate and get to the insights. We’ll intro PySpark and considerations in ETL jobs with respect to code structure and performance.
Comment:
production notes
delay=1.0
2016-07-16/15_12_30.ts
Apply:
15:12:30 - 15:15:18 ( 00:02:48 )
S:
15:12:30 -
E:
15:42:29
D:
00:29:59
(
End:
168.0)
show more...
vlc ~/Videos/veyepar/big_apple_py/pygotham_2016/dv/Room_CR4/2016-07-16/15_12_30.ts :start-time=00.0 --audio-desync=0
Raw File
Cut List
15:12:30
seconds: 0.0
Wall: 15:12:30
Duration
00:29:59
15:42:29
seconds: 168.0
Wall: 15:15:18
Comments:
mp4
mp4.m3u
dv.m3u
Split:
Sequence:
:
delete
2016-07-16/15_12_30.ts
Apply:
15:15:08 - 15:38:47 ( 00:23:38 )
S:
15:12:30 -
E:
15:42:29
D:
00:29:59
(
Start:
158.535797) (
End:
1577.0)
show more...
vlc ~/Videos/veyepar/big_apple_py/pygotham_2016/dv/Room_CR4/2016-07-16/15_12_30.ts :start-time=0158.535797 --audio-desync=0
Raw File
Cut List
15:12:30
seconds: 158.535797
Wall: 15:15:08
Duration
00:29:59
15:42:29
seconds: 1577.0
Wall: 15:38:47
Comments:
mp4
mp4.m3u
dv.m3u
Split:
Sequence:
:
delete
2016-07-16/15_12_30.ts
Apply:
15:38:47 - 15:42:29 ( 00:03:42 )
S:
15:12:30 -
E:
15:42:29
D:
00:29:59
(
Start:
1577.0)
show more...
vlc ~/Videos/veyepar/big_apple_py/pygotham_2016/dv/Room_CR4/2016-07-16/15_12_30.ts :start-time=01577.0 --audio-desync=0
Raw File
Cut List
15:12:30
seconds: 1577.0
Wall: 15:38:47
Duration
00:29:59
15:42:29
seconds: 0.0
Wall: 15:12:30
Comments:
mp4
mp4.m3u
dv.m3u
Split:
Sequence:
:
delete
Rf filename:
root is .../show/dv/location/, example: 2013-03-13/13:13:30.dv
Sequence:
get this:
check and save to add this
2016-07-16/15_12_30.ts
Veyepar
Video Eyeball Processor and Review