Hi
user
Admin Login:
Username:
Password:
Name:
Using pandas and pyspark to address challenges in processing and storing time series instrument data
--client
sfpy
--show
pybay23
--room bungalo_west 15323 --force
Next: 10 Python in Hardware & Embedded Systems: A Deep Dive
show more...
Marks
Author(s):
Aaron Wiegel
Location
Bungalo West
Date
oct Sun 08
Days Raw Files
Start
15:45
First Raw Start
15:25
Duration
00:25:00
Offset
0:19:11
End
16:10
Last Raw End
16:25
Chapters
00:00
0:00:11
0:12:33
0:20:13
0:28:42
Total cuts_time
28 min.
https://pybay.com/speakers/#sz-speaker-61dd28d8-78b9-4c39-9b10-11f62d83ab10
raw-playlist
raw-mp4-playlist
encoded-files-playlist
host
mp4
svg
png
assets
release.pdf
Using_pandas_and_pyspark_to_address_challenges_in_processing_and_storing_time_series_instrument_data.json
logs
Admin:
episode
episode list
cut list
raw files day
marks day
marks day
image_files
State:
---------
borked
edit
encode
push to queue
post
richard
review 1
email
review 2
make public
tweet
to-miror
conf
done
Locked:
clear this to unlock
Locked by:
user/process that locked.
Start:
initially scheduled time from master, adjusted to match reality
Duration:
length in hh:mm:ss
Name:
Video Title (shows in video search results)
Emails:
email(s) of the presenter(s)
Released:
has someone authorised pubication
Unknown
Yes
No
Normalise:
Channelcopy:
m=mono, 01=copy left to right, 10=right to left, 00=ignore.
Thumbnail:
filename.png
Description:
markdown
Time series data from scientific instruments for fermentation, environmental sensors, or spectroscopy often comes in proprietary or unusual formats that are require custom logic to process. In addition, processing data at scale is challenge since enterprise laboratory information management systems (LIMS) typically rely on transactional, row-oriented databases that are not designed to handle millions of records at a time. However, with clever use of pandas for unusually formatted files or pyspark (via Databricks) for large numbers of records, this data can be processed into cleaner, more useful forms for further analysis.
Comment:
production notes
2023-10-08/15_25_49.ts
Apply:
15:25:49 - 15:36:33 ( 00:10:44 )
S:
15:25:49 -
E:
15:55:48
D:
00:29:59
(
End:
644.0)
show more...
vlc ~/Videos/veyepar/sfpy/pybay23/dv/bungalo_west/2023-10-08/15_25_49.ts :start-time=00.0 --audio-desync=0
Raw File
Cut List
15:25:49
seconds: 0.0
Wall: 15:25:49
Duration
00:29:59
15:55:48
seconds: 644.0
Wall: 15:36:33
Comments:
mp4
mp4.m3u
dv.m3u
Split:
Sequence:
:
delete
2023-10-08/15_25_49.ts
Apply:
15:36:33 - 15:43:10 ( 00:06:37 )
S:
15:25:49 -
E:
15:55:48
D:
00:29:59
(
Start:
644.0) (
End:
1041.0)
show more...
vlc ~/Videos/veyepar/sfpy/pybay23/dv/bungalo_west/2023-10-08/15_25_49.ts :start-time=0644.0 --audio-desync=0
Raw File
Cut List
15:25:49
seconds: 644.0
Wall: 15:36:33
Duration
00:29:59
15:55:48
seconds: 1041.0
Wall: 15:43:10
Comments:
mp4
mp4.m3u
dv.m3u
Split:
Sequence:
:
delete
2023-10-08/15_25_49.ts
Apply:
15:43:10 - 15:43:15 ( 00:00:05 )
S:
15:25:49 -
E:
15:55:48
D:
00:29:59
(
Start:
1041.0) (
End:
1046.0)
show more...
vlc ~/Videos/veyepar/sfpy/pybay23/dv/bungalo_west/2023-10-08/15_25_49.ts :start-time=01041.0 --audio-desync=0
Raw File
Cut List
15:25:49
seconds: 1041.0
Wall: 15:43:10
Duration
00:29:59
15:55:48
seconds: 1046.0
Wall: 15:43:15
Comments:
mp4
mp4.m3u
dv.m3u
Split:
Sequence:
:
delete
2023-10-08/15_25_49.ts
Apply:
15:43:15 - 15:43:26 ( 00:00:11 )
S:
15:25:49 -
E:
15:55:48
D:
00:29:59
(
Start:
1046.0) (
End:
1057.0)
show more...
vlc ~/Videos/veyepar/sfpy/pybay23/dv/bungalo_west/2023-10-08/15_25_49.ts :start-time=01046.0 --audio-desync=0
Raw File
Cut List
15:25:49
seconds: 1046.0
Wall: 15:43:15
Duration
00:29:59
15:55:48
seconds: 1057.0
Wall: 15:43:26
Comments:
mp4
mp4.m3u
dv.m3u
Split:
Sequence:
:
delete
2023-10-08/15_25_49.ts
Apply:
15:43:26 - 15:55:48 ( 00:12:22 )
S:
15:25:49 -
E:
15:55:48
D:
00:29:59
(
Start:
1057.0)
show more...
vlc ~/Videos/veyepar/sfpy/pybay23/dv/bungalo_west/2023-10-08/15_25_49.ts :start-time=01057.0 --audio-desync=0
Raw File
Cut List
15:25:49
seconds: 1057.0
Wall: 15:43:26
Duration
00:29:59
15:55:48
seconds: 0.0
Wall: 15:25:49
Comments:
mp4
mp4.m3u
dv.m3u
Split:
Sequence:
:
delete
2023-10-08/15_55_49.ts
Apply:
15:55:49 - 16:03:29 ( 00:07:40 )
S:
15:55:49 -
E:
16:25:49
D:
00:30:00
(
End:
460.0)
show more...
vlc ~/Videos/veyepar/sfpy/pybay23/dv/bungalo_west/2023-10-08/15_55_49.ts :start-time=00.0 --audio-desync=0
Raw File
Cut List
15:55:49
seconds: 0.0
Wall: 15:55:49
Duration
00:30:00
16:25:49
seconds: 460.0
Wall: 16:03:29
Comments:
mp4
mp4.m3u
dv.m3u
Split:
Sequence:
:
delete
2023-10-08/15_55_49.ts
Apply:
16:03:29 - 16:11:58 ( 00:08:29 )
S:
15:55:49 -
E:
16:25:49
D:
00:30:00
(
Start:
460.0) (
End:
969.0)
show more...
vlc ~/Videos/veyepar/sfpy/pybay23/dv/bungalo_west/2023-10-08/15_55_49.ts :start-time=0460.0 --audio-desync=0
Raw File
Cut List
15:55:49
seconds: 460.0
Wall: 16:03:29
Duration
00:30:00
16:25:49
seconds: 969.0
Wall: 16:11:58
Comments:
mp4
mp4.m3u
dv.m3u
Split:
Sequence:
:
delete
2023-10-08/15_55_49.ts
Apply:
16:11:58 - 16:12:08 ( 00:00:10 )
S:
15:55:49 -
E:
16:25:49
D:
00:30:00
(
Start:
969.0) (
End:
979.0)
show more...
vlc ~/Videos/veyepar/sfpy/pybay23/dv/bungalo_west/2023-10-08/15_55_49.ts :start-time=0969.0 --audio-desync=0
Raw File
Cut List
15:55:49
seconds: 969.0
Wall: 16:11:58
Duration
00:30:00
16:25:49
seconds: 979.0
Wall: 16:12:08
Comments:
mp4
mp4.m3u
dv.m3u
Split:
Sequence:
:
delete
2023-10-08/15_55_49.ts
Apply:
16:12:08 - 16:25:49 ( 00:13:41 )
S:
15:55:49 -
E:
16:25:49
D:
00:30:00
(
Start:
979.0)
show more...
vlc ~/Videos/veyepar/sfpy/pybay23/dv/bungalo_west/2023-10-08/15_55_49.ts :start-time=0979.0 --audio-desync=0
Raw File
Cut List
15:55:49
seconds: 979.0
Wall: 16:12:08
Duration
00:30:00
16:25:49
seconds: 0.0
Wall: 15:55:49
Comments:
mp4
mp4.m3u
dv.m3u
Split:
Sequence:
:
delete
Rf filename:
root is .../show/dv/location/, example: 2013-03-13/13:13:30.dv
Sequence:
get this:
check and save to add this
2023-10-08/15_25_49.ts
2023-10-08/15_55_49.ts
Veyepar
Video Eyeball Processor and Review
