veyepar: Using pandas and pyspark to address challenges in processing and storing time series instrument data

Author(s):	Aaron Wiegel
Location	Bungalo West
Date	oct Sun 08	Days Raw Files
Start	15:45	First Raw Start	15:25
Duration	00:25:00	Offset	0:19:11
End	16:10	Last Raw End	16:25
Chapters	00:00 0:00:11 0:12:33 0:20:13 0:28:42
Total cuts_time	28 min.

https://pybay.com/speakers/#sz-speaker-61dd28d8-78b9-4c39-9b10-11f62d83ab10
raw-playlist raw-mp4-playlist encoded-files-playlist host mp4 svg png

assets release.pdf Using_pandas_and_pyspark_to_address_challenges_in_processing_and_storing_time_series_instrument_data.json
logs
Admin: episode episode list cut list raw files day marks day marks day image_files

State:
Locked:	clear this to unlock
Locked by:	user/process that locked.
Start:	initially scheduled time from master, adjusted to match reality
Duration:	length in hh:mm:ss
Name:	Video Title (shows in video search results)
Emails:	email(s) of the presenter(s)
Released:	has someone authorised pubication
Normalise:
Channelcopy:	m=mono, 01=copy left to right, 10=right to left, 00=ignore.
Thumbnail:	filename.png
Description:	Time series data from scientific instruments for fermentation, environmental sensors, or spectroscopy often comes in proprietary or unusual formats that are require custom logic to process. In addition, processing data at scale is challenge since enterprise laboratory information management systems (LIMS) typically rely on transactional, row-oriented databases that are not designed to handle millions of records at a time. However, with clever use of pandas for unusually formatted files or pyspark (via Databricks) for large numbers of records, this data can be processed into cleaner, more useful forms for further analysis. markdown
Comment:	production notes

2023-10-08/15_25_49.ts Apply: 15:25:49 - 15:36:33 ( 00:10:44 ) S: 15:25:49 - E: 15:55:48 D: 00:29:59 (End: 644.0) show more...

vlc ~/Videos/veyepar/sfpy/pybay23/dv/bungalo_west/2023-10-08/15_25_49.ts :start-time=00.0 --audio-desync=0

	Raw File	Cut List
	15:25:49		seconds: 0.0 Wall: 15:25:49
Duration	00:29:59
	15:55:48		seconds: 644.0 Wall: 15:36:33
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

2023-10-08/15_25_49.ts Apply: 15:36:33 - 15:43:10 ( 00:06:37 ) S: 15:25:49 - E: 15:55:48 D: 00:29:59 (Start: 644.0) (End: 1041.0) show more...

vlc ~/Videos/veyepar/sfpy/pybay23/dv/bungalo_west/2023-10-08/15_25_49.ts :start-time=0644.0 --audio-desync=0

	Raw File	Cut List
	15:25:49		seconds: 644.0 Wall: 15:36:33
Duration	00:29:59
	15:55:48		seconds: 1041.0 Wall: 15:43:10
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

2023-10-08/15_25_49.ts Apply: 15:43:10 - 15:43:15 ( 00:00:05 ) S: 15:25:49 - E: 15:55:48 D: 00:29:59 (Start: 1041.0) (End: 1046.0) show more...

vlc ~/Videos/veyepar/sfpy/pybay23/dv/bungalo_west/2023-10-08/15_25_49.ts :start-time=01041.0 --audio-desync=0

	Raw File	Cut List
	15:25:49		seconds: 1041.0 Wall: 15:43:10
Duration	00:29:59
	15:55:48		seconds: 1046.0 Wall: 15:43:15
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

2023-10-08/15_25_49.ts Apply: 15:43:15 - 15:43:26 ( 00:00:11 ) S: 15:25:49 - E: 15:55:48 D: 00:29:59 (Start: 1046.0) (End: 1057.0) show more...

vlc ~/Videos/veyepar/sfpy/pybay23/dv/bungalo_west/2023-10-08/15_25_49.ts :start-time=01046.0 --audio-desync=0

	Raw File	Cut List
	15:25:49		seconds: 1046.0 Wall: 15:43:15
Duration	00:29:59
	15:55:48		seconds: 1057.0 Wall: 15:43:26
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

2023-10-08/15_25_49.ts Apply: 15:43:26 - 15:55:48 ( 00:12:22 ) S: 15:25:49 - E: 15:55:48 D: 00:29:59 (Start: 1057.0) show more...

vlc ~/Videos/veyepar/sfpy/pybay23/dv/bungalo_west/2023-10-08/15_25_49.ts :start-time=01057.0 --audio-desync=0

	Raw File	Cut List
	15:25:49		seconds: 1057.0 Wall: 15:43:26
Duration	00:29:59
	15:55:48		seconds: 0.0 Wall: 15:25:49
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

2023-10-08/15_55_49.ts Apply: 15:55:49 - 16:03:29 ( 00:07:40 ) S: 15:55:49 - E: 16:25:49 D: 00:30:00 (End: 460.0) show more...

vlc ~/Videos/veyepar/sfpy/pybay23/dv/bungalo_west/2023-10-08/15_55_49.ts :start-time=00.0 --audio-desync=0

	Raw File	Cut List
	15:55:49		seconds: 0.0 Wall: 15:55:49
Duration	00:30:00
	16:25:49		seconds: 460.0 Wall: 16:03:29
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

2023-10-08/15_55_49.ts Apply: 16:03:29 - 16:11:58 ( 00:08:29 ) S: 15:55:49 - E: 16:25:49 D: 00:30:00 (Start: 460.0) (End: 969.0) show more...

vlc ~/Videos/veyepar/sfpy/pybay23/dv/bungalo_west/2023-10-08/15_55_49.ts :start-time=0460.0 --audio-desync=0

	Raw File	Cut List
	15:55:49		seconds: 460.0 Wall: 16:03:29
Duration	00:30:00
	16:25:49		seconds: 969.0 Wall: 16:11:58
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

2023-10-08/15_55_49.ts Apply: 16:11:58 - 16:12:08 ( 00:00:10 ) S: 15:55:49 - E: 16:25:49 D: 00:30:00 (Start: 969.0) (End: 979.0) show more...

vlc ~/Videos/veyepar/sfpy/pybay23/dv/bungalo_west/2023-10-08/15_55_49.ts :start-time=0969.0 --audio-desync=0

	Raw File	Cut List
	15:55:49		seconds: 969.0 Wall: 16:11:58
Duration	00:30:00
	16:25:49		seconds: 979.0 Wall: 16:12:08
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

2023-10-08/15_55_49.ts Apply: 16:12:08 - 16:25:49 ( 00:13:41 ) S: 15:55:49 - E: 16:25:49 D: 00:30:00 (Start: 979.0) show more...

vlc ~/Videos/veyepar/sfpy/pybay23/dv/bungalo_west/2023-10-08/15_55_49.ts :start-time=0979.0 --audio-desync=0

	Raw File	Cut List
	15:55:49		seconds: 979.0 Wall: 16:12:08
Duration	00:30:00
	16:25:49		seconds: 0.0 Wall: 15:55:49
Comments:
mp4 mp4.m3u	dv.m3u	Split: Sequence:: delete

Rf filename:	root is .../show/dv/location/, example: 2013-03-13/13:13:30.dv
Sequence:
get this:	check and save to add this