veyepar: Advanced Stream Processing on the Edge

Author(s):	Eduardo Silva
Location	Room 6
Date	jan Thu 16	Days Raw Files
Start	16:40	First Raw Start	error-in-template
Duration	0:45:0	Offset	None
End	17:25	Last Raw End
Chapters
Total cuts_time	None min.

https://lca2020.linux.org.au/schedule/presentation/88/
raw-playlist raw-mp4-playlist encoded-files-playlist mp4 svg png

assets release.pdf Advanced_Stream_Processing_on_the_Edge.json
logs
Admin: episode episode list cut list raw files day marks day marks day image_files

State:

Locked:

clear this to unlock

Locked by:

user/process that locked.

Start:

initially scheduled time from master, adjusted to match reality

Duration:

length in hh:mm:ss

Name:

Video Title (shows in video search results)

Emails:

email(s) of the presenter(s)

Released:

has someone authorised pubication

Normalise:

Channelcopy:

m=mono, 01=copy left to right, 10=right to left, 00=ignore.

Thumbnail:

filename.png

Description:

markdown

Logging is one of the ancient mechanism behavior to perform application or hardware analysis. In a new era of distributed systems at scale and connected embedded devices, data collection and processing becomes a real challenge; Logging has been forced to evolve and adapt to new needs.

In Data Analysis, logging is one of the key components to collect and pre-process data, usually, a logging mechanism goes through collect, parse, filter and centralize logs to a storage backend like a database, so data processing and analysis can be performed. This usually happens after the data has been aggregated and stored, but for real-time analysis needs, process the data while is still in motion brings a lot of advantages and this kind of approach is called Stream Processing.
 
What if it was possible to query your data using aggregation functions, windowing, and grouping results while the data was in motion and in-memory but on the edge side?.

In this presentation, we will go further and present an extended approach called 'Stream Processing on the Edge', where data is processed on the edge service or device, in a lightweight mode empowering features like anomaly detection (in the order of milliseconds) and Machine Learning in a distributed way using pure Open Source software.

Comment:

production notes