veyepar: Hands On Realtime Text Analytics

State:
Locked:	clear this to unlock
Locked by:	user/process that locked.
Start:	initially scheduled time from master, adjusted to match reality
Duration:	length in hh:mm:ss
Name:	Video Title (shows in video search results)
Emails:	email(s) of the presenter(s)
Released:	has someone authorised pubication
Normalise:
Channelcopy:	m=mono, 01=copy left to right, 10=right to left, 00=ignore.
Thumbnail:	filename.png
Description:	A common problem in large scale computing, is coordinating workers when they can be scattered across compute nodes. For workloads like this, atomic operators like increment and decrement reduce contention between distributed processes. In this talk I'll show a full text analysis tool which ranks words in the Twitter firehose. By storing each token in a key based on its characteristics, we can provide word rankings both globally, as well as over time and space. We'll show the running application, and take a tour through the code, as well as discuss cluster sizing and how it is impacted by variations in the input data stream. For instance a tweet in English from San Francisco might say "Go Giants" so counters for 2012:go and usa-sf:2012-07:giants (among a few dozen others) are incremented. Even using memory like this, the counts from a full corpus of English text would only take a few gigabytes to hold, making this architecture more efficient than a traditional index-and-rollup approach. markdown
Comment:	production notes

2013-05-16/13_16_26.dv Apply: 13:16:26 - 13:23:21 ( 00:06:55 ) S: 13:16:26 - E: 13:23:21 D: 00:06:55 show more...

vlc ~/Videos/veyepar/troy/nodepdx2013/dv/room_1/2013-05-16/13_16_26.dv :start-time=00.0 --audio-desync=0

2013-05-16/13_23_22.dv Apply: 13:23:22 - 13:47:03 ( 00:23:41 ) S: 13:23:22 - E: 13:47:03 D: 00:23:41 show more...

vlc ~/Videos/veyepar/troy/nodepdx2013/dv/room_1/2013-05-16/13_23_22.dv :start-time=00.0 --audio-desync=0

Rf filename:	root is .../show/dv/location/, example: 2013-03-13/13:13:30.dv
Sequence:
get this:	check and save to add this