veyepar: *** CANCELLED *** Zen: Introducing a pythonic, high-performance network analysis library

Author(s):	Derek Ruths
Location	106
Date	jul Wed 18	Days Raw Files
Start	15:30	First Raw Start	15:17
Duration	20:00	Offset	0:12:43
End	15:50	Last Raw End	15:55
Chapters	00:00
Total cuts_time	38 min.

raw-playlist raw-mp4-playlist encoded-files-playlist mp4 svg png

assets release.pdf Zen_Introducing_a_pythonic_highperformance_network_analysis_library.json
logs
Admin: episode episode list cut list raw files day marks day marks day image_files

State:

Locked:

clear this to unlock

Locked by:

user/process that locked.

Start:

initially scheduled time from master, adjusted to match reality

Duration:

length in hh:mm:ss

Name:

Video Title (shows in video search results)

Emails:

email(s) of the presenter(s)

Released:

has someone authorised pubication

Normalise:

Channelcopy:

m=mono, 01=copy left to right, 10=right to left, 00=ignore.

Thumbnail:

filename.png

Description:

markdown

Network datasets are becoming larger; network algorithms are becoming more computationally expensive; and interest in network research is becoming more widespread.  These and other factors are creating a widespread need for a network library that can both handle massive network datasets and still not sacrifice the conventions and designs that make a python library quick-to-learn, elegant, and easy-to-use.

In this talk, we will introduce the Zero-Effort Network (Zen) library.  Developed from scratch in a combination of python and cython, it aims to be the most memory- and computation-efficient network library available while still adhering to the conventions of good pythonic design.  Nearly all functions are implemented in cython, yielding C-level performance.  Currently in its first release, it implements a large suite of network-based functions including algorithms involved in finding shortest paths, clustering, community detection, and matchings.  Overall, these implementations outperform (sometimes by more than an order of magnitude) all popular network libraries (including both those that work with python and those that do not).

From the outset, we intended for Zen to be applicable to massive network datasets (e.g., hundreds of millions of edges).  In order to achieve this, the data structures involved in holding the networks themselves in memory have been designed to be very memory-efficient.  Furthermore, new storage formats have been designed that both minimizes the storage space required to store network connectivity on disk and maximizes the speed with which it can be loaded from disk into memory.  The result of this combination is that massive network datasets that previously could not be loaded and analyzed in python libraries can now be loaded in a matter of minutes.

Our aim in this presentation is to introduce the Zen library to the community: showing how it can be used to analyze datasets that have previously been beyond the reach of python-based network tools and highlighting the unique designs that allow it to achieve this high-performance without sacrificing pythonic conventions.  Attention will be given to discussing both the overarching design and use of the library and also those specific innovations that possible working with massive network datasets.

Comment:

production notes