veyepar: (re)implementing (Google) maglev in Rust

Author(s):	Robert Collins
Location	Guthrie Theatre CB06.03.28
Date	jan Wed 24	Days Raw Files
Start	11:40	First Raw Start	11:10
Duration	0:45:0	Offset	0:29:41
End	12:25	Last Raw End	12:40
Chapters	00:00 0:29:28
Total cuts_time	45 min.

http://lca2018.linux.org.au/schedule/presentation/77/
raw-playlist raw-mp4-playlist encoded-files-playlist host archive public tweet mp4 svg png

assets release.pdf reimplementing_Google_maglev_in_Rust.json
logs
Admin: episode episode list cut list raw files day marks day marks day image_files

State:

Locked:

clear this to unlock

Locked by:

user/process that locked.

Start:

initially scheduled time from master, adjusted to match reality

Duration:

length in hh:mm:ss

Name:

Video Title (shows in video search results)

Emails:

email(s) of the presenter(s)

Released:

has someone authorised pubication

Normalise:

Channelcopy:

m=mono, 01=copy left to right, 10=right to left, 00=ignore.

Thumbnail:

filename.png

Description:

markdown

A war story: Developer re-implements a Google whitepaper in Rust using netmap for networking.

In 2016 Google released a whitepaper about their small packet wire rate - 40Gbps even, highly available IP load balancer. I happened to have some weeks shortly after that happened where I was able to do whatever I wanted... and I decided to have some fun and implement an open source look-alike.

Maglev has some distinctive design choices - such as the use of userspace networking to achieve zero-copy forwarding - which I knew about but had not actively used, as well as doing direct server return - sending packets to backend servers wrapped in GRE, but traffic in the other direction is routed directly, so the traffic pattern is asymmetric.

Since I was young and foolish then, I thought it would be good to also deepen my knowledge of the Rust language at the same time... and develop it on my not-as-Linux-friendly-as-I'd-like Surface : so my Linux
environment was actually a Hyper-V VM.

This combination led to discovering some truely fantastic facilities in Rust, bugs in the Linux kernel and the netmap project, and exploring some of the awkwardness where very clever C datastructures don't map well to Rust's ownership model.

The current status of the project is that the core use case works: traffic can be forwarded using network ring buffers, without needing a dedicated network card, routed with zerocopy while preserving GRE wrappers. Higher level facilities like dynamic reconfiguration and backend health signalling are not yet implemented (but are designed).

In the talk I will cover the choices I made and why - rust vs the world, netmap vs DPDK, things I found surprising about doing userspace networking, detail the gritty details of doing userspace networking with Netmap from Rust, using Rust to parse and create network packets, as well as driving netlink from within Rust. I'll cover the bugs I had to fix in order to write the code I wanted to write.

Comment:

production notes