Author(s):	David Basden,Christopher Collins
Location	C001
Date	jan Tue 17	Days Raw Files
Start	10:30	First Raw Start	error-in-template
Duration	0:50:00	Offset	None
End	11:20	Last Raw End
Chapters
Total cuts_time	None min.

http://lca2012.linux.org.au/schedule/112/view_talk
raw-playlist raw-mp4-playlist encoded-files-playlist mp4 svg png

assets release.pdf Smashing_a_square_peg_into_a_round_hole_Automagically_building_and_configuring_Linux_systems_that_are_wildly_different.json
logs
Admin: episode episode list cut list raw files day marks day marks day image_files

State:	--------- borked edit encode push to queue post richard review 1 email review 2 make public tweet to-miror conf done
Locked:	clear this to unlock
Locked by:	user/process that locked.
Start:	initially scheduled time from master, adjusted to match reality
Duration:	length in hh:mm:ss
Name:	Video Title (shows in video search results)
Emails:	email(s) of the presenter(s)
Released:	Unknown Yes No has someone authorised pubication
Normalise:
Channelcopy:	m=mono, 01=copy left to right, 10=right to left, 00=ignore.
Thumbnail:	filename.png
Description:	Smashing a square peg into a round hole: Automagically building and configuring Linux systems that are wildly different. David Basden and Chris Collins Abstract You can’t walk two metres down the street without someone going on about how cool and hip The Cloud is these days, being able to spin up hundreds of identical Linux VMs easily. Even if you’re running your own cloud hardware, tools to build and configure lots of identical systems or VMs are a dime a dozen. But what if you every system you build is actually totally different? What if there is no “standard build” or even anything close, with different hardware, networking, software, distro, services, firewalls etc. every time, but you don’t want to spend all your time doing custom server builds and configuration? Working for a hosting provider, we have it even worse that that: Not only do most of our server builds have custom requirements, we also have to configure lots of our own individual systems to deal with monitoring, customer notifications, backups, fire-walling, accounting and system updates. Building a system and configuring the systems that support it by hand can take the best part of a day. Using some really cool algorithms, we’ve made some software that will take vague hand-waving from sales people and turn it into a deterministic set of build steps that can be automated, without a sysadmin being involved at all. We’ve also written some software that can hook-up scripts, modules and APIs between different servers securely with almost no overhead, and have it controllable from a single, trivial asynchronous API connection. Between the two of these we’ve gotten heterogeneous server builds that are fully automated, in minutes. The two bits are “make magic” and Orchestra. Rather than spinning up yet another bland VM copy, you get a beautiful and unique snowflake of Awesome. make magic The core of “make magic” derives the tasks required to build a system based on high level requirements. It takes a large pre-defined directed, acyclic dependency graph, and prunes it based on predicates generated from the high level requirements. The graph is then traversed to yield a list of tasks that can then be automated, with the traversal maintaining that any given task has already had it’s dependencies satisfied. It can be guaranteed to always output a valid, correct list of tasks for any set of input requirements. Once all this is done, “make magic” also keeps track of whether each task has been completed, or has failed etc, with a simple state machine. David is going to talk about some the really interesting underlying problems that come up when you have to automatically generate a really specific, and provably correct set of steps from hundreds of thousands of permutations based on some high level requirements. He’s then going to show how you can do this yourself with things you probably have lying around the home, as long as your home has a Linux box, and a network connection to download some python. make magic is written in pure python, but the talk should be accessible to people that don’t know it. Graph theory may be mentioned ‘cause it’s totally cool, but not in a way that is going to get in the way of people who have never heard of it. Orchestra Orchestra, the system to remotely execute all of the tasks that need to be automated, uses a very lightweight message-passing API over SSL. It can take a task and run it on either a single host, all of a set of hosts, or one host out of a pool. Unlike using ssh for remote control, Orchestra is asynchronous, and so lightweight that it can easily handle hundreds or thousands of concurrent jobs. It detects and propagates job failures, and deals gracefully with loss/reboots of servers it’s running jobs on. It’s packaged up for Debian and Redhat, with a core written in Go, and libraries/clients for shell, Python, Go (or pretty much anything else that can talk to a socket) Chris is going to talk about Orchestra, and show out how amazingly cool and simple it is, while it can at the same time be used to do really powerful things (generally all at the same time). Shut up and give me the code “make magic” is available at https://github.com/anchor/make-magic Orchestra is available at https://github.com/anchor/Orchestra Bio Chris and David both work for Anchor Systems in Sydney as Linux Sysadmins that spend way too much time doing development to code themselves out of a job. Chris designed and wrote Orchestra, which Anchor are letting him open source. David did much of the design and implementation of the original build algorithm magic. He then wrote the much cooler free software version “make magic”, which Anchor are also letting him release. David has been a Linux sysadmin for over 15 years, and has spend a large chunk of that time trying to figure out how to get hundreds of servers to be their own unique snowflakes while also not having to do stuff by hand. He will talk enthusiastically about what he thinks are the interesting parts of systems / algorithms / concepts that interest him, in the hope that people listening will find them interesting too. For as long as Chris can remember, he’s been fascinated by Computers. It was hardly a surprise to family and friends when he became a UNIX Systems Administrator early in his career. That was 12 years ago, and he hasn’t stopped yet. Since then he’s also been a developer on Linux based appliances, deployed computational clusters and reverse engineered USB devices. Target audience: sys admins markdown
Comment:	production notes