veyepar: Erlang in production: "I wish I'd known that when I started"

Author(s):	Bernard Duggan
Location	Studio
Date	jan Tue 17	Days Raw Files
Start	14:20	First Raw Start	error-in-template
Duration	0:50:00	Offset	None
End	15:10	Last Raw End
Chapters
Total cuts_time	None min.

http://lca2012.linux.org.au/schedule/113/view_talk
raw-playlist raw-mp4-playlist encoded-files-playlist mp4 svg png

assets release.pdf Erlang_in_production_I_wish_Id_known_that_when_I_started.json
logs
Admin: episode episode list cut list raw files day marks day marks day image_files

State:

Locked:

clear this to unlock

Locked by:

user/process that locked.

Start:

initially scheduled time from master, adjusted to match reality

Duration:

length in hh:mm:ss

Name:

Video Title (shows in video search results)

Emails:

email(s) of the presenter(s)

Released:

has someone authorised pubication

Normalise:

Channelcopy:

m=mono, 01=copy left to right, 10=right to left, 00=ignore.

Thumbnail:

filename.png

Description:

markdown

Erlang is gaining popularity as a language for developing robust, maintainable, concurrent systems for high load environments. Its list of features, including hot code loading, crash isolation and easy concurrency make it look extremely attractive, but any even moderately experienced developer knows there's no silver bullet.

In this talk we discuss our journey from being "Erlang newbies" to pushing a critical system out into M5's high-load production telephony environment. We will focus in particular on the various "lessons learned" from the experience; notably:

* Several great ways to crash the VM and bring down your whole system

Having your entire system in a VM can lead to a false sense of security. Things can go wrong, and when they do it will really hurt. We look at ways to break the VM and how to avoid them.

* Message queues – you really need to look after them

Message queues lie at the heart of Erlang's concurrency system, and for 99% of cases they fall into the "you don't have to care about the internals, they just work" category. We discuss how to avoid your system hitting the 1% of cases where they can cause serious pain.

* Running Erlang as a Unix-style service

One oft-overlooked shortcoming of Erlang is its lack of ability to be easily started up with a service (/etc/init.d) style interface. We take a brief look at erld, our in-house solution to the problem. [Note: We are presently awaiting management approval to release erld as open source – we hope to have this in the next couple of months, well before LCA rolls around].

* Hot code loading – it's not nearly as easy as it said on the brochure

Hot code loading promises little short of 100% uptime. We talk about how trying to get it working panned out in practice and how to maximise your chances of getting it to work in your system.

* Monitor your system like a hawk

Erlang makes it easy for problems to go un-noticed for quite a while before they suddenly cause serious damage. We look at some key areas you can monitor and how to do so.

This talk is meant to be anything but a "why Erlang sucks" talk; we still love Erlang and are pushing forward with moving all our core systems to use it. We now, however, have a much better understanding of its limitations and foibles than when we started out and we hope to help others avoid some of the pitfalls we have encountered.

Comment:

production notes