veyepar: 1.5MW and 3600 penguins in a room: Supercomputing at ANU

Author(s):	robin humble
Location	MCC3
Date	jan Thu 31	Days Raw Files
Start	15:40	First Raw Start	error-in-template
Duration	0:45:00	Offset	None
End	16:25	Last Raw End
Chapters
Total cuts_time	None min.

http://lca2013.linux.org.au/schedule/30284/view_talk
raw-playlist raw-mp4-playlist encoded-files-playlist mp4 svg png

assets release.pdf 15MW_and_3600_penguins_in_a_room_Supercomputing_at_ANU.json
logs
Admin: episode episode list cut list raw files day marks day marks day image_files

State:

Locked:

clear this to unlock

Locked by:

user/process that locked.

Start:

initially scheduled time from master, adjusted to match reality

Duration:

length in hh:mm:ss

Name:

Video Title (shows in video search results)

Emails:

email(s) of the presenter(s)

Released:

has someone authorised pubication

Normalise:

Channelcopy:

m=mono, 01=copy left to right, 10=right to left, 00=ignore.

Thumbnail:

filename.png

Description:

markdown

At the NCI National Facility at ANU we exclusively use Linux for supercomputing. This talk covers how we use and alter (and hopefully help to improve) Linux and other Open Source projects to create a fast and stable petascale machine.

Overview:
 - How we boot and run clusters at large scale using oneSIS, CentOS, root on Lustre, ...
 - How we tweak the kernel and tune the OS to keep them stable and efficient
 - Problems we've seen and areas we'd like to see improved

Our current supercomputer is 12k Nehalem cores with a 20GB/s filesystem and QDR InfinBand. The next machine (hopefully operational by LCA time) is 60k Sandy Bridge cores with a 100GB/s filesystem and FDR InfiniBand. Each of our new clusters is usually in the top 50 machines in the world.

oneSIS and the Lustre filesystem together with CentOS running vanilla and modified RHEL6 kernels are the primary components of our software stack. Uniquely the OS's for all 3600 nodes boot from a single copy of the OS that resides on the global Lustre filesystem. This gives us an easily updatable OS and guarantees all compute nodes are booting identical software.

High performance and predictability is a key focus. The main player in this game is numa awareness. We use cpusets extensively and enforce processor and memory binding to obtain ideal numa alignment and optimal and reproducible job performance. We have worked with the OpenMPI project to get binding included, and our users run jobs with binding on by default.

We are increasingly using cgroups and eventually may use all of cputime, freeze/thaw, acl, namespaces, and memcg - all tasks which our own resource manager software currently handles. We also have a history and interest in page migration which may again become useful to reduce machine fragmentation as each numa zone gets more cores.

Like many HPC sites we write our own job and cluster monitoring tools and these are available for other sites to use and modify, as are our changes to kernels and some InfiniBand tools. Unlike many sites we also write our own batch system.

I'll talk about our Linux improvement wishlist too - eg. Lustre integration into mainline so we can update kernels more aggressively and get shiny new stability. inode/dentry reclaim of deleted inodes with vfs_cache_pressure=0. Low overhead control groups. Better transparent huge pages. Improved virtual memory stability. md is awesome but zfs-on-linux might be better. OFED integration with mainline.

Comment:

production notes