veyepar: Challenges when Scaling: Adventures in Swift's Sharding

Author(s):	Matthew Oliver
Location	D4.303 Costa Theatre
Date	feb Wed 03	Days Raw Files
Start	16:35	First Raw Start	16:34
Duration	0:45:00	Offset	0:00:02
End	17:20	Last Raw End	17:20
Chapters	00:00
Total cuts_time	40 min.

https://linux.conf.au/schedule/30123/view_talk
raw-playlist raw-mp4-playlist encoded-files-playlist host archive tweet mp4 svg png

assets release.pdf Challenges_when_Scaling_Adventures_in_Swifts_Sharding.json
logs
Admin: episode episode list cut list raw files day marks day marks day image_files

State:

Locked:

clear this to unlock

Locked by:

user/process that locked.

Start:

initially scheduled time from master, adjusted to match reality

Duration:

length in hh:mm:ss

Name:

Video Title (shows in video search results)

Emails:

email(s) of the presenter(s)

Released:

has someone authorised pubication

Normalise:

Channelcopy:

m=mono, 01=copy left to right, 10=right to left, 00=ignore.

Thumbnail:

filename.png

Description:

markdown

Swift is an open-source, highly available, distributed, eventually-consistent object storage system.  As the project has matured, so has the need to store more and more data. Objects in swift are tracked in the containers they are placed and a container in essence is a SQLite database which is distributed throughout the cluster. As the number of objects in a container increases, so does latency meaning larger sites have moved to using expensive solid state drives (SSDs) for container storage, but there has to be a better way we can solve this in software, right?

For this presentation we'll cover the design decisions that have enabled Swift's success, and then show how these same decisions need to be revisited as we cater for the ever increasing size of containers.  We'll consider a proof of concept container sharding implementation that attempts to address a few of these limitations, allowing for larger datasets to be deployed on cheaper commodity hardware, splitting a container database over many databases and nodes. 
We won't just present the final solution, but we'll iterate over the different approaches tried, so that you can learn from our mistakes and lessons learnt.

Comment:

production notes