veyepar: Large Memory Management issues: Performance, Fragmentation, Movable objects and Huge Page overhead.

Author(s):	Christopher Lameter
Location	Great Hall CB01.05.009
Date	jan Mon 22	Days Raw Files
Start	11:35	First Raw Start	11:01
Duration	0:50:0	Offset	0:33:17
End	12:25	Last Raw End	12:31
Chapters	00:00 0:14:49
Total cuts_time	35 min.

http://lca2018.linux.org.au/schedule/presentation/197/
raw-playlist raw-mp4-playlist encoded-files-playlist host archive public tweet mp4 svg png

assets release.pdf Large_Memory_Management_issues_Performance_Fragmentation_Movable_objects_and_Huge_Page_overhead.json
logs
Admin: episode episode list cut list raw files day marks day marks day image_files

State:

Locked:

clear this to unlock

Locked by:

user/process that locked.

Start:

initially scheduled time from master, adjusted to match reality

Duration:

length in hh:mm:ss

Name:

Video Title (shows in video search results)

Emails:

email(s) of the presenter(s)

Released:

has someone authorised pubication

Normalise:

Channelcopy:

m=mono, 01=copy left to right, 10=right to left, 00=ignore.

Thumbnail:

filename.png

Description:

markdown

Memory management with larger memory sizes of around a couple of
terabyte leads to a number of issues that make life hard. This talk is
a look at the issues and at the work in progress on these issues.

First, huge pages are not supported for the page cache which leads to
numerous performance issues. It begins with the long time it takes to
just copy a file of a few gigs in size (minor compared to the size of
the memory in the system of a few terabytes after all...) to the
performance problems when accessing large amounts of 4k pages from a
memory mapped file. Since huge pages are not supported TLB trashing
often becomes an issue. Devices have issues writing large blocks of
data to disk due to large scatter gather lists having to be processed
in memory. Reclaim becomes slow since the system is literally churning
through billions of 4k page to figure out which ones to evict.

Second, huge pages cause a memory overhead that surprises users. The
kernel manages 4k chunks of the huge page in many cases and needs to
have metadata structures for each 4k page. Other metadata also exists.
This results in gigabytes of memory just not being present. The user
bought a system with a lot of memory but the Linux just cannot use it.

Third, fragmentation becomes an issue in particular if huge pages and
4k pages are frequently allocated and freed (like we have to because
the page cache does not support 4k after all).  So over time systems
have to be rebooted to allow the allocation of huge pages. Huge pages
are actually only a small problem because devices on large systems
often want to allocate large contiguous buffers for I/O (like
specialized networking devices using RDMA).

Comment:

production notes