veyepar: Large Memory Management issues: Performance, Fragmentation, Movable objects and Huge Page overhead.

Author(s):	Christopher Lameter
Location	Great Hall CB01.05.009
Date	jan Mon 22	Days Raw Files
Start	11:35	First Raw Start	11:01
Duration	0:50:0	Offset	0:33:17
End	12:25	Last Raw End	12:31
Chapters	00:00 0:14:49
Total cuts_time	35 min.

http://lca2018.linux.org.au/schedule/presentation/197/
raw-playlist raw-mp4-playlist encoded-files-playlist host archive public tweet mp4 svg png

assets release.pdf Large_Memory_Management_issues_Performance_Fragmentation_Movable_objects_and_Huge_Page_overhead.json
logs
Admin: episode episode list cut list raw files day marks day marks day image_files

State:
Locked:	clear this to unlock
Locked by:	user/process that locked.
Start:	initially scheduled time from master, adjusted to match reality
Duration:	length in hh:mm:ss
Name:	Video Title (shows in video search results)
Emails:	email(s) of the presenter(s)
Released:	has someone authorised pubication
Normalise:
Channelcopy:	m=mono, 01=copy left to right, 10=right to left, 00=ignore.
Thumbnail:	filename.png
Description:	Memory management with larger memory sizes of around a couple of terabyte leads to a number of issues that make life hard. This talk is a look at the issues and at the work in progress on these issues. First, huge pages are not supported for the page cache which leads to numerous performance issues. It begins with the long time it takes to just copy a file of a few gigs in size (minor compared to the size of the memory in the system of a few terabytes after all...) to the performance problems when accessing large amounts of 4k pages from a memory mapped file. Since huge pages are not supported TLB trashing often becomes an issue. Devices have issues writing large blocks of data to disk due to large scatter gather lists having to be processed in memory. Reclaim becomes slow since the system is literally churning through billions of 4k page to figure out which ones to evict. Second, huge pages cause a memory overhead that surprises users. The kernel manages 4k chunks of the huge page in many cases and needs to have metadata structures for each 4k page. Other metadata also exists. This results in gigabytes of memory just not being present. The user bought a system with a lot of memory but the Linux just cannot use it. Third, fragmentation becomes an issue in particular if huge pages and 4k pages are frequently allocated and freed (like we have to because the page cache does not support 4k after all). So over time systems have to be rebooted to allow the allocation of huge pages. Huge pages are actually only a small problem because devices on large systems often want to allocate large contiguous buffers for I/O (like specialized networking devices using RDMA). markdown
Comment:	production notes