veyepar: Reliable Linux Kernel Crash Dump with Micro-Controller Assistance

Author(s):	Vasant Hegde
Location	C1
Date	jan Tue 22	Days Raw Files
Start	10:45	First Raw Start	10:37
Duration	0:20:0	Offset	0:07:39
End	11:05	Last Raw End	11:07
Chapters	00:00 0:00:09
Total cuts_time	16 min.

https://2019.linux.conf.au/schedule/presentation/235/
raw-playlist raw-mp4-playlist encoded-files-playlist host archive public tweet mp4 svg png

assets release.pdf Reliable_Linux_Kernel_Crash_Dump_with_MicroController_Assistance.json
logs
Admin: episode episode list cut list raw files day marks day marks day image_files

State:

Locked:

clear this to unlock

Locked by:

user/process that locked.

Start:

initially scheduled time from master, adjusted to match reality

Duration:

length in hh:mm:ss

Name:

Video Title (shows in video search results)

Emails:

email(s) of the presenter(s)

Released:

has someone authorised pubication

Normalise:

Channelcopy:

m=mono, 01=copy left to right, 10=right to left, 00=ignore.

Thumbnail:

filename.png

Description:

markdown

Kdump is the standard mechanism to generate Linux kernel crash dumps. Reliability of kdump has always been a concern as it suffers from many possible failure modes -- rogue DMA, bad device state at the time of crash, dump capture kernel corruption, etc. Kdump, therefore, works on a best effort basis.

The IBM POWER9 processor has a number of satellite processors called PowerPC Processing Elements (PPEs) -- micro-controllers based on a scaled down version of embedded PowerPC processor. PPEs perform various boot and run-time activities for the normal working of the system. One of these PPEs is the Self-Boot Engine (SBE) which plays a significant part in the early boot of the Power processor.

In this presentation, we detail the concept, design, implementation and learning, from a framework that allows for guaranteed capture of the memory state of both the crashed Linux kernel and the OPAL firmware it runs on. On receipt of an indication that the OS/Firmware has crashed, the SBE triggers a memory preserving boot. It then passes control to the early boot firmware which then captures both the OPAL firmware and Linux kernel crash dumps. OPAL firmware then works in conjunction with Linux to extract this dump. Linux then formats the dumps appropriately so that existing analysis tools (crash/gdb) can be used on them. All the components involved to produce the dumps are Open Source.

We present various scenarios of software failure and how this framework is designed to work in all such cases, making this solution more robust than the standard kdump.

Comment:

production notes