Debugger For Mac Os



As security researchers, we often find ourselves needing to look deep into various kernels to fully understand our target and accomplish our goals. Doing so on the Windows platform is no mystery, as there have been countless well-written posts about kernel debugging setups. For macOS, however, the situation is slightly different.

There are many great posts describing how to set up kernel debugging between two machines, but all of them suggest that SIP (System Integrity Protection) should be disabled for kernel debugging. This creates a problem if we want to investigate the inner workings of macOS’s security mechanisms, since turning off SIP will also turn off most of the foundational security features of the operating system.

This blog post will describe a couple of setups that allow you to have SIP enabled while debugging.

Hopper is a disassembler + debugger meant for Windows, Linux, iOS and Mac OS. It is a cheap alternative for Hex rays IDA. Hopper is capable of showing Assembly code and Pseudo code at the same time so it is best for static analysis. Set Up Eclipse for Debugging on Mac OS X Set Up Eclipse for Debugging on Mac OS X. January 23, 2015 chad. I recently got an ST-LINK/V2 JTAG debugger from Mouser.com so I can play around with hardware debugging on Crazyflie 2.0. To do this more easily, I figured.

Our setup is as follows:

  • HOST: macOS Catalina 10.15.4 with supplemental update
  • GUEST: macOS Catalina 10.15.4 with supplemental update
  • VMware Fusion 11.5.3
  • Debugger: LLDB

The Good

We will start by debugging the original release version of the kernel, which is included by default on macOS. This is by far the easiest method we will see.

Our first step is to download the Kernel Debug Kit (KDK) from Apple’s Developer Downloads. Before we do that however, we need to determine the build version we are interested in. This can be accomplished with the following command on the guest VM:

% sw_vers
ProductName: Mac OS X
ProductVersion: 10.15.4
BuildVersion: 19E287

Listing 1 – Retrieving the kernel build version info

Once we know the BuildVersion number, we can download the corresponding KDK and install it on our host. It will be installed under /Library/Developer/KDKs/.

% ls -l /Library/Developer/KDKs/KDK_10.15.4_19E287.kdk/System/Library/Kernels/
total 212112
-rwxr-xr-x 1 root wheel 16030560 Mar 5 07:38 kernel
drwxr-xr-x 3 root wheel 96 Mar 5 07:38 kernel.dSYM
-rwxr-xr-x 1 root wheel 23795528 Mar 5 07:27 kernel.debug
drwxr-xr-x 3 root wheel 96 Mar 5 07:27 kernel.debug.dSYM
-rwxr-xr-x 1 root wheel 19329072 Mar 5 07:39 kernel.development
drwxr-xr-x 3 root wheel 96 Mar 5 07:39 kernel.development.dSYM
-rwxr-xr-x 1 root wheel 49436536 Mar 5 07:30 kernel.kasan

Listing 2 – KDK location

The next step is to enable remote debugging on our guest VM. Fortunately, VMware Fusion has a feature called gdb stub, which sets up a GDB server and allows a debugger to debug any VM (including Windows) using the GDB remote protocol. Since LLDB supports the GDB protocol, we can use this approach.

To enable the GDB stub for our guest VM, we need to add the following line to our virtual machine vmx configuration file:

Listing 3 – Enabling the VMware gdb stub

Next, we boot up the VM, launch lldb on our host, and issue the following commands to specify our kernel and also enable loading of symbol files:

(lldb) target create /Library/Developer/KDKs/KDK_10.15.4_19E287.kdk/System/Library/Kernels/kernel
(lldb) settings set target.load-script-from-symbol-file true
(lldb) gdb-remote 8864

Listing 4 – Starting LLBD with appropriate symbols

The first command will tell lldb where to find the kernel symbols. This command is not strictly necessary, as lldb will search the /Library/Developer/KDKs path and any other which is indexed by Spotlight, but it can be still a good practice in case the search fails.

The second command will tell lldb to load any scripts found inside the symbol (dSYM) directories. This is extremely useful, as these scripts typically extend the functionality of lldb. In the case of the kernel, we get about 400 new commands available to us.

The last command tells lldb which port the remote server is listening on. If we don’t specify the IP address or hostname, like here, it will connect to the localhost. By default, VMware listens on port 8864 on the localhost, so this is where we connect to.

If everything has gone well, we should be breaking into our VM and can start debugging. We can also break again at any point using the “CTRL+C” shortcut from the debugger.

(lldb) gdb-remote 8864
Kernel UUID: AB0AA7EE-3D03-3C21-91AD-5719D79D7AF6
Load Address: 0xffffff8002600000
Kernel slid 0x2400000 in memory.
(..)
Process 1 stopped
* thread #3, name = '0xffffff8009854a40', queue = 'cpu-0', stop reason = signal SIGTRAP
frame #0: 0xffffff800284dede kernel`machine_idle at pmCPU.c:181:3 [opt]
Target 0: (kernel) stopped.
(lldb) image list
[ 0] AB0AA7EE-3D03-3C21-91AD-5719D79D7AF6 0xffffff8002600000 /Library/Developer/KDKs/KDK_10.15.4_19E287.kdk/System/Library/Kernels/kernel
/Library/Developer/KDKs/KDK_10.15.4_19E287.kdk/System/Library/Kernels/kernel.dSYM/Contents/Resources/DWARF/kernel
(..)

Listing 5 – LLDB macOS kernel debugging with symbols

One thing to note is that if our VM and Host are running the same kernel, we could also use the basic kernel binary (/System/Library/Kernels/kernel) as a target. However, we would not have any access to the symbols. Autotune vst mac crack.

(lldb) target create /System/Library/Kernels/kernel
Current executable set to '/System/Library/Kernels/kernel' (x86_64).
(lldb) gdb-remote 8864
Process 1 stopped
* thread #1, stop reason = signal SIGTRAP
frame #0: 0xffffff8003b980f6
-> 0xffffff8003b980f6: rep stosb byte ptr es:[rdi], al
0xffffff8003b980f8: ret
0xffffff8003b980f9: add byte ptr [rax], al
0xffffff8003b980fb: add byte ptr [rax], al
Target 0: (kernel) stopped.
(lldb) image list
[ 0] AB0AA7EE-3D03-3C21-91AD-5719D79D7AF6 0xffffff8000200000 /System/Library/Kernels/kernel
(lldb) memory read 0xffffff8000200000
0xffffff8000200000: cf fa ed fe 07 00 00 01 03 00 00 00 02 00 00 00 ????......
0xffffff8000200010: 12 00 00 00 d0 0f 00 00 01 00 20 00 00 00 00 00 ..?... ...
(lldb) continue

Listing 6 – LLDB macOS kernel debuging without symbols

Debug Mac Os X Application

Finally, it is important to note that VMware uses hardware breakpoints by default, which limits us to four total. However, this limitation can be overcome by setting the hideBreakpoints setting in the vmx configuration file to FALSE, as shown below:

Listing 6 – Disabling VMWare use of hardware breakpoints

The Bad

In the previous section, we discussed how to set up macOS kernel debugging with the release version of the kernel. Nevertheless, it is important to mention that Apple also releases debug or development kernels as well. According to Apple, they are compiled with “additional assertions and error checking” and these are the ones that can stop and wait for a debugger to connect after initial startup.

For this setup, these kernels are not strictly needed as we will still use the previous feature of VMware. More specifically, the macOS kernel will not be responsible for the actual debugging. However, in case we do need these kernels for a special use case, this is how we can set this up.

Before we do anything else on our guest VM, we first need to temporarily disable SIP. The reason for this because the /System/ path is write-protected by SIP. We can do that by booting into recovery mode (CMD+R after turning on the VM), running the csrutil disable command, and rebooting.

Since the release of macOS Catalina, the / path is mounted as read-only as an additional protection beyond SIP. Therefore, we will need to make it writable so that we copy the debug kernel to the correct location.

Once rebooted, we mount the root directory as writable:

Listing 7 – Mounting the root directory as writable

Debugger For Macos

And copy the kernel of our choice (debug in this case):

Debugging Macos

Debugger For Mac Os
sudo cp /Library/Developer/KDKs/KDK_10.15.4_19E287.kdk/System/Library/Kernels/kernel.debug /System/Library/Kernels/

Listing 8 – Copying the debug kernel to the appropriate location

Then we need to invalidate the kernel cache. This is required because macOS doesn’t run the kernel binary directly, but rather as a prelinked kernel, which is built from the kernel and the kernel extensions. Normally, the prelinking happens when we install a new kernel extension or kernel, but not in this case. Here, we simply copy in the development or debug version of the kernel.

sudo kextcache -invalidate /
sudo kextcache -invalidate /Volumes/Macintosh HD

Listing 9 – Invalidating the kernel cache

Finally, we will need to set the NVRAM boot arguments to boot into the debug kernel instead of the regular one, and boot into recovery mode to turn SIP back on using the csrutil enable command.

Listing 10 – Setting the boot kernel

We can now run the same commands on our host as before, this time specifying the debug kernel.

(lldb) target create /Library/Developer/KDKs/KDK_10.15.4_19E287.kdk/System/Library/Kernels/kernel.debug
(lldb) settings set target.load-script-from-symbol-file true
(lldb) gdb-remote 8864

Debugger For Microsoft Edge

Listing 11 – Commands to start debug kernel debugging

At this point, we are able to perform macOS kernel debugging on a debug rather than a release version of the kernel.

Process 1 stopped
* thread #2, name = '0xffffff80158f0e28', queue = 'cpu-0', stop reason = signal SIGTRAP
frame #0: 0xffffff80052ee796 kernel.debug`machine_idle at pmCPU.c:181:3
Target 0: (kernel.debug) stopped.
(lldb) image list
[ 0] 16545FA7-C11F-3D9E-88FA-8DDB13E1A439 0xffffff8005000000 /Library/Developer/KDKs/KDK_10.15.4_19E287.kdk/System/Library/Kernels/kernel.debug
/Library/Developer/KDKs/KDK_10.15.4_19E287.kdk/System/Library/Kernels/kernel.debug.dSYM/Contents/Resources/DWARF/kernel.debug
[ 1] A6D59354-C9A1-3C61-87A7-C04DD74421B1 0xffffff7f8609f000 //System/Library/Extensions/corecrypto.kext/Contents/MacOS/corecrypto
(..)

Listing 12 – Debugging the macOS debug kernel

The Ugly

For completeness, we’ll briefly discuss how to set up kernel debugging using the macOS kernel instead of VMware gdb stub. This is the most commonly covered method elsewhere. We will also show that we can enable SIP despite the common misbelief that it has to be turned off.

The steps are essentially the same as before with a small difference. In this case, the NVRAM variables in the debugger box should be set as shown below:

Listing 13 – Setting the NVRAM variables for kernel debugging

In essence, these settings indicate that the debuggee can perform network-level debugging and break on interrupts. The interrupt is tricky, however, as we need to press CMD + OPTION + CONTROL + SHIFT + ESCAPE to break into the debugger. Furthermore, this has to be done on the target VM we are debugging. While others have had success in causing interrupts using this method, we were not as lucky. Instead, we made a keyboard shortcut in VMware Key Mappings using CMD + B for this combination.

“Windows” and “Alt” buttons as the target key mappings, however they translate into “Command” and “Option” respectively in macOS. VMware doesn’t provide an option to specify the macOS version of the keys.

Figure 1: VMware Key Mappings configuration

Finally, on the host machine we will use the kdp-remote command with our debugee IP address, instead of gdb-remote. Please note that before we issue this command, we will need to break into debugger on the debuggee.

Listing 14 – Using kdp-remote

Conclusion

In this post, we demonstrated three different ways of debugging the macOS kernel while determining that permanent disabling of SIP is not necessary. Due to the ease of setup, our preferred method is the first one we described, but others are available if the circumstances require them.

Additional resources:

  • Presentation: SyScan360 – Stefan Esser – OS X El Capitan sinking the SH/IP
About the Author

Csaba Fitzl has worked for 6 years as a network engineer and 8 years as a blue/red teamer in a large enterprise focusing on malware analysis, threat hunting, exploitation, and defense evasion. Currently, he is focusing on macOS research and working at OffSec as a content developer. He gives talks and workshops at various international IT security conferences, including Hacktivity, hack.lu, Troopers, SecurityFest, DEFCON, and Objective By The Sea. @theevilbit

For Developers‎ > ‎How-Tos‎ > ‎

Debugging Chromium on macOS

Resources:

Contents

  1. 4 Debugging the renderer process
  2. 11 Debugging in Release Mode

The Mac OS X Debugging Magic Technote contains a wealth of information about various debugging options built in to macOS.
IMPORTANT: By default, Xcode has the 'Load Symbols Lazily' preference set. As a result, any symbols not in the main static library (99% of our code) won't be visible to set breakpoints. The result is that you set breakpoints in the editor window and they're ignored entirely when you run. The fix, however, is very simple! Uncheck the 'Load Symbols Lazily' checkbox in the 'Debugging' panel in preferences. Now all your breakpoints will work, at the expense of a little longer load time in gdb. Well worth it, if you ask me.
ALSO IMPORTANT: If you include fast_build=1 in your GYP_DEFINES, there is an excellent chance the symbols you'll need for debugging will be stripped! You may save yourself a lot of heartache if you remove this, rerun gyp_chromium and rebuild before proceeding.

Disabling ReportCrash

macOS helpfully tries to write a crash report every time a binary crashes – which happens for example when a test in unit_tests fails. Since Chromium's debug binaries are huge, this takes forever. If this happens, 'ReportCrash' will be the top cpu consuming process in Activity Monitor. You should disable ReportCrash while you work on Chromium. Run man ReportCrash to learn how to do this on your version of macOS. On 10.8, the command is
launchctl unload -w /System/Library/LaunchAgents/com.apple.ReportCrash.plist
sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.ReportCrash.Root.plist


Yes, you need to run this for both the normal user and the admin user.

Processing Apple Crash Reports

If you get a Google Chrome crash report caught by ReportCrash/OS X, it will not have symbols (every frame will be ChromeMain). To get a symbolized stack trace, use the internal crsym tool by simply pasting the contents of an entire Apple crash report.

Debugging the renderer process

Xcode's built in gdb wrapper doesn't allow you to debug more than one process at once and doesn't deal very well with debugging Chrome's subprocesses directly. There are two different ways around this:

(a) Run Chrome in a single process

(NOTE: this option is not recommended any more -- Chrome's single-process mode is neither supported nor tested.)
  1. Edit the Executable settings for the Chromium app (make it the current executable, then choose Project > Edit Active Executable).
  2. Switch to the Arguments tab and press the '+' button under the arguments list
  3. Type '--single-process' in the list.
From now on Chromium will launch in single-process mode when invoked through this Xcode project, and the debugger will work fine. This obviously changes the apps behavior slightly, but for most purposes the differences aren't significant. If they are, though, you'll need to…

(b) or, Attach Xcode's debugger to a renderer process after launch

1. Launch the main executable from the Terminal (not through Xcode) and pass in the --renderer-startup-dialog flag on the command line. On macOS this causes the renderer to print a message with its PID and then call pause() immediately up on startup. This has the effect of pausing execution of the renderer until the process receives a signal (such as attaching the debugger).
e.g.
$ ~/dev/chrome//src/xcodebuild/Debug/Chromium.app/Contents/MacOS/Chromium --renderer-startup-dialog

[33215:2055:244180145280185:WARNING:/Users/Shared/bla/chrome/src/chrome/renderer/renderer_main.cc(48)] Renderer (33215) paused waiting for debugger to attach @ pid

So 33215 is the PID of the renderer process in question.
2. Open chrome.xcodeproj in Xcode and select Run -> Attach To Process -> Process ID .

Debugging out-of-process tests:

Similar to debugging the renderer process, simply attaching gdb to a out-of-process test like browser_tests will not hit the test code. In order to debug a browser test, you need to run the test binary with '--single_process' (note the underscore in single_process). Because you can only run one browser test in the same process, you're probably going to need to add --gtest_filter as well. So your command will look like this:
/path/to/src/xcodebuild/Debug/browser_tests --single_process --gtest_filter=GoatTeleporterTest.DontTeleportSheep

UI Debugging

For UI Debugging, F-Script Anywhere is very useful. Read https://sites.google.com/a/chromium.org/dev/developers/f-script-anywhere for more information.

Building with Ninja, Debugging with Xcode

Temporarily disabling the Sandbox

Disabling the sandbox can sometimes be useful when debugging, this can be achieved by passing the --no-sandbox flag on the command line. This will, for example, allow writing out debugging information to a file from the Renderer Process.
e.g.
$ ~/dev/chrome//src/xcodebuild/Debug/Chromium.app/Contents/MacOS/Chromium --no-sandbox

Tips on Debugging the Renderer Sandbox

Launch chrome with the --enable-sandbox-logging flag. This will cause a message to be printed to /var/log/system.log every time an operation is denied by the Sandbox (you can use Console.app to watch logfiles). This is really useful for debugging and can often provide an explanation for very puzzling problems.
You can also get the Sandbox to send a SIGSTOP to a process when the sandbox denies functionality. This allows you to attach with a debugger and continue the execution from where it left off:
$ sandbox-exec -p '(version 1) (allow default) (deny file-write* (regex 'foo') (with send-signal SIGSTOP))' touch foo

Breakpoints Not Getting Hit in gdb

If a breakpoint you set isn't causing the debugger to stop, try one of these solutions:
  • Uncheck 'Load symbols lazily' In the Xcode->Preferences->Debugging dialog.
  • Manually insert a call to Debugger() in the code, this will forcefully break into the Debugger.

Debugging in Release Mode

Preserving symbols in Release builds

Profiling tools like Shark and 'sample' expect to find symbol names in the binary, but in Release builds most symbols are stripped out. You can preserve symbols by temporarily changing the build process, by adding mac_strip_release=0 to your GYP_DEFINES, rerunning gclient runhooks, and rebuilding (changing this define only relinks the main binary, it doesn't recompile everything).
(The above 'Debugging in Release Mode' trick with the .dSYM file might work for Shark/sample too; I haven't tried it yet. —snej)

Using DTrace

jgm's awesome introductory article:
http://www.mactech.com/articles/mactech/Vol.23/23.11/ExploringLeopardwithDTrace/index.html
Defining static probes on macOS:
http://www.macresearch.org/tuning-cocoa-applications-using-dtrace-custom-static-probes-and-instruments
http://www.brendangregg.com/dtrace.html#Examples
http://blogs.sun.com/bmc/resource/dtrace_tips.pdf
DTrace examples on macOS: /usr/share/examples/DTTk
To get truss on macOS, use dtruss. That requires root, so I often sudo dtruss -p and attach to a running nonroot program.

Testing other locales

To test Chrome in a different locale, change your system locale via the System Preferences. (Keep the preferences window open so that you can change the locale back without needing to navigate through menus in a language you may not know.)

Memory/Heap Inspection

There are several low-level command-line tools that can be used to inspect what's going on with memory inside a process.
'heap' summarizes what's currently in the malloc heap(s) of a process. (It only works with regular malloc, of course, but Mac Chrome still uses that.) It shows a number of useful things:
  • How much of the heap is used or free
  • The distribution of block sizes
  • A listing of every C++, Objective-C and CoreFoundation class found in the heap, with the number of instances, total size and average size.
It identifies C++ objects by their vtables, so it can't identify vtable-less classes, including a lot of the lower-level WebCore ones like StringImpl. To work around this I temporarily added the 'virtual' keyword to WebCore::RefCounted's destructor method, which forces every ref-counted object to include a vtable pointer identifying its class.
'malloc_history' identifies the stack backtrace that allocated every malloc block in the heap. It lists every unique backtrace together with its number of blocks and their total size. It requires that the process use malloc stack logging, which is enabled if the environment variable MallocStackLogging is set when it launches. The 'env' command is handy for this:
$ env MallocStackLogging=1 Chromium.app/Contents/MacOS/Chromium
Then in another shell you run
$ malloc_history pid -all_by_size
Watch out: the output is big. I ran malloc_history on a fairly bloated heap and got 60MB of text.
'leaks' finds malloc blocks that have no pointers to them and are probably leaked. It doesn't require MallocStackLogging, but it's more useful if it's on because it can then show the backtrace that allocated each leaked block. (So far I've seen only trivial leakage in Chrome.)
'vmmap' shows all the virtual-memory regions in the process's address space. This is less useful since it doesn't say anything about individual malloc blocks (except huge ones) but it can be useful for looking at things like static data size, mapped files, and how much memory is paged out. I recommend the '-resident' flag, which shows how much of each allocation is currently paged into RAM. See the man page for details.
Notes:
  • These are not going to be very useful on stripped binaries, and they're less useful in release builds.
  • All of these except vmmap take several minutes to run, apparently because of the number of symbols in Chrome. They spend most of their time pegging one CPU down inside system code that's reading symbol tables from the binary. Be patient.
  • There are GUI apps in /Developer that do a lot of the same things, such as Instruments, MallocDebug and Shark. I (snej) personally find the command-line tools easier to understand, but YMMV.

Working with minidumps

CrMallocErrorBreak

If you are looking at a crash report that ends in CrMallocErrorBreak, then either a malloc or free call has failed with the given stacktrace. Chromium overrides the empty function malloc_error_break in macOS's Libc with CrMallocErrorBreak. The system calls this function as a debugging aide that we've made fatal because it catches useful memory errors. Specifically, CrMallocErrorBreak will be called (resulting in a crash) under the following circumstances:
  • Attempting to free a pointer that was not allocated.
  • Attempting to free the same pointer more than once.
  • Freeing a pointer of size 0.
  • Freeing an unaligned pointer.
  • An internal checksum of the object being freed does not match. This usually indicates heap corruption!
  • Invalid checksums on the small or tiny free list. The system maintains a list of small allocations that it reuses to speed up things like allocations in a loop. A checksum mismatch usually indicates a use-after-free, double-free, or heap corruption.
  • Extra-large allocation failures. Normally all failures to allocate go through CrMallocErrorBreak but are not fatal because that is the job of Chromium's OOM killer. Extra-large allocations go through a different path and are sometimes killed here instead.
If you get a crash report that that ends in CrMallocErrorBreak, it is likely not an issue with this feature. It is instead surfacing a (sometimes serious) bug in your code or other code that is stomping on your code's memory. Using Chromium's memory tools (ASan, HeapCheck, and Valgrind) is a good start, if you can reproduce the problem.

Enabling high-DPI (aka 'HiDPI' or 'Retina') modes on standard-DPI hardware.

Under macOS 10.7 and above it's possible to fake 'HiDPI' modes on standard-DPI hardware. This can be useful in testing up-scaling codepaths or high-DPI resources.
  1. Configure the OS to offer HiDPI modes:
    • EITHER follow Apple's instructions to enable high resolution modes:http://developer.apple.com/library/mac/#documentation/GraphicsAnimation/Conceptual/HighResolutionOSX/Testing/Testing.html.
    • OR run the command-line: sudo defaults write /Library/Preferences/com.apple.windowserver DisplayResolutionEnabled -bool YES
  2. Open the System Preferences -> Displays panel, select Scaled mode and scroll to the bottom to see modes marked '(HiDPI)'.
Looking for gdb? It's been replaced with lldb. Use that it instead.

Taking CPU Samples

A quick and easy way to investigate slow or hung processes is to use the sample facility, which will generate a CPU sample trace. This can be done either in the Terminal with the sample(1) command or by using Activity Monitor:
  1. Open Activity Monitor
  2. Find the process you want to sample (for 'Helper' processes, you may want to consult the Chrome Task Manager)
  3. Double-click on the row
  4. Click the Sample button in the process's information window
After a few seconds, the sample will be completed. For official Google Chrome builds, the sample should be symbolized using crsym. If you do not have access to crsym, save the entire contents as a file and attach it to a bug report for later analysis.
See also How to Obtain a Heap Dump.
Subpages (1):Building with Ninja, Debugging with Xcode