Here are a few notes on how tracing of compiled code works.
When a function is traced, a breakpoint instruction is placed at the start of the function, replacing the instruction that was there. (This is a :function-start breakpoint.) (This appears to be one instruction after the no-arg parsing entry point.) The breakpoint instruction is, of course, architecture-specific, but it must signal a trap_Breakpoint trap.
When the code is run, the breakpoint instruction is executed causing a trap. The trap handler runs HANDLE-BREAKPOINT to process it. After doing the appropriate processing, we now need to continue. Of course, since the real instruction has been replaced, we to run the original instruction. This is done by now inserting a new breakpoint after the original breakpoint. This breakpoint must be of the type trap_AfterBreakpoint. The original instruction is restored and execution continues from there. Then the trap_AfterBreakpoint instruction gets executed. The handler for this puts back the original breakpoint, thereby preserving the breakpoint. Then we replace the AfterBreakpoint with the original instruction and continue from there.
That’s all pretty straightforward in concept.
When tracing, additional information is needed. Breakpoints have the ability to run arbitrary lisp code to process the breakpoint. Tracing uses this feature.
When this breakpoint is reached, HANDLE-BREAKPOINT runs the breakpoint hook function. This function figures out where this function would return to and creates a new return area and replaces the original return address with this new address. Thus, when the function returns, it returns to this new location instead of the original.
This new return address is a specially created bogus LRA object. It is a code-component whose body consists of a code template copied from an assembly routine into the body. The assembly routine is the code in function_end_breakpoint_guts. This bogus LRA object stores the real LRA for the function, and also an indication if the known-return convention is used for this function.
The bogus LRA object contains a function-end breakpoint (trap_FunctionEndBreakpoint). When it’s executed the trap handler handles this breakpoint. It figures out where this trap come from and calls HANDLE-BREAKPOINT to handle it. HANDLE-BREAKPOINT returns and the trap handler arranges it so that this bogus LRA returns to the real LRA.
Thus, we can do something when a Lisp function returns, like printing out the return value for the function for tracing.
There are lots of internal details left out here, but gives a short overview of how this works. For more info, look at code/debug-int.lisp and lisp/breakpoint.c, and, of course, the various <foo>-arch.c files.