Here are a few notes on how tracing of compiled code works.
When a function is traced, a breakpoint instruction is placed at the
start of the function, replacing the instruction that was there.
(This is a :function-start breakpoint.) (This appears to be
one instruction after the no-arg parsing entry point.) The breakpoint
instruction is, of course, architecture-specific, but it must signal a
trap_Breakpoint trap.
When the code is run, the breakpoint instruction is executed causing a
trap. The trap handler runs HANDLE-BREAKPOINT to process it.
After doing the appropriate processing, we now need to continue. Of
course, since the real instruction has been replaced, we to run the
original instruction. This is done by now inserting a new
breakpoint after the original breakpoint. This breakpoint must be of
the type trap_AfterBreakpoint. The original instruction is
restored and execution continues from there. Then the
trap_AfterBreakpoint instruction gets executed. The handler
for this puts back the original breakpoint, thereby preserving the
breakpoint. Then we replace the AfterBreakpoint with the original
instruction and continue from there.
That’s all pretty straightforward in concept.
When tracing, additional information is needed. Breakpoints have the ability to run arbitrary lisp code to process the breakpoint. Tracing uses this feature.
When this breakpoint is reached, HANDLE-BREAKPOINT runs the
breakpoint hook function. This function figures out where this
function would return to and creates a new return area and replaces
the original return address with this new address. Thus, when the
function returns, it returns to this new location instead of the
original.
This new return address is a specially created bogus LRA object. It
is a code-component whose body consists of a code template copied from
an assembly routine into the body. The assembly routine is the code
in function_end_breakpoint_guts. This bogus LRA object stores
the real LRA for the function, and also an indication if the
known-return convention is used for this function.
The bogus LRA object contains a function-end breakpoint
(trap_FunctionEndBreakpoint). When it’s executed the trap
handler handles this breakpoint. It figures out where this trap come
from and calls HANDLE-BREAKPOINT to handle it.
HANDLE-BREAKPOINT returns and the trap handler arranges it so
that this bogus LRA returns to the real LRA.
Thus, we can do something when a Lisp function returns, like printing out the return value for the function for tracing.
There are lots of internal details left out here, but gives a short
overview of how this works. For more info, look at
code/debug-int.lisp and lisp/breakpoint.c, and, of
course, the various <foo>-arch.c files.