|
|
This chapter contains information about the compiler that every CMUCL user should be familiar with. Chapter 5 goes into greater depth, describing ways to use more advanced features.
The CMUCL compiler (also known as Python, not to be confused with the programming language of the same name) has many features that are seldom or never supported by conventional Common Lisp compilers:
|
Functions may be compiled using compile, compile-file, or compile-from-stream.
This function compiles the function whose name is name. If name is nil, the compiled function object is returned. If definition is supplied, it should be a lambda expression that is to be compiled and then placed in the function cell of name. As per the proposed X3J13 cleanup ``compile-argument-problems'', definition may also be an interpreted function.
The return values are as per the proposed X3J13 cleanup ``compiler-diagnostics''. The first value is the function name or function object. The second value is nil if no compiler diagnostics were issued, and t otherwise. The third value is nil if no compiler diagnostics other than style warnings were issued. A non-nil value indicates that there were ``serious'' compiler diagnostics issued, or that other conditions of type error or warning (but not style-warning) were signaled during compilation.
The CMUCL compile-file is extended through the addition of several new keywords and an additional interpretation of input-pathname:
The return values are as per the proposed X3J13 cleanup ``compiler-diagnostics''. The first value from compile-file is the truename of the output file, or nil if the file could not be created. The interpretation of the second and third values is described above for compile.
input-pathname
- If this argument is a list of input files, rather than a single input pathname, then all the source files are compiled into a single object file. In this case, the name of the first file is used to determine the default output file names. This is especially useful in combination with block-compile.
- :output-file
- This argument specifies the name of the output file. t gives the default name, nil suppresses the output file.
- :error-file
- A listing of all the error output is directed to this file. If there are no errors, then no error file is produced (and any existing error file is deleted.) t gives "name.err" (the default), and nil suppresses the output file.
- :error-output
- If t (the default), then error output is sent to *error-output*. If a stream, then output is sent to that stream instead. If nil, then error output is suppressed. Note that this error output is in addition to (but the same as) the output placed in the error-file.
- :verbose
- If t (the default), then the compiler prints to error output at the start and end of compilation of each file. See *compile-verbose*.
- If t (the default), then the compiler prints to error output when each function is compiled. See *compile-print*.
- :progress
- If t (default nil), then the compiler prints to error output progress information about the phases of compilation of each function. This is a CMUCL extension that is useful mainly in large block compilations. See *compile-progress*.
- :trace-file
- If t, several of the intermediate representations (including annotated assembly code) are dumped out to this file. t gives "name.trace". Trace output is off by default. See section 5.12.5.
- :load
- If t, load the resulting output file.
- :block-compile
- Controls the compile-time resolution of function calls. By default, only self-recursive calls are resolved, unless an ext:block-start declaration appears in the source file. See section 5.7.3.
- :entry-points
- If non-null, then this is a list of the names of all functions in the file that should have global definitions installed (because they are referenced in other files.) See section 5.7.3.
- :byte-compile
- If t, compiling to a compact interpreted byte code is enabled. Possible values are t, nil, and :maybe (the default.) See *byte-compile-default* and See section 5.9.
[Variable]
*compile-print*[Variable]
*compile-progress*These variables determine the default values for the :verbose, :print and :progress arguments to compile-file.
This function is similar to compile-file, but it takes all its arguments as streams. It reads Common Lisp code from input-stream until end of file is reached, compiling into the current environment. This function returns the same two values as the last two values of compile. No output files are produced.
|
CMUCL supports the with-compilation-unit macro added to the language by the proposed X3J13 ``with-compilation-unit'' compiler cleanup. This provides a mechanism for eliminating spurious undefined warnings when there are forward references across files, and also provides a standard way to access compiler extensions.
This macro evaluates the forms in an environment that causes warnings for undefined variables, functions and types to be delayed until all the forms have been evaluated. Each keyword value is an evaluated form. These keyword options are recognized:
:override
- If uses of with-compilation-unit are dynamically nested, the outermost use will take precedence, suppressing printing of undefined warnings by inner uses. However, when the override option is true this shadowing is inhibited; an inner use will print summary warnings for the compilations within the inner scope.
- :optimize
- This is a CMUCL extension that specifies of the ``global'' compilation policy for the dynamic extent of the body. The argument should evaluate to an optimize declare form, like:
See section 4.7.1(optimize (speed 3) (safety 0))- :optimize-interface
- Similar to :optimize, but specifies the compilation policy for function interfaces (argument count and type checking) for the dynamic extent of the body. See section 4.7.2.
- :context-declarations
- This is a CMUCL extension that pattern-matches on function names, automatically splicing in any appropriate declarations at the head of the function definition. See section 5.7.5.
|
Warnings about undefined variables, functions and types are delayed until the end of the current compilation unit. The compiler entry functions (compile, etc.) implicitly use with-compilation-unit, so undefined warnings will be printed at the end of the compilation unless there is an enclosing with-compilation-unit. In order the gain the benefit of this mechanism, you should wrap a single with-compilation-unit around the calls to compile-file, i.e.:
Unlike for functions and types, undefined warnings for variables are not suppressed when a definition (e.g. defvar) appears after the reference (but in the same compilation unit.) This is because doing special declarations out of order just doesn't work---although early references will be compiled as special, bindings will be done lexically.(with-compilation-unit () (compile-file "file1") (compile-file "file2") ...)
Undefined warnings are printed with full source context (see section 4.4), which tremendously simplifies the problem of finding undefined references that resulted from macroexpansion. After printing detailed information about the undefined uses of each name, with-compilation-unit also prints summary listings of the names of all the undefined functions, types and variables.
This variable controls the number of undefined warnings for each distinct name that are printed with full source context when the compilation unit ends. If there are more undefined references than this, then they are condensed into a single warning:
When the value is 0, then the undefined warnings are not broken down by name at all: only the summary listing of undefined names is printed.Warning: count more uses of undefined function name.
|
One of Python's unique features is the level of source location information it provides in error messages. The error messages contain a lot of detail in a terse format, to they may be confusing at first. Error messages will be illustrated using this example program:
The main problem with this program is that it is trying to add 3 to a symbol. Note also that the functions roq and ploq aren't defined anywhere.(defmacro zoq (x) `(roq (ploq (+ ,x 3)))) (defun foo (y) (declare (symbol y)) (zoq y))
|
The compiler will produce this warning:
In this example we see each of the six possible parts of a compiler error message:File: /usr/me/stuff.lisp In: DEFUN FOO (ZOQ Y) --> ROQ PLOQ + ==> Y Warning: Result is a SYMBOL, not a NUMBER.
In this example, the file, definition and original source are identical for the two messages, so the compiler omits them in the second message. If consecutive messages are entirely identical, then the compiler prints only the first message, followed by:File: /usr/me/stuff.lisp In: DEFUN FOO (ZOQ Y) --> ROQ ==> (PLOQ (+ Y 3)) Warning: Undefined function: PLOQ ==> (ROQ (PLOQ (+ Y 3))) Warning: Undefined function: ROQ
where repeats is the number of times the message was given.[Last message occurs repeats times]
If the source was not from a file, then no file line is printed. If the actual source is the same as the original source, then the processing path and actual source will be omitted. If no forms intervene between the original source and the actual source, then the processing path will also be omitted.
|
The original source displayed will almost always be a list. If the actual source for an error message is a symbol, the original source will be the immediately enclosing evaluated list form. So even if the offending symbol does appear in the original source, the compiler will print the enclosing list and then print the symbol as the actual source (as though the symbol were introduced by a macro.)
When the actual source is displayed (and is not a symbol), it will always be code that resulted from the expansion of a macro or a source-to-source compiler optimization. This is code that did not appear in the original source program; it was introduced by the compiler.
Keep in mind that when the compiler displays a source form in an error message, it always displays the most specific (innermost) responsible form. For example, compiling this function:
(defun bar (x)
(let (a)
(declare (fixnum a))
(setq a (foo x))
a))
gives this error message:
This error message is not saying ``there's a problem somewhere in this let''---it is saying that there is a problem with the let itself. In this example, the problem is that a's nil initial value is not a fixnum.In: DEFUN BAR (LET (A) (DECLARE (FIXNUM A)) (SETQ A (FOO X)) A) Warning: The binding of A is not a FIXNUM: NIL
|
The processing path is mainly useful for debugging macros, so if you don't write macros, you can ignore the processing path. Consider this example:
Compiling results in this error message:(defun foo (n) (dotimes (i n *undefined*)))
Note that do appears in the processing path. This is because dotimes expands into:In: DEFUN FOO (DOTIMES (I N *UNDEFINED*)) --> DO BLOCK LET TAGBODY RETURN-FROM ==> (PROGN *UNDEFINED*) Warning: Undefined variable: *UNDEFINED*
(do ((i 0 (1+ i)) (#:g1 n))
((>= i #:g1) *undefined*)
(declare (type unsigned-byte i)))
The rest of the processing path results from the expansion of do:
(block nil
(let ((i 0) (#:g1 n))
(declare (type unsigned-byte i))
(tagbody (go #:g3)
#:g2 (psetq i (1+ i))
#:g3 (unless (>= i #:g1) (go #:g2))
(return-from nil (progn *undefined*)))))
In this example, the compiler descended into the
block, let,
tagbody and return-from to reach the progn printed as the actual source. This is a
place where the ``actual source appears in explanation'' rule was
applied. The innermost actual source form was the symbol *undefined* itself, but that also appeared in the
explanation, so the compiler backed out one level.
|
There are three levels of compiler error severity:
Violation of type declarations, or
In the language of the Common Lisp standard, these are situations where the compiler can determine that a situation with undefined consequences or that would cause an error to be signaled would result at runtime.
|
The compiler handles errors that happen during macroexpansion, turning them into compiler errors. If you want to debug the error (to debug a macro), you can set *break-on-signals* to error. For example, this definition:
(defun foo (e l)
(do ((current l (cdr current))
((atom current) nil))
(when (eq (car current) e) (return current))))
gives this error:
In: DEFUN FOO (DO ((CURRENT L #) (# NIL)) (WHEN (EQ # E) (RETURN CURRENT)) ) Error: (during macroexpansion) Error in function LISP::DO-DO-BODY. DO step variable is not a symbol: (ATOM CURRENT)
|
The compiler also handles errors while reading the source. For example:
The ``at 2'' refers to the character position in the source file at which the error was signaled, which is generally immediately after the erroneous text. The next line, ``(,/\foo)'', is the line in the source that contains the error file position. The ``/\ '' indicates the error position within that line (in this example, immediately after the offending comma.)Error: Read error at 2: "(,/\foo)" Error in function LISP::COMMA-MACRO. Comma not inside a backquote.
When in Hemlock (or any other EMACS-like editor), you can go to a character position with:
Note that if the source is from a Hemlock buffer, then the position is relative to the start of the compiled region or defun, not the file or buffer start.M-< C-u position C-f
After printing a read error message, the compiler attempts to recover from the error by backing up to the start of the enclosing top-level form and reading again with *read-suppress* true. If the compiler can recover from the error, then it substitutes a call to cerror for the unreadable form and proceeds to compile the rest of the file normally.
If there is a read error when the file position is at the end of the file (i.e., an unexpected EOF error), then the error message looks like this:
In this case, ``starting at 14'' indicates the character position at which the compiler started reading, i.e. the position before the start of the form that was missing the closing delimiter. The line "(defun test ()" is first line after the starting position that the compiler thinks might contain the unmatched open delimiter.Error: Read error in form starting at 14: "(defun test ()" Error in function LISP::FLUSH-WHITESPACE. EOF while reading #<Stream for file "/usr/me/test.lisp">
|
There is some control over the verbosity of error messages. See also *undefined-warning-limit*, *efficiency-note-limit* and *efficiency-note-cost-threshold*.
This variable specifies the number of enclosing actual source forms that are printed in full, rather than in the abbreviated processing path format. Increasing the value from its default of 1 allows you to see more of the guts of the macroexpanded source, which is useful when debugging macros.
[Variable]
*error-print-level*These variables are the print level and print length used in printing error messages. The default values are 5 and 3. If null, the global values of *print-level* and *print-length* are used.
This macro defines how to extract an abbreviated source context from the named form when it appears in the compiler input. lambda-list is a defmacro style lambda-list used to parse the arguments. The body should return a list of subforms that can be printed on about one line. There are predefined methods for defstruct, defmethod, etc. If no method is defined, then the first two subforms are returned. Note that this facility implicitly determines the string name associated with anonymous functions.
|
A big difference between Python and all other Common Lisp compilers is the approach to type checking and amount of knowledge about types:
|
If the compiler can prove at compile time that some portion of the program cannot be executed without a type error, then it will give a warning at compile time. It is possible that the offending code would never actually be executed at run-time due to some higher level consistency constraint unknown to the compiler, so a type warning doesn't always indicate an incorrect program. For example, consider this code fragment:
(defun raz (foo)
(let ((x (case foo
(:this 13)
(:that 9)
(:the-other 42))))
(declare (fixnum x))
(foo x)))
Compilation produces this warning:
In this case, the warning is telling you that if foo isn't any of :this, :that or :the-other, then x will be initialized to nil, which the fixnum declaration makes illegal. The warning will go away if ecase is used instead of case, or if :the-other is changed to t.In: DEFUN RAZ (CASE FOO (:THIS 13) (:THAT 9) (:THE-OTHER 42)) --> LET COND IF COND IF COND IF ==> (COND) Warning: This is not a FIXNUM: NIL
This sort of spurious type warning happens moderately often in the expansion of complex macros and in inline functions. In such cases, there may be dead code that is impossible to correctly execute. The compiler can't always prove this code is dead (could never be executed), so it compiles the erroneous code (which will always signal an error if it is executed) and gives a warning.
Type warnings are inhibited when the extensions:inhibit-warnings optimization quality is 3 (see section 4.7.) This can be used in a local declaration to inhibit type warnings in a code fragment that has spurious warnings.This function can be used as the default value for keyword arguments that must always be supplied. Since it is known by the compiler to never return, it will avoid any compile-time type warnings that would result from a default value inconsistent with the declared type. When this function is called, it signals an error indicating that a required keyword argument was not supplied. This function is also useful for defstruct slot defaults corresponding to required arguments. See section 5.2.5.
Although this function is a CMUCL extension, it is relatively harmless to use it in otherwise portable code, since you can easily define it yourself:
(defun required-argument () (error "A required keyword argument was not supplied."))
|
With the default compilation policy, all type assertions1 are precisely checked. Precise checking means that the check is done as though typep had been called with the exact type specifier that appeared in the declaration. Python uses policy to determine whether to trust type assertions (see section 4.7). Type assertions from declarations are indistinguishable from the type assertions on arguments to built-in functions. In Python, adding type declarations makes code safer.
If a variable is declared to be (integer 3 17), then its value must always always be an integer between 3 and 17. If multiple type declarations apply to a single variable, then all the declarations must be correct; it is as though all the types were intersected producing a single and type specifier.
Argument type declarations are automatically enforced. If you declare the type of a function argument, a type check will be done when that function is called. In a function call, the called function does the argument type checking, which means that a more restrictive type assertion in the calling function (e.g., from the) may be lost.
The types of structure slots are also checked. The value of a structure slot must always be of the type indicated in any :type slot option.2 Because of precise type checking, the arguments to slot accessors are checked to be the correct type of structure.
In traditional Common Lisp compilers, not all type assertions are checked, and type checks are not precise. Traditional compilers blindly trust explicit type declarations, but may check the argument type assertions for built-in functions. Type checking is not precise, since the argument type checks will be for the most general type legal for that argument. In many systems, type declarations suppress what little type checking is being done, so adding type declarations makes code unsafe. This is a problem since it discourages writing type declarations during initial coding. In addition to being more error prone, adding type declarations during tuning also loses all the benefits of debugging with checked type assertions.
To gain maximum benefit from Python's type checking, you should always declare the types of function arguments and structure slots as precisely as possible. This often involves the use of or, member and other list-style type specifiers. Paradoxically, even though adding type declarations introduces type checks, it usually reduces the overall amount of type checking. This is especially true for structure slot type declarations.
Python uses the safety optimization quality (rather than presence or absence of declarations) to choose one of three levels of run-time type error checking: see section 4.7.1. See section 5.2 for more information about types in Python.
|
When the value for the speed optimization quality is greater than safety, and safety is not 0, then type checking is weakened to reduce the speed and space penalty. In structure-intensive code this can double the speed, yet still catch most type errors. Weakened type checks provide a level of safety similar to that of ``safe'' code in other Common Lisp compilers.
A type check is weakened by changing the check to be for some convenient supertype of the asserted type. For example, (integer 3 17) is changed to fixnum, (simple-vector 17) to simple-vector, and structure types are changed to structure. A complex check like:
will be omitted entirely (i.e., the check is weakened to *.) If a precise check can be done for no extra cost, then no weakening is done.(or node hunk (member :foo :bar :baz))
Although weakened type checking is similar to type checking done by other compilers, it is sometimes safer and sometimes less safe. Weakened checks are done in the same places is precise checks, so all the preceding discussion about where checking is done still applies. Weakened checking is sometimes somewhat unsafe because although the check is weakened, the precise type is still input into type inference. In some contexts this will result in type inferences not justified by the weakened check, and hence deletion of some type checks that would be done by conventional compilers.
For example, if this code was compiled with weakened checks:
and myfun was passed a foo, then no type error would be signaled, and we would try to multiply a simple-vector as though it were a float (with unpredictable results.) This is because the check for bar was weakened to structure, yet when compiling the call to bar-a, the compiler thinks it knows it has a bar.(defstruct foo (a nil :type simple-string)) (defstruct bar (a nil :type single-float)) (defun myfun (x) (declare (type bar x)) (* (bar-a x) 3.0))
Note that normally even weakened type checks report the precise type in error messages. For example, if myfun's bar check is weakened to structure, and the argument is nil, then the error will be:
However, there is some speed and space cost for signaling a precise error, so the weakened type is reported if the speed optimization quality is 3 or debug quality is less than 1:Type-error in MYFUN: NIL is not of type BAR
See section 4.7.1 for further discussion of the optimize declaration.Type-error in MYFUN: NIL is not of type STRUCTURE
|
Since Python does much more comprehensive type checking than other Lisp compilers, Python will detect type errors in many programs that have been debugged using other compilers. These errors are mostly incorrect declarations, although compile-time type errors can find actual bugs if parts of the program have never been tested.
Some incorrect declarations can only be detected by run-time type checking. It is very important to initially compile programs with full type checks and then test this version. After the checking version has been tested, then you can consider weakening or eliminating type checks. This applies even to previously debugged programs. Python does much more type inference than other Common Lisp compilers, so believing an incorrect declaration does much more damage.
The most common problem is with variables whose initial value doesn't match the type declaration. Incorrect initial values will always be flagged by a compile-time type error, and they are simple to fix once located. Consider this code fragment:
Here the variable foo is given an initial value of nil, but is declared to be a fixnum. Even if it is never read, the initial value of a variable must match the declared type. There are two ways to fix this problem. Change the declaration:(prog (foo) (declare (fixnum foo)) (setq foo ...) ...)
or change the initial value:(prog (foo) (declare (type (or fixnum null) foo)) (setq foo ...) ...)
It is generally preferable to change to a legal initial value rather than to weaken the declaration, but sometimes it is simpler to weaken the declaration than to try to make an initial value of the appropriate type.(prog ((foo 0)) (declare (fixnum foo)) (setq foo ...) ...)
Another declaration problem occasionally encountered is incorrect declarations on defmacro arguments. This probably usually happens when a function is converted into a macro. Consider this macro:
Although legal and well-defined Common Lisp, this meaning of this definition is almost certainly not what the writer intended. For example, this call is illegal:(defmacro my-1+ (x) (declare (fixnum x)) `(the fixnum (1+ ,x)))
The call is illegal because the argument to the macro is (+ 4 5), which is a list, not a fixnum. Because of macro semantics, it is hardly ever useful to declare the types of macro arguments. If you really want to assert something about the type of the result of evaluating a macro argument, then put a the in the expansion:(my-1+ (+ 4 5))
In this case, it would be stylistically preferable to change this macro back to a function and declare it inline. Macros have no efficiency advantage over inline functions when using Python. See section 5.8.(defmacro my-1+ (x) `(the fixnum (1+ (the fixnum ,x))))
Some more subtle problems are caused by incorrect declarations that can't be detected at compile time. Consider this code:
(do ((pos 0 (position #\a string :start (1+ pos))))
((null pos))
(declare (fixnum pos))
...)
Although pos is almost always a fixnum, it is nil at the end of
the loop. If this example is compiled with full type checks (the
default), then running it will signal a type error at the end of
the loop. If compiled without type checks, the program will go into
an infinite loop (or perhaps position will
complain because (1+ nil) isn't a sensible
start.) Why? Because if you compile without type checks, the
compiler just quietly believes the type declaration. Since pos is always a fixnum, it
is never nil, so (null
pos) is never true, and the loop exit test is optimized away.
Such errors are sometimes flagged by unreachable code notes (see
section
5.4.5), but it is still important to initially compile any
system with full type checks, even if the system works fine when
compiled using other compilers.
In this case, the fix is to weaken the type declaration to (or fixnum null).3 Note that there is usually little performance penalty for weakening a declaration in this way. Any numeric operations in the body can still assume the variable is a fixnum, since nil is not a legal numeric argument. Another possible fix would be to say:
(do ((pos 0 (position #\a string :start (1+ pos))))
((null pos))
(let ((pos pos))
(declare (fixnum pos))
...))
This would be preferable in some circumstances, since it would
allow a non-standard representation to be used for the local pos variable in the loop body (see section 5.11.3.)
In summary, remember that all values that a variable ever has must be of the declared type, and that you should test using safe code initially.
|
The policy is what tells the compiler how to compile a program. This is logically (and often textually) distinct from the program itself. Broad control of policy is provided by the optimize declaration; other declarations and variables control more specific aspects of compilation.
|
The optimize declaration recognizes six different qualities. The qualities are conceptually independent aspects of program performance. In reality, increasing one quality tends to have adverse effects on other qualities. The compiler compares the relative values of qualities when it needs to make a trade-off; i.e., if speed is greater than safety, then improve speed at the cost of safety.
The default for all qualities (except debug) is 1. Whenever qualities are equal, ties are broken according to a broad idea of what a good default environment is supposed to be. Generally this downplays speed, compile-speed and space in favor of safety and debug. Novice and casual users should stick to the default policy. Advanced users often want to improve speed and memory usage at the cost of safety and debuggability.
If the value for a quality is 0 or 3, then it may have a special interpretation. A value of 0 means ``totally unimportant'', and a 3 means ``ultimately important.'' These extreme optimization values enable ``heroic'' compilation strategies that are not always desirable and sometimes self-defeating. Specifying more than one quality as 3 is not desirable, since it doesn't tell the compiler which quality is most important.
These are the optimization qualities:
|
The extensions:optimize-interface declaration is identical in syntax to the optimize declaration, but it specifies the policy used during compilation of code the compiler automatically generates to check the number and type of arguments supplied to a function. It is useful to specify this policy separately, since even thoroughly debugged functions are vulnerable to being passed the wrong arguments. The optimize-interface declaration can specify that arguments should be checked even when the general optimize policy is unsafe.
Note that this argument checking is the checking of user-supplied arguments to any functions defined within the scope of the declaration, not the checking of arguments to Common Lisp primitives that appear in those definitions.
The idea behind this declaration is that it allows the definition of functions that appear fully safe to other callers, but that do no internal error checking. Of course, it is possible that arguments may be invalid in ways other than having incorrect type. Functions compiled unsafely must still protect themselves against things like user-supplied array indices that are out of bounds and improper lists. See also the :context-declarations option to with-compilation-unit.
|
Since Common Lisp forbids the redefinition of standard functions4, the compiler can have special knowledge of these standard functions embedded in it. This special knowledge is used in various ways (open coding, inline expansion, source transformation), but the implications to the user are basically the same:
might turn into(nthcdr 4 foobar)
or even(cdr (cdr (cdr (cdr foobar))))
(do ((i 0 (1+ i))
(list foobar (cdr foobar)))
((= i 4) list))
If nth is closed coded, then
might stay the same, or turn into something like:(nth x l)
In general, open coding sacrifices space for speed, but some functions (such as car) are so simple that they are always open-coded. Even when not open-coded, a call to a standard function may be transformed into a different function call (as in the last example) or compiled as static call. Static function call uses a more efficient calling convention that forbids redefinition.(car (nthcdr x l))