Last update: $Date: 2000/06/26 20:10:11 $
Licence: this document is placed in the public domain, for unencumbered use, modification and distribution. I ask however that modified versions carry a notice describing the changes made, the author of the changes, and a pointer to this original version. This is a request which I cannot enforce other than by writing rude things about people who don't.
Your comments (on the document, preferably, not the licence) are always appreciated.
Warning: Contents under pressure.
I kept hearing nice things about Lisp. Speak to anyone who programmed real systems prior to the widespread adoption of Unix (if your local stores don't stock them, try ``The Unix-Haters' Handbook''), and you find that a typical Lisp system has (had) amazing programmer productivity tools: online hypertexted manuals, cross-reference things, call-graph generators, code coverage generators, an advanced debugger, incremental development and testing utilities, and so forth. Ten years later, gdb on Linux doesn't work properly with threads and shared libraries, Emacs can't indent Perl correctly, a new exploit caused by a programmer's careless use of fixed-length string buffers is found every week in some utility or other, weird shared library things happen when you don't expect them to, and the program taking most of the available memory on my computer is a glorified help file viewer with a tcp sockets interface stuck on it. This is progress?
``The Lisp programming environment'' has the potential to be a lot better than ``the Unix programming environment''. Right now it doesn't come that way out of the box (for values of `it' constrained to `free libre Lisp setups that work on Linux/x86') and if you're new to Lisp, setting it up so it does is about as much fun as, say, learning sendmail.cf rewrite rules without use of the Bat book.
So, the aim of this document isn't to tell you how to program Lisp (there are some good books on that, see the References section at the end), it's to provide a roadmap towards a comfortable free Lisp programming environment based on Linux/x86. I'm trying to cover the gaps in the textbook where it says ``your local wizard will be able to tell you how to do this'' - not that I qualify on either count, but the experience of figuring it out without one is still fresh in my mind.
A number of syntactic conventions have been adopted throughout this document. Most of them are pretty obvious, but just in case:
To compile filename.lisp, type (compile-file "filename")you should be able to work out that (compile-file "foo") is what you type to compile foo.lisp. Frankly, if you're fluent in English and that wasn't obvious anyway, you probably have more pressing problems than learning Lisp.
$ id uid=500(dan) gid=500(dan) groups=500(dan),100(users) * (+ 1 2) 3 # /sbin/reboot
That last one doesn't seem to produce much output. Odd, that.
To get much value from this document, you will need a basic understanding of Linux or other Unix, and of Emacs. You should know what to do when I write "copy the resulting file to /usr/lib/cmucl/lisp.core", and you should know what to do when I write "evaluate this by pressing M-C-x" or "add this to your .emacs"
Obviously. To set up a Lisp environment like mine, you need a Linux system like mine. That means libc 6.0 (glibc 2.0) or 6.1 (2.1). Suitable distributions include
If you're using Glibc 2.1 you need to be running linux 2.2.13 (or maybe 2.3.x) and to have configured the kernel for 1Gb max physical memory not 2Gb. This advice applies to cmucl 2.4.17; newer versions have since been released so I may be out of date.
The four basic options for ANSI-ish Common Lisp on Linux are
The choice between CMUCL and CLISP is mostly a size/speed tradeoff, except for some `oddball' uses - CLISP has very fast bignum support but a bit missing from its CLOS support. I don't attempt here to give arguments for one or the other, so go and decide which you'd rather use, and come back here if you choose CMUCL.
CMUCL is my choice for fast execution and near-ANSI compliance. It's written in Common Lisp in a rather unportable fashion, so if you want to compile it yourself you'll need a working binary package installed first anyway. It's currently at version 18b, but you're less likely to find that actual version anywhere than you are to find the Debian-packaged version which is basically the same. Debian `slink' (glibc 2.0-based) has 2.4.8, `potato' (2.1-based) has 2.4.20. Pick the one appropriate to your libc version, and if your system doesn't understand Debian packages, alien can translate them for you.
If you don't use the Debian package system, you will need alien to convert the CMUCL packages into a format you can use. It's available from http://kitenet.net/programs/alien/
Common Lisp has an ANSI standard (X3.226-1994, for the curious). It's distributed on laminar carbon media (a.k.a. paper) and it costs money.
The Common Lisp HyperSpec(TM) is not officially that standard, but except in cases of error or HTML deficiency, contains exactly the same text with the addition of extensive hypertext-based cross-references.
Get a copy (unless you have a lightning-fast internet link, that is, in which case you can use the one at Harlequin). It's invaluable.
I don't wish to start an editor flamewar, but if you don't use Emacs you won't get much out of this document.
I've used FSF Emacs 19 and 20 and XEmacs 20 for Lisp. All of them play nice, so choose one based on your political sympathies; everyone else has. The only relevant technical difference is that version 5.8 of ILISP (see below) doesn't seem to produce pulldown menus when used with XEmacs. I can't remember key bindings, so I use FSF Emacs.
Currently at version 5.10.1. This is a much better Lisp mode than the default Emacs one, and can be found at http://ilisp.cons.org/. Xemacs also bundles an older version, but the Xemacs people and the ILISP people are presently talking to each other about getting this resolved.
Install the packages. You almost certainly want at least cmucl-defsystem-2.4.x and cmucl-2.4.x, plus your choice of core image (-normal, -safe or -small); if you have a fast network link you may well find a use for the rest of it too. You don't need hemlock if you have Emacs (possibly also vice versa, but I haven't used it, I don't know).
Run /usr/sbin/cmuclconfig. To minimise time spent faffing around at the beginning of a Lisp session, it's possible to dump extra bits of code into a Lisp image so it starts up with them already resident (it's probably also a win in paging terms, but I don't know). This concept will be familiar to anyone who's watched Emacs build. Anyway, this script allows you to select the bits you want for your own custom `core' file. I'd recommend that you include defsystem, at least. Once you have a core file you will need to put it somewhere that Lisp can find it. I usually just copy it over the top of the original (I bet dpkg loves me); the alternative is to set the environment variable $CMUCLLIB to whatever directory you did put the core file in.
Unless your distribution is old or broken, it should have come with an acceptable Emacs setup. According to the FSSTND, Emacs will install architecture-independent data files into /usr/share/emacs/version/lisp. If you aren't already on familiar terms with Emacs at the elisp level, you might well end up being by the time you've done any Lisp - you'll probably find yourself living in the editor. If you elected not to install the *.el files the first time around, you might want to now. If space is a concern you can gzip them and have Emacs uncompress them whenever you visit one - FSF Emacs has jka-compr and XEmacs has toggle-auto-compression for this purpose
Assuming ILISP 5.9 or later, it installs fairly painlessly (read the INSTALLATION file) unless you have an XEmacs with an older version installed already. It might install painlessly even if so, but I haven't tried..
Once you've done that, it comes with installation instructions, which FSF Emacs users can follow and XEmacs users may have to improvise around (as it's already installed). My advice would be to paste all/most of the ilisp.emacs file into your .emacs, at least until you know what it does. The segment below is probably about the minimum necessary.
(autoload 'run-ilisp "ilisp" "Select a new inferior LISP." t)
(autoload 'cmulisp "ilisp" "Inferior CMU Common LISP." t)
(setq cmulisp-program "/usr/bin/lisp")
(setq cmulisp-local-source-directory "/usr/src/cmucl/cmucl/")
(set-default 'auto-mode-alist
(append '(("\\.lisp$" . lisp-mode)) auto-mode-alist))
(add-hook 'lisp-mode-hook
'(lambda ()
(require 'ilisp)
(setq indent-tabs-mode nil)))
;;; the cmucl-specific stuff in ilisp runs after the hook defined
;;; above, and overrides the binary extension to be "sparcf", which is not
;;; what we want. The hook below runs after that, so we can set it back
;;; again
(add-hook 'cmulisp-hook
(lambda ()
(setq ilisp-init-binary-extension "x86f")
(setq ilisp-init-binary-command "(progn \"x86f\")")
(setq ilisp-binary-extension "x86f")))
I don't intend to go into too much elisp detail here; this is supposed to be about Common Lisp not Emacs. But you may well find yourself writing or acquiring small elisp functions here and there, and needing a place to keep them. Your options are: (1) paste them all into your .emacs, or (2) create a directory somewhere, add it to your load-path then save them each in their own file (or according to some logical division, at any rate) and use load (for straightforward Lisp) or require (for things that include a provide form) to load them when needed.
If you use XEmacs 20, you can put these files into ~/.xemacs/lisp and they will be found automatically. FSF Emacs users will need to create their own directory and add it to theload path by hand : (setq load-path (append '("~/elisp") load-path)) in .emacs is an appropriate incantation.
Once everything is installed, you should be able to open Emacs, type M-x cmulisp RET to start the environment, then visit foo.lisp. Type
(+ 1 2)and press M-C-x to have Lisp evaluate it. The number '3' should be displayed in the echo area (that's the bottom line in the Emacs frame). If it produces more than one line of output -- and that example probably won't, unless it triggers a bout of garbage collection -- Emacs will split the frame into two windows and display the answer in the new one. Thus:
(format nil "One line~%Two lines")M-C-x that, and see it show the returned string. Now C-c 1 to make the output window go away.
While you're here, try the various `documentation' keys. Position the cursor on 'format' and press
The first thing you need to know is how to get out of the debugger. To demonstrate this, we need to get into the debugger:
* (/ 1 0) Arithmetic error DIVISION-BY-ZERO signalled. Operation was KERNEL::DIVISION, operands (1 0). Restarts: 0: [ABORT] Return to Top-Level. Debug (type H for help) (KERNEL::INTEGER-/-INTEGER 1 0) Source: Error finding source: Error in function DEBUG::GET-FILE-TOP-LEVEL-FORM: Source file no longer exists: target:code/numbers.lisp. 0]
0] is the debugger prompt. If you cause another error, you could get 0]] which means that you're in the debugger twice. You exit by typing q to abort everything and go back to the top level, or the number of one of the listed restarts - in this case there is only the one restart 0 which will abort back to the toplevel anyway.
(If you don't want a recursive debug session, you can use the debugger command flush (or related variable debug::*flush-debug-errors*) to toggle whether errors while in the debugger are ignored or cause another level of debugger. The default setting is T (ignore errors) when running from the command-line, NIL (recursive debug) when running with ILISP)
If you are repeatedly evaluating a form with errors from a Lisp source buffer, note that Emacs doesn't exit the debugger on its own, but just carries on (conceptually, it's entering the subsequent forms at the 0] prompt). This doesn't affect too much usually, but does mean that
Help! 11 nested errors. KERNEL:*MAXIMUM-ERROR-DEPTH* exceeded(which you can probably ignore - it means that CMUCL has dropped back to the * prompt on its own),
For tracing the flow of execution of a program, the standard CL macros TRACE and UNTRACE exist, and work like this
* (defun fac (n) (if (> n 1) (* n (fac (1- n))) 1))
FAC
* (trace fac)
(FAC)
* (fac 3)
0: (FAC 3)
1: (FAC 2)
2: (FAC 1)
2: FAC returned 1
1: FAC returned 2
0: FAC returned 6
6
*
Sometimes CMUCL refuses to trace compiled functions, complaining about :function-end breakpoints. I don't know why it does this. The simplest workaround is to redefine the fuction as interpreted - place the cursor on the function definition in the source buffer and M-C-x. Face it, you're now printing out two lines of text every time it gets called, the speed slowdown from running the interpreted version is not exactly going to be critical.
[TBD: is there a flashy GUD-style debugger for Emacs?]
There are many kinds of file system. Common Lisp provides facilities for referring to and manipulating files which minimize the program-visible differences between them.
As programmers we have the choice of representing filenames as strings (namestrings), or as pathnames. Pathnames are implementation-independent structured objects with six components: a host, a device, a directory, a name, a type, and a version. The mapping onto the underlying filesystem(s) is defined by the implementation: in CMUCL it is documented in section 2.13 of the User Manual
* (make-pathname :directory "/etc" :name "inetd" :type "conf") #p"//etc/inetd.conf"
Logical pathnames effectively provide a `hook' into the translation process for the programmer, so can be used to avoid hardcoding non-portable paths into programs. There is one caveat with logical pathnames, however - their behaviour is defined only for uppercase names (which on Unix will get translated to lowercase) so require some degree of work if you want to refer to files with uppercase in their names.
* (setf (logical-pathname-translations "FOO")
'(("**;*.*.*" "/full/path/to/foo/**/*.*.*")))
* (translate-logical-pathname (make-pathname :host "FOO" :directory "BAR" :name "BAZ" :type "LISP"))
#p"/full/path/to/foo/bar/baz.lisp"
The preferred way of maintaining these is to keep a file containing the list of translations for FOO in #p"library:FOO.translations" and use load-logical-pathname-defaults when you need it.
Namestrings are defined for logical paths as well as for physical ones - the standard read/print syntax for a namestring is #p followed by the namestring representation. #p is a reader macro that creates a pathname object
* (type-of #p"FOO:etc;inetd.conf") LOGICAL-PATHNAME
As a convenience to the user, PARSE-NAMESTRING (which is the thing that handles #p syntax) will uppercase its argument when dealing with a logical pathname. MAKE-PATHNAME might not, however. That is, it doesn't in CMUCL at present, but that will probably change.
defsystem is to Lisp as make is to C. Which is to say,
CMUCL comes with a portable defsystem implementation (if you installed it as recommended above, anyway), which originally came from Mark Kantrowitz's ``Portable Utilities for Common Lisp''.
You can check that your Lisp has defsystem loaded by evaluating *features* to see if it includes :MK-DEFSYSTEM - if not, see the previous section.
It actually turns out to be quite simple. Your project is a system, which has components. It's defined in a file called projectname.system using the defsystem macro. Once you have that, then you can do
* (compile-system 'my-system)and/or
* (load-system 'my-system)
Here's the my-system definition, for completeness
(defsystem my-system
:source-pathname "/home/dan/src/http-lisp/"
:components ((:file "socket")
(:file "daemon"))
:depends-on nil)
For the most part, defsystem requires relative pathnames. Be aware though that where it uses absolute names it doesn't get on too well with logical pathnames. My system definitions tend to contain things like
:source-pathname #.(translate-logical-pathname #p"src:rinaldo;")to get around this.
You have three options for IP networking:
(let* ((socket (ext:connect-to-inet-socket "localhost" 80))
(stream (system:make-fd-stream socket :input t :output t
:buffering :none)))
(format stream "HEAD / HTTP/1.0~%~%")
(do ((l (read-line stream nil) (read-line stream nil)))
((not l) nil)
(format t "~S~%" l)))
"HTTP/1.1 200 OK^M"
"Date: Tue, 12 Jan 1999 01:40:20 GMT^M"
"Server: Apache/1.2.5^M"
"Last-Modified: Sun, 10 Jan 1999 14:46:12 GMT^M"
"ETag: \"48808-2b51-3698bd34\"^M"
"Content-Length: 11089^M"
"Accept-Ranges: bytes^M"
"Connection: close^M"
"Content-Type: text/html^M"
"^M"
NIL
or more latterly The CMUCL User Guide is pretty good on this (mostly Chapter 8, Alien Objects but also see 2.5 Garbage Collection ). Fundamentally, you can call anything from Lisp that you can call in C. In practice, you might have to be careful for anything that involves threads or signals, but read the user guide and see how you get on. A fairly straightforward example is in pcre.lisp, an interface to Philip Hazel's Perl-Compatible Regular Expresion library.
A neat feature which is said to be better than some of the commercial CLs is the finalize function. This allows you to allocate foreign objects and then forget about them. When they become garbage the GC will call your function that knows how to free the object.
Multithreading in CMUCL is currently done with user-level threads, although there is work in progress to take advantage of kernel threads. The code is in src/code/multi-proc.lisp, with an interface "based roughly on the CLIM-SYS spec. and support needed for cl-http".
You might want to try running
* (setf mp::*idle-process* mp::*initial-process*) * (mp::start-lisp-connection-listener :port 3456 :password "foobar")then you can get a listener by telnetting to localhost port 3456 and supplying the password (including the quotes).
See http://ww.telent.net/cliki/CORBA for up-to-the-minute news.
There are a number of options here
* (load "library:subsystems/clm-library") * (load "library:subsystems/clx-library") * (/ 1 0)to see the Motif-based debugger...
The "One True Lisp" graphics interface is CLIM. People are working on free-clim, but in the meantime GARNET and CLIO/CLUE are probably the two options most worth considering.
For more timely information, and some other options, see http://ww.telent.net/cliki/Graphics%20Toolkit
These are (some of) the CL books I bought, borrowed, or somehow have access to. The opinions expressed about them are mine alone; please take them only for what they're worth (about 5p each, at last valuation)
Probably the most often recommended ANSI CL book for beginners. I haven't actually read it properly, but people whose opinions I trust - including two of the early guinea pigs for this document - like it, and on that basis I'd recommend it.
``Should be on the bookshelf of every CL programmer'' - my copy isn't usually, as it tends to be open on my desk. Excellent book.
Excellent. Look at the comments at the location linked above; they're all true. This may be an opportune moment to plug the rest of Peter Norvig's web site too.
Basically, this is the manual for the MOP, and if you want to do metathings you're going to struggle without it.Be aware however: it's not really intended as an introduction to CLOS - if you aren't familiar with CLOS already you might prefer something like "Object Oriented Programming in Common Lisp" by Sonya Keene. Disclaimer: I haven't tried it, but it's favourably thought of on comp.lang.lisp and elsewhere
I've not actually looked at it much since; I already can program ... If I sound lukewarm about this book, I probably am - compare with ANSI Common Lisp or Norvig, and decide for yourself.
I'm not suggesting that you need all of these; I have a real struggle going past a computer bookstore without buying something, but that doesn't mean you have to have. If you want a second opinion, Amazon have a reasonable summary of Lisp books. Note that despite anything they say, `Common Lisp: the Language' is not the official specification for ANSI CL: see the HyperSpec instead. Also, you may want to avoid buying it and use the copy on the Web as listed above.
All the mistakes in this document are mine. Credit for the bits which are right should also go to Hannu Rummukainen, Paul Foley, Paolo Amoroso, Peter VanEynde, Raymond Toy, Sam Steingold and others