Documentation: Fast dumping and loading of data

Handling persistence / marshaling and unmarshaling / pickling There are several ways of saving and restoring state in CMUCL:

  1. Ensure that your data structures know how to print themselves readably, by defining print-functions for your structures, and by specializing the PRINT-OBJECT generic function for your classes. You can then use the Common Lisp printer to save objects in ASCII form, and the Common Lisp reader to load them. This technique is likely to be very slow on large amounts of data, since the printer and reader are not very efficient. It will be particularly slow when serializing integers and floats, since the conversion from binary to ASCII format and back is very inefficient. However, this is the only technique that saves your data in a portable format.
  2. Ensure that all your objects are externalizable by specializing the generic function MAKE-LOAD-FORM for your classes and structures. You can then use COMPILE-FILE to create a FASL file for your data. Section 3.2.4.4 of CLtS specifies the constraints on portable externalization using MAKE-LOAD-FORM.
  3. Save a lisp image, using the CMUCL function EXT:SAVE-LISP. You can restart this image later, using the -core commandline option to CMUCL. This method is easy to implement and fast, but doesn't allow you to be selective about what you save; you serialize the entire lisp world.
  4. Use the following CMUCL-specific binary loading and dumping code, which is similar to the preceding technique, but doesn't require you to define MAKE-LOAD-FORM for everything.

Portability of dumps

There is a compromise between efficiency and portability of your dumps: the most efficient techniques are those that save data in a form that is closest to the in-memory representation used by CMUCL, but this in-memory representation changes as CMUCL evolves. The only technique that creates a portable dump (that you can read in another Common Lisp implementation on a different platform) is the first: saving to ASCII. All the other techniques save data in binary format, and so will not be portable to another hardware platform (consider for example big-endian/little-endian issues). The least portable technique is saving an image; since CMUCL images are tightly bound to their runtime, you won't be able to use a saved image with a future release of CMUCL.

The two other techniques are a little less tied to the exact CMUCL version number, but you shouldn't count on them producing data that is portable from one release to another. This is because the FASL file format can change occasionally, which will mean that an old FASL file will no longer be readable by the new CMUCL. The fastest way to dump binary data to disk (and retrieving it) under cmucl is through the following code, posted some time ago by Pierre Mai on cmucl-help. It uses the compiler's fasl (fast loading) facility:

(defvar *internal-value-passer*)
(defun bindump (object filename)
  (let ((file (c::open-fasl-file (pathname filename) nil t)))
    (unwind-protect
        (let ((c::*coalesce-constants* nil)
              (c::*dump-only-valid-structures* nil)
              (c::*cold-load-dump* t))
          (c::dump-fop 'lisp::fop-normal-load file)
          (c::dump-object `(setq *internal-value-passer* ',object) file)
          (c::dump-fop 'lisp::fop-eval-for-effect file))
      (c::close-fasl-file file nil)))
  t)
(defun binload (filename)
  (let ((*internal-value-passer* nil))
    (load filename :verbose nil :print nil)
    *internal-value-passer*)))

The object argument to bindump can be pretty much anything. I've tried with structs, lists of vectors and symbols etc. it just works.

By Pierre Mai and Eric Marsden.