dl-runtime: a patch for using primitives from shared libraries with OCaml 3.00

Copyright (C) 2000 Alain Frisch 

  distributed under the terms of any license that suits your needs and complies
  to the restrictions of the LGPL (for the runtime patch) and the QPL
  (for the compiler patch)

email: Alain.Frisch@ens.fr
web:   http://www.eleves.ens.fr:8080/home/frisch


--------------------------------------------------------------------------
Description

This package consists of a patch to the OCaml 3.00 system. The
modified bytecode interpreter (ocamlrun) can load external C functions
(aka primitives) from shared libraries, so that it is no longer needed
to build a custom runtime system.

NOTE
 * You will need the whole OCaml source tree to build the modified
 * version of the system. The patch assume you have then official
 * release 3.00.
 *
 * If you experience any problem to apply the patch, please don't 
 * hesitate to contact me.


In the standard OCaml system, each bytecode program has a primitive
table. The bytecode interpreter must be linked with exactly this
ordered set of primitives. Two solutions are natural:

- always use -custom (implied by the new autolink feature)

     ->  the linking is slow (it has to build a new runtime system)
     ->  the executable is large, no more portable

- build a huge runtime system with all the libraries you'll ever need
  and use -with-runtime when linking a bytecode program

     ->  when a new library is added, all the program have to be
         compiled again
     ->  the startup time of the huge runtime system may be important


With the patch, every runtime system may import primitives from
a shared library to fulfill the needs of the bytecode program. It
still has its own primitive table (statically linked). In addition,
the order in the primitive table does not matter any more: the runtime
system look up every needed primitive in the set of statically linked
primitives + shared libraries.

There are two ways to specify which library to import:

- give this information via the runtime system command line ("-l..."):
    ocamlrun -l/usr/local/lib/ocaml/libunix_so.so my_program

- give the Caml linker a set of shared library to consider at runtime
  ("-dl ...")
    ocaml -noautolink -o my_program unixL.cma my_program.cmo -dl /usr/local/lib/ocaml/libunix_so.so

Technically, this second solution adds to the bytecode file a section
listing all the "-dl ..." options.  The "-dl ..." options are not
recorded for library autolink (-ccopt and -cclink are). This would
have changed the structure of .cmo files.

The shared libraries are searched in the standard way (man dlopen).

The runtime system merges this list and the one from its command line.
It is possible to ignore the list of shared library specified in the
bytecode with the option "-i" :
LD_LIBRARY_PATH=/usr/local/lib/ocaml ocamlrun -i -llibunix_so.so my_program


The new runtime system should be able to read standard bytecode program.

--------------------------------------------------------------------------
Requirements

You will need the OCaml compiler 3.00 source tree
(http://caml.inria.fr/ocaml/index.html).

The patch will most probably fail to work under non-Unix like systems.

--------------------------------------------------------------------------
Building the new system:

- make apply_patch  : copy the ocaml source tree and apply the patch

  !!! You have to edit the Makefile and fill out the
      OCAML_SRC = ... line 

Then you should cd into ocaml-dl and build the ocaml system as usual
(configure; make world; make install; ...)

Shared versions of the unix, str and graphics C support libraries
are built: libunix_so.so,   libstr_so.so,  libgraphics_so.so.


In the demo subdirectory, you can find examples. test_libs demonstrate
the use of unix, str and graphics without building a new runtime system.


--------------------------------------------------------------------------
Problems:

- no specific support for native code: it should be possible
  to use shared libraries without any problem, using the standard
  unix dynamic loader

- no support for -output-obj

- interactions with Dynlink and with the toplevel not thoroughly tested

- performance penalty:
   * runtime startup: the system has to lookup every primitive; I think
     that the way it is implemented makes it fast enough;
   * at each primitive call: maybe an extra indirection; this could be
     avoided by "threading" the primitive addresses in the bytecode;
     I'm not sure it is worth the effort;
   * compile time: no penalty; actually, avoiding the creation of a
     specific runtime system speeds up a lot the compilation.

--------------------------------------------------------------------------
Acknowledgments:

* Nicolas George submitted a similar solution to the Caml Mailing List.

The main differences between the two patches are:
- I don't use a "primitive database";
- the shared libraries may be specified to the linker, and
  overidded at runtime;
- the compiler is patched to accept primitives which are not present
  in the runtime system.


* The OCaml team: it was a pleasure to work on their extremely well
and clearly written code.