Kawa is a set of Java classes useful for implementing dynamic languages, such as those in the Lisp family. Kawa is also an implementation of near-R5RS Scheme using these classes, and which compiles Scheme to the bytecode instructions of the Java Virtual Machine. This paper discusses the various issues involved in implementing Scheme using an abstract machine designed for a very different language. and how Kawa solves these problems. For information on getting and using Kawa, see the Kawa home page. This document is based on a paper presented November 1998. Kawa has seen many changes since then, but alas, this document has not kept pace. While Java is a decent programming language, the reason for the Java explosion is largely due to the Java Virtual Machine, and the many packages for it. The JVM allows programs to be distributed easily and efficiently in the form of portable bytecodes, which can run on a wide variety of architectures and in web browsers. These advantages are largely independent of the Java language, which is why there have been a number of efforts to run other languages on the JVM, even though the JVM is very clearly designed and optimized for Java. Kawa is both a toolkit for compiling other languages into Java bytecodes, and an implementation of the Scheme language implemented in Java. Kawa also incorporates Qexo, an implementation of the XQuery language, and JEmacs, a partial implementation of the Emacs Lisp extension language. The benefits of this hybrid Scheme/Java environment include: Provides a higher-level, more dynamic programming interface than Java, with support for scripting and interactive read-eval-print loops. Extends the Java benefits of "write once run anywhere" to Scheme programs, including portable bytecode distribution and web applets, though the use of standard Java bytecodes. Full integration between Java and Scheme: Scheme programs can call Java methods, and Java methods can call Scheme procedures. Scheme programs benefit from the extensive efforts in improved Java implementations, optimizations, environments, and tools. Scheme programs get access to the large library of standard Java classes. Starting in 1995 Cygnus (on behalf of the Free Software Foundation) developed Guile, an implementation of Scheme suitable as a general embedding and extension language. Guile was based on Aubrey Jaffar's SCM interpreter; the various Guile enhancements were initially done by Tom Lord. In 1995 we got a major contract to enhance Guile, and with our client we added more features, including threads (primarily done by Anthony Green), and internationalization. The contract called for a byte-code compiler for Guile, and it looked like doing a good job on this would be a major project. One option we considered was compiling Scheme into Java bytecodes and executing them by a Java engine. The disadvantage would be that such a Scheme system would not co-exist with Guile (on the other hand, we had run into various technical and non-technical problems with Guile that led us to conclude that Guile would after all not be strategic to Cygnus). The advantage of a Java solution was leveraging off the tools and development being done in the Java space, plus that Java was more likely to be strategic long-term. The customer agreed to using Java, and I started active development June 1996. As a base, I used the Kawa Scheme interpreter written by R. Alexander Milowski. He needed an object-oriented Scheme interpreter to implement DSSSL , a Scheme-like environment for expressing style, formatting, and other processing of SGML documents. DSSSL is an subset of pure Scheme with some extensions. Kawa 0.2 was a simple interpreter which was far from complete. It provided a useful starting point, but almost all of the original code has by now been re-written. Kawa 1.0 was released to our customer and the Net September 1996. Development has continued since then, at a less intense pace! The long-term goal is an object-oriented environment that harmoniously integrates Scheme, Java, XQuery, and other languages. There are three basic ways one might implement some programming language X using Java: One could write an interpreter for X in Java. First parse the source into an internal abstract syntax tree, and then evaluate it using a recursive eval function. The advantage of using Java rather than C or C++ is having garbage collection, classes, and the standard Java class library makes it easier. The obvious down-side of the interpreter solution is speed. If your interpreter for language X is written in Java, which is in turn interpreted by a Java VM, then you get double interpretation overhead. You could write a compiler to translate language X into Java source code. You need to define a mapping for language X constructs into equivalent Java constructs, and then write a program that writes out a parsed X program using corresponding Java constructs. A number of implementations take this approach, including NetRexx, and various extended Java dialects. This only gives you the single a single (Java VM) layer of interpretation. On the other hand, most of the efforts that people are making into improving Java performance will benefit your implementation, since you use standard Java bytecodes. The biggest problem with that approach is that it is an inherently batch process, and has poor responsiveness. Consider a read-eval-print-loop, that is the ability for a user to type in an expression, and have it be immediately read, evaluated, and the result printed. If evaluating an expression requires converting it to a Java program, writing it to a disk file, invoking a separate java compiler, and then loading the resulting class file into the running environment, then response time will be inherently poor. This hurts exploratory programing, that is the ability to define and update functions on the fly. A lesser disadvantage is that Java source code is not quite as expressive as Java bytecodes. While bytecodes are very close to Java source, there are some useful features not available in the Java language, such as goto. Debugging information is also an issue. Alternatively, you could directly generate Java bytecode. You can write out a .class file, which can be saved for later. You also have the option of writing to an internal byte array, which can be immediately loaded as a class using the java.lang.ClassLoader.defineClass method. In that case you can by-pass the file system entirely, yield a fast load-and-go solution, which enables a very responsive read-eval-print loop. This solution is the best of both worlds. The main problem is that more code needs to be written. Fortunately, by using Kawa, much of that work has already been done. I will discuss the compiler later, but first we will give an overview of the run-time environment of Kawa, and the classes used to implement Scheme values. Java has primitive types (such as 32-bit int) as well reference types. If a variable has a reference type, it means that it can contain references (essentially pointers) to objects of a class, or it can contain references to objects of classes that extend (inherit from) the named class. The inheritance graph is rooted (like Smalltalk and unlike C++); this means that all classes inherit from a distinguished class java.lang.Object (or just Object for short). Standard Scheme has a fixed set of types, with no way of creating new types. It has run-time typing, which means that types are not declared, and a variable can contain values of different types at different times. The most natural type of a Java variable that can contain any Scheme value is therefore Object, and all Scheme values must be implemented using some class that inherits from Object. The task then is to map each Scheme type into a Java class. Whether to use a standard Java class, or to write our own is a tradeoff. Using standard Java classes simplifies the passing of values between Scheme functions and existing Java methods. On the other hand, even when Java has suitable built-in classes, they usually lack functionality needed for Scheme, or are not organized in any kind of class hierarchy as in Smalltalk or Dylan. Since Java lacks standard classes corresponding to pairs, symbols, or procedures, we have to write some new classes, so we might as well write new classes whenever the existing classes lack functionality. The Scheme boolean type is one where we use a standard Java type, in this case Boolean (strictly speaking java.lang.Boolean). The Scheme constants #f and #t are mapped into static fields (i.e. constants) Boolean.FALSE and Boolean.TRUE. On the other hand, numbers and collections are reasonably organized into class hierarchies, which Java does not do well. So Kawa has its own classes for those. The next sections will give skeletal definitions of the classes used to to represent Scheme values. Kawa has a hierarchy of collection classes, which extend the Java2 Collections framework. (It is possible to build Kawa so it does not require the Java2 Collections classes, which are not available in JDK 1.1.x.) interface Sequence { ...; abstract public int size(); abstract public Object get(int i); } Classes that implement Sequence include lists, vectors, and strings. class FString implements Sequence { ...; char[] value; } Used to implement fixed-length mutable strings (array of Unicode character). This is used to represent Scheme strings. class FVector implements Sequence { ...; Object[] value; } Used to implement fixed-length mutable general one-dimensional array of Object. This is used to represent Scheme vectors. public class LList extends Sequencw { ...; protected LList () { } static public LList Empty = new LList (); } Used to represent Scheme (linked) lists. The empty list '() is the special static value List.Empty. Non-empty-lists are implemented using Pair objects. public class Pair extends LList { ...; public Object car; public Object cdr; } Used for Scheme pairs, i.e. all non-empty lists. public class PairWithPosition extends Pair { ...; } Like Pair, but includes the filename and linenumber in the file from which the pair was read. Future plans include more interesting collection classes, such a sequences implemented as a seekable disk file; lazily evaluated sequences; hash tables; APL-style multi-dimensional arrays; stretchy buffers. (Many of these ideas were implemented in my earlier experimental language Q -- see and . class Environment { ...; } An Environment is a mapping from symbols to bindings. It is used for the bindings of the user top-level. There can be multiple top-level Environments, and an Environment can be defined as an extension of an existing Environment. The latter feature is used to implement the various standard environment arguments that can be passed to eval, as adopted for the latest Scheme standard revision, R5RS. Nested environments were also implemented to support threads, and fluid bindings (even in the presence of threads). Environments will be integrated into a more general name-table interface, which will also include records and ECMAScript objects. Symbols represent identifiers, and do not need much functionality. Scheme needs to be able to convert them to and from Scheme strings, and they need to be interned (which means that there is a global table to ensure that there is a unique symbol for a given identifier). Symbols are immutable and have no accessible internal structure. Scheme symbols are reprented using interned Java Strings. Note that the Java String class implements immutable strings, and therefore cannot be used to implement Scheme strings. However, it makes sense to use it to implement symbols, since the way Scheme symbols are used is very similar to how Java Strings are used. The method intern in String provides an interned version of a String, which provides the characters-to-String mapping needed for Scheme strings. Scheme defines a numerical tower of numerical types: number, complex, real, rational, and integer. Java has primitive unboxed number types (such as int), just like C, and also has some wrapper classes that are basically boxed versions of the unboxed number types. Specifically, the standard Java number classes are not organized in any particularly useful hierarchy, except that they all inherit from Number. Kawa implements the full tower of Scheme number types, using its own set of sub-classes of the abstract class Quantity, a sub-class of Number we will discuss later. public class Complex extends Quantity { ...; public abstract RealNum re(); public abstract RealNum im(); } Complex is the class of abstract complex numbers. It has three subclasses: the abstract class RealNum of real numbers; the general class CComplex where the components are arbitrary RealNum fields; and the optimized DComplex where the components are represented by double fields. public class RealNum extends Complex { ...; public final RealNum re() { return this; } public final RealNum im() { return IntNum.zero(); } public abstract boolean isNegative(); } public class DFloNum extends RealNum { ...; double value; } Concrete class for double-precision (64-bit) floating-point real numbers. public class RatNum extends RealNum { ...; public abstract IntNum numerator(); public abstract IntNum denominator(); } RatNum, the abstract class for exact rational numbers, has two sub-classes: IntFraction and IntNum. public class IntFraction extends RatNum { ...; IntNum num; IntNum den; } The IntFraction class implements fractions in the obvious way. Exact real infinities are identified with the fractions 1/0 and -1/0. public class IntNum extends RatNum { ...; int ival; int[] words; } The IntNum concrete class implements infinite-precision integers. The value is stored in the first ival elements of words, in 2's complement form (with the low-order bits in word[0]). There are already many bignum packages, including one that Sun added for JDK 1.1. What are the advantages of this one? A complete set of operations, including gcd and lcm; logical, bit, and shift operations; power by repeated squaring; all of the division modes from Common Lisp (floor, ceiling, truncate, and round); and exact conversion to double. Consistency and integration with a complete numerical tower. Specifically, consistency and integration with fixnum (see below). Most bignum packages use a signed-magnitude representation, while Kawa uses 2's complement. This makes for easier integration with fixnums, and also makes it cheap to implement logical and bit-fiddling operations. Use of all 32 bits of each big-digit word, which is the expected space-efficient representation. More importantly, it is compatible with the mpn routines from the Gnu Multi-Precision library . The mpn routines are low-level algorithms that work on unsigned pre-allocated bignums; they have been transcribed into Java in the MPN class. If better efficiency is desired, it is straight-forward to replace the MPN methods with native ones that call the highly-optimized mpn functions. If the integer value fits within a signed 32-bit int, then it is stored in ival and words is null. This avoids the need for extra memory allocation for the words array, and also allows us to special-case the common case. As a further optimization, the integers in the range -100 to 1024 are pre-allocated. Many operations are overloaded to have different definitions depending on the argument types. The classic examples are the functions of arithmetic such as +, which needs to use different algorithms depending on the argument types. If there is a fixed and reasonably small set of number types (as is the case with standard Scheme), then we can just enumerate each possibility. However, the Kawa system is meant to be more extensible and support adding new number types. The solution is straight-forward in the case of a one-operand function such as negate, since we can use method overriding and virtual method calls to dynamically select the correct method. However, it is more difficult in the case of a binary method like +, since classic object-oriented languages (including Java) only support dynamic method selection using the type of the first argument (this). Common Lisp and some Scheme dialects support dynamic method selection using all the arguments, and in fact the problem of binary arithmetic operations is probably the most obvious example where multi-dispatch is useful. Since Java does not have multi-dispatch, we have to solve the problem in other ways. Smalltalk has the same problems, and solved it using coercive generality: Each number class has a generality number, and operands of lower generality are converted to the class with the higher generality. This is inefficient because of all the conversions and temporary objects (see ), and it is limited to what extent you can add new kinds of number types. In double dispatch the expression x-y is implemented as x.sub(y). Assuming the (run-time) class of x is Tx and that of y is Ty, this causes the sub method defined in Tx to be invoked, which just does y.subTx(x). That invokes the subTx method defined in Ty which can without further testing do the subtraction for types Tx and Ty. The problem with this approach is that it is difficult to add a new Tz class, since you have to also add subTz methods in all the existing number classes, not to mention addTz and all the other operations. In Kawa, x-y is also implemented by x.sub(y). The sub method of Tx checks if Ty is one of the types it knows how to handle. If so, it does the subtraction and returns the result itself. Otherwise, Tx.sub does y.subReversed(x). This invokes Ty.subReversed (or subReversed as defined in a super-class of Ty). Now Ty (or one of its super-classes) gets a chance to see if it knows how to subtract itself from a Tx object. The advantage of this scheme is flexibility. The knowledge of how to handle a binary operation for types Tx and Ty can be in either of Tx or Ty or either of their super-classes. This makes is easier to add new classes without having to modify existing ones. The DSSSL language is a dialect of Scheme used to process SGML documents. DSSSL has quantities in addition to real and integer numbers. Since DSSSL is used to format documents, it provides length values that are a multiple of a meter (e.g. 0.2m), as well as derived units like cm and pt (point). A DSSSL quantity is a product of a dimension-less number with an integral power of a length unit (the meter). A (pure) number is a quantity where the length power is zero. For Kawa, I wanted to merge the Scheme number types with the DSSSL number types, and also generalize the DSSSL quantities to support other dimensions (such as mass and time) and units (such as kg and seconds). Quantities are implemented by the abstract class Quantity. A quantity is a product of a Unit and a pure number. The number can be an arbitrary complex number. public class Quantity extends Number { ...; public Unit unit() { return Unit.Empty; } public abstract Complex number(); } public class CQuantity extends Quantity { ...; Complex num; Unit unt; public Complex number() { return num; } public Unit unit() { return unt; } } A CQuantity is a concrete class that implements general Quantities. But usually we don't need that much generality, and instead use DQuanity. public class DQuantity extends Quantity { ...; double factor; Unit unt; public final Unit unit() { return unt; } public final Complex number() { return new DFloNum(factor); } } public class Unit { ...; String name; // Optional. Dimensions dims; double factor; } A Unit is a product of a floating-point factor and one or more primitive units, combined into a Dimensions object. The Unit may have a name (such as kg), which is used for printing, and when parsing literals. public class BaseUnit extends Unit { ...; int index; } A BaseUnit is a primitive unit that is not defined in terms of any other Unit, for example the meter. Each BaseUnit has a different index, which is used for identification and comparison purposes. Two BaseUnits have the same index if and only if they are the same BaseUnit. public class Dimensions { BaseUnit[] bases; short[] powers; } A Dimensions object is a product and/or ratio of BaseUnits. You can think of it as a data structure that maps every BaseUnit to an integer power. The bases array is a list of the BaseUnits that have a non-zero power, in order of the index of the BaseUnit. The powers array gives the power (exponent) of the BaseUnit that has the same index in the bases array. Two Dimensions objects are equal if they have the same list of bases and powers. Dimensions objects are interned (using a global hash table) so that they are equal only if they are the same object. This makes it easy to implement addition and subtraction: public static DQuantity add (DQuantity x, DQuantity y) { if (x.unit().dims != y.unit().dims) throw new ArithmeticException ("units mis-match"); double r = y.unit().factor / x.unit().factor; double s = x.factor + r * y.factor; return new DQuantity (s, x.unit()); } The Unit of the result of an addition or subtraction is the Unit of the first operand. This makes it easy to convert units: (+ 0cm 2.5m) ==> 250cm Because Kawa represents quantities relative to user-specified units, instead of representing them relative to primitive base units, it can display quantities using the user's preferred units, rather than having to use prmitive units. However, this does make multiplication and division a problem The actual calculation (finding the right Dimensions and multiplying the constant factors) is straight-forward. The difficulty is that we have to generate a new compound Unit, and print it out in a reasonable fashion. Exactly how this should best be done is not obvious. Scheme has procedures that are first-class values. Java does not. However, we can simulate procedure values, by overriding of virtual methods. class Procedure { ...; public abstract Object applyN (Object[] args); public abstract Object apply0(); ...; public abstract Object apply4 (Object arg1, ..., Object arg4); } We represent Scheme procedures using sub-classes of the abstract class Procedure. To call (apply) a procedure with no arguments, you invoke its apply0 method; to invoke a procedure, passing it a single argument, you use its apply1 method; and so on using apply4 if you have 4 arguments. Alternatively, you can bundle up all the arguments into an array, and use the applyN method. If you have more than 4 arguments, you have to use applyN. Notice that all Procedure sub-classes have to implement all 6 methods, at least to the extent of throwing an exception if it is passed the wrong number of arguments. However, there are utility classes Procedure0 to Procedure4 and ProcedureN: class Procedure1 extends Procedure { public Object applyN(Object[] args) { if (args.length != 1) throw new WrongArguments(); return apply1(args[0]); } public Object apply0() { throw new WrongArguments();} public abstract Object apply1 (Object arg1); public Object apply2 (Object arg1, Object arg2) { throw new WrongArguments();} ...; } Primitive procedures can be written in Java as sub-classes of these helper classes. For example: public class force extends Procedure1 { public Object apply1 (Object arg1) throws Throwable { if (arg1 instanceof Promise) return ((Promise)arg1).force (); return arg1; } } The Kawa compiler used to compile each user-defined procedure into a separate class just like the force function above. Thus a one-argument function would be compiled to a class that extends Procedure1, and that the body of the function compiled to the body of an apply1 method. This has the problem that compiling a Scheme file generates a lot of classes. This is wasteful both at run-time and in terms of size of compiled files, since each class has some overhead (including its own constant pool). Early versions of Kawa were written before Sun added reflection to Java in JDK 1.1. Now, we can use reflection to call methods (and thus functions) not known at compile-time. However, invoking a function using reflection is a lot slower than normal method calls, so that is not a good solution. The next sections will discuss what Kawa does instead. Each Scheme function defined in a module is compiled to one or more Java methods. If it's a named top-level function, the name of the method will match the name of the Scheme function. (If the Scheme name is not a valid Java name, it has to be mangled.) An anonymous lambda expression, or a non-toplevel named function, gets a generated method name. A function with a fixed number of parameters is compiled to a method with the same number of parameters: (define (fun1 x y) (list x y)) Assuming the above is in mod1.scm, the generated bytecode is equivalent to this method: public static Object fun1 (Object x, Object y) { return MakeList(x, y); } The method can be an instance method or a static method, depending on compilation options. Here we'll assume it is static. To compile a call to a known function in the same module, it is easy for Kawa to generate static method invocation. In certain cases, Kawa can search for a method whose name matches the function, and invoke that method. If the function has parameter type specifiers, they get mapped to the corresponding Java argument and return types: (define (safe-cons x (y :: <list>)) :: <pair> (cons x y)) The above compiles to: public static gnu.lists.Pair safeCons (Object x, gnu.lists.LList y) { return Cons(x, y); } A function with optional parameters is compiled to a set of overloaded methods. Consider: (define (pr x #!optional (y (random))) (+ x y)) This gets compiled to two overloaded method, one for each length of the actual argument list. public static Object pr (Object x) { return pr(x, random()); public static Object pr (Object x, Object y) { return Plus(x, y)); If there is a rest-parameter list that get compiled to either an Object[] or LList parameter. The method name gets an extra $V to indicate that the function takes a variable number of parameters, and that extra parameters should be passed as a list or array to the last method parameter. For example this Scheme function: (define (rapply fun . args) (apply fun args)) This get compiled to: public static Object rapply$V(Object fun, LList args) { return apply(fun, args); } You can declare in Scheme that the rest parameter has type <Object[]>, in which case the method rest parameter is Object[]. Kawa compiles a Scheme module (a source file, or a standard-alone expression) to a Java class, usually one that extends ModuleBody. class ModuleBody { ... } Top-level forms (including top-level definitions) are treated as if they were nested inside a dummy procedure. For example assume a Scheme module mod1.scm: (define (f x) ...) (define (g x) ...) (do-some-stuff) This gets compiled to class mod1 extends ModuleBody implements Runnable { public Object f(Object x) { ... } public Object g(Object x) { ... } public Procedure f = ???; /* explained later */ public Procedure g = ???; public void run() { define_global("f", f); define_global("g", g); do_some_stuff(); } } When a file is loaded, an instance of the compiled class is created, and the run is invoked. This add the top-level definitions to the global environments, and runs any top-level expressions. Alternatively, using the --module-static command-line flag generates a static module: class mod1 extends ModuleBody { public static Object f(Object x) { ... } public static Object g(Object x) { ... } public static Procedure f = ???; public static Procedure g = ???; static { define_global ("f", f); define_global ("g", g); do_some_stuff(); } } In this case the top-level actons (including definitions) are performed during class initialization. A Java method represents the actions of a Scheme function, and calling a known Scheme function is easily implemented by invoking the method. However, Scheme has first-class functions, so we need to be able wrap the Java method as an Object that can be passed around, and called from code where the compiler that doesn't know which function will get called at run-time. One solution is to use Java reflection, but that has high overhead. Another solution (used in older versions of Kawa) is to compile each Scheme function to its own class that extends Procedure, with an applyN method that evaluates the function body; this incurs the overhead of a class for each function. The solution (as with all other problems in Computer Science [David Wheeler]) is to add an extra level of indirection. Every function in a module gets a unique integer selector. The utility ModuleMethod class is a Procedure that has a method selector code plus a reference to the ModuleBody context: class ModuleMethod extends Procedure { ModuleBody module; int selector; String name; public Object apply1(Object arg1) { return module.apply1(this, arg1); } public ModuleMethod(ModuleBody body, int selector, String name) { ... } } class ModuleBody { public Object apply1(Object x) { throw Error(); } } The compiler generates a switch statement to map selector indexes to actual methods. Thus the previous example generates (in static mode): class mod1 extends ModuleBody { public static f(Object x) { ... } public static g(Object x) { ... } public static Procedure f = new ModuleMethod(this, 1, "f"); public static Procedure g = new ModuleMethod(this, 2, "g"); static { define_global ("f", f); define_global ("g", g); do_some_stuff(); } public Object apply1(ModuleMethod proc, Object x) { switch (proc.selector) { case 1: return f(x); case 2: return g(x); default: return super.apply1(proc, this); } } } The symbol g resolves to the Procedure value of mod1.g. Invoking its apply1 method calls the method in ModuleMethod, which calls the 2-argument apply1 method in mod1. This switches on the selector 2, so we end up calling the g method. This is more expensive than calling g directly, but far less expensive than using reflection. When a language combines first-class nested functions with lexical scoping (as Scheme does), then we have the problem that an inner function can reference a variable from an other scope, even when that outer scope has exited. In this simple example we say that the inner function f2 captures the variable a from the outer function f1: (define (f1 a) (define (f2 b) (list a b)) (cons a f2)) The standard solution uses a closure to represent a function together with the environment of captured variables. Kawa does this by using the same ModuleBody mechanism used above for first-class functions. class foo extends ModuleBody { public Procedure f1 = new ModuleMethod(this, 1, "f1"); public Object f1 (Object a) { foo$frame1 frame = new foo$frame1(); frame.a = a; return cons(frame.a, frame.f2); } public Object apply1(ModuleMethod proc, Object x) { switch (proc.selector) { case 1: return f1(x); default: return super.apply1(proc, this); } } This is as dicussed earlier, except for the body of the f1 functions. It create a new inner module or frame. The parameter a is copied to a field in the frame, and any references to the parameter are replaced by a reference to the field. The inner module is implemented by this class: public class foo$frame1 extends ModuleBody { Object a; public Procedure f2 = new ModuleMethod(this, 1, "f2"); public Object f2 (Object b) { return list(this.a, b); } public Object apply1(ModuleMethod proc, Object x) { switch (proc.selector) { case 1: return f2(x); default: return super.apply1(proc, this); } } This mechanism again requires an extra indirection when an inner function is called. We also require a distinct frame class for each scope that has one or more variables captured by some inner scopes. At run-time, we need to allocate the frame instance plus ModuleMethod instances for each inner function (that does capture an outer variable), when we enter the scope for the frame. It should be possible to use general-purpose (sharable) frame classes for the common case that only a few variables are captured; however, I have to investigated that optimization. Aside: The original Java language definition did not support nested functions. However, it did have objects and classes, and it turns out the objects and first-class functions are similar in power, since a closure can be represented using an object and vice versa. The inner classes added to Java in JDK 1.1 are an object-oriented form of first-class functions. The Java compiler translates the nested classes into plain objects and non-nested classes, very much like Kawa represents nested Scheme functions. This section documents how Kawa implemented closures years ago. It is included for historical interest. Kawa used to implement a closure as a Procedure object with a static link field that points to the inherited environment. Older versions of Kawa represented the environment as an array. The most recent version uses the Procedure instance itself as the environment. Let us look at how this works, starting with a very simple example: (define (f1 a) (define (f2 b) (list a b)) (cons a f2)) This gets compiled to the bytecode equivalent of: class F1 extends Procedure1 { public Object apply1(Object a) { // body of f1 F2 heapFrame = new F2(); heapFrame.a = a; return Cons.apply2(heapFrame.a, heapFrame); } } class F2 extends Procedure1 { // F2 closureEnv = this; Object a; public Object apply1(Object b) { // body of f2 return List.apply2(this.a, b); } } Note that the instance of F2 that represents the f2 procedure contains both the code (the apply1 methods), and the captured instance variable a as a Java field. Note also that the parent function f1 must in general use the same field instance when accessing a, in case one or the other function assigned to a using a set!. Next, a slightly more complex problem: (define (f3 a) (define (f4 b) (cons a b)) (define (f5 c) (cons a c)) (cons a f5)) In this case all three functions refers to a. However, they must all agree on a single location, in case one of the functions does a set! on the variable. We pick f4 as the home of a (for the simple but arbitrary reason that the compiler sees it first). class F3 extends Procedure1 { public Object apply1(Object a) { // body of f3 F4 heapFrame = new F4(); heapFrame.a = a; return Cons.apply2(heapFrame.a, new F5(heapFrame)); } } class F4 extends Procedure1 { // F4 closureEnv = this; Object a; public Object apply1(Object b) { // body of f4 return Cons.apply2(this.a, b); } } class F5 extends Procedure1 { F4 closureEnv; public F5 (F4 closureEnv) { this.closureEnv = closureEnv; } public Object apply1(Object c) { // body of f5 return Cons.apply2(closureEnv.a, c); } } If a variables is captured through multiple levels of nested functions, the generated code need to follow a chain of static links, as shown by the following function. (define (f6 a) (define (f7 b) (define (f8 c) (define (f9 d) (list a b c d)) (list a b c f9)) (list a b f8)) (list a f7)) That gets compiled into bytecodes equivalent to the following. class F6 extends Procedure1 { public Object apply1(Object a) { // body of f6 F7 heapFrame = new F7(); heapFrame.a = a; return List.apply2(heapFrame.a, heapFrame); } } class F7 extends Procedure1 { Object a; public Object apply1(Object b) { // body of f7 F8 heapFrame = new F8(this); heapFrame.b = b; return List.apply3(this.a, heapFrame.b, heapFrame); } } class F8 extends Procedure1 { Object b; F7 staticLink; public F8(F7 staticLink) { this.staticLink = staticLink; } public Object apply1(Object c) { // body of f8 F9 heapFrame = new F9(this); heapFrame.c = c; return List.apply4(staticLink.a, this.b, heapFrame.c, heapFrame); } } class F9 extends Procedure1 { Object c; F8 staticLink; public F9(F8 staticLink) { this.staticLink = staticLink; } public Object apply1(Object d) { // body of f9 return List.apply4 (staticLink.staticLink.a, staticLink.b, this.c, d); } } Handling tail-recursion is another complication. The basic idea is to divide the procedure prologue into the actions before the loop head label, and those after. (Note that allocating a heapFrame has to be done after the head label.) Handling inlining also requires care. Kawa has various hooks for inlining procedures. This can allow substantial speedups, at the cost of some generality and strict standards-compliance, since it prevents re-assigning the inlined procedures. Most of these hooks work by having the compiler notice that a name in function call position is not lexically bound, yet it is declared in the (compile-time) global scope. The most powerful and low-level mechanism works by having the compiler note that the procedure implements the Inlineable interface. That means it implements the specical compile method, which the compiler calls at code generation time; it can generate whatever bytecode it wants. This is a way for special procedues to generate exotic bytecode instructions. This hook is only available for builtin procedures written in Java. Another mechanism uses the Java reflective facilities. If the compiler notices that the class of the procedure provides a static method with the right name (apply), and the right number of parameters, then it generates a direct call to that static method. This is not inlining per se, but it does by-pass the (currently significant) overhead of looking up the name in the global symbol-table, casting the value to a procedure, and then making a virtual function call. Also, because the procedure is replaced by a call to a statically known method, that call could actually be inlined by a Java bytecode optimizer. Another advantage of calling a known static method is that the parameter and return types can be more specific than plain Object, or even be unboxed primitive types. This can avoid many type conversions. The Kawa compiler generates a suitable apply method for all fixed-arity procedures that do not require a closure, so this optimization is applicable to a great many procedures. Finally, Kawa has preliminary support for true inlining, where a procedure that is only called in one place except for tail-calls, is inlined at the call-site. I plan to add an analysis pass to detect when this optimization is applicable. For now, there is a special case to handle the do special looping form, and these are now always implemented in the natural way (as inlined loops). The named let cannot always be implemented as an inlined loop, so implementing that equally efficiently will need the planned analysis phase. This is describing a work in progress. To handle general tail-calls, and to be able to select between overloaded methods, we split a function call into two separate operations: The match operation is given the actual parameters, and matches them against the formal parameters. If the right number and types of arguments were given, a non-negative integer return code specifies success; otherwise a negative return code specifies a mis-match. On success the arguments are saved in the argument save area of the CallContext. The apply operation performs the actual work (function body) of the called function. It gets the actual parameters from the CallContext, where match previously saved it. These are the stages of compilation: Reading The first compilation stage reads the input from a file, from a string, or from the interactive command interpreter. The result is one or more Scheme forms (S-expressions), usually lists. If reading commands interactively, only a single form is read; if reading from a file or string, all the forms are read until end-of-file or end-of-string; in either case, the result is treated as the body of a dummy function (i.e. a ModuleBody). Semantic analysis The source form is rewritten into an Expression object, specifically a ModuleExp. This stage handles macro expansion and lexical name binding. It figures out which local variables are captured by an inner function, and hence need to be heap-allocated. (Other variables are stack-allocated in the Java local variable frame.) Various optimizations are done, including selection of closure representations. Code generation The resulting ModuleExp is compiled into one or more byte-coded classes. This is done by invoking the virtual compile method recursively on the Expressions, which generates instructions (using the bytecode package) to evaluate the expression and leave the result on the Java operand stack. At the end we ask the bytecode package to write out the resulting classes and methods. They can be written to a file (for future use), or into byte arrays in memory. Loading The compiled bytecodes are loaded into the Kawa run-time. In the case of code that is compiled and then immediately executed, the compiled code can be immediately turned into Java classes using the Java ClassLoader feature. (That is how the read-eval-print loop works.) An instance of the compiled sub-class of ModuleBody is created and run, which normally produces various side-effects. The abstract Expression class represents partially processed expressions. These are in principle independent of the source language, though there are still some Scheme assumptions wired in. class Expression { ...; public abstract Object eval (Environment e); public abstract void compile (Compilation comp, Target targ); } The eval method evaluates the Expression in the given Environment. The compile method is called when we are compiling the body of a procedure. It is responsible for generating bytecodes that evaluate the expression, and leave the result in a result specified by the Target parameter. This is usually the Java evaluation stack, but we will go into more detail later. class QuoteExp extends Expression { ...; Object value; public QuoteExp(Object val) { value = val; } public Object eval(Environment env) { return value; } public void compile (Compilation comp, Target target) { comp.compileConstant (value, target); } } A QuoteExp represents a literal (self-evaluating form), or a quoted form. class ReferenceExp extends Expression { ...; Symbol symbol; Declaration binding; } A ReferenceExp is a reference to a named variable. The symbol is the source form identifier. If binding is non-null, it is the lexical binding of the identifier. class ApplyExp extends Expression { ...; Expression func; Expression[] args; } An ApplyExp is an application of a procedure func to an argument list args. class ScopeExp extends Expression { ...; ScopeExp outer; // Surrounding scope. public Declaration add_decl(Symbol name) { ...Create new local variable... } } A ScopeExp is a abstract class that represents a lexical scoping construct. Concrete sub-classes are LetExp (used for a let binding form) and LambdaExp. class LambdaExp extends ScopeExp { ...; Symbol name; // Optional. Expression body; int min_args; int max_args; } The Scheme primitive syntax lambda is translated into a LambdaExp, which represents anonymous procedures. Each LambdaExp is compiled into a different bytecoded class. Invoking eval causes the LambdaExp to be compiled into a class, the class to be loaded, an instance of the class to be created, and the result coerced to a Procedure. Other sub-classes of Expression are IfExp (used for conditional expressions); BeginExp (used for compound expressions); SetExp (used for assignments); and ErrorExp (used where a syntax error was found); The translation phase takes a top-level form (or body), and generates a ModuleExp, which is a top-level expression. This is done using a Translator, which keeps track of lexical bindings and other translation state. class Translator { ...; public Expression rewrite(Object exp) { ... } public Expression syntaxError (String message) { ... } } The rewrite method converts a Scheme source form to an Expression. The syntaxError method is called when a syntax error is seen. It prints out the current source filename and line number with the given message. In addition to handling special forms (such as lambda), the rewrite phase also handles name resolution of identifiers to their lexical declarations. This is complicated because a variable reference may appear before the declaration that defines it, for example in a letrec. The solution is to use two phases: The first scan sub-phase looks for declarations, macro-expanding outermost macro applications to determine if the result is a declaration. The second rewrite sub-phase takes the result of the scan phase, using the declarations that scan produced, and rewrites any deferred forms. The phases are actually interleaved, because a Scheme body may contain internal definitions. Thus to rewrite a body we first have to scan it for definitions, before we rewrite the result. We won't go into further details about the scan phase. class Syntax { ...; public abstract Expression rewriteForm (Object obj, Translator tr); } The rewrite method in Translator checks for syntactic keywords and macros. If the car of a call is a Syntax or if it is a Symbol that is bound to a Syntax, then the rewriteForm method of the Syntax is called. As an example, this trivial class implements quote: class quote extends Syntax { ...; public Expression rewriteForm (Object form, Translator tr) { // Error-checking is left out. return new QuoteExp((((Pair) form).cdr).car); } } (The actual implementation of quote is more complex, since it has to interpolate forms bound to pattern variables.) A Macro is a Syntax where rewriteForm calls a transformer function: class Macro extends Syntax { ...; Procedure transformer; Expression rewriteForm (Object form, Translator tr) { return tr.rewrite(expand(form, tr)); } Object expand (Object form, Translator tr) { // Much simplified. return transformer.apply1(form); } } When Kawa sees a define-syntax, it creates a Macro that contains the transformer resulting from the transformer expression. The latter may be a lambda expression; that is normally the case when using syntax-case. Alternatively, the transformer expression may be a syntax-rules form. That gets compiled to a SyntaxRules object: This contains an encoded representation of the patterns and templates in the syntax-rules. class SyntaxRules extends Procedure1 { ...; SyntaxRule[] rules; public Object apply1 (Object arg) { Translator tr = getCurrentTranslator(); Object[] v = new Object[maxVars]; for (int i = 0; i < rules.length;) { SyntaxRule r = rules[i++]; if (r.match (obj, v)) return r.execute(v, tr); } return tr.syntaxError ("no matching syntax-rule"); } } A problem when writing a macro is that identifiers you use internally in the macro might conflict with identifiers that appear at the macro call site. User-supplied code might accidentally reference a temporary variable in the macro expansion or vice versa. A macro system is hygienic when it automatically separates the names in the macro (and the macro definition context) from those in the macro expansion context. Conceptually, this is done by magically renaming certain names. Macros defined using the standard syntax-rules are by default hygienic, as are by default those defined using the syntax-case system provided by Kawa and some other Scheme implementations. It is possible to override hygiene when using syntax-case, though we won't go into that. The following discussion will focus on syntax-case, since we can view syntax-rules as a short-hand syntax for combinating a syntax-case with a syntax template for each alternative. The key to understanding syntax-case is that it uses syntax objects, which combine the actual Scheme source form (a list, symbol, or literal) with its syntactic context. A syntax object is implemented in two ways. An implicit syntax object is just the Scheme source form. In that case the syntactic context is implicitly the current lexical context. An explicit syntax object is implemented using SyntaxForm: class SyntaxForm { Object form; TemplateScope scope; } The form is the orginal form from the Scheme reader, while the scope field identifies the lexical context. A SyntaxForm object is created when a syntax template (i.e. a syntax expression) is evaluated; the scope field is the scope of the syntax form within the macro definition, which may be unrelated to the lexical scope of the expansion context. If a syntax form doesn't contain any pattern variables, then the result is a single SyntaxForm object that wraps the syntax argument expression. However, any pattern variables are returned as-is. This may require some destructuring of the template, rather like quasi-quotation. Consider this example: (syntax-case form () ((_ e1 e2) (syntax (list a e1 e2 b c)))) The syntax form is compiled as if it were (using the cons* function from SRFI-1): (cons* (syntax list) (syntax a) e1 e1 (syntax b c)) Note that in Scheme code you'd have to write e1 and e2 as (syntax e1) and (syntax e2), since you're only allowed to reference pattern variables in syntax templates. However, syntax on a pattern variable is implemented as the identity function. When the Kawa compiler sees a syntax, it translates the template into a SyntaxTemplate object. This object contains a compact encoding of the template, plus a reference to the current lexical scope. Evaluating the syntax form translates into invoking the execute method of the SyntaxTemplate. The execute creates a fresh TemplateScope instance, when is used for the scope field of any generated SyntaxForm objects. The pattern in a syntax-case or syntax-rules is matched against a syntax object, which can be an explicit SyntaxForm object, an implicit syntax object (a plain Scheme value), or a mixture of the two. The destructuring specified by the pattern may need to decend into a SyntaxForm. Consider for example this syntax object: `(,(syntax a) ,(syntax (b c d)) f) being matched against this pattern: (x (y . z) . r) Then we get these bindings: x: (syntax a) y: (syntax b) z: (syntax (c d)) r: (f) The scope fields of y and z are taken from the original (syntax (b c d)). Note that the last element of the original list is a plain Scheme form - i.e. an implicit syntax form, so that's what the pattern variable is bound to, and that's what get inserted into any template that references r. The Kawa compiler compiles a syntax-case into a SyntaxPattern object. A single rule in a syntax-rules is compiled to a SyntaxRule, which is a combination of a SyntaxPattern with a SyntaxTemplate. The whole syntax-rules is compiled to a SyntaxRules, which is basically a collection of SyntaxRule objects that implements Procedure1 so it can be used as the transformer of a define-syntax macro. A TemplateScope is created when a syntax template is expanded. class TemplateScope extends LetExp { } Initially, a TemplateScope doesn't contain any declarations, but it may gain implicit alias declarations as explained below. In addition, a TemplateScope inherits declarations from the parent scope, which is set to the context of the macro definition. Kawa's rewrite stage translates an input syntax object to an Expression. The initial syntax object is an implicit syntax object, consisting of a form from the Scheme reader. If we're processing the output from a macro, then the syntax object may contain SyntaxForm objects. To rewrite a SyntaxForm we temporarily change the current scope to the TemplateScope. We then rewrite the form value of the SyntaxForm, using that TemplateScope. This means that identifiers will be looked up in the TemplateScope. This normally doesn't have direct declarations, so effectively we're searching the TemplateScope's parent, which is the syntactic context of the macro definition, as desired. The current scope is restored to the orginal scope when we're done rewriting the SyntaxForm. (Setting and restoring the current scope isn't implemented very efficiently, though it probably doesn't matter in normal code.) Things get more complicated if the syntax template creates new definitions, but using names coming from the expansion site rather than the template. Consider this example: (define-syntax mac (syntax-rules () ((mac v1 init exp) (let ((v1 init)) (let ((i 2)) (list exp i)))))) (define j 10) (define k (mac i 1 (+ i j))) ;; => (11 2) Here we have a pair of nested let-expressions, both of which end up declaring variables named i. The list-expression references the inner i literally. After macro-expansion the exp also references i, but its syntactic scope comes from the macro application, so it should match the i declaration whose name also comes from the macro application - which is that of the outer let. This is how it is done: When rewriting the outer let we create a normal LetExp. Rewriting the inner let also creates a LetExp, nested in the normal way. However, the name of this inner i declaration comes from the macro template, so we need to hide it from any code not from the same template. This is where we use the TemplateScope: We place the declaration for the inner i in the TemplateScope, rather than in the inner LetExp where it actually belongs. When we rewrite the first operand of the list, it uses the normal scoping, so it doesn't see the inner i, since that's in the TemplateScope. On the other hand the second operand of list is a SyntaxForm whose scope is the TemplateScope, so we change the current scope to that TemplateScope. This hides the outer i and makes the inner i visible. When we're done rewriting the inner let, we move the inner i declaration out of the TemplateScope (which gets garbage collected) and into the inner LetExp where it stays for the rest of compilation. (The implementation doesn't actually move the declaration. Instead, it initially creates the declaration with a null name in the LetExp and an alias that references to it from the TemplateScope. When we're done, it sets the name from the alias.) Many people think of Scheme, Lisp, and ECMAScript as interpreted languages, though many of these languages have compilers. What these languages do have is eval - that is a command that at run-time takes a source program, and evaluates it. They may also have an interactive read-eval-print interface. For such uses a traditional interpreter is easiest and most responsive. Therefore, high-end Lisp systems traditionally provide both a compiler and an interpreter. Such duplication is expensive, in terms of size, development effort, and testing. If one has load-and-go capabilities, that is the abilility to efficiently load a compiled program into a running application, then one can simply implement eval as a compile followed by a load. When we compile to Java bytecodes, we create one or more files in the .class format. The standard Java class java.lang.ClassLoader has a method defineClass that takes a byte array laid out in the format of a .class, and from it dynamically creates a new class in the existing Java run-time. (This facility is used for applets downloaded accross the Network.) Kawa uses this scheme to implement eval, and it works well. Because ClassLoader.defineClass takes an array, rather than a file, we can compile and load entirely inside the Kawa run-time, without having to go via the filesystem for temporary files, as a traditional compiler batch does. The result is near-instant response. There is a tradeoff, though. Doing a compile+load is a very heavy-duty operation, compared to a simply interpreting an expression. It creates a lot of temporary objects. Worse, it also creates some temporary classes, and some Java implementations do not garbage collect unused classes. Kawa uses a compromise strategy. If the Expression is simple, it is interpreted directly, using the Expression.eval. Otherwise, it is compiled. Simple expressions include literals, (global) variable access, assignment, and function application. Implementing eval in those cases is trivial. Expressions that define new local bindings (such as lambda expressions and let forms) do not implement eval. If the user types in such an expression, it is wrapped inside a dummy function, compiled to bytecodes, and immediately executed. This is to avoid dealing with lexical binding in the evaluator. A ModuleExp represents a top-level form: class ModuleExp extends LambdaExp { ...; public Object eval_module (Environment env) { if (body_is_simple) // Optimization return body.eval (env); Object v = eval (env); return ((ModuleBody) v).run (env); } } ModuleExp is a sub-class of LambdaExp, since it is actually a dummy function created by wrapping the top-level forms in an implicit lambda. The eval_module method evaluates the top-level forms. If the body is not simple, it invokes the eval in LambdaExp (which invokes the compiler). The result of eval is a ModuleBody, which we can run. A Compilation object manages the classes, methods, and temporary state generated as a result of compiling a single top-level ModuleExp. class Compilation { ...; ClassType[] classes; boolean immediate; public ClassType addClass (LambdaExp lexp, String name) { ... } public ClassType(ModuleExp exp, ...) { ...; addClass (exp, ...); } } Each Compilation may create one or more ClassType objects, each of which generates the bytecodes for one class. Each ClassType is generated from a LambdaExp, including the top ModuleExp. The boolean immediate is true if we are compiling for immediate loading, and is false if the target is one or more .class files. The addClass method does all the work to compile a given LambdaExp. It creates a ClassType, adds it to Compilation's classes array, and generates Method objects for the constructor and the main applyX method. Once the applyX Method has been created, addClass emits some bytecodes to set up the incoming parameters, and then invokes the virtual compile method on the body of the LambdaExp, which generates the code that does the actual work of the procedure. The Compilation constructor gets a ModuleExp, which it passes to addClass. The compile method of LambdaExp (which gets called for all lambdas except the dummy top-level) also calls addClass to generate the class corresponding to the lambda, and then it emits instructions to create a new instance of the generated Procedure class, and pushes it on the Java stack. Most operations in the Java VM leave their result on the VM stack, where they are available for succeeding operations. The obvious and general way to compile an expression is therefore to generate bytecode instructions that leave the result (in the form of a Object reference) on the stack. This handles most cases quite well, but we can do better. We specify a Target parameter when invoking the compile method; the Target specifies where to leave the result. public abstract class Target { ...; public abstract void compileFromStack (Compilation comp, Type stackType); public static final Target Ignore = new IgnoreTarget(); } An Expression's compile method does not have to handle all the kinds of Targets, as long as it can generate code to leave the result on the VM stack, and then invoke compileFromStack, which is responsible for moving the result to the actual target. The simplest Target is an IgnoreTarget. It is used when the result of an expression will be ignored, but we still need to evaluate it for possible side-effects. The implementation of IgnoreTarget.compileFromStack just emits an instruction to pop a value from the VM stack. Expressions that have no side-effects can check if the target is an IgnoreTarget, and then immediately return. This saves a useless push-pop pair. The usual Target is an StackTarget. This specifies that an expression should leave the result on the VM stack. Normally, the type of the result is Object, but a StackTarget can specify some other expected type, when that can be determined. The implementation of StackTarget.compileFromStack is also trivial: If the type of the result on the stack is a sub-type of the expected target type, nothing needs to be done; otherwise, it generates code to do the type conversion. Things get more interesting when we come to ConditionalTarget. public class ConditionalTarget extends Target { ...; public Label ifTrue, ifFalse; } A ConditionalTarget is used when compiling the test expression in a conditional. The expression is evaluated as a boolean value; if the result is true, control transfers to ifTrue; otherwise control transfers to ifFalse. Using ConditionalTarget makes it straight-forward to generate optimal code for nested conditionals, including and and or macros, and (when inlining) functions such as not and eq?. The ClassType and Method classes are in a separate gnu.bytecode package, which is an intermediate-level interface to code generation and Java .class files. It is essentially independent of Scheme or the rest of Kawa. class ClassType extends Type { ...; CpoolEntry[] constant_pool; Method methods; // List of methods. Field fields; // List of fields. public Field addField (String name, Type type, int flags) { ...Create new field... } public method addMethod(String name,...) { ...Create new method... } public void writeToStream (OutputStream stream) { ... } public void writeToFile(String filename) { ... } public byte[] writeToArray() { ... } } The ClassType class is the main class of the bytecode package. It manages a list Fields, a list of Methods, and the constant pool. There are utility methods for adding new fields, methods, and constant pool entries. When the ClassType has been fully built, the writeToFile method can be used to write out the contents into a file. The result has the format of a .class file . Alternatively, the class can be written to an internal byte array (that has the same layout as a .class file) using the writeToArray method. The resulting byte array may be used by a ClassLoader to define a new class for immediate execution. Both of the these methods are implemented on top of the more general writeToStream. Each method is represented by a Method object. class Method implements AttrContainer { ...; Type[] arg_types; Type return_type; Attribute attributes; } An AttrContainer is an object that contains zero or more Attributes. The Java .class file format is quite extensible. Much of the information is stored in named attributes. There are standard attributes, but an application can also define new ones (that are supposed to be ignored by applications that do not understand them). Each class file may have a set of top-level attributes. In addition, each field and method may have attributes. Some standard attributes may have nested sub-attributes. public abstract class Attribute { ...; AttrContainer container; String name; } An Attribute's container specifies who owns the attribute. The attribute also has a name, plus methods to gets its size, write it out, etc. The most interesting (and large) standard Attribute occurs in a method and has the name "Code". It contains the actual bytecode instructions of a non-native non-abstract method, and we represent it using CodeAttr. class CodeAttr extends Attribute { ...; Variable addLocal(Type t, String name) { ... } public void emitLoad(Variable var) { ... } public void emitPushInt(int i) { ... } public void putLineNumber(int lineno) { ... } } As an example of the level of functionality, emitPushInt compiles code to push an integer i on stack. It selects the right instruction, and if i is too big for one of the instructions that take an inline value, it will create a constant pool entry for i, and push that. The method addLocal creates a new local variable (and makes sure debugging information is emitted for it), while emitLoad pushes the value of the variable on the stack. Kawa calls putLineNumber to indicate that the current location corresponds to a given line number. These are emitted in the .class file, and most Java interpreters will use them when printing a stack trace. We use gnu.bytecode mainly for generating .class files, but it also has classes to read .class files, and also classes to print a ClassType in readable format. The combination makes for a decent Java dis-assembler. There are other toolkits for creating or analyzing .class files, but gnu.bytecode was written to provide a lot of support for code generation while having little overhead. For example, some assemblers represent each instruction using an Instruction instance, whereas CodeAttr just stores all the instruction in a byte array. Using a linked list of Instructions may be more object-oriented, and it does make it easier to do peep-hole optimizations, but the time and space overhead compared to using an array of bytes is huge. (If you do need to do peephole optimizations, then it makes sense to use a doubly-linked list of Instructions, but to use that in conjunction with CodeAttr. You will in any case want a byte-array representation for input and output.) A Scheme quoted form or self-evaluating form expands to a QuoteExp. Compiling a QuoteExp would seem a trivial exercise, but it is not. There is no way to embed (say) a list literal in Java code. Instead we create a static field in the top-level class for a each (different) QuoteExp in the body we are compiling. The code compiled for a QuoteExp then just needs to load the value from the corresponding static field. The tricky part is making sure that the static field gets initialized when the top-level class is loaded to the value of the literal. This is easy when compiling for immediate execution: after the compiled class has been loaded and initialized, we use reflection to set the field to the literal value. When compiling to a class file, things are harder. The basic idea is that for: (define (foo) '(3 . 4)) we compile the equivalent of: class foo extends Procedure0 { Object static lit1 = Pair.make(IntNum.make(3), IntNum.make(4)); public Object apply0() { return lit1; } } When the compiled class foo is loaded, we do: Class fooCl = Class.forName("foo"); Procedure fooPr = (Procedure) fooCl.newInstance (); // Using foo: Object result = fooPr.apply0 (); How does the Kawa compiler generate the appropriate Pair.make expression as shown above? In earlier versions of Kawa, a class whose instances could be literals implemented the Compilable interface. Then the compiler just called the methods in Compilable, and these would generate the code needed to re-create the literal. This had the advantage that the compiler was not limited to a few classes of literals it knew about. One problem is that it caused cross-package dependencies, since any class that could be a literal has to implement gnu.expr.Compilable. Another problem was that writing the Compilable methods required knowing how gnu.bytecode works. The key insight is that saving compile-time literals and then restoring them at run-time is a kind of object persistence, similar to that offered by Java serialization. One option considered was to use standard serialization, but that required some way to store the serialized data in a class file. The article Long-Term Persistence for JavaBeans suggested an alternative. I realized that using Externalizable classes provided the more abstract serialization we needed. Let us start with a summary: An object that can be a literal must implement the java.io.Externalizable interface. When the compiler need to generate a reference to a literal value, it calls the object's writeExternal method. The writeExternal method expects an argument that implements the java.io.ObjectOutput interface. The actual argument is a LitTable object owned by the compiler. A writeExternal method can call the standard ObjectOutput methods, such as writeObject, writeInt, and so on. The LitTable remembers the values that were passed to it in these writeObject, writeInt, etc calls, and creates an argument list from those values. When the writeExternal returns, the LitTable will take the argument list, and look for a matching constructor in the literal's class, using reflection. It will also look for methods named make or set. If found, the compiler will generate a call to the matching method. Simple values in the argument list will be pushed on the JVM stack directly; object references will use values generated using previous calls to writeExternal. Here are more details on how the compiler selects which method to invoke to re-create an object. If the compiler detects a recursive writeObject call with the object as an argument while writeExternal is called on the object, then the object cyclically references itself. In that case, the compiler must take care to construct the object so that the cycle gets reproduced at run-time. This is done by first generating a call to the default constructor, and saving the result in a static field. The argument list is evaluated, with recursive references using the saved static field. Then we call a method to properly initialize the object using the evaluated argument list. The compiler looks for a method named set with a parameter list matching the argument list. If there is no such method, the compiler gives up. (In the future, we could use JavaBeans introspection, or look for a Java2 ObjectStreamField to determine how to set the properties.) If an object does not have a cycle, the compiler first looks for a matching static method named make. If it does not find one, it looks a matching constructor. In the latter case, the compiler must also check to see if there is a zero-argument readResolve method, which Java 2 serialization uses to replace an object by a canonical object. (Kawa does not require Java 2, but LitTable does respect this Java 2 extension to serialization.) As a last resort, the compiler looks for a default constructor followed by set, as in the cycle case. The writeExternal method of some classes may generate a variable number of values. Therefore, if a constructor or method takes an array parameter, then LitTable will consider it as matching the argument list if the latter contains an int (written using writeInt) followed by that number of arguments of the component type of the array. (One could argue that this feature is an unneeded kludge, since there is the alternative that writeExternal could just call writeObject with a single array argument, instead of calling writeObject a variable number of times. However, that would expose the internal array into the externalization protocol, which is wrong.) Advantages of this implementation include: No explicit dependency on any class in the the gnu.expr package. All a class needs to be used in literals is to implement Externalizable, and provide a matching method for creating object instances. No need to write special code to handle literals. Literals are re-created using efficient code at class initialization time. Automatically handles cycles and duplicate references. (Standard Scheme does not support self-referential constants, but for example Common Lisp does. See section 25.1.4 Similarity of Constants in .) Only requires standard JDK 1.1 features. Java supports reflection, that is the ability to determine and examine the class of an object, and use the class at run-time to extract fields and call methods using names specified at run-time. Kawa, like some other Scheme implementations, also supports reflection. It seems plausible to represent a type using a java.lang.Class object, since that is what the Java reflective facility does. Unfortunately, there are at least three reasons why Kawa needs a different representation: We may need to refer to classes that do not exist yet, because we are in the process of compiling them. We want to be able to specify different high-level types that are represented using the same Java type. For example, we might want to have integer sub-ranges and enumerations (represented using int), or different kinds of function types. We want to associate different conversion (coercion) rules for different types that are represented using the same class. Kawa represents types using instances of Type: public abstract class Type { ...; String signature; // encoded type name int size; public final String getName() { ... } public boolean isInstance(Object obj) { ... } public void emitIsInstance(CodeAttr c) { ... } } The method isInstance tests if an object is a member of this type, while emitIsInstance is called by the compiler to emit a run-time test. Note that the earlier mentioned ClassType extends Type. Kawa follows the convention (used in RScheme and other Scheme dialects) that identifiers of the form <typename> are used to name types. For example Scheme vectors are members of the type <vector>. This is only a convention and these names are regular identifiers, expect for one little feature: If such an identifier is used, and it is not bound, and typename has the form of a Java type, then a corresponding Type is returned. For example <java.lang.String[]> evaluates to a Type whose values are references to Java arrays whose elements are references to Java strings. As a simple example of using type values, here is the definition of the standard Scheme predicate vector?, which returns true iff the argument is a Scheme vector: (define (vector? x) (instance? x <vector>) The primitive Kawa function instance? implements the Java instanceof operation, using Type's isInstance method. (In compiled code, if the second operand is known at compile-time, then the compiler uses Type's emitIsInstance method to generate better code.) The traditional benefits of adding types to a dynamic language include better code generation, better error checking, and machine-checkable interface documentation. These benefits require either type inference or (optional) type declarations. Kawa does allow the types of parameters to be declared, and does some very simple local type inference. In some cases this lets unboxed values (such as raw Java int) be passed from one function to another without having to allocate an object. While Kawa has a framework for working with types, it does need a more systematic approach. Kawa includes the record extension which allows a new record type to be specified and created at run-time. It is implemented by creating a new ClassType with the specified fields, and loading the class using ClassLoader.defineClass. The record facility consists of a number of functions executed at run-time. Many people prefer an approach based on declarations that can be more easily analysed at compile-time. (This is why the record facility was rejected for R5RS.) A more declarative and general class definition facility is planned but not yet implemented. A number of things need to happen before we evaluate a module, and the order these things are done is important. Here we give an overview. Creation of literal values. In immediate mode, we want to use the exact same literal values as in the source forms. Since we cannot embed the values in the bytecode, we have to pass them in from the run-time environment. As in non-immediate mode, the compiler generates fields for the literals, but does not generate code to set the fields. Instead, it uses reflection to set the fields. This has to be done after the class is initialized, which means we cannot place anything that depends on the literals in the class initializer. (Alternatively, the class initializer could access the literals table using a thread-local variable, and use that to initialize the literal fields.) Initializing public fields for exported declarations. Each expored declarations has an associated public field that can be accessed by other modules. These are usually final fields. Evaluating the top-level forms aka the model-level statements and expressions. In a static module, this code is in the class initializer (check this). In a non-statric module (including immeediate mode), this is placed in run method. In a static method, these actions are also done in the class initializer. Many implementations of a high-level language provide an interface to functions written in a lower-level language, usually C. Kawa has such a Foreign Function Interface, but the lower-level language it targets is Java. A PrimProcedure is a Procedure that invokes a specified Java method. public class PrimProcedure extends ProcedureN { ...; Method method; Type retType; Type[] argTypes; } The following syntax evaluates to a PrimProcedure such that when you call it, it will invoke the static method named method-name in class class with the given arg-types and result-type: (primitive-static-method class method-name return-type (arg-type ...)) When such a function is called, Kawa makes sure to convert the arguments and result between the Scheme types and Java types. For example: (primitive-static-method <java.lang.Character> "toUpperCase" <char> (<char>)) This is a function that converts a Scheme character (represented using a <kawa.lang.Char> object), to a Java char, applies the standard java.lang.Character.toUpperCase method, and converts the result back to a Scheme character. Normally, the Java reflection features are used to call the specified method. However, if the primitive-static-method is used directly in the function position of an application, then the compiler is able to inline the PrimProcedure, and emit efficient invokestatic bytecode operations. That is the usual style, which is used to define many of the standard Scheme procedures, such as here char-upcase: (define (char-upcase ch) ((primitive-static-method <java.lang.Character> "toUpperCase" <char> (<char>)) ch)) Similar forms primitive-virtual-method and primitive-interface-method are used to generate virtual method calls and interface calls, while primitive-constructor is used to create and initialize a new object. You can access instance and static fields of an object using similar macros. For example, to get the time-stamp from an Event, do: ((primitive-get-field <java.lang.Event> "when" <long>) evt) Kawa also has low-level operations for working with Java arrays. All these primitive operations are inlined to efficient byte code operations when the compiler knows that the procedure being called is a primitive; otherwise, the Java reflection features are used. Scheme has a few features that do not map easily into Java. We discussed closures earlier; next we will discuss tail-call-elimination, continuations, and multiple return values. R5RS defines a procedure values, which allows an expression or a function to yield multiple (or zero) values, rather then being restricted to a single result. However, the multiple values can only be passed to the call-with-values procedure. There is no automatic coercion from a multiple values to a single value, as in Common Lisp. This makes it easy to implement multiple values in a way that does not slow down code that does not use the feature, which is most Scheme code. Kawa uses a helper class Values: class Values { ... private Object[] vals; public static final Values empty = new Values(new Object[0]); } The values procedure just creates a new Values object using its arguments. However, if there is only a single value, it returns the value unchanged. If there are zero values, Values.empty is returned. (This value is the same as #!void, which in Kawa is used for the result of a definition or assignment, and whose print-out is suppressed.) The Values instance is returned, with no special handling. The only part of Kawa that needs to know about Values is the call-with-values procedure; it needs to check if it was passed a Values object, or just a regular Scheme value. This implementation satisfies the goal of no extra overhead for programs that do not use multiple values, and the cost is reasonable for programs that do. It is not as good a returning the multiple results on the stack, as some Scheme and Lisp implementations do, but that is not do-able in a Java context. Another implementation would be needed if we want the Common Lisp behavior where multiple values are automatically coerced to the first value in single-value contexts. One way to implement that would be move the apply methods to always take an extra "continuation" parameter. Then values can check the kind of continuation it is returning to. Having the return context explicitly passed has other uses too, though it adds some extra overhead to the common case. Scheme continuations capture the current execution state. They can be implemented by copying the stack, but this requires non-portable native code. Kawa continuations are implemented using Java exceptions, and can be used to prematurely exit (throw), but not to implement co-routines (which should use threads anyway). class callcc extends Procedure1 { ...; public Object apply1(Object arg1) { Procedure proc = (Procedure) arg1; Continuation cont = new Continuation (); try { return proc.apply1(cont); } catch (CalledContinuation ex) { if (ex.continuation != cont) throw ex; // Re-throw. return ex.value; } finally { cont.mark_invalid(); } } } This is the Procedure that implements call-with-current-continuation. It creates cont, which is the current continuation, and passes it to the incoming proc. If callcc catches a CalledContinuation exception it means that proc invoked some Continuation. If it is our continuation, return the value passed to the continuation; otherwise re-throw it up the stack until we get a matching handler. The method mark_invalid marks a continuation as invalid, to detect unsupported invocation of cont after callcc returns. (A complete implementation of continuations would instead make sure the stacks are moved to the heap, so they can be returned to an an arbitarry future time.) class Continuation extends Procedure1 { ...; public Object apply1(Object arg1) { throw new CalledContinuation (arg1, this); } } A Continuation is the actual continuation object that is passed to callcc's argument; when it is invoked, it throws a CalledContinuation that contains the continuation and the value returned. class CalledContinuation extends RuntimeException { ...; Object value; Continuation continuation; public CalledContinuation (Object value, Continuation cont) { this.value = value; this.continuation = cont; } } CalledContinuation is the exception that is thrown when the continuation is invoked. Scheme requires that tail-calls be implemented without causing stack growth. This means that if the last action of a procedure is another function call, then the called function's activation frame needs to be discarded before the new function's frame is allocated. In that case, unbounded tail-recursion does not grow the stack beyond a bounded size, and iteration (looping) is the same as tail-recursion. Making this work is easy using a suitable procedure calling convention, but this is difficult to do portably in Java (or for that matter in C), since implementing it efficiently requires low-level procedure stack manipulation. Compiler optimizations can re-write many tail-calls into gotos. The most important case is self-tail-calls or tail recursion. Kawa rewrites these to be a simple goto to the start of the procedure, when it can prove that is safe. Specifically, it does optimize Scheme's standard looping forms do and named-let. Implementing general tail-calls and continuations require being able to manipulate the procedure call stack. Many environments, including the Java VM, do not allow direct manipulation of stack frames. You have the same problem if you want to translate to portable C, without assembly language kludges. Hence, you cannot use the C or Java stack for the call stack, but instead have to explicitly manage the call graph and return addresses. Such re-writing has been done before for ML and Scheme . In Java we have the extra complication that we do not have function addresess, and no efficient way to work with labels. Instead, we can simulate code labels by using switch labels. This is more overhead than regular method calls, so the regular Procedure interface discussed earlier will probably remain the default. Thus some procedures use the regular calling convention, and others the CPS (Continuation Passing Style) calling convention. The rest of this section explains the planned CPS calling convention. public abstract class CpsFrame { CpsFrame caller; int saved_pc; } Each CpsFrame represents a procedure activation. The caller field points to the caller's frame, while saver_pc is a switch label representing the location in the caller. There is a single global CpsContext which owns the generalized call stack. There may be many CpsContext if there are multiple threads, and in fact one CpsContext is allocated each time a regular (non-CPS) method calls a procedure that uses the CPS calling convention. public class CpsContext { CpsFrame frame; int pc; Object value; Object run() { while (frame != null) frame.do_step(this); return value; } } Each CpsContext has a frame which points to the currently executing procedure frame, and pc which is a case label for the code to be executed next in the current procedure. The result of a function is left in the value field. All of these these fields may be imagined as global (or per-thread) registers, which is how you would ideally like to implement a CPS calling convention if you had access to machine registers. The frame, pc, and value fields simulate the frame pointer register, the program counter, and the function result register in a typical computer. After creating a CpsContext with an initial frame and pc, you would call run, which uses the do_step method to execute each step of a function until we return from the initial frame with a final value. Consider a simple Scheme source file, which defines two functions: (define (f) (g) (h)) (define (g) ...) This would get compiled into: public foo extends CpsFrame { void do_step(CpsContext context) { CpsFrame fr; switch (context.pc) { case 0: // top-level code define("f", new CpsProc(this, 1); define("g", new CpsProc(this, 3); return; case 1: // beginning of f // do a (non-tail) call of g: fr = g.allocFrame(context); fr.caller = this; fr.saved_pc = 2; context.frame = fr; return; case 2: // then do a tail call of h: fr = h.allocFrame(context); fr.caller = this.caller; fr.saved_pc = this.saved_pc; context.frame = fr; return; case 3: /* beginning of g */ ...; } } } The entire code of the Scheme compilation unit is compiled into one large switch statement. Case 0 represents the top-level actions of the program, which defines the functions f and g. Next comes the code for f, followed by the (omitted) code for g. When f is called, a new foo frame is allocated, and the context's pc is set to 1, the start of f. The body of f makes two function calls, one a non-tail function, and finally a tail-call. Either call allocates a CpsFrame and makes it the current one, before returning to the the main loop of CpsContext's run method. The regular (non-tail) call saves the old current frame in the new frame's return link. In contrast, the tail call makes the return link of the new frame be the old frame's return link. When we return then from do_step, the old frame is not part of the call chain (unless it has been captured by callcc), and so it has become garbage that can be collected. At the time of writing, the CPS calling convention has not been implemented, but I am filling in the details. It has some extra overhead, but also a few side benefits. One is that we compile an entire source file to a single Java class, and it is more convenient when there is a one-to-one correspondence between source files and binary files (for example in Makefiles). Another exciting possibility is that we can write a debugger in pure Java, because we can run do_step until some condition (break-point), examine the CpsFrame stack, and optionally continue. In one sense benchmarking Kawa is meaningless, because performance will vary so much depending on the underlying Java implementation. One can even take the compiled bytecode files and translate them to native code, factoring out the Java interpreter altogether. For example, the Gcc-based Java implementation GCJ ; can speed up compiled Kawa code substantially. While the goal of Kawa is not maximum performance, it would be reassuring if its speed is at least comparable to other Scheme implementations. So I took three of the old Gabriel benchmarks (as I found them in the Stalin distribution, by Jeffrey Mark Siskind Qobi). I ran them on an UltraSparc running Solaris 2.6. I compared Kawa, running under JavaSoft's JDK 1.1.5, with two representative Scheme implementations: Guile is the Free Software Foundation's Scheme implementation. It is an interpreter based on the heavily-tuned SCM (written by Aubrey Jaffer). Scheme48 (written by Richard Kelsey and Jonathan Rees) compiles to a Scheme-specific bytecode, which is then interpreted (Note that Kawa is doing the various inlining optimizations mentioned earlier; this makes a substantial difference, but could break some unusual programs.) The Puzzle benchmark is based on the old Forrest Baskett benchmark, and uses vectors and do-loops extensively. Traverse creates and traverses a tree structure. Takl is the Takeuchi function using lists as counters. Kawa Scheme48 Guile Puzzle 21s 14s 67s Traverse 18s 14s 46s Takl 81s 36s 87s These results show Kawa beating Guile on all benchmarks, the first two substantially. Scheme48 is faster, but by less than a factor of two (on average). It is interesting that Kawa does as well as it does, using a bytecode designed for Java, as opposed to Scheme48's bytecodes tailored specifically to Scheme. The main current priorities of Kawa are making it fully compatible with standard (R5RS) Scheme, implementing debugging facilities, and making the ECMAScript support usable. The major tasks for R5RS-compatibility are the rewrite to support general continuations and tail-calls, plus a redesign of how macros are implemented. Implementing ECMAScript requires moving Scheme-specific code out of the Kawa core. We also need a more general interface to plug in new parsers, pre-defined functions, data types, and output formatting. That will make it easier to add new languages and dialects. Of special interest is re-implementing some of the ideas and syntax from my earlier Q language . These include a line-oriented syntax with fewer parentheses, and high-level sequence and array operations (as in APL). Also of interest is support for Emacs Lisp. This would require an extensive library to implement the Emacs data types (such as buffers and windows), in addition to the challenges of the Emacs Lisp language itself (it has different data types and name binding rules than Scheme), but may be a good way to build a next-generation Emacs. There is very preliminary threads support in Kawa. It provides an interface to Java threads that looks somewhat like delay, except that the delayed expression is evaluated in a new thread. (The model is similar to to the futures concept of MultiScheme , but there is no implicit force, at least yet.) Some of the core classes (such Environment and Translator) now support threads with optionally separate top-level environments. An interface to graphics primitives is needed. The new Swing toolkit seems like a more powerful base then the old Abstract Windowing Toolkit. Kawa is a solid implementation of Scheme with many features. It is portable to any environment that can run Java applications. It has active use and development, a 75-member mailing list, and is used for a number of different projects. Most people seem to be using it as a scripting language for Java packages. Other people just prefer to use Scheme, but have to co-exist with Java. Bothner88 Per Bothner Ph.D. thesis, Department of Computer Science, Stanford University 1988 Budd91Arith Timothy Budd Journal of Object-Oriented Programming 3(6) 11-22 February 1996 CommonLisp2 Guy L. Steele Jr. Second edition Digital Press and Prentice-Hall 1990 DSSSL International Standards Organization 1996 International Standard ISO/IEC 10179:1996(E) Dybvig93 R. Kent Dybvig Robert Hieb Carl Bruggeman Lisp and Symbolic Computation, 5(4):295-326 1993 ECMAScript ECMA GccJava Per Bothner IEEE Compcon 1997 Proceedings 174-178 February 1997 See also gmp Torbjörn Granlund 1996 (Gmp and its manual are available on most GNU archives.) Ingalls86 Daniel Ingalls ACM SIGPLAN Notices 21(11) 347-349 November 1986 JavaSpec James Gosling Bill Joy Guy Steele Addison-Wesley 1996 JavaVMSpec Tim Lindholm Frank Yellin Addison-Wesley 1996 Kaffe Tim Wilkinson Kawa Per Bothner Miller87 James Miller Ph.D. thesis, Department of Electrical Engineering and Computer Science, MIT 1987 MLtoC David Tarditi Peter Lee Anurag Acharya ACM Letters on Programming Languages and Systems 1992 1(2) 161-177 R5RS Richard Kelsey William Clinger Jonathan Rees (editors) 1998 RScheme Donovan Kolbly Paul Wilson others SGML International Standards Organization Waddell99 Oscar Waddell R. Kent Dybvig In Conference Record of the Twenty Sixth Annual ACM Symposium on Principles of Programming Languages 203-213 January 1999