Reflections on the Future History of Arming Bears

Although I had been exposed to Common Lisp through working with Daniel Pezley and others in the early 1990s on the "Virtual Reality Operating Systems" VEOS, and its successor MOSES, not much of it really stuck me as particually profound. While I understand the basic concepts of macros and the condition system, I was frankly too inexperienced of a programmer to really appreciate them in context. I did Physics/Math as an undergraduate, thinking somehow that one didn't need "Computer Science" to use computers.

I have been using Emacs as my primary editor since early 1988, mainly so I could look at two files simultaenously on a 1200 baud modem, but basically treated Emacs Lisp as this slightly odd configuration language that I would cargo-cult utility from copying from USENET posts.

I (re)discovered Common Lisp in 2005, having become quite a disillusioned Java programmer. have quit programming if I hadn't found and devoured Peter Seibel's Practical Common Lisp.

http://www.gigamonkeys.com/book/

In 2021, after working professionaly as a programmer for over three decades, I am a much more experienced coder, perhaps armed with enough lambdas to be dangerous. I've read a lot more of classical Computer Science, and try to keep abreast of contemporary topics. My research interests have switched more to topics in Computer Science than tha of pure Physics, especially those that involve the relationship of computational complexity to the physics we observe in our local pocket of the Multiverse. Sometimes I jokingly refer to myself as a mathematician slumming as a software developer.

ABCL originated as the extension language for a the J Editor, an Emacs-like environment written in entirely in Java started sometime in the late 1990s, apparently written entirely by Peter Graves.

http://armedbear-j.sourceforge.net/

By 2005, Peter Graves announced to the mailing list that he would like would like to have the J editor work well enough to concentrate on XCL. At this point, ABCL was largely the work of Peter and Andras Simon, and very close to a conforming implementation. Notably issing the long form of CLOS method combination, a manual to count as accompanying documentation, and a whole slew of minor ANSI incompatibilies.

Erik Hüelsmman, Ville Voultaine, Alessio Stalla, myself, and later, for a real MOP, Rudolf Schlatte worked over the implementation.

Initially, Erik did most of the heavy lifting fixing major gaps in the implementation as well as the release engineering. Alessio added support for extensible sequences. Ville provided numerous fixes. I sort of hung on for dear life, learning what was broken by reading the Common Lisp HyperSpec. Eventually, I was able to understand things well enough to take over the release engineering, which I continue to do to the present day.

The implementation was close to for suitable values of "close". It wasn't until 2011 that the community was able to declare the implementation as ANSI conforming in Amsterdam at ECLM 2011.

The implementation initially ran on Java 5, and we were able to port to Java 6, Java 7, and Java 8 with minimal effort.

ABCL had no other Java dependencies than Ant to run a buildfile (which I maintained). There was a Lisp hosted build that used a working CL implementation to script Java invocations, but it wasn't so common for a Java developer looking to run ABCL. We retain the lack of other dependencies to this day, so ABCL is addressable as a single, svelte ~10Mib jar file for the full implementation (including the compiler and ASDF). We have an additional collection of useful "Beyond ANSI" extensions consisting of Common Lisp-only code which we call "abcl-contrib" that is packaged separately.

https://search.maven.org/artifact/org.abcl/abcl/1.8.0/jar

https://search.maven.org/artifact/org.abcl/abcl-contrib/1.8.0/jar

ABCL-CONTRIB is packaged separately to 1) place an upper bound on the download needed to run the basic implementation, and 2) to allow more permissive licensing than the base implementation, which is GPLv2 with a classpath exception clause.

https://github.com/armedbear/abcl/blob/master/doc/packaging-abcl.org

https://github.com/armedbear/abcl/blob/master/COPYING

In general, we prioritized correctness of the implementation over efficiency in our development.

An amazing piece of code that runs more places than you would think. In addition to the java platform compatibility for a large number of implementations (recent forks in the Valley, in China), it does have an interpreter expressed in the Java code that has been successfully booted on the .NET CLR runtime, and other platforms that are to implement enough of the java.lang.* contracts.

And ABCL1 is slow. Slow to start up. Slow to compile. Slow to load things. But once it gets going, due to the ability to have many different GC implementations "pluggable" via the hosting JVM, it can be remarkably efficient. And it can address currently "exotic" processor architectures such as terrabytes of really, really fast memory with hardward support for many simultaneous threads of execution.

1. Implement atomic compare and swap operations. Technically, CaS could be implemented in openjdk8, but it would require major rewrites to everything that descends from org.armedbear.lisp.LispObject. openjdk9 introduced an interface for memory references which one may dynamically (and atomically!) switch under an existing reference, and the interface for CaS operations was generally cleaned up to a mature state. openjdk9 and openjdk10 are both EOL'd, so openjdk11 becomes a good baseline target platform.

https://github.com/armedbear/abcl/issues/92

1. Utilize improvements in JVM bytecode that have been added, notably support for dynamic method invocation in JSR-292 "Supporting Dynamically Typed Languages".

https://jcp.org/en/jsr/detail?id=292

Re-entrant, generate code for dynamic method sites via invokedynamic instruction.

Adding atomic compare and Swap operations will "complete" the usabilty of ABCL on terrabyte heaps.

There are other APIs present as well that would be bring features to the platform, most notably continued replacement of basic I/O routines with the abstractions afforded by NIO2

https://en.wikipedia.org/wiki/Non-blocking_I/O_(Java)

1. Given a CL:PATHNAME, open a stream of bytes from its CL:TRUENAME.

   Such a resource will either contain one or more top-level forms, or be a zip archive. Confusingly, in both of these cases, the CL:PATHNAME will have a PATHNAME-TYPE of "abcl". The case of containing one-or-more top-level forms is currently only used to package the system in system 'abcl.jar'.

   […]
     -rw-r--r--      3660   4-Jun-2020  15:21:30  org/armedbear/lisp/time.abcl
     -rw-r--r--      3448   4-Jun-2020  15:21:30  org/armedbear/lisp/time_1.cls
     -rw-r--r--      1150   4-Jun-2020  15:21:30  org/armedbear/lisp/time_2.cls
     -rw-r--r--      1536   4-Jun-2020  15:21:30  org/armedbear/lisp/time_3.cls
     -rw-r--r--      1353   4-Jun-2020  15:21:30  org/armedbear/lisp/time_4.cls
     -rw-r--r--      3231   4-Jun-2020  15:21:30  org/armedbear/lisp/time_5.cls
     -rw-r--r--       883   4-Jun-2020  15:21:30  org/armedbear/lisp/time_6.cls
     -rw-r--r--     16115   4-Jun-2020  15:21:30  org/armedbear/lisp/top-level.abcl
     -rw-r--r--      1565   4-Jun-2020  15:21:30  org/armedbear/lisp/top_level_1.cls
     -rw-r--r--      1385   4-Jun-2020  15:21:30  org/armedbear/lisp/top_level_10.cls
     -rw-r--r--      3402   4-Jun-2020  15:21:30  org/armedbear/lisp/top_level_11.cls
     -rw-r--r--      2861   4-Jun-2020  15:21:30  org/armedbear/lisp/top_level_12.cls
   […]
   

   M Filemode      Length  Date         Time      File
   - ----------  --------  -----------  --------  -------------------------
     -rw-rw-rw-      3229  11-Jun-2020  09:01:44  __loader__._
     -rw-rw-rw-      3714  11-Jun-2020  09:01:44  binding_tmp1U1MRERI_1.cls
     -rw-rw-rw-      2445  11-Jun-2020  09:01:44  binding_tmp1U1MRERI_2.cls
     -rw-rw-rw-      2174  11-Jun-2020  09:01:44  binding_tmp1U1MRERI_3.cls
     -rw-rw-rw-      1433  11-Jun-2020  09:01:44  binding_tmp1U1MRERI_4.cls
   - ----------  --------  -----------  --------  -------------------------
                    12995                         5 files
   

2. If the resource is a zip archive, open it, access the forms contained within it.

   (merge-pathnames "__loader__._" zip-archive-pathname)
   

3. Read and evaluate the form(s). They perform the actual loading of the fasl.

   The forms present in the loader artifact will be a bunch of SETQs to set up source properties and one or more SYSTEM::GET-FASL-FUNCTION calls which actually loads the classes emitted by the ABCL1 compiler.

   To load a given class, an array of primitive bytes ("byte[]") is loaded into memory from the archive. This array is passed to the java.lang.ClassLoader.defineClass() function to actually instantiate the class in the JVM.

   "; -*- Mode: Lisp -*-"
   (SYSTEM:INIT-FASL :VERSION 43)
   (SETQ SYSTEM:*SOURCE* #P"/Users/evenson/quicklisp/dists/quicklisp/software/alexandria-20200427-git/alexandria-1/arrays.lisp")
   
   (SETQ SYSTEM:*FASL-LOADER* (SYSTEM::MAKE-FASL-CLASS-LOADER "org.armedbear.lisp.arrays_tmpOKDK4UI"))
   (SYSTEM:%IN-PACKAGE "ALEXANDRIA")
   (PROGN (SYSTEM:PUT '#1=COPY-ARRAY '#2=SYSTEM::SOURCE (CONS '((:FUNCTION #1# ) "/Users/evenson/quicklisp/dists/quicklisp/software/alexandria-20200427-git/alexandria-1/arrays.lisp" 
   #3=25) (GET '#1#  '#2#  #4=NIL))) (SYSTEM:FSET '#1#  (SYSTEM::GET-FASL-FUNCTION SYSTEM:*FASL-LOADER* 
   0) #3#  '(#5=ARRAY &KEY (ELEMENT-TYPE (ARRAY-ELEMENT-TYPE #5# )) (#6=FILL-POINTER 
   (AND (ARRAY-HAS-FILL-POINTER-P #5# ) (#6#  #5# ))) (ADJUSTABLE (ADJUSTABLE-ARRAY-P 
   #5# ))) "Returns an undisplaced copy of ARRAY, with same fill-pointer and
   adjustability (if any) as the original, unless overridden by the keyword
   

The current compiler emits an entire class for each top-level DEFUN form in a given source unit. This simplified constructing the compiler, but for each class, one has to seek() from the beginning to load each class.

With a suitable abstractions, we should be able to instrument all of these top-level forms into a single Java class which can be injected into the JVM in one fell swoop.

openjdk6 onwards allows "type inference", but the Java class files must contain such information. ABCL1 doesn't.

ABCL1 emits class files whose version is 49.3 (Java 5)

https://docs.oracle.com/javase/specs/jvms/se8/html/jvms-4.html#jvms-4.10

1. Verification by type checking must be used to verify class files
whose version number is greater than or equal to 50.0.

2. Verification by type inference must be supported by all Java Virtual
Machine implementations, except those conforming to the Java ME CLDC
and Java Card profiles, in order to verify class files whose version
number is less than 50.0.

https://docs.oracle.com/javase/specs/jvms/se8/html/jvms-4.html#jvms-4.10

If, and only if, a class file's version number equals 50.0, then if
the type checking fails, a Java Virtual Machine implementation may
choose to attempt to perform verification by type inference (§4.10.2).

We only implement the "type checking" verification. Current JVMs are largely optimized for "type inference". At some point, loading classes via "type checking" will no longer be possible on future JVMS

Should be easier if we have a single class in our emitted fasl representation?

The ABCL1 compiler

Involves untangling all the lexical and dynamic scope variables that keep its state.

If we re-write it "not to use specials", then we should be able to use multiple parallel threads to perform the loading

1. Implement atomic compare and swap operations
2. Fork the current compiler, re-writing it to be re-entrant
3. Implement writing type verification byte code

This would form a decent platform to start figuring out how to use invokedynamic to make switching call sites more efficient. Currently, the biggest win here would be for CLOS method dispatch.