Yesterday, John McCarthy, the inventor of the LISP programming language passed away. It’s been a bad month for computer guys. First Steve Jobs, then Dennis Ritchie, creator of C, now McCarthy.
LISP was one of my early great loves in programming languages. I learned it first in college (prepping for an AI course). Perhaps it was the intensely well written “LISP Bible” Guy Steele’s Common LISP: The Language or Paul Graham’s mind opening On Lisp. Then at MIT my advisors were Patrick Winston (author of LISP) and Rod Brooks. No surprise I took to it. From Paul Graham I learned that programing languages didn’t need to be static, but could become fluid langages shaped to the specific domain or your task. From Rod I learned to roll my own.
For nearly two decades I was a diehard LISP advocate. I even forced all my programers to code three Crash Bandicoot and four Jak & Daxter games in custom LISP dialects that I wrote the compilers for. This started with our fighting game Way of the Warrior (Rod Brooks voices a character in the game) where I used MacCL to create a LISP syntax state machine to pcode compiler. Then from Crash Bandicoot a much more elaborate language for coding all the gameplay objects called GOOL. In Jak & Daxter I went full on crazy and wrote a native compiler for an object oriented full featured Scheme language called GOAL. We wrote 98% of four Jak & Daxter games in it, including the vector unit assembly.
One of the interesting things about LISP is that it’s actually a pretty easy language to parse, interpret, and compile. This isn’t actually an accident as the S-expression syntax was initially chosen for it’s machine regularity (in those early days of underpowered mainframes). Newer languages are syntactically much more complicated. Ironically most normal programmers, being human, seem to find the more complicated syntax easier and the “simple” S-expression syntax confusing (being backward much of the time to normal human convention). I always found it unambiguous, but go figure. It’s also precisely this regularity that makes the awesome macrology of LISP possible and has allowed the language to remain relevant despite its advanced age.
But by the mid 2000s I started doing the kind of programming I used to do in LISP in Ruby. It’s not that Ruby is a better language (although it is a good one), but mostly it was the momentum factor and the availability of modern libraries for interfacing with the vast array of services out there. Using the crappy unreliable or outdated LISP libraries — if they worked at all — was tedious. Plus the LISP implementations were so outmoded. It was very hard to get other programers (except a couple enthusiasts) to work that way. And ugh, those old CMCL and ACL Garbage Collection code/algo’s were (at least when I last used them in 2006) so awful. In ACL I’d get these LispMachine-like multi-hour GCs.
Ruby had a great book (I put big stock in that) and struck a decent compromise. It’s type system and object model are better (or at least more modern) than Common LISP anyway. The syntax is more inconsistent, and the macro model nowhere near as good. In Ruby you can manually build up strings and feed them into the interpreter, which is equivalent to simple backquote. But you can’t easily do the kind of cool nested constructions that are trivial in LISP.
But it turns out. Libraries and implementation matter a lot. Momentum too. Ruby has momentum, people supporting it who aren’t older than me (and I’m not a young programmer anymore, started in 1980!) Still, you can feel lots and lots of LISP influence in all the new runtime typed languages (Ruby, Python, etc). And 30 years later, listeners still rule! Using a language without a listener is like walking without legs. I pity the C, C++, Java only type programmer.
For more info on my video game career, click here.
For what I’m up to now, click here.
All Things Andy Gavin 
[…] This is part of a now lengthy series of posts on the making of Crash Bandicoot. Click here for the PREVIOUS or for the FIRST POST. I also have a newer post on LISP here. […]
An extra note:
In 2006 I wrote a complex multi-threaded socket and web server that talked to MySQL. I wrote it first in ACL, ported it to CMCL, then to Ruby, and then parts of it (with another programmer) to C (ultra high performance after that). So I got a head to head comparison for the same task.
As to libraries in 2006 (and Ruby libs are only better now). It was easy in Ruby to find libs to talk to third party APIs like Twitter, Facebook, Photobucket, etc. None of this existed in CL. They might be quirky, but they were there.
In ACL/CMCL “core” libraries like sockets and database access tended to be missing a lot of “fringe” (not really that fringe) features like transactions, multiple database support, enums, bigints, etc. But worse than that they tended to mysteriously hang under moderate volume. I found this true on both ACL and CMCL, different libs.
The equivalent Ruby libs, like ActiveRecord, had some SERIOUS quirks, and were missing some of those “fringe” features (I added a lot of them like multi-db, enums, and bigints). But fundamentally, they were more modern in design and reliable. ActiveRecord almost never crashed or hung. Yeah, it did some crazy and stupid things, and performance was a problem. But it didn’t hang. When writing real production code mysterious hang/crash/corruption bugs are just a deal killer.
This accurately reflects my experience trying to use Common Lisp as a modern programming language.
I love the language and know it can do these things, but it’s really missing on the library, tools and environment front. It’s probably all there, but it’s very arcane to get it all working.
The difference between getting a full Ruby environment up and running with external packages vs. doing the same in Common Lisp is night and day.
Thanks for the post Andy!
Yep. Sadly not in the language design per se, but just in the level of support and the number of people working on using it in a modern context. Lots of people use Ruby as a web backend, so there are countless libraries for doing simple but tedious stuff (like interfacing with APIs). These days, there are so many systems out there, that if you have to spend several days writing an authentication and communication layer for each service you have to talk to you’ll never get your “real” work done.
I’ve always been fascinated about GOAL, and there’s precious little written about it. Hopefully you’ll cover it in some future post. 🙂
I have some articles, but I need to format them at some point.
Learning about GOOL and GOAL from TheChaosEngine.com forums made me start learning Common Lisp through Paul Graham’s ANSI Common Lisp book, and few others.
What are your thoughts on clojure?
People keep asking me that, but I haven’t looked at it yet. I need to.
Clojure is a pretty compelling Lisp for practical work – it neatly solves the “library problem” that has plagued other Lisps by allowing you to use all the libraries in the huge Java/JVM ecosystem.
On top of that you also get some innovative concurrency capabilities, decent functional programming capabilities and some “modern Lisp” enhancements like built-in syntactic support for sets, vectors and hashmaps etc,
My experience (after about a year of using Clojure) is that it’s a very good language all round – currently would be my No.1 choice for serious server-side projects.
I’ll have to look for sure.
I’m still but a fledgling LISPer. First learned CommonLISP for my AI course a few years ago. The library problem was one of the original reasons I shied away from CL for practical side projects (even though I loved my experience with it).
Recently I’ve been delving into Clojure and have been loving it. I’m finding the libraries I need, if I can’t find them in idiomatic Clojure I can find them in straight Java and if not in straight Java I can utilize Java’s JNA to hook into C libraries.
A great visionary and Inventor of Artificial Intelligence and Lisp paases away. May his soul rest in peace.
Dr.A.Jagadeesh Nellore(AP),India
E-mail: anumakonda.jagadeesh@gmail.com
“And it’s type system and object model are better than CL anyway. ”
Would be nice if you could expand on this, as it surprises me that an [ex-]Lisper would not prefer CLOS above all! 🙂
CLOS was clunky in practice. It was often hard to tell which methods would get called when things got complicated. Don’t get me wrong, still way better than most object systems, but it had it’s moments. Most of the problems came when you were changing your class design. I often had to dump the LISP image and reload to get things into a clean state.
The library situation has gotten significantly better in the last couple of years. With SBCL there is also a very solid and reliable free implementation.
Andy,
I was also surprised to read some of the Ruby vs. Lisp thoughts and always thought GOAL and GOOL were really cool. I was fortunate in coming to the Common Lisp community in late 2008 and with a mentor that was more familiar with the current state of the art in Lisp libraries and tooling. It was also my first serious programming language. I was 22 at the time.
Since then, things have improved massively. Quicklisp, Clozure Common Lisp or SBCL and SLIME make for a phenomenal free toolset that I’ve been quite productive in and have yet to find an equal for. I cannot compare it to Clojure or Ruby with authority and don’t presume to tell you that you should spend a bunch of time playing with it…but if you check back in, I think you might be pleasantly surprised. Thanks for writing about your hacking exploits. It’s always an enjoyable read.
Best Regards,
Brit Butler
[…] never been very partial to. It’s like Ruby, but 100x uglier and more primitive. I am a LISP (and Ruby) guy after all. Oh well. This is easy peasy programming, so I’ll just suck it up. Using something like […]