Utena: A system to fulfil my desire for maximalist computing

Table of Contents

Introduction

Utena is a system that wants to empower an individual developer to achieve as much as possible with as little effort as possible. This principle of empowerment can be realised through techniques such as program generation, introspection, meta-media, automatic memory management, and interactive debugging. This is not the same as dangerous and downright unsafe capabilities that could be considered to be `empowering', such as access to raw memory.

This is because we believe that empowering individual developers to have more leverage, and achieve more with their system that otherwise would not be possible, will enable them to to empower other poeple to do the same. Other poeple who may not necessarily be programmers by profession. This is a necessary step to break the hierarchy between the producers and consumers of software.

We use the term Maximalism to distance ourselves from minimalists who state the equation ``less is more'', but fail to demonstrate it. There is a fine line to tread in what we actually simplify; to have simplified any software, we must provide sufficient functionality in the software. Otherwise one will inevitably have to use more software to achieve their aims, which would neutralise any gains in simplicity.1 Therefore to achieve this aim we must carefully analyse how to capture a range of functionality with a simple meta-medium.

A result, this means that we want to more closely scrutinise a compromise commonly made when designing programming languages whereby concerns in language implementation (for instance appeals to simplicity or performance) are simply passed through to the language itself, to become first class concerns for every programmer and every program written in the language. We argue these compromises can increase the complexity of even the simplest of programs.3

The Principles

To alleviate our dissatisfaction with available programming languages we set out some principles for the Utena programming system and virtual machine.

The Message Send Meta-Medium and transparent modularity

At the basis of the environment there is just one single notion, the message send to an object.

Newspeak introduces the term message-based programming to refer to a style of programming where all name are late bound and there is no global namespace.4 Similarly to Newspeak, Utena requires that all operations be expressed as messages and all message selectors be late bound. Therefore all dependencies are always mediated by an interface that is the set of message selectors the client is able to send to that dependency. We refer to this property as Transparent modularity. In a step further, Utena provides a syntax agnostic platform for programs to run upon by embedding this principle into the virtual machine; any machine instruction must not depend on the implementation of any primitive object.

Concurrency

We want all programs to be written in direct style, where I/O is blocking by default, avoiding function colour like async/await. Any solution must also consider interactive condition handling, as another criticism to be had with async/await is its requirement for a distinct error handling system.

While nothing concrete has been specified here yet, in prototypes of Utena activation records are backlinked via an escape continuation. In this model, condition handlers can be established between activation records. This backlinked stack should also reasonably allow us to implement preemption without relying on heavy system threads or an OS scheduler. Some context is provided for why we arrived at these requirements later in this article.

Mutability & Identity

We want to encourage the use of immutable structures, but not necessarily enforce them. This means recommending all slots be only readable by default. In addition to this, we want to move towards immutable cons cells and strings.5 On top of this we want to use Baker's model for identity6 and explore the concept of automatic identity management7. This focus on immutability will pay dividends when it is time to start transporting objects and classes.

Classes and class definition mixins

NOTE: This is uncharted territory.

In another idea borrowed and then extended from Newspeak, class definition mixins are immutable descriptions of a class, its methods, slots and an inheritance clause. We take the class definition mixin and use it is a general class expression as we simply treat nested class definition mixins as a slot description in the enclosing class with a class expression as the initialization expression for that slot, which in turn is evaluated when the enclosing class's constructor is called.

We keep both classes and class expressions immutable in order to avoid a situation where a class expression that as been transported to multiple machines is needed to be synchronized during an update. Instead we introduce a notion of superseding for both class expressions and classes which should also allow for finer control over upgrading instances of a given class. This also means that class expressions and classes are no longer responsible or needed to be trusted to track their dependants, and this means that tracking class dependants needs to be carried out by components in the system.

Syntax and language

Utena VM is supposed to be a substrate VM for any capability based dynamic-interactive programming language. What this means is that unless you're somehow using Newspeak,4 you're going to need a new programming language. For us, while designing and prototyping Utena we naturally began to express ourselves using a lisp called Sourcerer, as then we don't have to edit a parser each time we change our minds about the system.

Projects and Progress

Related writing

Implementation

Gnuxie's Technical Project

Utena VM was first envisaged when I was studying at University.

Initially what sent me on a path of looking elsewhere was my dissatisfaction with concurrency in Common Lisp. Presently, the only implementation offering lightweight "green threads" or fibers is CMUCL. This was problematic to me because I previously had been writing lots of asynchrnous I/O-bound code while working with the Matrix protocol. Doing any asynchronous programming in Common Lisp without support from the implementation for fibers or green-threads forces you to abandon dynamic variables and the condition system, as these are intrinsically tied to the stack model. What this meant to me is losing the most significant benefit of working with Common Lisp: interactive error handling & debugging.

Something I had previously attempted to do while working with Matrix was create a Matrix server implementation in Common Lisp, which also sent me into thinking about partitioning and distributed computing. Matrix itself was also making me think about how to distribute software between machines in a safe and trustless way.

As part of my bachelor's degree, I would produce a technical project so that my independent learning could be assessed. It's ultimately these experiences that led me to decide to prototype a virtual machine for my technical project, and it is this project that planted the seed in my mind that would later grow into the idea of Utena VM.

The prototype was relatively simple, I had no prior experience in language VMs at this point, and desining the VM for the semantics that were interesting to me at the time, was much more important to me. The design was mostly informed by the Self VM and this version of Utena would use a prototype-based object system like Self too. In the end the machine was capable of running a simple program counting from 0 to 5 and then returning back to the host. I didn't have a higher level language and Utena instructions were represented as byte-code at this point. So I had assembled this program by hand. While this would normally be a trivial program, running it in Utena requires a great deal of the VM to function: blocks (closures), numeric and boolean primitives, assignment, all of the machine instructions (including semantics for the different self sends and slot lookup algorithms), and accessing then calling the escape continuation of the activated record.

You can read the report here, and the original prototype is here. There's a small mistake towards the end of the report where, I forget to delete my own comments (or fail to mark them as comments while editing depending on how you want to look at it), but maybe you won't even spot it, I didn't until years later.

Hopes and Dreams

Hopes and dreams is an attempt to create a bootstrap interpreter for Utena VM, but it must be stressed that Utena is still being designed.

  • Implementation differences

    Learning from my mistakes from the original prototype, I would instead use concrete instructions in the form of an IR for the machine and generate IR from a guest language, that I called Sourcerer. For Sourcerer itself I would take inspiration from the S-expressionists and read Sourcerer into Concrete Syntax Trees that would save us a lot of effort when debugging Sourcerer programs as the CSTs allow us to track the source of nodes. The interpreter will no longer use recursion in the host stack.

  • Utena now has classes

    There was an attempt in the original prototype to bring Bracha's modules as objects over to the prototypical object system of Self. While I haven't written an academic analysis on why I stopped doing that, one of the major issues is that object initialisation in Self happens within the context on a lobby object (though, in reality it can be any enclosing environment). As of writing, I cannot actually remember the entire series of events leading to its abandonment, but I do remember it seeming much more problematic to me at the time relative to Newspeak's nested class model. One of the problems I do recall is that there is not a clear why for slots to evolve with prototypes. There isn't a straight forward way to add/remove slots and change representation, even if another indexing method is provided than identifying objects by class, it would not be clear how to identify all of the objects that are targeted by a change, and the problem is also worse if you have transported some objects to another machine. If I do make another evaluation of Self VS Newspeak in the future, it will be essential to consider the work that was done on the Self transporter,8 which I encourage you to read anyways as it gave birth to one of my favourite images.

    So the first big change was adopting Newspeak's class model, with some modifications to ensure classes and class-definition-mixins are immutable. Doing so ensures that even if only just transport class-definition-mixins, we will be able to do so entirely trustlessly without dependencies.

Timeline

When What
Oct 2020 Gnuxie started background research for the technical project
Jun 2021 Gnuxie has finished the technical project and report and graduates
Jun 2021 Gnuxie published the previous version of this page
Jul 2021 Gnuxie starts work at Element, work on Utena slows down
Aug 2021 Gnuxie & Hayley start collaborating on the Utena specification
Sep 2021 Gnuxie starts work on hopes-and-dreams
Aug 2022 Gnuxie & Hayley publish Zero Feet
Dec 2022 Hayley publishes The next 700 virtual machines
Jan 2023 Gnuxie & Hayley update this page, adding additional context

Acknowledgements

I want to thank Hayley for being the best co-conspirator I could ask for, and you couldn't ask for really.

I want to thank the Self team for being so uncompromising, they would probably be questioned into failure if attempted today, Self will eternally inspire me.

I want to thank Gilad Bracha for being awesome.

I want to thank Robert Strandh & Gilbert Baumann for guiding both me and Hayley, and showing there is an alternative to the current hegemony of programming languages.

Footnotes:

1

Neutralising at best; interoperability and interfacing between more software systems may require more code and more cognitive overhead.2

2

Jan Heering & Paul Klint, Towards monolingual programming environments https://dl.acm.org/doi/abs/10.1145/3318.3321

3

Cooperative of Applied Language, I don't want to go to Checl-C https://applied-langua.ge/posts/i-dont-want-to-go-to-chel-c.html

4

Ministry of Truth, Modules as Objects in Newspeak https://bracha.org/newspeak-modules.pdf

5

Henry Baker, The Legacy of Lisp slides: https://www.international-lisp-conference.org/2005/media/baker-slides.pdf audio: https://www.international-lisp-conference.org/2005/media/baker-audio.mp3 Fun fact, this is I think the only source for Baker's voice that we have?

6

Henry Baker, Equal rights for functional objects or, the more things change the more they are the same https://dl.acm.org/doi/10.1145/165593.165596

8

David Ungar, Annotating Objects for Transport to Other Worlds https://dl.acm.org/doi/10.1145/217838.217845