More implementation details on the LuaJIT integration


(Théo Degioanni) #1

This document goes into more details on how to implement a LuaJIT integration into the engine as per the scripting RFC.

LuaJIT

LuaJIT is a runtime virtual machine for the 5.1 version of the Lua programming language. It is relatively easy to integrate, supports both JIT (Just-In-Time) compilation for increased performance and interpreted mode for platforms that require it (such as iOS). Its fast (JIT-aware) FFI system allows us to create advanced interfaces between LuaJIT scripts and the engine.

Environment

Lua is a very flexible language. The general philosophy behind this language driver implementation is to create a context where all Lua functions would be pure functions, with little to no side effects.

Pure Functions

These are defined in the sense of mathematics or functional programming: a function that takes immutable inputs and returns values, without interacting with anything else other than the arguments that have been passed.

Notice that the run function in an Amethyst Rust system can be considered as pure, as mutability on resources and self can be mathematically described as taking immutable resources and self and returning a new version of those values on every call. The key difference with functions that use globals is that the scope of the mutation is entirely and statically controlled.

Lua Functions

A Lua function using solely local variables has all the characteristics of a pure function. Therefore, disallowing globals would suffice to create the desired context. All Lua functions are associated with an environment table, containing all the globals the function can access. Like all Lua tables, this environment table can receive a metatable: therefore it is possible to preemptively restrict its access. The advantages of controlling this table are great:

  • We can prevent the creation of new global variables, creating a purely functional context.
  • We can select what Lua standard libraries the script can access, and add our own. Anything that is not a parameter in a Lua function is accessed through the environment table, so we effectively have absolute control. The exact list of modules we will expose should be determined in practice.
  • We can control the mutability of values.
  • We can offer meaningful and clear runtime error messages when an isolation rule is broken, potentially redirecting the programmer to documents explaining the special rules in Amethyst scripting.

You can find a complete list of metatable events here.

Lua Examples

This Lua script from deepmind demonstrates an application of using Lua environment tables to prevent the creation of global variables. Note that they apply their restrictions to the entire environment, but Lua 5.1 lets us tune it for specific functions, giving us much flexibility. Here is another example I made for you because I love you:

-- Here we create global variables
a = 4
b = 0

-- This will be the environment table the function will have.
-- We want our function to only be able to call the "print" function,
-- so we fill the environment with the global "print" function.
controlledEnv = { print = print }

-- We also want to ban the creation of new global variables.
setmetatable(controlledEnv, {
    __newindex = function (table, key, value)
        error("A Lua script cannot declare a new global variable.")
    end,
})

-- We declare our function
function f()
    b = 5 -- Lua variables are global by default
    print(a)
end

f() -- prints 4
print(b) -- prints 5

-- We apply the new environment to the function
setfenv(f, controlledEnv)

f() -- errors out while trying to set b

When importing a Lua script (a system, a state function, anything), we simply have to build an environment table like this and apply it. LuaJIT takes care of the rest, including optimizing it all so the metatable has no cost.

Note that if a script requires a global state, they can simply register it as a specs resource. Please also note that the require system to import modules in Lua is a typical Lua function we can overwrite, so users will also be able to use that safely.

Parallelism

Lua is fundamentally single-threaded. We can still provide parallel execution by creating a virtual machine pool. Just like a typical thread pool, the virtual machine pool would execute specific functions with specified parameters. As we are dealing with pure functions, there is no memory to track between machines, making them stateless. In other words, any virtual machine can run any script at any time, so the difference between a virtual machine pool and a thread pool in this context requires no special treatment and has no consequences for the user.

The LuaJIT FFI lets us pass as function arguments arbitrary memory pointers, which means invoking a scripted function is easy and fast on any virtual machine, no matter where the previous execution of that script happened.

In other words, we can swap virtual machines as much as we want without a care in the world; they are fungible. We can take advantage of all the sweet sweet research that has been made on thread pools in that situation too.

In practice, a Lua scripted system would be a specs System (or equivalent) that grabs a VM from the pool and runs its script on it, passing a pointer to the fetched resources. We only need to allocate one VM per core, which does not require a lot of memory. No book keeping, the VM just is a magic one-shot runner.

Abstraction

While Lua is dynamically typed, the advanced binding generation process described in the RFC is not useless in this context. As explained in the RFC, a call in the engine is a typical FFI call with types and signatures described as a C header. The language driver can however insert itself between the scripting user and the engine to provide nicer looking abstractions.

In the case of Lua, we can extend some of the types with utility features using metatables, such as making iterators iterable or indexed elements indexable with the [] syntax. This would be done by appending a metatable to all values of the appropriate type. As we will be reusing the same metatable for all instances of a type, this will be very fast (amusingly, as a metatable is still a Lua table, we can configure the metatable of the metatable so it is immutable). Note that using the same metatable for all instances of the same type allows us to perform very efficient type checking for the _add, _sub, etc metatable events as we simply need to compare the metatable pointers to know if the types match.

Tooling

You might have noticed that all this abstraction is pure Lua, we have no fundamentally external operation to perform on the script to wrap it up properly. That means integrating Lua IDE utilities should be simple.