Igneous: A path to a usable engine


(Jacob Kiesel) #1

Introduction

Usability in Amethyst needs improvement. We ask a lot of our users in the first few minutes of using the engine, and it’s been repeatedly cited both internally and externally as one of the biggest reasons to not use the engine right now.

Knowledge of all of the following things are required to get started with Amethyst

  • Lifetimes
  • Associated types
  • Traits with lifetimes
  • The fact that we implement traits for generic tuples
  • Read/Write requires Default, this is usually fine but can be a sneaky requirement
  • Having a system subscribe to events is mildly awkward and disjointed.
  • Defining System dependencies sometimes requires crawling the source code to find the specific magic string.

These are our biggest barriers to cleaner more usable code.

So what can we do about them?

Well a lot of this comes from our reliance on specs. It’s a fantastic piece of software, but the inability of it to cater to our needs specifically is starting to hurt us. Which is why I propose a fork from specs, to create an ECS tailor made for our purposes. The tentative name I’ve given this project is igneous, as igneous rock is the foundation upon which Amethyst crystals grow.

So what changes would we make to specs if it existed entirely to serve Amethyst?

Lifetimes and Traits with lifetimes

It has been demonstrated that there is no known use case for an API that supports non-static lifetimes in the dispatcher. If we change all of the references in specs with these lifetimes to lifetime 'static then we can completely eliminate this learning barrier.

Associated Types

It’s important for the ECS to know in advance what kind of resource access a system is going to need, otherwise it can’t schedule the system properly. Thus far, we’ve used tuples and associated types to accomplish this. Our current approach has several drawbacks.

  • Uses Read and Write instead of & and &mut, introducing new types needlessly
  • Implementing the SystemData trait on tuples places an arbitrary limit on how many parameters a system can take. Right now that limit is 32. This can be expanded by nesting your tuples, but that’s kind of a silly workaround.
  • Confusing when Read and Write need to be swapped out for ReadExpect and WriteExpect
  • Resource initialization is disjointed and resources cannot have dependencies on other resources. Additionally System::setup methods can’t use a resource if it’s initialized by another system later in the chain.

So what can we do about this?

It’s important to remember associated types exist for a reason. Otherwise we’d have to hurt performance, and make some assumptions about dynamically returned types. But they also exist primarily as an implementation detail, which means they don’t have to be a cognitive load on the user.

We’ve created a proof of concept in the past demonstrating a system can be automatically generated from a function with procedural macros. We decided not to go with this approach because it meant that if your system needed to store caching information, or an event subscription ID you’d have to use a different approach and syntax. However, if we alter that approach we can support a system storing its own data. We keep most of our existing structure, but instead we add the procedural macro attribute to a member function, taking &mut self or &self.

In one fell swoop we’ve removed the need for associated types in code, implementing SystemData on tuples, Read, Write, ReadExpect, and WriteExpect.

Resource/System Initialization

We’ve demonstrated that data can have dependencies, much like a system can. So, I believe it’s time to make that an explicit part of our API. Using a similar procedural macro approach to what was described above, we can mark a function returning Self as an initialization function, automatically turning the parameters to the function into data dependencies*. In this new model you can even think of Systems as a resource, they just also have an associated function that gets executed every dispatch.

* If a set of resources creates a circular dependency, or any other kind of dependency chain that can’t be resolved, the ECS will panic.

Misc. .join() improvement

This syntax doesn’t always make sense to the uninitiated and places an arbitrary limit on how many things you can join across, so I propose altering this to be a macro join!(&foo, &mut bar) which would automatically generate the join iterator in-line. This removes the limit on how many things can be joined and is a bit more consistent with general Rust syntax. We use something similar in the Hello World of Rust.

fn main() {
    println!("Hello world!");
}

Review

So let’s review, of the things discussed above what’s been handled?

  • Lifetimes
  • Associated types
  • Traits with lifetimes
  • The fact that we implement traits for generic tuples
  • Read/Write requires Default, this is usually fine but can be a sneaky requirement
  • Having a system subscribe to events is mildly awkward and disjointed. (Handled in Resource/System initialization)
  • Defining System dependencies sometimes requires crawling the source code to find the specific magic string.

Just one thing left. System dependencies. I think we can resolve this pretty simply. Rather than relying on magic strings, we just use the TypeId internally for the same purpose, and require System to implement Any. Since we’re already removing lifetimes above, this doesn’t introduce any unnecessary requirements as anything 'static is also Any.

Hey what about scripting? We can’t use macros outside of Rust.

Implementing approaches with similar benefits for scripting will likely require us to make tie-ins specific to the scripting language being used. They may not have the same performance advantages that native Rust code would, but then again we’re not expecting scripting to be incredibly fast. We’d need to implement the ECS macros in the scripting language (maybe using functions or whatever other language specific tools are available), and have those implementations tie in to the Rust implementation of the concepts. Likely at a performance disadvantage.

Closing

I’d like feedback on these ideas. I expect it to be controversial because it leans on macros, and introduces huge changes to our workflow however I am requesting that everyone please give it a chance. Usability is something we’ve been hurting for a lot, and we’ve done it all in the name of performance. I believe we can achieve similar performance, greater usability, and a more robust API with these approaches.


(Joël Lupien) #2

Add a thing where each system’s run method automatically trigger a thread_profile!(“system_SYSTEM_NAME”)


(Fletcher) #3

If we do this, I’d much prefer that we do a proper design and architecture doc before a line of code is written.

Please.

Please.


(Jacob Kiesel) #4

I agree, this is mostly just to get the conversation started.


(Jacob Kiesel) #5

@jojolepro asked for an example of how this would change the engine, so here’s my vision

Original

#[derive(new, Debug, Default)]
pub struct NoClipToggleSystem<T>
where
    T: Send + Sync + Hash + Eq + Clone + 'static,
{
    #[new(default)]
    event_reader: Option<ReaderId<InputEvent<T>>>,
}

impl<'a, T> System<'a> for NoClipToggleSystem<T>
where
    T: Send + Sync + Hash + Eq + Clone + 'static,
{
    type SystemData = (
        Entities<'a>,
        Read<'a, EventChannel<InputEvent<T>>>,
        WriteStorage<'a, Transform>,
        ReadStorage<'a, GlobalTransform>,
    );

    fn run(
        &mut self,
        (entities, events, mut transforms, mut _global_transforms): Self::SystemData,
    ) {

    }
    fn setup(&mut self, res: &mut Resources) {
        Self::SystemData::setup(res);
        self.event_reader = Some(
            res.fetch_mut::<EventChannel<InputEvent<T>>>()
                .register_reader(),
        );
    }
}

New

#[derive(Debug)]
pub struct NoClipToggleSystem<T>
where
    T: Send + Sync + Hash + Eq + Clone + 'static,
{
    event_reader: ReaderId<InputEvent<T>>,
}

impl<T> NoClipToggleSystem<T>
where
    T: Send + Sync + Hash + Eq + Clone + 'static,
{
    #[system_run]
    fn run(
        &mut self,
        entities: &EntitiesRes,
        events: &EventChannel<InputEvent<T>>,
        transforms: &mut Storage<Transform>,
        _global_transforms: &Storage<GlobalTransform>,
    ) {

    }

    #[igneous_setup]
    fn setup(event_channel: &mut EventChannel<InputEvent<T>>) -> Self {
        Self {
            event_reader: event_channel.register_reader(),
        }
    }
}

(Jacob Kiesel) #6

I’d like to tack on a sub-proposal. No more manual component registration. Using the inventory crate comboed with a macro we can implement Component and submit it to a component inventory in the same swoop. It’d end up looking like this:

impl_component!(FooComponent, DenseVecStorage<Self>)

(David LeGare) #7

I am wildly in favor of the changes proposed here. Ergonomics is a major issue with Amethyst currently, but (as you’ve noted), there’s really no technical blockers preventing us from providing a more user-friendly API.

I especially think the proposals to use compile-time codegen via proc macros to cut down on boilerplate is exactly the right approach.

Yes please! This would also be tremendously beneficial for the editor and dynamic prefabs, the designs for both of which also require manual registration of component types.


(doomy) #8

I’m very much a beginner to Amethyst, and ergonomic changes would be hugely beneficial. I’m sure I’m not alone in feeling that getting started is very daunting. So many concepts are introduced at once and it can be difficult to understand the core of Amethyst, especially for those not familiar with ECS.

I’m not knowledgeable enough to comment on the specifics of Xaeroxe’s post, but improving ergonomics with little to no hit to performance would be great for beginners.


(Jacob Kiesel) #9

Since a more concrete design document has been requested (@fletcher) here’s some additional info on the implementation.

How is this connected to the rest of the code?

It is additive, and the changes proposed thus far could arguably be implemented as a wrapper over specs, however I would discourage this as some future proposals I have in mind will require modification of the underlying specs implementation. (I’d like to improve the parallelism of the dispatcher for example)

What external deps does it have?

We’d have to replace two very important deps in the Amethyst ecosystem. specs and specs-derive with igneous. Aside from this, no additional third party dependencies should be introduced. igneous would be kept under and maintained by the Amethyst github organization. igneous would also automatically export an internal proc_macro crate called igneous_proc_macro.

What are the requirements for it to be considered done?

igneous_proc_macro

  • Implement system_run attribute, which takes a member function as an input and operates on system structures.
  • Implement igneous_setup attribute, which takes a member function as an input and operates on resource and system structures.

igneous

  • Replace Join trait with join! macro.
  • Refactor DispatcherBuilder::with to no longer take an instance or name, and instead accept a type as the system definition, and accept a sequence of types as system dependencies. The dependencies portion would likely be easiest to implement as a macro which returns a slice of TypeId from the the Any trait. Example usage:
.with::<Foo>(deps!(Bar, Baz))
  • Add impl_component! macro which would provide a Component implementation and submit it to the component inventory, which Amethyst can iterate on initialization to register all components.

Will this cause breaking changes?

Yes. If things go according to plan all systems, resources, and components will need to be refactored.

Hey what happened to removing the lifetimes?

I determined that this can’t be done at a low level, however we can conceal the presence of the lifetimes within the macros, so end user impact is still the same.


(Marco Alka) #10

I love the changes proposed here. It’s not a seldom thing for me to forget registering a new component, which always leads to a run-time fatal error. I’d love to see this solution come true, getting rid of run-time errors because I neglected something. Also, the new code looks really simple in comparison to the old code, so that’s a huge plus.

I have one question on my mind, though: What about nitric? What is @torkleyy’s opinion? I understand that nitric is not ready for action, yet, however since you plan changes to specs, how big would they be? Will they be so big, that designing them into nitric or a fork of it might be a better option?

Sorry if this is a dumb question, because you already might have talked about it on Discord or some other channel. I don’t really have the time to follow them all the time :wink:


(Thomas Schaller) #11

All these months I’ve tried to do just that. I’ve asked for feedback multiple times and I kept maintaining Specs since Amethyst still uses it.

I’m aware there are several flaws, which is the whole reason I started nitric. I think there are some good ideas here, which I’d love to integrate into nitric.

You were the first who proposed to make resource initialization more explicit, and there are efforts to do that. I think it‘s sad you never brought all this up in that thread.

I agree. The current proposal for that is just a free standing new function, in case you aren‘t aware.

I really like the approach of the system_run attribute, and I‘d love to take it into Specs.

Unfortunately, I did not even know about @Xaeroxe‘s plans. I would have appreciated the help and these ideas.

I know I‘m not as active right now, which is mostly because I‘ve got three exams every week, but the fact that I‘m working on two ECS implementations none of which are going to be used makes me sad.


(Erlend Sogge Heggen) #12

I’m strongly in favor of building Igneous on top of nitric, and migrating to that (step-by-step or in one fell swoop, we’ll see what’s easier) new foundation once it is just too useful to be deferred any longer. It’s more work, but it’s a unifying objective rather than a divisive one, which a fork would be (albeit to a fairly small degree).

It would have been easier to digest this large proposal in bite-sized pieces :sushi:, but it’s out here now and there’s lots of useful ideas to pick apart and digest.

We’ve had a slightly rocky start but on the whole I’m very excited to see this discussion progress. In my 15 years of designing things, “improved usability” is practically never the wrong thing to be working on.


Related idea:

I had a though about a simple graphic I’d love to see as a first time amethyst user.

Imagine the TOC for the Rust book, with sections shown in different highlights.

Like green, yellow and orange, depending on how necessary-reading it is

And our goal for the engine would always be to have as many sections as possible marked “not necessary”

Like lifetimes for example.

If anyone has a fairly good idea of what parts of the book would be highlighted with which color, please show and tell :pray:


(Jacob Kiesel) #13

@torkleyy and I have been discussing our visions for the Amethyst ECS and trying to determine how we can make them work together. There is more to come on this, and we’ll be coming forward with our findings once we’re done.


(Joël Lupien) #14

This will not work, especially with generics.

Concerning the macro names, they are not consistent. Both should start with system_.

Also, did you really not ask torkleyy before proposing to fork his crate and replace it in a project he is also working in??


(Jacob Kiesel) #15

That inconsistency is intentional, as igneous_setup is also intended for use with resources.

That’s a good point that I hadn’t considered, I’ll have to think about that.

In my defense the proposal was made to him at the same time it was made to others, but yes that’s what I did. Torkleyy and I have come to the conclusion we haven’t been sharing ideas enough and we’ve also had several misunderstandings along the way. We’re working through those.


(Joël Lupien) #16

Okay :+1:

Filler filler filler filler

We need to remove the minimum message length lol

:heart: :heart: :heart: :heart: :heart: :heart: :heart: :heart: :heart: :heart:


(Erlend Sogge Heggen) #17

When all you want to do is express approval, just hit the Like button instead :heart:


(Jacob Kiesel) #18

Hi guys!

I made a blog post about this: https://xaeroxe.github.io/ecs-rambling/
Pasted in-full below.


ECS ramblings

Introduction

This post is mainly written for people familiar with Amethyst and I won’t be spending a lot of time going over the history of the project and how we got to where we are today.

A small summary and some reminders is in order though.

For those who aren’t familiar I have opinions on how an ECS should be implemented for Amethyst.

We have three names flying around for our ECS project so I’ll provide a quick summary of them here.

  • specs The OG amethyst ECS, this has been kicking around for a long time. It’s got a few problems that have manifested for Amethyst in unfortunate ways which inspired two spin off/reboot projects.
  • nitric @torkleyy’s spin on an ECS, developed here on gitlab.
  • igneous My proposal for a fork of specs that can be used to solve problems with it. Doesn’t actually exist beyond a few drafts and proposals.

Unifying new development work

When I first wrote the igneous proposal I hadn’t spoken with torkleyy as much as I should have about my ideas. I’m hoping that with the creation of this blog I can get better at sharing my ideas. So I’ll start with my perspective of what’s happened since the igneous post.

Torkleyy reached out to me to express interest in unifying our efforts. Nitric has some interesting ideas and enhancements that igneous didn’t, but some of those ideas and enhancements have made integrating some igneous features more difficult. So we’re working towards solutions that express the best of both worlds as best we can.

There’s a lot of nuance and smaller ideas that I’m not quite as interested in covering here especially because I consider them unproblematic. So I’ll cover the most important ideas.

Automatic component registration

At one point I wanted to automatically register component types to lift a cognitive load from the user. This has proven difficult, mainly in that components with generics are very important, but also incomplete types. The type becomes “complete” as soon as the generics are filled, but I haven’t yet figured out how to submit the type to the inventory only when the component type is complete. I’ll keep pondering this, and if any readers have ideas about how this can be done and can make a proof of concept please let me know.

Macros to generate Systems using simpler syntax

This is at the top of my priority list, but has been difficult to implement due to…

Support for registering multiple components and resources of the same type with alternate keys

Making this work with macros to generate systems has been difficult, mainly because my existing proposals have been highly focused around only needing a type.

Here’s some alternatives I’m considering

proc_macro separated keys

struct System {
    caching_data: i32,
    other: u32,
    stuff: u64,
}

#[system_run("foo", "bar", "baz")]
fn run(system: &mut System, foo_resource: &Foo, bar_resource: &mut Bar, baz_resource: &Baz) {
    // System body here
}

This works pretty well and solves most of our problems, my only concern is the fact that keys are separate from the types syntactically. That might not be such a big deal though.

decl_macro keys inline

struct System {
    caching_data: i32,
    other: u32,
    stuff: u64,
}

system_run! {
    fn run(system: &mut System, foo_resource: &Foo;"foo", bar_resource: &mut Bar;"bar", baz_resource: &Baz;"baz") {
        // System body here
    }
}

Oh hey! The keys are now adjacent to the types. That’s neat. This approach has problems of its own though.

  • There’s more magic happening behind the scenes here which can make the error messages of this setup confusing
  • I used a semicolon to separate the type and key here, but that has a few known collisions with existing Rust syntax. May or may not be a problem if the decl macro can identify a “complete type” vs an incomplete one.
  • Even if you use a different delimiter that doesn’t collide you can’t guarantee Rust won’t use that delimiter in the future.

Just cut the keys

This is another option we’re considering. there is some utility in the newtype design pattern. and it would drastically simplify the implementation.

Resource and data dependencies aka Resource/System initialization

There’s a lot of similarities between this problem and the system macros, to the point where I’ll declare this problem automatically solved as soon as we pick a solution for that one.

Misc .join() improvement mentioned in igneous proposal

Still on track! No reason we can’t do this.

Dynamic system graph support and cutting the gaps in the shred dispatcher

I think both torkleyy and I lack concrete plans for this as of right now, but we’re very eager to make this happen.

What’s the name?

We’ve decided to go with nitric since it’s the older name.


DRAFT: Amethyst 0.11.0 release