Please keep in mind there’s a single client and a single server. Horizontally scalable solutions are the wrong tool for the job here imo.
To give an example, in the Unity editor there are a number of tools that use data from a running game:
- Frame debugger (rendering debugger)
- Scene hierarchy visualizer
- Inspector (entity-specific)
- Animation state machine debugger
- Audio mixer
- VFX graph
- Sequencing timeline (Cinemachine)
- Physics debugger
These tools have very varied data needs and based on my experience with Unity, the single-threadedness of the application greatly impacts usability. Let’s say the scene hierarchy visualizer needs 10ms to process the data and render a frame in a large scene. If it uses the same copy of data as the running game, you will end up having 10ms less for actually running the game, leading to lower FPS.
What I propose is to have a query engine that allows to extract only the data required to render a frame in a certain tool. This limits the impact on the game loop to the time it takes to fetch the required data and then the heavier processing can take place in the tool’s own context. A poorly written tool that still only fetches the data it needs will thus have much less impact on the FPS of the game, even if the tool itself is slow.
I do not propose a full state synchronization between an external tool and the game engine. I think it’s better for tools to fetch only the data they need as they request it, and to have no “state maintenance” - just fetch a new state every frame.
I realize that what I proposed is not strictly on-topic so maybe I should move it to another thread. On topic though, the strongest reason to use a multi-process approach is that it’s the only way to make the editor debugging tools connect to remotely running games on a phone or console. With in-process state access, you can only use these debugging tools when running on something the editor can run on.
Alright that all makes sense. I still have two issues.
Who’s going to write and maintain that? Parsing SQL alone is a huge task let alone efficiently compiling it and executing it. Furthermore we might not be using specs forever, if nitric takes off I’d rather us not have so many development hours into specs. I’d be much happier with a Rust API used to generate binary queries though. Something with a really minimal and simplistic API surface area, so we aren’t on the hook for implementing all of SQL.
fetch a new state every frame.
We were talking earlier about hundreds of thousands of entities. Do we want to copy all of those every frame?
As for who will write and maintain it, I’m not sure. Maybe some kind souls from the Rust gamedev community can step forward and share in the vision of improving the debugging experience for
shred. I’m sure it will be useful for others too.
I’m not sure what the difference is for
nitric, but I don’t think it would be necessary to couple a query engine that hard with
specs specifically. Maybe you are familiar with
LINQ in C#? It works with general iterators and would work fine for the use-case I am proposing, although there is a difference in that LINQ is compiled into an application whereas this would be accessible remotely. Maybe it would be acceptable to compile “editor data queries” into the application instead of parsing it over the RPC.
As for writing an SQL parser, it’s not required to implement all of any SQL standard. Just enough to get our use-cases working. There are multiple parsers in the rust ecosystem in various states:
Some produce rather acceptable ASTs that may be useful when implementing a backend.
I don’t particularly care that specifically SQL is used as an interface, but SQL is very powerful and a language most people already know, so it’s quite nice in that way. It may also open up for “ad-hoc” game state queries which is quite interesting as it doesn’t exist in any other engine that I know of.
None of the tools I listed need data for hundreds of thousands of entities. If the filtering can be done in the game process then the result set will be much smaller. For example, a scene hierarchy visualizer might run a query to fetch the names of all entities with a
None parent to get the root objects. For expanded nodes, it would find entities with parents in a list of all expanded nodes.
When it comes to high performance I tend to think “lower level is better” as it typically involves less overhead. I’d categorize SQL as fairly high level.
Now on the other hand I’ve also come to see that maybe we do require a bit more concurrency than I’d previously imagined. So I think the best of both worlds solution would likely involve the following
- load a dylib with user code
- sandbox execution of it in another thread so it can’t kill the editor thread
- For computations and displays that require large reads of game data, such as a debugger or profiler, achieve this by first copying the data necessary to complete the computation, starting the computation in a separate thread, then updating the relevant UI when it’s complete. For especially lengthy computations this has the bonus benefit that we don’t need to perform tool rendering updates on every frame. We could do it every second or so say.
The overarching aspect of what I’m trying to say is that, thanks to Rust, we can achieve great concurrency and keep everything low level and simple.
Can’t sandbox against a process exit call. Would we simply tell the user to guard against that?
Yet you use Rust Zero-overhead abstractions is front and center.
SQL is a data query language. It only defines data operations and it is up to the implementation to make it fast. It can be JIT compiled and produce optimal assembly with zero overhead. It can be translated to another language. It can be interpreted. It is up to the implementation to implement the semantics and make it fast. Modern SQL databases are very fast at processing millions of rows.
You can’t load a dylib on a phone or console. How will you run debugging tools for these targets?
Yes this sounds good, but why predicate this part?
“For computations and displays that require large reads of game data”
Do it for everything. Make it the default and make it fast. Otherwise you may implement things with an architecture that turns out to be slow later, like Unity.
Needs more research to see if it’s possible, though I would consider “just don’t do that” to be an acceptable solution here.
If something can be provably shown to not cost CPU cycles at runtime then I take no issue with it. I believe SQL would take CPU cycles at runtime. Furthermore, “Modern SQL databases” have decades of development put into them, which we don’t have the time or manpower to replicate.
That’s a good point, I hadn’t considered that. I think it’s pretty dependent on how these devices would interface with the host machine. Needs more research.
That seems unnecessary. More threads sometimes come with more overhead than they’re worth, the cost of transferring data, the cost on the OS scheduler. A great example of this is that our renderer used to be multi-threaded, until I benchmarked it and proved we actually got better performance on a single thread. Our CPU demands simply weren’t great enough, and splitting the task apart resulted in a lot more GPU communication than was necessary, slowing things down. I don’t want to force a computational model on a tool if that model just isn’t fit for it. I’d like each tool to be able to decide how great its needs are and behave accordingly.
Afaik, there’s no guarding against that, though the Mobile Platform point supersedes this.
RPC can be proxied over a network. That was one of the initial reasons for using RPC.
It’s not just "user explicitly calls
abort" that we need to be worried about, though. Segfaults, double panics, and other error cases in game code can lead to a process-wide abort that could take the editor down with the game. I would personally like to make such a failure case impossible, such that there’s no way for the game code to crash the editor.
That said, there might be a slightly different hybrid solution: Run a second host process as part of the editor that loads the game as a dylib. This could own all the game state and be in charge of reloading the dylib on change, and could also contain any editor-specific query logic needed to display game state. I don’t know if that approach would massively better than the original proposal of running the game in a completely separate process, but it does allow us to get some of the advantages in the dylib approach while keeping the robustness of the multi-process approach.
I can’t see iOS allowing this, though that’s an assumption from knowing how strict it can be.
For platform development, the editor-host process would be baked into the game binary. You can’t do hotloading on device (as far as I know), so it would effectively fallback to the originally proposed multi-process solution.
One thing I don’t understand: what would the benefit be for hot-reloading vs the game reconnecting to the editor immediately on start? The state in-game would be lost, unless temporarily backed up in the editor, then loaded in afterwards.
LINQ proves that a form of SQL can be transformed into host language expressions and compiled using the regular toolchains, therefore being effectively zero overhead at runtime. But yes, I know SQL databases are exceedingly complex. Interestingly enough it is primarily because they also implement very complex concurrency models and try to extract maximum performance at the same time. SQLite is one of the smaller ones, though still very featureful. It implements an entire SQL database including some complex concurency problems and is only at 128.9 KSLOC (excluding whitespace and comments).
But yeah, I realize it is not a small project and starting with hardcoded data extraction is fine.
It is more overhead to transfer the data and use multiple contexts, yes. But it is more scalable across CPU cores and puts less load on the game loop since editor code would spend less time with access to the game state, which I believe is what is important to optimize for.
I’m not sure I agree that each tool should be able to decide how to access game state independently… It is my experience that the “Tragedy of the Commons” will strike and people will code according to the path of least resistance, possibly resulting in a tangled mess of an architecture. Can you give some examples where this approach would be worse for a specific tool?
Dylib loading potentially allows us to reload game code while preserving state. A way that I’ve seen this done is by having the host process own the game state data, and pass a pointer to that state data when it calls into the game code each frame. This allows us to reload the game code without having to do anything terribly expensive to save and restore the game state.
That said, the only ways of doing this that I’ve seen have been terribly memory unsafe. I’d prefer a solution for hot-reloading that doesn’t involve easily-triggered segfaults in game code, so I’d need a bit more convincing in order to want to follow a dylib approach.
Dylib hot reloading has some intersection with the Scripting RFC. But in this case, you are proposing that all of the user-compiled part of the Amethyst engine be hot reloaded, right?
After giving this some thought I think a networked solution (or IPC, take your pick) would probably be best as it’s easiest to use over a wide variety of interfaces. I’d still prefer a compact binary messaging scheme for it though, ideally with delta computation so that we only have to transmit data that has changed. It’s going to be pretty complicated to implement, however I believe it’s the only solution that has a chance at fulfilling all of the various requirements.
Multiprocess with a common protocol to abstract whether it is querying a server on the same host or one across the Internet, please. This would allow live, shared debugging of an application.
Specs has an
is_modified, but it works on if a particular Component was retrieved through a
Write<T>. Without heavy abstractions imposed on every component’s Write access, there is no way to tell if a value was actually updated, unless we copied every Component that is used with Write access for checking against.
Also, in order to prevent editor desync, we have to Write values at the beginning of the Dispatch, then Read at the end, so every tracked component will be marked at modified. (This is something I was going to do tomorrow)