Sorting Through the Mio Mess


(Fletcher) #1

My patience with this topic is very low at this point, so I apologize in advance for being somewhat curt.

Core Issue

On Windows, Mio does not work properly. See here for additional context. The end result is packet loss on Windows is much greater than on macOS or Linux. The effect of this will depend mostly on the application.

In an FPS, it might result in rubberbanding if the packet contained player position updates, for example.

Laminar’s Goals

When we started this, the goal was to build a low-level networking library that was rock-solid, reliable, stable, and fast. People using this library and building games or other applications on top of it should not have to worry about performance drops or other non-deterministic behavior across OSes.

Windows has the bulk of the gaming market. My personal opinion is that we are not being true to our goals if we ignore such a glaring discrepancy.

Below I will present the options I’ve thought of to deal with this, and hopefully we can have a productive discussion. We’ve been mired in this issue for weeks trying to decide if it should even be dealt with. We need to make a decision and move on.

Options

Here we go!

Option 1: Ignore It

If our reliability system is good enough, and we make developers aware of the issue so they can code around it, we could potentially ignore this issue for now and hope it will be fixed in Mio in the future.

Option 2: Fix Mio Ourselves

This would require some work, but we could dedicate a chunk of time to fixing the Windows subsystem of Mio, and try to contribute it back. My guess is this would take a few weeks.

Option 3: Stay With Threads

Since we are UDP based, we only need 3 threads:

  1. Listener thread
  2. Thread that processes and multiplexes the packets
  3. Thread that monitors for timeouts

This has the advantage of being dead simple to code, and is proven to work on all OSes. On the downside, we have to have mutexes or figure out a lockless data structure to manage connection status.

Option 4: Conditional Compilation

We could use Mio for macOS and Linux, and use the Windows OS API directly and switch it to Mio if it ever gets fixed. Windows has async sockets that would work fine and are much simpler to work with than iocp.

Option 5: Remove Mio

We could remove Mio entirely and code using the OS APIs directly. This has the benefit of removing a lot of dependencies and layers of abstraction. The downside is that it is a lot of coding.

That’s All

Those are the ideas I have. Please put in more if you have some. We just need to get this resolved and move on.


The constant cycle of rewriting and how we can put it to an end
The constant cycle of rewriting and how we can put it to an end
(Lucio Franco) #2

I personally prefer this route and if one of us wants to spend time working on mio windows I think this would allow us to continue moving forward.


(Erlend Sogge Heggen) #3

Is there a Mio issue where we can track progress on this?


(Lucio Franco) #4

Yup this one is it, though I expect this issue list will grow. I have been following the progress on it from mio side.


(Thomas Schaller) #5

Short question: is 2 more or less work than 5?


(Jacob Kiesel) #6

This is an uninformed opinion and might be useless, but I think you should go with Option 5. In specs we used rayon for pooling of threads, and while this worked at first we’ve rapidly been running into all the ways rayon isn’t tuned for our needs. (See the issue tracker for “High CPU usage in empty project”). For the nitric rewrite @torkleyy and I are planning on building our own thread scheduling manually to make more effective use of the hardware. This leads into my core idea that we should consider using for future development.

Stop writing large amounts of code around highly ambitious general purpose dependencies

Nearly all of our major rewrite tasks have been because we decided we didn’t like how some dependency of ours was interfacing with the hardware. We’re a game engine, it’s okay to get low level and platform specific. We should be doing this more in Amethyst directly so that we’re not dependent on projects that aren’t beholden to us. This way, our hardware interfaces can be built for our needs.

I’ll admit this advice hasn’t been well thought out, which is why I’d like feedback on it before we start using it.


(Erlend Sogge Heggen) #7

By the way, improving windows support is part of their GSoC ideas, so someone from Amethyst could try pitch that as a project.

Improve Mio windows support

Mio is a low level abstraction on top of the operating system’s evented I/O APIs. It is used by Tokio to integrate with the operating system and perform the I/O operations. The current Windows implementation is not ideal. A better solution is outlined in piscisaureus/wepoll.

Expected outcomes

The windows Mio implementation is rewritten using the strategy used by wepoll.

Skills

  • Rust
  • Windows networking

Difficulty level

Medium


(David LeGare) #8

I’m going to go out on a limb on this one second what @Xaeroxe said. I also haven’t dug into the specifics of mio or its issues on Windows, but I’ve heard it mentioned a number of times that the mio’s representation of async I/O is fundamentally at odds with how Window’s async sockets work. While I generally vote in favor of using established platform abstractions, it’s important to note that not all abstractions will work for all use-cases.

That said, it may not be necessary to drop mio entirely. As I understand it, the core paradigm issue is one of push-based async vs pull-based: mio is based off of the unix model of polling sockets, whereas Windows async sockets provide callbacks to notify the process of completion. We could continue to use mio on unix-like systems, and then build a push-based abstraction (and call it wio to be annoying), then have Amethyst abstract over the two approaches. That would still require us Amethyst to account for two different forms of I/O, but it would make the abstraction less leaky without needing us to bind to system APIs directly.

btw, sorry if any of this is wildly inaccurate or not an appropriate suggestion for this issue. Like I said I’ve only been lightly following the issues with mio on Windows so I apologize if I’m speaking out of line!


Also, on a more general note, I think this can be applied to what @Xaeroxe was saying:

I absolutely agree that we shouldn’t be trying to use existing libraries and abstractions if they don’t work for our particular use case. I imagine a lot of general purpose solutions don’t work well for the specific needs of a game engine, so it’s totally valid to build a more tailored solution. And even when we do that, we can still release those alternative solutions as independent crates! Chances are the alternate thread pool implementation being built for nitric would make sense to pull out into its own crate, and I expect the same would be true for whatever bespoke I/O solution we’d end up building.


(Jacob Kiesel) #9

I have more to say on ambitious dependencies, but I don’t want to hijack this post so I’m going to write a blog post and post it in a new topic.


(Fletcher) #10

I also prefer option 5, and I look forward to @Xaeroxe’s post.


(Timon) #11

Test applications could be found here

it includes:

  1. mio_test.rs; wich is a stripped down version of lamiar.
  2. tokio_test.rs; which is a tokio implementation.

Those can be exdented to monitor performance. tokio_test.rs is already finished for a great part and mio_test.rs needs to be tested.

We ran tests with tokio and we got those results:

Linux:

Endpoint 0 metrics: EndpointMetrics { num_sent: 1000000, num_bytes_sent: 11000000, num_received: 0, num_bytes_received: 0 }
Endpoint 1 metrics: EndpointMetrics { num_sent: 0, num_bytes_sent: 0, num_received: 997457, num_bytes_received: 10972027 }
Test took 2618 ms.

Windows

Timon

Endpoint 0 metrics: EndpointMetrics { num_sent: 1000000, num_bytes_sent: 11000000, num_received: 0, num_bytes_received: 0 }
Endpoint 1 metrics: EndpointMetrics { num_sent: 0, num_bytes_sent: 0, num_received: 1000000, num_bytes_received: 11000000 }
Test took 40204 ms.

Amethyst VM

Endpoint 0 metrics: EndpointMetrics { num_sent: 1000000, num_bytes_sent: 11000000, num_received: 0, num_bytes_received: 0 }
Endpoint 1 metrics: EndpointMetrics { num_sent: 0, num_bytes_sent: 0, num_received: 1000000, num_bytes_received: 11000000 }
Test took 37606 ms.

jstnlef

Endpoint 0 metrics: EndpointMetrics { num_sent: 1000000, num_bytes_sent: 11000000, num_received: 0, num_bytes_received: 0 }
Endpoint 1 metrics: EndpointMetrics { num_sent: 0, num_bytes_sent: 0, num_received: 1000000, num_bytes_received: 11000000 }
Test took 18960 ms.

Mac

jstnlef

Endpoint 0 metrics: EndpointMetrics { num_sent: 1000000, num_bytes_sent: 11000000, num_received: 0, num_bytes_received: 0 }
Endpoint 1 metrics: EndpointMetrics { num_sent: 0, num_bytes_sent: 0, num_received: 1000000, num_bytes_received: 11000000 }
Test took 4215 ms.

(Gray Olson) #12

Hi I just wanted to jump in and say that improving mio on Windows is an asked GSoC project for Tokio (see https://tokio.rs/gsoc/)! If anyone that’s part of Amethyst, is interested in networking, and is eligible for GSoC, this could be an awesome opportunity to both resolve this issue, open a line of communication with the Tokio folks, and get paid to work on an important thing for Amethyst :smiley:


(Lucio Franco) #13

So to circle back, I agree we should get low level but mio works amazingly well for linux and osx and a few other platforms. The amount of work to rewrite that and test everything doesn’t make sense to duplicate when this type of api is common. In my mind it makes sense for something like specs because that is very specialized and core to the amethyst product. Epoll and its stuff isnt core to our product. That said, there has been work on the nodejs/libuv to push epoll based apis forward in windows. https://github.com/piscisaureus/wepoll

This has been mentioned by carllerche as how mio will get the windows rewrite. I personally think this is the best path forward in the sense that it will scale and support our usecase, already works and would be a small project to integrate it into mio. Not only this, our mio implementation is simpler meaning we can effectively test it a lot easier and achieve the result of having a very stable library a lot quicker.

That said, we can revert back with option 5 but a lot of work that @jstnlef and @TimonPost have put into simplifying will be gone. Which is something im not a huge fan of.

I think on our current path 1 turning into 2 is the best option. We can use this time to get better surface area test coverage and work on the amethyst portions as we are not blocked by windows. We can still use it for clients as games are not throughput bound applications in the case that laminar will be used.

Edit: For clarification, I mean not blocked in the terms that at the speed showed in the benches we should be able to no receive packet loss. That said, we should probably figure out if the 26 packets/ms are enough.


(Fletcher) #14

One thing we often forget is that we are trying to build a game engine. If an external dependency doesn’t work, it isn’t in our best interests to wait months in the name of cooperation.

The core point is that the current solution is very flawed on the operating system that the vast majority of our users use.

This is not a difficult piece of software to write without mio, and we won’t lose the work @TimonPost and @jstnlef have done. The bottom line is that for our needs right now, mio doesn’t work. We can’t plan our work based on the potential future of external dependencies we have no control over; that will just set us back months.

It is fine to make use of other parts of the Rust ecosystem when they work and when they make sense for our use case. In this case, it doesn’t. Our mission is to make a game engine, not spend months trying to get an external dependency to a basic functional level.


(Justin LeFebvre) #15

I actually agree with both @fletcher and @LucioFranco (as usual). To @fletcher’s point, we are trying to build a game engine and we don’t want to be bogged down by external dependencies which may or may not get around to implementing the features we need. However, I personally feel it is silly to spend time writing a bunch of code from scratch when we can spend a smaller amount of time forking/fixing a well used library and modify it for our needs.


#16

One thing that I was thinking of when starting working with nodejs again is that node has insanely too many dependencies and make your app super fat, for web maybe it doesn’t matter, but if you take a look at electron apps, you add one dependency and you may add a few mb of dependencies. Saying that as was stated, we are making a game engine and IMHO we should use as less dependencies as possible, and seems that mio is not a critital one (I didn’t read the laminar source nor know anything specific about it, so my opinion may be not fit the case) or at least is not so hard/time consuming if we implement it directly as was pointed earlier, also other possible benefit could be less abstraction, better results (again this is just speculation).

So my point is basically remove as much dependencies we can and implement our own code where a dependency is not really needed or can be replaced with a simplified/specific code that fit our needs.

PD: I’m a bit too obsessed with having my app/web/game as thin as possible, I really think that app size is overrated nowadays, just check out electron apps vs native, or unity games, the engines used by them are huge for my taste and most of amethyst projects will be small, at least in the beginning, also unity is aware of this and are slowly shifting to a modular approach, but that’s another topic that we already cover by having rust to be modular.

Sorry if my post is a bit vage and not well informed about the subject.


(Lucio Franco) #17

Just to clarify mio is a very lightweight abstraction, it only comes with 2 dependences not included in amethyst and one of those dependencies will be apart of the std lib in the coming months. The issues with dependencies is that a game engine has to do a lot of things, so there wont really be a way to reduce them unless we totally reimplement all the libraries. But at that point it will take roughly the same amount of time to compile it and probably roughly the same size of binary. Though I do think there are a bunch of areas we can cut down on dependencies.


(Marco Alka) #18

Reduce amount compiled? #FeatureGates. Amethyst is very modular, and Feature Gates allow a more fine-grained selection of things which are important for the application, cutting down on compile time and binary size.

Back on topic, my 2 cents:

  • What’s the difference between 2 and 4? Except that someone would fix mio outside of mio, hence not contribute back… which sounds a bit dumb
  • 3 doesn’t sound ideal, but I am not skilled enough to really compare it
  • 5 would mean implementing, testing and maintaining a lot of things which are already present in mio.
  • Imho, 1 doesn’t really seem to be a blocker for now (Amethyst is far from being stable, and development could continue on other platforms), and if mio is a good match for Amethyst, and Windows will likely get fixed this year, why not stay with it and document the Windows networking issues as to-be-resolved-later. If no mio contributor cares about Windows once Amethyst is more stable, I don’t think that it is a lot more trouble than it would be now to go with 4, or even 5.

(Jacob Kiesel) #19

I just want to re-iterate dependency reduction isn’t about compile times or file sizes, those are at best secondary gains. Dependency reduction is about making sure we’re building on a solid foundation built for us, that we can tweak if we have to. I’ve been here for over 2 years and during that entire time I’ve seen a group of developers flitting from one new toy to the next, rewriting things constantly when we discover the dependencies we’re using aren’t serving our needs as well as we like, and we can’t easily just go and fix them, because we have to navigate the social and political circles of their project and cater to their users.

I know we can save a lot of time by using external code if it fills our needs, but when we discover it doesn’t instead we end up burning a lot more time.


(Fletcher) #20

For extra clarification, it isn’t known if the packet loss is proportional or absolute. If the server is sending 1000 packets in some time quantum, and the client only receives 26, we have 74% PL, which will render a game unplayable. On the other hand, if the server sends 26 and the client receives 26, 0% PL, and no problem.

@jstnlef It isn’t a binary choice of recode everything or nothing. There’s certainly parts that can be reused, particularly because we use so little of mio.