Mp-patch
Visual
Mp formation flight, two sessions on the same pc:
Mp refueling session video (with recorded planes):
victor refueling two lightnings
The first video with two mp pilots, one from canada, the other from france:
A bit of history
Long time ago, we were recording some refueling action, and it became evident that the lag was affecting our respective position, the refueling planes were very close to the tanker , but the tanker saw them quite far, as can be seen in this vidéo taken from the tanker.
Then started an attempt to compensate for the lag, and smooth jitter problems (rubber band effect).
Where it can be helpful is when we need to flight close, with no rubber band effect, e.g:
- Refueling with a mp tanker
- Towing a glider, without broking the rope because of rudder band effect
- Having an aerobatic patrol
Currently this work well locally (with 127.0.0.1 as server) very smooth with YASim planes and a little less with JSBSim, for predictive time in the 0.2 s.
The code
The patch found it's way in next, a bit simplified but working (no acceleration sent), you can find it by using FG development version :D
How to use it
Start the patched fgfs, with the patched datas, and use the multiplayer > lag-adjust menu to set it working. You need to have the master switch checked, and "apply to close mp" to have the close mp plane put in the future.
Both the time and distance sliders allow you to play with a general delay offset, and the distance within the mp planes start to be displayed in the future. A good start is the mpserver's ping value (need some test flight to be more precise).
For a starter, start fg with 127.0.0.1 as server, start the patch (with the "apply to close mp" checkbox, and play with the time slider to see the result on yourself as mp plane.
Pilots using this patch
If you want to try the lag correction, or just see the result on a formation flight, tag along the pilots using a recent enough FG version, and doing some close flight stuff (formation, refueling etc ...)
Parts involved
In order to compensate for the lag, one need to predict the plane position ahead of time, for that you need to address somehow the jitter issue, find a way to know the lag between the players, and to send plane FDM parameters which allow the prediction to be done.
Predictive system
The existing predictive system was used in very few case, most probably only when a player quit, and fg continued to interpolate 5s, then keep the last position 5s more before removing the plane.
I first tried to use the existing code, and the "few Euler steps" were quite good if the speed vector tag along the plane longitudinal axis, but degrade then we got sideslip, or big alpha.
For now i separated the position, and the orientation, having only position, speed and acceleration used for determinate the position in the future, with a simple trajectory equation, and using the rotation terms to guess the current orientation (for now i keep the old equation for this, but I'm not satisfied with the result.)
In case of jitter the old code was coping fast with the new lag value to keep the plane in the time between packets, causing the "rubber band effect", so in the current code, the response to jitter is very slow, with "jump" if we exceed 1s, this is part of what need some tune from real mp flight. I did some test using the timestamp from mp players and assuming the clocks were sync, but the result were not so smooth as the current code.
FDM properties
The properties used by the mp code are:
- A timestamp, telling the time from the sender's side
- The position, in ECEF coordinates
- The speed, what i use is the speed in ECEF, expressed in body coordinates, for YASim i needed to remove the wind component (bug #201)
- The acceleration vector: initially not sent, the result was a good behaviour in straight flight, but a jumpy plane when taking "g". the YASim one seems to be quite fine, but for JSBSim, i need a better acceleration, seems i don't have the ECEF acceleration for now.
- The plane's orientation
- The angular velocity
- The angular acceleration (didn't check if it was sent)
FPS
After some tests, the time needed to have mp plane in sync with the main aircraft depend of fps (it's done using 127.0.0.1 as server, then using the slider to adjust the lag correction).
I think that the mp position is taken from the precedent frame, while the current main aircraft position is from the current frame.
This make the "in the future" plane vibrate if the fps are not steady, and this will need something in the code.
Mp transmission rate
The same as above, a change in speed rate from the sender, change the lag correction needed, i tried to adjust this by using a "0.48lag" coefficient, taken from a rough interpolation. this is biased when the plane is reported to have 10 pps, and networks and mpservers problems make it only 0.5 pps :) , so this term will probably disappear in future version (if there are any, anyone want to help ???)
Time acceleration
Something i saw just a few days ago is that in case the mp player is using time acceleration, we need to change the speeds and accelerations accordingly, otherwise the mp plane will be seen jumping between factor 1 speed portions. This was taken care when sending the paquets, only version 3.6 and after will be ok.
Server ping
Here's the most crucial part, we have to somehow know the time travel to and back from the server. For now, i just give some ping value in the nasal file (lag_adjust.nas), so before the flight you need to ping the server to set the value.
A better idea should be to have time information exchange between the mp pilots and the server, so as to have a real time lag value, and a good guess of the jitter.
A way to deal with this is to change the time reference used, instead of the sim time from the start, the utc time could be used, to give a common reference to all user (and this could be "quite" good using something like a ntp slaving on each fg PC). Even if in reality there is a little offset between each user leading for some to compensate more than other, this would provide good synch, without the need of server/interserver pings. ntp slave can be done on linux, mac or windows easily with dedicaced software, so it's not something to be implemented in FG.
Jitter
Jitter is when the lag value change.
It can be seen as the difference between the sent timestamp and our clock change, for now this is addressed by a slow response to the drift, and a jump if the value is to big, it allow the sender's clock to drift a little, and make the rubber band phenomenon far less noticeable, but still a little when in very close flight.
Inter server lag (and jitter)
We have a simple way to know the lag to the server we are connected to, but no simple way to know the inter server lag. In case the mp pilots are on 2 different mp server, we should compensate for the inter server lag, as it's far for be done , it's best to flight on formation using the same server.
Currently we don't even know on which servers are the mp pilots connected!
Notice that this could be addressed using ntp (see above).
VRP plane position
We are using the CG position, velocity and acceleration to guess the plane's futur, so we need to know the cg position relative to the vrp, and then add some calcul to obtain the cg real position, and the to have the in futur plane vrp position. for this we need to transmit the cg position.
Utility
Spectator mode
Here we try to completely avoid doing prediction, by displaying the others further in the past.
This can be used when no interaction is needed, like in case of an UFO filming.
Known bugs
- The behaviour is strange if some mp pilots are using time acceleration (doing jump each new packet) - should be ok now for mp planes using the futur 3.6 FG - .
- YASim planes are dancing on the ground :) with predictive system on - should be ok, i disabled prediction for small velocity value -.
- Without transmitted acceleration, the plane is more jumpy in high g figures, it's a bit better with.
- It make you addict to the patch :D (but it's a feature)
TODO list, with no time limit :)
Spectator mode
The idea is to display the mp planes "very late", with a strong jitter annulation, to stay in the "interpolation between packets" zone.
This can be used when you don't need to interact with others mp pilots, and need very smooth mp behaviour:
- Camera plane in a mp session
- Linux tag demo ;)
Lag and jitter
- Lag: find a way to have the server ping from fg, instead of letting the user tune the correction value.
JSBSim
- Acceleration vector suitable for the task: partially done. Currently the acceleration provided make the plane in the future "shaky", except if i head east or west , in which case mp display is very smooth. Got to use east-west oriented runways for aerobatic demos. :) I strongly suspect a problem of geocentric/geodetic rotation.
YASim
- Find acceleration suitable: done, but not included into FG yet
Timestamp
This is the biggest problem, currently the timestamp is of arbitrary origin (=0 when you start FG), and we have no way to use it to sync the mp planes with the same time reference.
Add to this the fact that when the frame time is too long (excessing the "max-sim-time-per-frame") the time stamp make a pause, so it's not a real time clock, eg when loading 3d stuff for age.
It could be better to use a realtime timestamp, with users synchronised via ntp or a gps :) . edit 2018: that's what the new WIP does, i hope it should allow a "realtime" mode.
furthermore, the timestammp is not attached to the fdm state, ie we send the current timestamp with the fdm state computed a frame before. in case of unregular fps, this lead to unsteady mp plane. We could address this by sending our position juste after running the fdm instead of before, this would reduce the latence by a frame time, eg at 20fps, 50ms better. edit 2018, this is not the case anymore, position sending is now done after the fdm is computed.
FG integration
It went into next, curently without acceleration being sent, and without a way to add an offset per mp plane (can be added later, as this is nasal stuff). I hope this way flight test will be easier!
2018 WIP
here are some informations on the "next" mp patch started in winter 2017 :) (for now it's only on my private branch mp on sourceforge). It's for my personnal reference first as i only code every few years :) but can be used to help understand what's going on...
timestamp
there's a new clock, kept on sync with the wall clock, this address the previous devious behaviour (non continuous and non linear clock with pause or time acceleration). If users are using ntpd to sync their times (based on time servers, or by e.g. a gps with pulse mode), i expect a "real time" can be introduced, where no lag or jitter is considered, but instead we just assume the other player to be in sync (or quite close) and we compute its position for the time we are in. In recent FG, the mp packet is sent after the fdm is run, so there's no more latency here, and the timestamp used is in sync with the fdm state.
lag and jitter
Assuming both sender and receiver's clocks are more or less running at the same flow, the challenge now is to find the lag value hidden in the jitter noise. there are two jitter sources:
- the network itself introduce some jitter.
- new packets are checked each new frame, so they can be waiting from nearly the frame dt, compared to a basic ping, it add like 0.5dt when we average the lag value. notice that for very low fps < send rate, the value is clamped to 0.51/send rate. as there are more packets arrived in one frame time.
The WIP code use a more statistical approach, we consider a mean value for the lag, which is considered the time the mp player packet arrive. Then depending the lag tuning, we compute for a more in the past or in the future time with a fixed time offset.
real time mode
using ntpd or similar to be sync to utc time should allow to not care about lag and jitter, and simply compute the mp plane's position for the time we are in. This is not tested yet, linux works fine with that, and on windows i managed to get a ntp daemon running, not sure how well it work, and the wallclock precision is no more than one millisecond (in the std::chrono implementation of system time on windows). That's not a big deal if some players have to compensate a bit more, specially if that simplify a lot the lag correction.
Notice the master-slave protocol could be improved this way, with pc in sync locally, and if we predict the plane position in the slave (not done)
not real time mode
in this case, we can't assume horloges in sync, so we need a bit of talk between players, with nasal i guess, if there is no noticeable clock drift, we can set an offset between two players, and made them talk to part the lag value in two, and agree on their respective offset. this is simple for 2 players, but is bit more difficult to set up with more, so for now it's only made by hand :) I'm not too sure how this will be done, one player is designated the master lag setter and other comply, or using a 3rd player to give other 2players difference as seen on the mp server ??
plane's position prediction
the current code is a bit heavy for high future value, and each time a new packet arrive, there's a little jump:
- because the plane can have diverted from the predicted position (more visible with high g turns), i don't use acceleration for now.
- because the position precision can be up to the meter as based on latitude and longitude value with a limited resolution (double?) it's a bit annoying in very close flight.
We'll see what come in the code for this, after some flights tests for sure ...
packet send rate
The player tell in mp protocol at wich speed it send it's packets, and if we want to avoid predicting the mp planes position, we have to take this in account. In this case the planes's position and float props are simply interpolated between two known positions (integer are dealt without interpolation, recently :) ).
But, the assumed send rate is not allways what we got in the reveiver side :)
- there can be lot's of udp loss, depending network charge and QoS
- depending the respective mp servers, we can only have the lazy relai once per second, or even less (i've seen one packet every3 or so second)
that's why the WIP code keep a watch on the received rate.
2021-2022 winter offensive
The bad weather season is coming again, and so come the time to start again a bit of c++ :)
From mp sessions with the last FlightGear, using clock synchronization for the lag compensation work well, but only for some particular cases: pilots need to use ntp, or tune the clock offset to be in a 300ms range (usually lag is in the 80ms range for same continent pilots) fps and packet send rate have to be at a correct value, as the code go off the real time mode once we go over 300ms prediction time, making planes jumpy. most of the pilots don't know they can improve the formation flight experience, and to be honest the lag menu is not in a friendly end user state and need a rewrite.
- on the c++ side:
the real time mode test need to be done only when receiving a new mp packet, not each frame as of now, and we need an hysteresis here to switch between real time mode and fixed lag compensation mode. big frames may give the impression the lag made a big gap (it can be on sending side, or receiving side) so we nedd a way to follow the lag value by omitting this case. the lag value take age to stabilize after a clock offset change, need to have a reset when change happen, either on sending or receiving side. having a filtered acceleration send could help the predicting code, instant acceleration is pretty bad as the tests showed earlier.
- on the fgdata side
we need a simple way to select a player as time reference, if we don't use ntp, so that the clock offset is found automatically, either by comparing of lag player's value and calculating the middle point, or by a fixed lag compensation if that's not possible. not sure we should keep the system displaying planes later when a bit far to improve visual keeping only a "spectator" mode can be enough. keeping the planes in an interpolated zone should be done from the pps value, it's nonsensical to try to predict 2s for planes being received at 0.5 pps