Howto:Record, analyze and replay multiplayer flights with network tools

You can record, analyze and replay flights with network tools by recording and working on UDP packets sent through the multiplayer protocol between your computer and a multiplayer server.

This article will cover some uses in multiplayer problems diagnostics and a way to record (and replay) a multiplayer session.

The truth is I never found what I needed in FlightGear itself, but just a little work with some network tools made my day.

I agree this is not easy to use, need some time to master the tools, and root access to the network interface (so take care ;) ), but it works, as can be seen in this video!

Some examples of utilization

• Debriefing/training of formation flight: Your aerobatic formation can review the flight, and find who broke the diamond figure in the top of the loop :) You can even fly again with the recorded multiplayer flight to improve your skills.
• Make a film by recording the planes one by one, without any Internet connection.
• Having some "multiplayer action files" available, for example to replay them in a FlightGear demo, or if you do not have internet but want some other planes than the silly AI models.
• Save a multiplayer event and do video capture after the event. This way you can make multiple video takes and do a nice editing.
• UDP traffic analysis, to see the effect of internet transmission on other multiplayer pilot's packets (loss/jitter)

Network tools

• Wireshark: A network traffic sniffer/analyzer available for all platforms. We will use it to record the FlightGear UDP traffic.
• tcpreplay: A suit of programs available on Mac and Linux, and maybe on Windows by using Cygwin. It will be used to change some UDP packet properties and to send back the traffic to another FlightGear session.

Captureing FlightGear network traffic

Assuming we are connected to a multiplayer server and that the server and your FlightGear session receive on UDP port 5000, we can record all the UDP packets going to a port 5000.

This way we will record our outgoing packets and the ingoing packets from the other multiplayer pilots.

We start Wireshark, then select the network interface used, and finally set a filter and apply it, for example:

UDP.dstport == 5000 && !(icmp.type)

Finally, we stop the capture and save the file as a .cap file.

What do we have now? We have the flight recorded at the network level!

With wireshark we can now extract some pilot(s), use a certain time interval of traffic, or see certain traffic properties.

With the help of tcpreplay/tcprewrite, we can send the multiplayer traffic back to a running FlightGear session.

Extracting files for individual pilots

In Wireshark it is possible to extract each pilot flight individually. Using the filter looks like:

frame contains "callsign"

and to exclude a pilot:

!frame contains "callsign"

This way you can have each pilot's flight separated. If needed you can merge again 2 or more pilots in a file, if you for example you have recorded a diamond formation and you want to practice the left wingman position. Either you remove the existing one for the .cap file, or you merge the 3 others again, after having done individual .cap files.

Wireshark come with mergecap to merge multiple .cap files. Taking care the chronological order is respected. The provided editcap can be used too if you need to add a time offset to a plane to address recorded latency issues.

It is sometimes difficult to have lot of matching schedules when you are training in an aerobatic team, and we have here a good way to train with other mp pilots without being connected. You can even practice by file exchange: The leader make a flight, send the .cap file to the second pilot which add his flight, then pass to the third pilot etc.

Analyzing UDP packets loss/jitter

Received Multiplayer packet rate

Here is a wireshark analysis showing the multiplayer packet rate for different multiplayer pilots from a recorded multiplayer session.

• The black curve got a nice 10 packets/s, our internet link was good and is maybe connected to the same multiplayer server.
• The red and pink were pretty chaotic. Their packets are often lost and I guess they are jittery in FlightGear. They were probably connected to an other multiplayer server.
• The blue and green show less than 1 packet/s, this can be explained by some connection losses between servers. By default the multiplayer server send at 1 packet/s to the other multiplayer server just to "inform" them which planes they host, and start relaying all the packets only once other multiplayer server send them planes in vicinity. In this case our multiplayer server probably did not relay to their server, and we only got the "lazy relay" from their server.
  

Replay the flight to our FlightGear session

Ok, we have a recorded flight saved as .cap file and want to see it again in FlightGear. That's where the tricky parts start. :)

Limitations of tcpreplay

To make things short, tcpreplay can not send directly to FlightGear if we are on the same computer.

Until now I either use an other PC to send traffic back to my FlightGear PC, or i send to the internet box's external IP address, with a redirection to my pc for the port used in its firewall.

According to the tcpreplay FAQ, it could work if tcpreplay runs in a virtualized machine, but I did not give it a go.

tcprewrite magic

Next is that the UDP packets. To come back to our PC, we need some editing in the UDP packets:

• The MAC address of the sender must match the MAC of the PC sending the recorded flight.
• The destination MAC address must match the receiving PC, or depending the case the MAC address of the router.
• The destination and sender IP address must match the IP address of the 2 PC involved.

We make the change in one go with tcprewrite, here's a practical example. We have:

• The computer running FlightGear, MAC address 00:11:22:33:44:55, IP address 192.168.1.12 .
• The computer we will use to send the recorded packets: MAC address 99:88:77:66:55:44, IP address 192.168.1.17

It is up to you to find your MAC and IP address. You can use wireshark on some network traffic between the two, or ifconfig on Linux etc. Here is the "magic" line,

tcprewrite -C --enet-smac=99:88:77:66:55:44 --enet-dmac=00:11:22:33:44:55 
-S 0.0.0.0/0:192.168.1.17 -D 0.0.0.0/0:192.168.1.12 -i recorded.cap -o readytosend.cap

where:

• -C force a checksum recalculation. If not the packets are not valid and are not sent.
• --enet-smac gives the mac address of the sender
• --enet-dmac same with the receiver
• -S changes source IP address
• -D does the same for destination
• -i input file (the previously recorded flight)
• -o output file

tcprewrite do not need to be root to be used, but using tcpreplay to see the recorded traffic again need root access to use the network interface.

Now we start FlightGear with Multiplayer enabled by sending to a server (it can be a MPserver or the address of a valid local host on your network, for example 192.168.0.17).

Finally we start replaying the flight, on the dedicaced PC:

tcpreplay -i eth0 readytosend.cap

Be sure to use the correct name for the network interface. ifconfig/ipconfig is your friend here!

Tcpreplay works fine this way if we don't need a very good time flow sync, as it end being late, dépending on the cpu usage mostly. I didn't notice this issue at first, but this became evident using lag statistics available recently in next, so to keep a good sync between tcpreplay and flightgear, using a core affinity, setting a priority higher and preferably on a not overloaded PC/CPU core makes things better, so we got something like that on linux:

taskset 02 nice -n -15 tcpreplay -x 1.000065  -K -i enp0s10  /rab/refuel/test.pcapng

The "-x" option (time acceleration) ensure a good time sync over the replay (find yourself the best value for your config :) ), i just need to adjust the time offset for the mp clock, to have replayed planes in sync with my session, considered as "real time" buddys.

Give it a try

The "mont aiguille", in the French Vercors Massif.

if you don't have time to make a record yourself, you can use this flight in the French Vercors Massif (3.4M). Start at LFLU, runway 01 with the UFO or in a plane capable of more than 400 kts, and follow the leading F-4N in a Vercors Massif trip, followed by a Lightning, a F-16 and a F-14B. It is better with a detailed scenery like "La France d'el maxo".

Just decompress it, and use tcprewrite to match your config.

Related content

Wiki articles

• Generic protocol - A flexible way to log properties locally.
• Interfacing FlightGear - Various ways to interface FlightGear.
• JSBSim Logging - Logging JSBSim internal properties locally.
• Logging properties - The simplest way to log properties locally.

External links

• wireshark.org.