User:Hooray/SandBox/FlightGear Performance and Run-Time Footprint Debate
 |
Work in progress This article or section will be worked on in the upcoming hours or days. See history for the latest developments. |
Background
|
Thorsten and I, well, we're both sitting in front of hardware worth $2k-3k US (~18 months ago), with ~10+ gb of RAM, 2 gb of VRAM, 8 cores etc and I'll be the first to admit that this may very well kinda "mask" such issues, and that end-users may run into issues that we may not perceive as critical, or that we may never even notice at all.
|
|
|
|
Yes, there are issues. FG isn't perfect, it's rough around the edges, the GUI is far from optimized,...
|
|
|
|
The lesson I take from all this (and related discussions) that I will be in the future very careful to make any rendering code available on GIT which isn't guaranteed fool-proof and might lead to less than perfect visual sometimes. If the general mood is that we bash the project for a release early, release often policy, then I will stop with it.
|
|
|
|
(Originally, there actually was a conceptual separation i.e. WRT to master/next branches, but this hasn't been used in years, despite having been established by Tim Moore as such. But these days, we're simply lacking the manpower to do so.) |
|
|
|
Poor performance doesn't necessarily have to do much with graphics - what you are describing are fluctuations in frame rate and frame spacing (latency, the time required to create a single frame). And here, it's frequently a bunch of other subsystems that are the culprit, or that at least contribute to the whole issue in a combined fashion. While it is true that many things can be disabled or customized, this doesn't apply to all features - and there are many non-optional features/subsystems that cannot be easily customized or disabled yet, but it's something that will happen over time. However, while it's "graphics" that many people tend to consider the cause, it's more likely to be a sympton, because that's what people perceive to be slow, and the problem. Utlimately, frame rate and frame spacing are really just symptoms though - there are many other subsystems that contribute to performance issues.
|
|
Symptoms
|
|
Why do I have frame-rates of 8 - 60 sometimes, depending (mostly) on local weather and all settings to maxx. And here comes the part I mean to talk about: Why does my GPU jumps to 70°C in a minute or so, while my 2CPUs run gently with 50% usage each. (threading : auto)
— chris_blues (Thu Apr 18). Can FG computations be made more balanced?.
(powered by Instant-Cquotes) |
|
|
|
The problem isn't completely on my hardware, but mostly on the consumption of most features in FG. One day the game runs fine, then a 777-S is added and there goes my 4 GiB of RAM. It's a very nice aircraft but it consumes too much ram, not even the 777 PMDG consumes that amount. |
|
|
|
I'd love to have the money to buy a new pc and use it for many things and Flightgear, but I don't. I also don't think most people flying a free Simulator have the required computer power to run the game as it's requesting since about 2 years ago, but since I have a choice of either spending lots of money to buy a new PC just for Flightgear (because all of my other 54 games run fine at High settings) or buy a payware simulator, I chose to buy a simulator that runs fine with my current hardware.
|
|
|
|
I don't need to max everything in order not to be able to play it. The default settings by themselves are enough to make the game unplayable, at least on Linux. FG's memory consumption is high and the scenery is too detailed. Either that, or it's a combo of mistakes that make the game not work for everybody. For instance, I saw a picture in this forum showing a part of the scenery: it was formed of what, little squares of 40x40cm? Even smaller? That might be OK in short distances but the screenshot (for what I can remember) showed the scenery as quite far. No one gives that much attention to ground level details, no one will zoom out till they can see ants on the ground.
|
|
|
|
In fact I tested X-Plane and it works so well I couldn't believe, because the developers say "use 16GB of RAM and multiple processors". That's not quite the case, as I spoke to one of it's developers he/she said the game auto-adjusts itself to the hardware it's been run on. But even so, I didn't notice any difference from the ad videos they put up on Youtube.
|
|
v2.0 Scenery
|
|
The numbers say that without any doubt, 2.0 scenery is what uses all the memory, nothing else - not hires textures, not trees.
|
|
|
|
Do not use the 2.0 scenery (because it is an early, imperfect work in progress release) and watch the issues disappear. I confirmed with my 'Yeah, pretty much' that I recommend putting up a warning that you need lots of memory to use the 2.0 scenery and that I would recommend stay with 1.0 if you don't have that.
|
|
|
|
I don't think it's fair to blame any particular subsystem for these issues. The fact is, the project has to keep evolving and transforming in order to remain relevant, and this means things can sometimes break in the process, regressions rear ugly heads, some users become unhappy, but overall the project is moving forward and commits are regularly made by core contributions, which I think at some point becomes essential to maintain confidence in the future of the project, as we can see in many other open-source code bases, rather than stagnate and become increasingly irrelevant in today's fast moving market, as it's never a good idea to throw away perfectly good opportunities for progress, even though it could inspire distrust and even drive away some potential users, as long as the core contributors are kept reasonably satisfied and are not compelled to move away from the project.
|
|
|
|
The scenery doesn't have a LOD system yet, yes. I gather the philosophy was we need a viable scenery first before we can simplify it down to implement a LOD system - so we have the first step done, and the second is in progress. What you're upset about is that the second step isn't done yet and the scenery team was stupid enough to listen to people like me and gave us what they had so far. But people needed to start with the first. |
|
|
|
There's nothing wrong with FG as far as I know - it will probably run fine on your computer and deliver quite decent visuals if you customize it. There's a lot wrong with the way the information how to customize it to your needs is documented and transmitted. And this is where the republic of heaven thing could come in - rather than complain and write support PNs for the kingdom to come, you (all) could help to assemble, improve and spread this information.
|
|
|
|
I had banned myself from KSFO because I would get to swap even before seeing anything and now I tried and fgfs starts and is usable in my 32 bit debian system with 2GB of RAM. Simple planes and no multiplayer though, but is really great improvement.
|
|
|
|
You do not need to buy a new computer or more memory to have very pleasant visuals of FG if you have a halfway decent GPU. I have plenty of forum screenshots in the old scenery to demonstrate that.
|
|
|
|
The scenery people know that. It needs way too much memory, especially the line features (roads, rivers,...) eat it like mad. A third to half is zero area triangles which could be culled. There's no LOD system yet. People are working on it.
|
|
|
|
You can use 1.0 scenery and lots of visibility (4 GB will buy you 200 km if you de-activate trees and buildings). You can use hires scenery and lower visibility (4 GB will perhaps buy you 40-50 km). And anything else. Obviously, if you're flying an airliner, your choice might be different than if you're flying a single prop.
|
|
|
|
revert to the 1.0 scenery. On a by now 6 year old 32bit computer with 4 GB memory, I could run that with shader settings at what is now medium quality and visibility ranges of 120 km with 20+ fps.
|
|
|
|
the 2.0 scenery is heavy - FG 3.2 really isn't. You're confusing the framework with the data - FG is just the framework, its resource dependence crucially depends on what data you feed it with, i.e. what scenery you use or what aircraft. FG 2.0 would be just as memory-consuming with the new scenery and the most recent top-notch aircraft (if it would run that is, there was a format change for the scenery).
|
|
|
|
An obvious case is scenery 2.x. Yes it is heavy, yes it is released early (understandably as Thorsten argues), yes one can warn people to revert to 1.x temporarily. But do not leave it at that. Maybe there is being worked hard at smart solutions, but that is not really becoming obvious to people that read the forum. Understandably they get frustrated by that.
|
|
|
|
For the scenery issue, and actually every issue, i think it is important for a developer to ask himself: "what would a user want/need". I fly high fl300-400, so i might need some 200+ mi visibility to see some earth. Hardware (memory) wise i can only set some 60 mi visibility, so at altitude i fly in plain fog. At low altitude one has a horizon of maybe 20 mi, so the heavy scenery is not a problem at whatever visibility setting. But at high altitude one gets more tiles. The current problem is that the tiles are as detailed as at low altitude, which is totally unnecessary. So the logical second step after the early release of scenery 2.x would be to load lower res tiles in memory at high altitude. I am quite sure that such solution is also used by x-plane, which explains the better experience there.
|
|
|
|
There's a story to that, and that is that for years we (certainly I and many others) have been asking the scenery team again and again to release something of the new world scenery, even if it's not perfect, so that we have something to work with. They've finally done so - and we got something that's not very perfect - with zero-area triangles, possibly too high details on roads, heavy on memory,... so now that's a base to improve upon.
|
|
|
|
I'm getting seriously frustrated about the whole scenery thing. About the release management which discarded all the good evidence about memory issues. About users who can't find it in them to cut the scenery people some slack and acknowledge that the 2.0 scenery is a step towards something better - not the end product, but something to work with while something new is rolled out, and that using 1.0 apparently isn't an option. About the misinformation that is spread because people can't be bothered to verify information. About long term contributors who run around and claim that we have a huge problem in the code, when in fact all investigations have shown that most issues go away when reverting to 1.0 scenery. And about the general notion that nobody would care about the whole thing, when in fact plenty of people are working on the issues.
|
|
|
|
Making scenery is a completely different branch of coding from rendering scenery. Even if I would not add any more rendering options, I would have zero qualification working on scenery. You can't simply assume that you can reassign people to any task - a 3d modeler isn't going to be a coder if you ask him to, a graphics artist isn't going to become a rendering specialist. Within each area of specialization, people do fix bugs before adding options.
|
|
Memory Footprint
|
|
Memory footprint has been a recurring theme on the mailing list and Rebecca has demonstrated it to be almost exclusively relate to the scenery. The scenery team is busy getting rid of extra zero area triangles for the next scenery release. James has pushed the changes to the tile cache for better memory footprint. People are working on a terrain LOD scheme. Prior to the 3.0 release, you can find my messages pushing for a clear statement that the 2.0 scenery as is is rather memory intense and that we recommend it should only be used on systems with plenty of memory and that we should keep the 1.0 scenery available. There are forum messages by yours truly all over the place explaining that the first measure in case of any trouble with FG 3.0 out of the box should be to revert to the old scenery.
|
|
CPU vs. GPU work
|
|
If you have a halfway decent GPU, when you ask a rendering task (texture lookups, vector transformations, matrix multiplications,...) , the additional performance of 8 CPUs is pretty negligible. The GPU does these tasks in lightning speed.
— Thorsten (Fri Apr 19). Re: Can FG computations be made more balanced?.
(powered by Instant-Cquotes) |
|
|
|
on a high-end GPU (which I assume you have), the single determining factor for the framerate seems to be the vertex count of clouds, buildings and trees (in this order). So rendering overcast sky makes a huge difference to rendering clear sky, rendering a city worth of random buildings makes a huge difference to rendering a desert, and rendering dense forests makes a difference to rendering a few patches of green. With all details to max, we probably burn 99% of the total computational load in rendering (it's about 1200 lines of instructions per pixel 30-60 times per second), all other flight related tasks like computing aerodynamics are small in comparison.
— Thorsten (Fri Apr 19). Re: Can FG computations be made more balanced?.
(powered by Instant-Cquotes) |
|
|
|
the only thing that I can identify as affecting my framerate is the vertex pipeline. Currently, with the rest I can pretty much do what I like. The Concorde with it's gazillions of systems runs as fast as a glider. The high-fideltity (or at least quite complex FDM) T4T Spitfire runs as fast as any other plane. Switching background Nasal on and off makes no bloody difference for anything. Only what enters the vertex pipeline does. Or lots and lots of semi-transparent stacks in the fragment pipeline - they can eventually kill as well.
— Thorsten (Fri Apr 19). Re: Can FG computations be made more balanced?.
(powered by Instant-Cquotes) |
|
|
|
The vast majority of the computing load is generated due to rendering, i.e. if you have GPU support (as with any modern graphics card) it doesn't touch the CPU at all. The framerate is usually determined exclusively by the GPU capability - distributing CPU load across several cores may be elegant conceptually, but doesn't make anything run faster.
— Thorsten (Thu Mar 15). Re: [SUGGESTION] Multi-core FlightGear support.
(powered by Instant-Cquotes) |
|
|
|
Thorsten's system is unfortunately not representative of the overall situation for all users - because not everybody has a $ 3k workstation-style laptop at home with 8 64 bit cores (hyperthreading included), 16gb of RAM and a NVIDIA 670M 2gb SLI configuration. We need to be careful not to compare apples and oranges here, I have no doubt that Nasal code or poorly coded C++ code doesn't really show up on such a system.
|
|
|
|
performance issues like these are not obvious on high-end machines, no matter if it's framerate, frame spacing or the Nasal garbage collector - on a 3-4 ghz multicore system, there's usually enough horsepower available to make up for architectural shortcomings - but that only hides the real problem. And it's a well-known issue that computers used by developers are usually high-end machines, which end up masking problems encountered by end users, who may very well use much less powerful hardware.
|
|
Our Main Loop & HLA
|
|
have heard on these forums from various people that FG does not utilise all the resources available to it very effectively.
|
|
|
|
Basically, your GPU is a massively parallel special-purpose cluster of "CPUs" (depending on the hardware, hundreds of tiny cores) that can do certain mathematical computations extremely well/fast by partioning the task at hand and handing each job to another work queue, which may be processed by hundreds of "dedicated cores sitting on the GPU" in parallel. Basically, your GPU is super computer (certainly when compared to the computing power of the 80s or 90s).
|
|
|
|
the idea is not to use idle CPU cores for rendering related stuff - but rather optimize the FG<->OSG pipeline such that more main loop resources become available, so that certain tasks can be offloaded onto other cores. For example, OSG is able to run certain rendering-related optimizations asynchronously, outside the main loop. But that requires a paradigm change, because of the way FG makes currently use of OSG in a fairly sub-optimal fashion, even though Thorsten is right in saying that this particular issue doesn't show up on extremely fast systems with high-speed CPUs.
|
|
|
|
The point of the whole FGViewer/HLA effort is having a pure renderer and moving everything not rendering related to separate entities, that may run in separate threads or processes, without affecting the visualization of the simulation. That is a key technique used in professional full motion simulators, and in the simulation industry in general. Even without fully understanding the benefits, you will certainly come to appreciate the fact that you will be able to run many computations that don't directly affect frame rate or frame spacing/latency anymore.
|
|
|
|
I think HLA is a great idea for many reasons, because it addresses many unnecessary sources of bottlenecks. But I don't think it addresses all bottlenecks, in particular I don't think it addresses mine. Mathias has made it abundantly clear that he's not interested in high-end shader effects, but that he cares for multiple display setups and similar things. It's also become quite apparent to me that he doesn't really follow how our high-end effects actually are implemented and what they do. So my impression is that HLA will be a good thing for many purposes - when you're CPU limited for any reason, for MP, for multi-display setups,..., but I don't see how it will address situations where the bottleneck is shader code.
— Thorsten (Fri Apr 19). Re: Can FG computations be made more balanced?.
(powered by Instant-Cquotes) |
|
About Testing
Lack of Feedback from End Users
|
|
From where I stand, we're mainly lacking willingness of users to actually run tests under guidance. If people can't be bothered to report their system specs or what they're actually doing in detail, then I can't see how they'd report more internal diagnostics.
|
|
|
|
many people don't even know how to get/deploy and use a nightly snapshot, let alone actually provide useful/actionable data, "useful" in the sense that developers can actually act upon that data
|
|
|
|
The mechanism as such is clear and sound: We have the Jenkins nightly GIT binaries on distribution, we distribute release candidates, these should lead to test data on a wide set of systems, which them comes back and enables people to take action.
|
|
|
|
I don't know - personally, I believe we simply couldn't deal with the results: Just look at some of the recent developments in this area, such as the integrated "Crash Reporting" system: We've been seeing dozens of users making extensive use of this - and then it became obvious that the SINGLE recipient of all these bug reports simply couldn't deal with the sheer number of reports, because we never established any mechanism to process all the feedback at a larger scale. Or look at the issue tracker and the number of "critical" bugs there - which I think kinda proves the point that we, as a project, cannot deal with too much feedback. Even though that may not necessarily apply to other contributors, like yourself. But the few people who do not just have 2-3 "pet projects", but who maintain the whole simulator, cannot deal with the sheer amoung of feedback. Thus, I think that built-in diagnostics and automated regression testing would help us more in the long term, i.e. kinda like the whole "benchmark" debate we've been seeing for years. The final result could then be a "flight recorder tape" that includes important run-time info, i.e. stats but also log files. |
|
|
|
Case in point - it'd be interesting to find out why FG in junior-s system can drop from 50 fps down to 5. I've never ever seen this, I don't know anyone who has, it'd be important to find out what's the issue and see whether we can address it - but rather than a structured bug report, or anything I could try to help to diagnose, what I see of this is 'FG is becoming unusable, how do I retire from the forum'.
|
|
Poor Quality of Bug Reports
|
|
I'm not a developer so, please, don't expect me to give the detail you want. I'm an end user, that's all. What I know is that there ***IS*** something wrong with FG because it's getting slow over time. What I do know is that a plane with a level of detail of the 777-S should NOT eat that much ram. What I do know is that there are other simulator which renders a lot more scenery than FG and yet they don't eat too much RAM. What I do know is that sims like FSX will make the scenery as low detail as possible when you get far from it. Hell, if you're 500m from a mountain it'll start to get such low polygon count it'll actually look like a 90's game. Does it matter? No, because probably most people don't pay much attention to what's far.
|
|
Proposals
|
|
In cases of flightgear being heavy on resources, i have often seen comments like: "lower the settings". Often and surely temporarily that can be a valid solution. But in some cases i think: do not leave it at that statement, because surely a smart solution can be found.
|
|
FlightGear light ?
|
|
Moreover, I've always been in favour of trying to clearly propose the user a low-end and a high-end version of FG, evolving with the ongoing versions. And I mean clearly, not by having to spend time understanding sliders
|
|
|
|
we simply don't have the manpower to really "produce" too many different versions - people have also been suggesting to come up with versions featuring less complex scenery/aircraft, or possibly an aircraft/scenery-specific "mod", e.g. "HeliGear" or "CombatGear". The thing is, these ideas are good in theory, but they are tying up a lot of manpower.
|
|
|
|
I don't quite see how we could come up with different versions, binaries and packages of FG easily - but we could certainly prepare the main binary to become increasingly configurable, so that it can be used for such purposes - it would even be possible to use built-in FG dialogs in an external window, running in a standalone process - e.g. for an instructor console, but also for a Canvas-driven TerraGear GUI.
|
|
|
|
I looked at the "FGCanvas" branch I started last year to see how many subsystems can be easily made optional these days. Primarily with a focus on aircraft & scenery subsystems - the idea is to put those into dedicated subsystem groups, so that resource utilization can be OPTIONALLY tracked per group. This would make it possible to overload operator new at the SGSubsystem/Group-level to track most allocations and provide /some/ stats.
|
|
Resource Utilization Tracking
|
|
I don't think it's fair to all "blame" it all on more TerraGear progress - the underlying issue is a known one, and we've seen recurring patterns here long before World Scenery 2.0, it's just the concrete manifestation that differs over time, but otherwise the culprit is that we 1) don't provide any optional diagnostics to tell what's going on in terms of CPU/GPU/RAM/VRAM utilization, and 2) that we don't do any headless regression testing when adding new features. So I can kinda relate to people feeling "locked out". We're certainly far from being user-friendly, if people have to be around for half a decade to turn FG into a usable piece of software
|
|
|
|
I also find it more "exciting" to do some fancy Canvas display than fixing the Nasal/GC - and that seems to apply to roughly half a dozen of people around here who obviously have the mental capacity and expertise to address such issues, no matter if it's an alternate Nasal/GC scheme, or an improved threading architecture - still, we're all motivated by things having to be fun, and there's really only a single guy who seems to "enjoy" doing the unglamorous work of fixing other people's ugly code all over the place, and that guy is kinda busy with allowing aircraft to be switched at run-time
|
|
|
|
that's basically a matter of providing some kind of dedicated monitoring function, analogous to the existing system monitor, which also tracks other resources, such as RAM/VRAM - at least optionally.
|
|
|
|
don't agree at all that the question whether we need integrated monitoring mechanisms is a matter of "strategy" as long as we're expecting non-developers to make GOOD bug reports. Obviously someone familiar with Nasal, GLSL and C++ doesn't need such tools, but these people can usually also provide good bug reports without any hand-holding. But the instant we expect our end-users to make useful bug reports, we need to integrate and provide optional diagnostics - analogous to the system/performance monitor or the OSG on-screen stats, otherwise we'll keep on having these discussions for another 5+ years probably. |
|
|
|
I am not interested in providing end-user support, or in dealing with poor bug reports, I am interested in providing people with means to provide us with good information that we can act upon, and I don't want this to be specific about a single technology, let alone a single subsystem like AW/ALS etc, or even require the people who originally wrote the code to be around to make sense of an issue. But I never said anywhere that I wanted to integrate a profiler or debugger necessarily - those are fairly low-level tools and most end-users couldn't deal with them either. In fact, there's an integrated profiler, which while optional, is being used by fellow coders - and I was the one who provided the corresponding patch. Yet, I fully realize that is has not much of a merit when it comes to dealing with end-users. |
|
|
|
I find it kinda short-sighted to limit, or even just associate, the whole debate with TerraGear and scenery development. Anybody who's been around for a few years, knows that we've had a plethora of non-scenery features that ended up gobbling memory, even though I do agree that mentioning random buildings in this context was probably not very smart, I just did so, because it's a feature that has basically zero to do with TG. In other words, we're having issues regardless of TG, and I don't think it's fair to really blame it all on TG, which is a huge, complex and intimidating code base, so I don't envy the few people involved in working with it.
|
|
Feature Scaling
|
|
Well why can we use a profile systems for users like me? ... Example..at the start a option to put all FG options in basic. |
|
|
|
the main challenge is that FlightGear was never designed to do any form of explicit memory management, as per Stuart's comments here:
|
|
|
|
Thus, while the idea to scale features up/down, depending on available resources, the majority of the legacy code in FG (especially anything unrelated to OSG/rendering) would need to be explicitly re-architected to 1) track memory allocations & freeing, and to also adjust subsystems and their features accordingly.
|
|
OpenCL Support
|
|
Also, eventually adopting OpenCL support  would be a great way for some projects like local weather to parallelize more and more code.
|
|
Moving some Nasal code to other cores
|
|
some people have repeatedly expressed an interest in eventually running a weather simulation asynchronously from the main loop, i.e. either multi-threaded or in a separate process (the HLA interface was mentioned by Mathias and Torsten).
|
|
Feature Scaling
|
|
This isn't the first person/contributor to state pretty clearly that they're kinda disappointed with the way FG has evolved over the years, some others stated pretty clearly that even a recent version of X-Plane looks and works better than FG 3.x using the same hardware - so I guess we should take this for what it is, i.e. feedback demonstrating that people no longer know how to scale down FlightGear's features. Admittedly, I also find myself having to do quite a bit of tweaking these days, despite having fairly powerful hardware. Stating that all these are optional features and that people "merely" need to disable those, is as short-sighted as the whole multi-threading debate that we've been having since ~2006 ... Obviously, the "minimal startup profile" should still work for people, and it should easily provide ~60 fps, but that's not what people want FG to look like, and there are other products that simply provide a better experience.
|
|
|
|
So, I'd say we do have a problem, but what people consider the problem is usually just a symptom and not the real culprit. But that should be understandable, most non-coders cannot easily understand all the factors involved here. |
|
|
|
To see for yourself, start up in a place without any scenery, keep FG running for several hours and then see the valgrind report (one of those tools that you refuse to use), while it's true that valgrind outupt is misleading in the context of OSG's ref_ptr's (smart pointers, that manage memory), we do have a number of memory-leaking features, including particles and even effects.
|
|
|
|
Things like a slow Nasal/GC scheme are those that you cannot easily track down, without using tools like a profiler, debugger or leaks checking tool.
|
|
|
|
I never even questioned that our current issue is scenery related, I just said that we've been seeing other issues, and I pointed out that a few of us provided patches to recognize such issues much earlier (the v2.0 scenery issue was RANDOMLY identified by Atlas core developer Brian Schack if you remember). Had we integrated the corresponding resource tracking patches that we discussed (and posted) 4+ years ago, we would have had a much better chance of detecting such issues - even regardless of them being specific to scenery, the Nasal GC or whatever.
|
|
Optional Features
|
|
All it takes to run the recent FG version on old hardware is to not use optional features and dial down the configuration. The project is most definitely not going into any direction making it impossible for some people to use FG - there's huge concern over backward compatibility on the devel list.
|
|
|
|
I mean, in that context FG is just a platform which runs the data you throw at it. Someone decided to develop that 777-S at high detail, and then it consumes more memory. Should the devel team veto detailed aircraft based on the notion that they might not run on all systems? Or could we perhaps trust users to select aircraft based on what runs on their system and what they want to have?
|
|
|
|
I agree wrt to valgrind, specifically if you want to profile, use callgrind and see in pretty colours how much time a piece of code takes to run. Requires lots of patience though.
|
|