Difference between revisions of "Nasal performance and benchmarks"

From FlightGear wiki
Jump to: navigation, search
m (References: add pointers to related articles/efforts, some of these might actually be helpful for anyone working on this in the future)
m (Regression tests: https://sourceforge.net/p/flightgear/mailman/attachment/4D8CF843.9080302%40gmail.com/2/)
 
(12 intermediate revisions by 2 users not shown)
Line 7: Line 7:
  
 
= Benchmark add-on =
 
= Benchmark add-on =
The benchmark add-on will be published [https://sourceforge.net/p/flightgear/fgaddon/HEAD/tree/trunk/Addons/Benchmark here].
+
The benchmark add-on can be downloaded [https://sourceforge.net/p/flightgear/fgaddon/HEAD/tree/trunk/Addons/Benchmark here].
It is basically a collection of nasal snippets to be tested.
+
It is basically a collection of nasal snippets to be tested.  
 +
A minimal GUI (menu item, simple dialog box) allows to start some benchmarks.
 +
 
 +
A benchmark test runs some function multiple time (repeat count) and measures the time taken (result).
 +
The results of any benchmark is currently printed to the console, so you have to start FlightGear with console window to see this.
 +
(The repeat count should be configurable via the GUI, this is on my to do list.)
  
 
= Performance analysis =
 
= Performance analysis =
== 2019-01 ==
+
Performance of Nasal snippets is tested by means of the Benchmark add-on which makes use of the debug.benchmark_time() function. The default repeat count for tests is 10000 (ten thousand). Depending on your hardware some tests may block your Flightgear many seconds. On Windows you may see the usual warnings about FlightGear not reacting any more.
to be written
+
In the next section available benchmarks are described. Test results follow below.
  
-->[[Category:Nasal]]
+
Results (time taken to complete a benchmark test) obviously depend on the hardware used, so the relevant information is not the absolute time but the ratio when comparing two code snippets.
 +
Also, one should repeat benchmarks a couple of times and see if results are in the same order of magnitude. Benchmarks should be run only after FlightGear is fully ready, e.g. scenery and aircraft model loaded etc.
  
== References ==
+
== currently available benchmarks ==
  
=== Articles ===
+
=== simpleClass ===
 +
Object oriented programming (OOP, "programming with classes") gives several advantages (readability, re-usability and easier maintainance of code, to give just some examples) but might generate slightly slower code.
 +
For most real-world cases this should be neglectible, nevertheless it is a simple, easy to understand test scenario.
 +
 
 +
<syntaxhighlight lang="nasal">
 +
var simpleClass = {
 +
    new: func {
 +
        var obj = {
 +
            parents: [simpleClass],
 +
            value: 0,
 +
        };
 +
        return obj;
 +
    },
 +
   
 +
    set: func(value) { me.value = value; },
 +
    get: func { return me.value; },
 +
};
 +
 
 +
var c = simpleClass.new();
 +
 
 +
var test_get_method = func {
 +
    var x = c.get();
 +
}
 +
var test_get_direct = func {
 +
    var x = c.value;
 +
}
 +
var test_set_method = func {
 +
    c.set(1);
 +
}
 +
var test_set_direct = func {
 +
    c.value = 1;
 +
}
 +
</syntaxhighlight>
 +
 
 +
=== property tree access ===
 +
As the property tree is the central data exchange in FlightGear it is worth to know how to efficiently access it.
 +
This benchmarks compares the getprop() agains the props.Node.getValue().
 +
 
 +
<syntaxhighlight lang="nasal">
 +
var path = "/_benchmark/getprop/prop";
 +
var myprops = [];
 +
append(myprops, props.getNode(path, 1));
 +
 
 +
var test_getvalue = func {
 +
    myprops[0].getValue();
 +
}
 +
var test_setvalue = func {
 +
    myprops[0].setValue(21);
 +
}
 +
 
 +
var test_getprop = func {
 +
    getprop(path);
 +
}
 +
var test_setprop = func {
 +
    setprop(path, 42);
 +
}
 +
 
 +
var init = func {
 +
    for (var i = 0; i < repeat; i += 1) {
 +
        append(myprops, props.getNode(path~i, 1));
 +
    }
 +
}
 +
printResult("test_getNode", debug.benchmark_time(init, repeat));
 +
printResult("test_getvalue", debug.benchmark_time(test_getvalue, repeat));
 +
printResult("test_getprop", debug.benchmark_time(test_getprop, repeat));
 +
</syntaxhighlight>
 +
 
 +
=== Canvas API - Text class ===
 +
tbd
 +
 
 +
== Results 2019-01 ==
 +
 
 +
=== simpleClass ===
 +
Repeat count = 10000
 +
 
 +
Interpretation: using class methods is fine.
 +
{| class="wikitable"
 +
|-
 +
! Test !! Time [s] !! Comment
 +
|-
 +
| get via class.get() || 0.015 ||
 +
|-
 +
| get via class.variable || 0.006 ||
 +
|-
 +
| set via class.set()|| 0.011 ||
 +
|-
 +
| set via class.variable || 0.004 ||
 +
|}
 +
 
 +
=== Property access ===
 +
Repeat count = 100,000
 +
 
 +
It seems that any operation involving a property path (string) is relativly slow.
 +
If you have to access a property more than once, e.g. in an update loop, you should create a props.Node object and use object.getValue().
 +
 
 +
{| class="wikitable"
 +
|-
 +
! Test !! Time [s] !! Comment
 +
|-
 +
| node.setValue() || 0.126 || ~25% faster
 +
|-
 +
| node.getValue() || 0.125 || ~25% faster
 +
|-
 +
| setprop() || 0.178 ||
 +
|-
 +
| getprop() || 0.164 ||
 +
|}
 +
 
 +
=== Canvas API - Text class ===
 +
Optimized canvas API with caching and props.Node (not merged to FGDATA at the time of writing). Older implementation below for reference.
 +
Repeat = 50,000
 +
 
 +
{| class="wikitable"
 +
|-
 +
! Test !! Time [s] !! Comment
 +
|-
 +
| setText() same value || 0.046 ||
 +
|-
 +
| setText() different value || 0.561 ||
 +
|-
 +
| Old
 +
|-
 +
| setText() same value || 0.340 ||
 +
|-
 +
| setText() different value || 0.692 ||
 +
|-
 +
| setTextFast() same value || 0.192 ||
 +
|-
 +
| setTextFast() different value || 0.541 ||
 +
|-
 +
| updateText() same value || 0.050 ||
 +
|-
 +
| updateText() different value || 0.704 ||
 +
|}
 +
 
 +
[[Category:Nasal]]
 +
 
 +
= References =
 +
 
 +
== Regression tests ==
 +
* [https://github.com/andyross/nasal/tree/master/misc original regression tests (Andy Ross)]
 +
* [http://sourceforge.net/p/flightgear/mailman/message/27262122/ GC tracking discussion] (ThorstenB, [https://sourceforge.net/p/flightgear/mailman/attachment/4D8CF843.9080302%40gmail.com/2/ patch])
 +
 
 +
== Articles ==
 
* [[Howto:Canvas Path Benchmarking]]
 
* [[Howto:Canvas Path Benchmarking]]
 
* [[FlightGear Benchmark]]
 
* [[FlightGear Benchmark]]
 +
* [[Testing]]
 
* [[Feature Scaling]]
 
* [[Feature Scaling]]
 
* [[Built-in Profiler]]
 
* [[Built-in Profiler]]
 +
* [[Resource Tracking for FlightGear]]
 +
* [[How the Nasal GC works]]
  
=== See also ===
+
== See also ==
 
+
* [[User:Philosopher/Howto:Write Optimized Nasal Code]]
 
* [[User:Philosopher/Nasal introspection]]
 
* [[User:Philosopher/Nasal introspection]]
 
* [[User:Philosopher/Optimization findings]]
 
* [[User:Philosopher/Optimization findings]]
 
* [[Improving Nasal]]
 
* [[Improving Nasal]]
 +
* [[Nasal Maintenance]]
  
 
=== Discussions ===
 
=== Discussions ===
  
 
{{Appendix}}
 
{{Appendix}}

Latest revision as of 14:54, 28 January 2019

This article is a stub. You can help the wiki by expanding it.

This article shall collect best practices for coding in Nasal based on regular performance tests.

Motivation

Over the years many things have been improved so some old findings regarding performance are not true anymore. Regular benchmarking and updating should help the FlightGear community to have a clear picture of which code constructs perform well. Sometimes performance and "good" coding style (e.g. readability, re-usability, maintainability) might be in conflict.

Benchmark add-on

The benchmark add-on can be downloaded here. It is basically a collection of nasal snippets to be tested. A minimal GUI (menu item, simple dialog box) allows to start some benchmarks.

A benchmark test runs some function multiple time (repeat count) and measures the time taken (result). The results of any benchmark is currently printed to the console, so you have to start FlightGear with console window to see this. (The repeat count should be configurable via the GUI, this is on my to do list.)

Performance analysis

Performance of Nasal snippets is tested by means of the Benchmark add-on which makes use of the debug.benchmark_time() function. The default repeat count for tests is 10000 (ten thousand). Depending on your hardware some tests may block your Flightgear many seconds. On Windows you may see the usual warnings about FlightGear not reacting any more. In the next section available benchmarks are described. Test results follow below.

Results (time taken to complete a benchmark test) obviously depend on the hardware used, so the relevant information is not the absolute time but the ratio when comparing two code snippets. Also, one should repeat benchmarks a couple of times and see if results are in the same order of magnitude. Benchmarks should be run only after FlightGear is fully ready, e.g. scenery and aircraft model loaded etc.

currently available benchmarks

simpleClass

Object oriented programming (OOP, "programming with classes") gives several advantages (readability, re-usability and easier maintainance of code, to give just some examples) but might generate slightly slower code. For most real-world cases this should be neglectible, nevertheless it is a simple, easy to understand test scenario.

var simpleClass = {
    new: func {
        var obj = {
            parents: [simpleClass],
            value: 0,
        };
        return obj;
    },
    
    set: func(value) { me.value = value; },
    get: func { return me.value; },
};

var c = simpleClass.new();

var test_get_method = func {
    var x = c.get();
}
var test_get_direct = func {
    var x = c.value;
}
var test_set_method = func {
    c.set(1);
}
var test_set_direct = func {
    c.value = 1;
}

property tree access

As the property tree is the central data exchange in FlightGear it is worth to know how to efficiently access it. This benchmarks compares the getprop() agains the props.Node.getValue().

var path = "/_benchmark/getprop/prop";
var myprops = [];
append(myprops, props.getNode(path, 1));

var test_getvalue = func {
    myprops[0].getValue();
}
var test_setvalue = func {
    myprops[0].setValue(21);
}

var test_getprop = func {
    getprop(path);
}
var test_setprop = func {
    setprop(path, 42);
}

var init = func {
    for (var i = 0; i < repeat; i += 1) {
        append(myprops, props.getNode(path~i, 1));
    }
}
printResult("test_getNode", debug.benchmark_time(init, repeat));
printResult("test_getvalue", debug.benchmark_time(test_getvalue, repeat));
printResult("test_getprop", debug.benchmark_time(test_getprop, repeat));

Canvas API - Text class

tbd

Results 2019-01

simpleClass

Repeat count = 10000

Interpretation: using class methods is fine.

Test Time [s] Comment
get via class.get() 0.015
get via class.variable 0.006
set via class.set() 0.011
set via class.variable 0.004

Property access

Repeat count = 100,000

It seems that any operation involving a property path (string) is relativly slow. If you have to access a property more than once, e.g. in an update loop, you should create a props.Node object and use object.getValue().

Test Time [s] Comment
node.setValue() 0.126 ~25% faster
node.getValue() 0.125 ~25% faster
setprop() 0.178
getprop() 0.164

Canvas API - Text class

Optimized canvas API with caching and props.Node (not merged to FGDATA at the time of writing). Older implementation below for reference. Repeat = 50,000

Test Time [s] Comment
setText() same value 0.046
setText() different value 0.561
Old
setText() same value 0.340
setText() different value 0.692
setTextFast() same value 0.192
setTextFast() different value 0.541
updateText() same value 0.050
updateText() different value 0.704

References

Regression tests

Articles

See also

Discussions

References