Nasal Unit Testing Framework: Difference between revisions

Jump to navigation Jump to search
m
clever wiki ...
m (clever wiki ...)
 
(24 intermediate revisions by 5 users not shown)
Line 1: Line 1:
{{infobox subsystem
|image =
|name =Nasal Unit Testing Framework
|started= 02/2014 (stalled)
|description = Unit Testing support for Nasal
|status = RFC
|maintainers  = F-JYL, dbelcham, Hooray, Philosopher
|developers = dbelcham, F-JYL (since 02/2014),
}}
{{Template:Nasal Navigation}}


== Status 06/2013 ==
== Status 06/2013 ==
The intent was to explore the idea of being able to run isolated unit tests against Nasal scripts. The idea was being able to do something like this:
We do not currently have any established unit testing framework for Nasal. For the time being, this whole discussion is just about coming up with the requirements and a possible design.
The intent is to explore the idea of being able to run isolated unit tests against Nasal scripts. The idea was being able to do something like this:
 
See also {{flightgear commit|d7a680}}.


=== Concept ===
=== Concept ===
<pre>
<syntaxhighlight lang="bash">
standalone-nasal.exe ATR72-FMC-tests.nas
standalone-nasal.exe ATR72-FMC-tests.nas
</pre>
</syntaxhighlight>


and see output like this:
and see output like this:
<pre>
<syntaxhighlight>
When_calculating_deflection_that_is_greater_than_the_provided_maximum
When_calculating_deflection_that_is_greater_than_the_provided_maximum
the_provided_maximum_should_be_returned (failed: expected 180 was 99)
the_provided_maximum_should_be_returned (failed: expected 180 was 99)
Line 15: Line 30:
When_calculating_deflection_from_33_degrees_to_90_degrees
When_calculating_deflection_from_33_degrees_to_90_degrees
the_result_should_be_67_degrees (failed: expected 67 was 66)
the_result_should_be_67_degrees (failed: expected 67 was 66)
</pre>
</syntaxhighlight>




=== The standalone Nasal interpreter ===
=== The standalone Nasal interpreter ===
Based on what I read about the current stand-alone interpreter this should be fairly easy to do. That should now be possible, the nasal-standalone branch builds successfully for Windows:
Based on what I read about the current stand-alone interpreter this should be fairly easy to do. That should now be possible, the nasal-standalone branch builds successfully for Windows (make sure to have boost available for building cppbind):


Here's the Nasal standalone interpreter as part of SimGear: https://gitorious.org/~hooray/fg/hoorays-simgear/commits/topics/nasal-standalone
Here's the Nasal standalone interpreter as part of SimGear: {{gitorious source|proj=fg|repo=hoorays-simgear|branch=topics/nasal-standalone|view=shortlog}}
Just check out the branch named "nasal-bin".


To build it:
To build it:
<pre>
<syntaxhighlight lang="bash">
    cd $SG_SRC
cd $SG_SRC
    mkdir BUILD
mkdir BUILD
    cd BUILD
cd BUILD
    cmake ../ -DENABLE_TESTS=ON -DSIMGEAR_HEADLESS=ON -DENABLE_SOUND=OFF -D_WIN32=1
cmake ../ -DENABLE_TESTS=ON -DSIMGEAR_HEADLESS=ON -DENABLE_SOUND=OFF -DENABLE_LIBSVN=OFF
    make
make
</pre>
</syntaxhighlight>
 
People not using git, can also directly download the source code: https://gitorious.org/fg/hoorays-simgear/archive-tarball/topics/nasal-standalone
(Use 7Zip or IZArc to extract the tarball. Then follow the build instructions (you can also use the cmake GUI for those).


Basically
Basically


*create and use a separate build folder, separate from the source tree
* create and use a separate build folder, separate from the source tree
* configure the build via -DENABLE_TESTS=ON -DSIMGEAR_HEADLESS=ON -DENABLE_SOUND=OFF -D_WIN32=1
* configure the build via -DENABLE_TESTS=ON -DSIMGEAR_HEADLESS=ON -DENABLE_SOUND=OFF -DENABLE_LIBSVN=OFF
* build
* build
* report back  
* report back  


Let us know if there are still any windows-specific build errors so that we can fix the config file. It should give you a "nasal-bin.exe" in $SG_BUILD/simgear/nasal/ that runs just Nasal scripts, no FG APIs whatsoever.
Note that there's no need to actually install anything (make install), because we are just using the SimGear library to build a standalone nasal-bin binary, nothing else.
 
Let us know if there are still any windows-specific build errors so that we can fix the config file. It should give you a "nasal-bin.exe" in $SG_BUILD/simgear/nasal/ that runs just Nasal scripts, no FG APIs whatsoever - you '''need''' to pass a valid Nasal script when running the file.


You should be able to "make test" to run a bunch of standard Nasal tests from the original Nasal repository.
You should be able to "make test" to run a bunch of standard Nasal tests from the original Nasal repository, which are to be found in $SG_SRC/nasal/tests: {{gitorious source|proj=fg|repo=hoorays-simgear|view=tree|branch=topics/nasal-standalone|path=simgear/nasal/tests}}


Okay, I have checked that the build actually works for Windows using the MingW compiler - I am getting a nasal-bin.exe, which I cannot test currently because I don't have a Windows VM available: http://www.speedyshare.com/MbBTA/download/nasal-bin.exe
Meanwhile, the build actually works for Windows using the MingW compiler - providing a nasal-bin.exe, which I cannot test currently because I don't have a Windows VM available: http://www.speedyshare.com/MbBTA/download/nasal-bin.exe{{dead link}}


To run it, open a shell (START/EXECUTE cmd/command) and go to the folder of the binary, add a simple Nasal script and run "nasal-bin script.nas"
To run it, open a shell (START/EXECUTE cmd/command) and go to the folder of the binary, add a simple Nasal script and run "nasal-bin script.nas", i.e. use one of the scripts in: {{gitorious source|proj=fg|repo=hoorays-simgear|view=tree|branch=topics/nasal-standalone|path=simgear/nasal/tests}}


=== Roadmap ===
=== Roadmap ===
Line 54: Line 69:
* build a set of Nasal scripts that provide stubs for native fg calls (getprop/setprop/etc) {{Not done}}
* build a set of Nasal scripts that provide stubs for native fg calls (getprop/setprop/etc) {{Not done}}
* build out testing script with the ability to verify values and report failures to the console {{Not done}}
* build out testing script with the ability to verify values and report failures to the console {{Not done}}
* send a patch upstream, so that a standalone Nasal interpreter is included in each upcoming release {{Not done}}


It should be doable to teach the nasal-bin.exe to check $FG_ROOT, and use that if available to load a semi-plausible FG environment (API-wise) - using some fancy metaprogramming tricks, most of the default APIs could probably be wrapped, without too much manual work involved. Philosopher could be truly instrumental here, because he really has a deep understanding of some of the more esoteric tricks that can be done in Nasal space, referring to advanced uses of compile(), bind(), call(), closure() and caller() - which make metaprogramming a fantastic experience. Basically, familiarity with this handful of APIs, can save tons of time: http://plausible.org/nasal/
Once we have those three things we would be able to write and execute tests independent of FG and still have them be meaningful. The key thing to remember is that this would be only for isolated unit tests. For integration tests (verifying that different systems, whether 2 different scripts/methods/application, work together correctly) we would need to think about a different approach.


There's quite a lot of stuff possible in Nasal, that nobody ever used in FG - Philosopher has started writing a bunch of tutorials, for example see: http://wiki.flightgear.org/Nasal_Meta-Programming
It should be doable to teach the nasal-bin.exe to check $FG_ROOT, and use that if available to load a semi-plausible FG environment (API-wise) - using some fancy meta-programming tricks, most of the default APIs could probably be wrapped, without too much manual work involved. Philosopher could be truly instrumental here, because he really has a deep understanding of some of the more esoteric tricks that can be done in Nasal space, referring to advanced uses of compile(), bind(), call(), closure() and caller() - which make meta-programming a fantastic experience. Basically, familiarity with this handful of APIs, can save tons of time: http://plausible.org/nasal/


And you can take a look at some of the scripts in the standalone branch, which support fancy constructs like dependency resolution using import("foo"); but also completely sandboxed/wrapped environments: https://gitorious.org/~hooray/fg/hoorays-simgear/trees/topics/nasal-standalone/simgear/nasal/tests
There's quite a lot of stuff possible in Nasal, that nobody ever used in FG - Philosopher has started writing a bunch of tutorials, for example see: [[Nasal Meta-Programming]]
 
And you can take a look at some of the scripts in the standalone branch, which support fancy constructs like dependency resolution using import("foo"); but also completely sandboxed/wrapped environments: {{gitorious source|proj=fg|repo=hoorays-simgear|view=tree|branch=topics/nasal-standalone|path=simgear/nasal/tests}}


These are regression tests developed by the Nasal developer himself, so not true unit testing - but only regression tests for the interpreter itself.
These are regression tests developed by the Nasal developer himself, so not true unit testing - but only regression tests for the interpreter itself.


=== Contributing ===
=== Contributing ===
If possible, new code should be contributed through gitorious, ideally even a branch of FG_ROOT, because that will make it easier to direclty integrate such a unit testing system with all the code we got in $FG_ROOT/Nasal.
If possible, new code should be contributed to the maintainers, ideally even a branch of FG_ROOT, because that will make it easier to directly integrate such a unit testing system with all the code we got in $FG_ROOT/Nasal.


== Problem ==
== Problem ==
Line 81: Line 99:


Wrapping APIs is simple to do in Nasal, too - without even requiring C/C++ changes, a standalone testbed could be scripted in Nasal like this:
Wrapping APIs is simple to do in Nasal, too - without even requiring C/C++ changes, a standalone testbed could be scripted in Nasal like this:
<syntaxhighlight lang="php">
<syntaxhighlight lang="nasal">
 
var tree = {};
var tree = {};


var isalpha = func(n) n >= `a` and n <= `z` or n >= `A` and n <= `Z`;
var isdigit = func(n) n >= `0` and n <= `9`;
var sanitize = func(p) {
    if (!p) die();
    if (p[0] == `/`) p = substr(p, 1, nil);
    parts = split("/", p);
    for (var i=0; i<size(parts); i+=1) {
        if (parts[i] == "") {
            if (i == size(parts)-1)
                parts = parts[:i-1];
            else parts = parts[:i-1] ~ parts[i+1:];
            i-=1;
        } else {
            for (var j=0; j<size(parts[i]); j+=1) {
                if (parts[i][j] == `[`) break;
                if (parts[i][j] != `-` and !isalpha(parts[i][j]) and
                    parts[i][j] != `_` and !isdigit(parts[i][j]) and
                    parts[i][j] != `.`) die("bad character in name "~parts[i]~" at index "~j~".");
            }
            if (j == size(parts[i]))
                parts[i] ~= "[0]";
            elsif (parts[i][-1] != `]`) die("bad index specifier in string "~parts[i]~".")
        }
    }
    var p = "/";
    foreach (var part; parts)
        p ~= part;
    return p;
}
# wrappers for the FG setprop/getprop APIs:
# wrappers for the FG setprop/getprop APIs:
var setprop = func(p, value) tree[p] = value;
var setprop = func(p, value) tree[ sanitize(p) ] = value;
var getprop = func(p) return tree[p] or 0.00;
var getprop = func(p) return tree[ sanitize(p) ];
# some tests:
 
var path = ["/foo/bar", "/foo[0]/bar[0]","/foo[0]/bar[0]/", "/foo/bar[0]","/foo[0]/bar/","/foo[0]/bar/"];
var value = "MyUniqueValue";
setprop(path[0], value);
foreach(var p; path)
  if (getprop(p) != value) die("sanitize() implementation is broken");
print("sanitize() looks good!\n");


# init your tree:
# init your tree:
Line 96: Line 155:


We would need to use custom script-specific wrappers, instead of the main FG/Nasal APIs and modules - so that your Nasal code *never* uses the APIs directly, that way you can easily have different implementations - i.e.
We would need to use custom script-specific wrappers, instead of the main FG/Nasal APIs and modules - so that your Nasal code *never* uses the APIs directly, that way you can easily have different implementations - i.e.
<syntaxhighlight lang="php">
<syntaxhighlight lang="nasal">
var debug_profile = {};
var debug_profile = {};
var runtime_profile = {};
var runtime_profile = {};
Line 110: Line 169:


print( current_profile.systime() );
print( current_profile.systime() );
# or simply override the global symbols during initialization:
var systime = current_profile.systime;
print( systime() );
</syntaxhighlight>
</syntaxhighlight>


Line 123: Line 187:
Ultimately what I'd like to have is an implementation of something like the jUnit/xUnit/nUnit testing frameworks. Tonight's goal will be to hack out a rough implementation that allows for isolation of the property tree.
Ultimately what I'd like to have is an implementation of something like the jUnit/xUnit/nUnit testing frameworks. Tonight's goal will be to hack out a rough implementation that allows for isolation of the property tree.


== Integration ==
== Integration & Adoption ==
Unit testing support in Nasal would certainly be beneficial - but it would need to be added to FG at a library-level, i.e. in $FG_ROOT/Nasal, so that people have to use it, and have an advantage when using it - sort of like the RoR example you mentioned previously.
Unit testing support in Nasal would certainly be beneficial - but it would need to be added to FG at a library-level, i.e. in $FG_ROOT/Nasal, so that people have to use it, and have an advantage when using it - sort of like the RoR example you mentioned previously.


Line 131: Line 195:


People would only be likely to actually use that if they have a corresponding developers background, so it would need to be designed right into the framework and touch lots of places in $FG_ROOT and $FG_AIRCRAFT - I only see a handful of aircraft developers here who would go that route and actually have the mental capacity, and developer mentality to see the merits here.
People would only be likely to actually use that if they have a corresponding developers background, so it would need to be designed right into the framework and touch lots of places in $FG_ROOT and $FG_AIRCRAFT - I only see a handful of aircraft developers here who would go that route and actually have the mental capacity, and developer mentality to see the merits here.
Probably,a handful of people would be able to use it, but if it's well documented, and if it actually supports features not provided otherwise, it could gain traction - so it would need to be more compelling than the current workflow obviously.
A unit testing framework is definitely going to be useful for $FG_ROOT as a whole, not just aircraft/instrument developers. Obviously, one of the first steps will be documenting the whole thing with tutorials, so that people can start adopting it.

Navigation menu