Howto:Serializing Nasal data structures: Difference between revisions

From FlightGear wiki
Jump to navigation Jump to search
(Undo revision 117325 by Necolatis (talk))
 
(6 intermediate revisions by 2 users not shown)
Line 1: Line 1:
{{Stub}}
{{Stub}}
== Objective ==
== Objective ==
# serialization: convert a Nasal data structure to a string using valid Nasal syntax
# serialization: convert a Nasal data structure (vector, hash, scalar) to a string using valid Nasal syntax
# deserialization: compile that string to a temporary Nasal function, call it to retrieve the original data structure
# deserialization: compile that string to a temporary Nasal function, call it to retrieve the original data structure


Line 9: Line 9:
var serialized = serialize(myVector); # [1,2,3,4,5]
var serialized = serialize(myVector); # [1,2,3,4,5]
unserialize(serialized);
unserialize(serialized);
<syntaxhighlight>
</syntaxhighlight>


== Background ==
== Background ==


Nasal's hash syntax is already close enough to JSON that Nasal could even serialie a property tree without requiring much work, analogous to props.getValues() which is already traversing a property tree node and which turns it into a Nasal data structure (hash).
Nasal's hash syntax is already close enough to JSON that Nasal could even serialize a property tree without requiring much work, analogous to props.getValues() which is already traversing a property tree node and which turns it into a Nasal data structure (hash).


<ref>{{cite web
<ref>{{cite web
Line 66: Line 66:


var serializeVector = func(vector) {
var serializeVector = func(vector) {
retVal = "[";
    retVal = "[";
foreach(var item; vector) {
    foreach(var item; vector) {
retVal ~= serialize(item) ~ ',';
    retVal ~= serialize(item) ~ ',';
}  
}  


Line 75: Line 75:


var serializeHash = func(hash) {
var serializeHash = func(hash) {
var retVal = "{";
    var retVal = "{";
foreach(var member; keys(hash)) {
    foreach(var member; keys(hash)) {
retVal ~= serialize(member) ~ ',';
    retVal ~= serialize(member) ~ ',';


}
}
Line 86: Line 86:


var serialize = func (data) {
var serialize = func (data) {
var type = typeof(data);
    var type = typeof(data);
if (type == "scalar") return data;
    if (type == "scalar") return data;
if (type == "vector") return serializeVector(data);
    if (type == "vector") return serializeVector(data);
if (type == "hash") return serializeHash(data);
    if (type == "hash") return serializeHash(data);
if (type == "ghost" or type == "func") return "nil";
    if (type == "ghost" or type == "func") return "nil";
die("cannot serialize unsupported data type:", type);
    die("cannot serialize unsupported data type:", type);
}  
}  


Line 100: Line 100:
# populate the vector with elements with [index, index^2]
# populate the vector with elements with [index, index^2]
for (var i=0;i<=40;i+=1) {
for (var i=0;i<=40;i+=1) {
append(myVector,[i, i*i]);
    append(myVector,[i, i*i]);
}
}


Line 123: Line 123:


== Finally ==
== Finally ==
the original Nasal example, already contains a helper function to serialie a Nasal type:<ref>{{cite web
the original Nasal example, already contains a helper function to serialize a Nasal type:<ref>{{cite web
   |url    =  https://forum.flightgear.org/viewtopic.php?p=257123#p257123  
   |url    =  https://forum.flightgear.org/viewtopic.php?p=257123#p257123  
   |title  =  <nowiki> Re: String manipulation and file I/O (pm2thread) </nowiki>  
   |title  =  <nowiki> Re: String manipulation and file I/O (pm2thread) </nowiki>  
Line 131: Line 131:
   |script_version = 0.40  
   |script_version = 0.40  
   }}</ref>
   }}</ref>
We can also look at the implementation of debug.dump() and props.dump() respectively.


http://plausible.org/nasal/sample.nas
http://plausible.org/nasal/sample.nas
Line 172: Line 174:
}
}
</syntaxhighlight>
</syntaxhighlight>


== References ==
== References ==
{{Appendix}}
{{Appendix}}

Latest revision as of 21:41, 10 February 2019

This article is a stub. You can help the wiki by expanding it.

Objective

  1. serialization: convert a Nasal data structure (vector, hash, scalar) to a string using valid Nasal syntax
  2. deserialization: compile that string to a temporary Nasal function, call it to retrieve the original data structure


var myVector = [1,2,3,4,5];
var serialized = serialize(myVector); # [1,2,3,4,5]
unserialize(serialized);

Background

Nasal's hash syntax is already close enough to JSON that Nasal could even serialize a property tree without requiring much work, analogous to props.getValues() which is already traversing a property tree node and which turns it into a Nasal data structure (hash).

[1]


Another possibility would be using a subset of JSON, i.e. to serialize/unserialize Nasal data structures as scalars, vectors and hashes - which should be pretty straightforward. The format would then be plain text, and it would just be evaluated via Nasal's compile()/call() APIs to turn data structures back into Nasal types.[2]

you could then also directly use io.load_nasal() to load a Nasal file from disk (think mydata.nas) and for serialiation purposes, write to the same file - just make sure that it is valid Nasal[3]


depending on how much can be saved using a space-time or time-memory tradeoff, you could compute lookup tables using Nasal vectors/hashes (or both) and serialie those to disk - e.g. using JSON, which is basically equivalent to the syntax used by Nasal. You would basically compute your lookup tables once and then use a nested for-loop to read the lookup tables and serialize/write those to a custom file in $FG_HOME/Export, at that point you can then copy the whole thing into a folder where it can be loaded at runtime using the helpers available in io.nas (using JSON would be much less verbose than using PropertyList/XML, but obviously not as efficient as using a binary serialization format). Basically, this would be equivalent to programmatically creating a complex texture and storing that on disk to save reduce the runtime overhead (treating the texture as a multi-dimensional lookup table). https://en.wikipedia.org/wiki/Space%E2%80%93time_tradeoff

https://en.wikipedia.org/wiki/JSON[4]

Proof of concept

Nasal Console

var serializeVector = func(vector) {
    retVal = "[";
    foreach(var item; vector) {
    retVal ~= serialize(item) ~ ',';
} 

return retVal~"]";
}

var serializeHash = func(hash) {
    var retVal = "{";
    foreach(var member; keys(hash)) {
    retVal ~= serialize(member) ~ ',';

}

return retVal ~"}";
}


var serialize = func (data) {
    var type = typeof(data);
    if (type == "scalar") return data;
    if (type == "vector") return serializeVector(data);
    if (type == "hash") return serializeHash(data);
    if (type == "ghost" or type == "func") return "nil";
    die("cannot serialize unsupported data type:", type);
} 


# create an empty vector
var myVector = [];

# populate the vector with elements with [index, index^2]
for (var i=0;i<=40;i+=1) {
    append(myVector,[i, i*i]);
}



# debug.dump(myVector);

# serialize the data structure to a string that is valid Nasal syntax
var serialized = serialize(myVector);

# debug.dump(serialized);

# next, compile the serialized string to a Nasal function 
var compiled = compile(serialized);
# actually call the function, and store the return value 
var result = compiled();

# dump the deserialized return value to the console
debug.dump(result);

Finally

the original Nasal example, already contains a helper function to serialize a Nasal type:[5]

We can also look at the implementation of debug.dump() and props.dump() respectively.

http://plausible.org/nasal/sample.nas

##
## A rockin' metaprogramming hack.  Generates and returns a deep copy
## of the object in valid Nasal syntax.  A warning to those who might
## want to use this: it ignores function objects (which cannot be
## inspected from Nasal) and replaces them with nil references.  It
## also makes no attempt to escape special characters in strings, which
## can break re-parsing in strange (and possibly insecure!) ways.
##
dump = func(o) {
    result = "";
    if(typeof(o) == "scalar") {
        n = num(o);
        if(n == nil) { result = result ~ '"' ~ o ~ '"'; }
        else { result = result ~ o; }
    } elsif(typeof(o) == "vector") {
        result = result ~ "[ ";
        if(size(o) > 0) { result = result ~ dump(o[0]); }
        for(i=1; i<size(o); i=i+1) {
            result = result ~ ", " ~ dump(o[i]);
        }
        result = result ~ " ]";
    } elsif(typeof(o) == "hash") {
        ks = keys(o);
        result = result ~ "{ ";
        if(size(o) > 0) {
            k = ks[0];
            result = result ~ k ~ ":" ~ dump(o[k]);
        }
        for(i=1; i<size(o); i=i+1) {
            k = ks[i];
            result = result ~ ", " ~ k ~ " : " ~ dump(o[k]);
        }
        result = result ~ " }";
    } else {
        result = result ~ "nil";
    }
    return result;
}

References

References
  1. Hooray  (Oct 9th, 2016).  Re: Why not use Erlang instead of .
  2. Hooray  (Sep 12th, 2015).  Re: String manipulation and file I/O (pm2thread) .
  3. Hooray  (Sep 12th, 2015).  Re: String manipulation and file I/O (pm2thread) .
  4. Hooray  (Aug 12th, 2017).  Re: Space Shuttle .
  5. Hooray  (Sep 12th, 2015).  Re: String manipulation and file I/O (pm2thread) .