Howto:Serializing Nasal data structures

This article is a stub. You can help the wiki by expanding it.

Objective

1. serialization: convert a Nasal data structure (vector, hash, scalar) to a string using valid Nasal syntax
2. deserialization: compile that string to a temporary Nasal function, call it to retrieve the original data structure

var myVector = [1,2,3,4,5];
var serialized = serialize(myVector); # [1,2,3,4,5]
unserialize(serialized);

Background

Nasal's hash syntax is already close enough to JSON that Nasal could even serialize a property tree without requiring much work, analogous to props.getValues() which is already traversing a property tree node and which turns it into a Nasal data structure (hash).

Another possibility would be using a subset of JSON, i.e. to serialize/unserialize Nasal data structures as scalars, vectors and hashes - which should be pretty straightforward. The format would then be plain text, and it would just be evaluated via Nasal's compile()/call() APIs to turn data structures back into Nasal types.

you could then also directly use io.load_nasal() to load a Nasal file from disk (think mydata.nas) and for serialiation purposes, write to the same file - just make sure that it is valid Nasal

depending on how much can be saved using a space-time or time-memory tradeoff, you could compute lookup tables using Nasal vectors/hashes (or both) and serialie those to disk - e.g. using JSON, which is basically equivalent to the syntax used by Nasal. You would basically compute your lookup tables once and then use a nested for-loop to read the lookup tables and serialize/write those to a custom file in $FG_HOME/Export, at that point you can then copy the whole thing into a folder where it can be loaded at runtime using the helpers available in io.nas (using JSON would be much less verbose than using PropertyList/XML, but obviously not as efficient as using a binary serialization format). Basically, this would be equivalent to programmatically creating a complex texture and storing that on disk to save reduce the runtime overhead (treating the texture as a multi-dimensional lookup table). https://en.wikipedia.org/wiki/Space%E2%80%93time_tradeoff

https://en.wikipedia.org/wiki/JSON

Proof of concept

Nasal Console

var serializeVector = func(vector) {
    retVal = "[";
    foreach(var item; vector) {
    retVal ~= serialize(item) ~ ',';
} 

return retVal~"]";
}

var serializeHash = func(hash) {
    var retVal = "{";
    foreach(var member; keys(hash)) {
    retVal ~= serialize(member) ~ ',';

}

return retVal ~"}";
}


var serialize = func (data) {
    var type = typeof(data);
    if (type == "scalar") return data;
    if (type == "vector") return serializeVector(data);
    if (type == "hash") return serializeHash(data);
    if (type == "ghost" or type == "func") return "nil";
    die("cannot serialize unsupported data type:", type);
} 


# create an empty vector
var myVector = [];

# populate the vector with elements with [index, index^2]
for (var i=0;i<=40;i+=1) {
    append(myVector,[i, i*i]);
}



# debug.dump(myVector);

# serialize the data structure to a string that is valid Nasal syntax
var serialized = serialize(myVector);

# debug.dump(serialized);

# next, compile the serialized string to a Nasal function 
var compiled = compile(serialized);
# actually call the function, and store the return value 
var result = compiled();

# dump the deserialized return value to the console
debug.dump(result);

Finally

the original Nasal example, already contains a helper function to serialize a Nasal type:

We can also look at the implementation of debug.dump() and props.dump() respectively.

http://plausible.org/nasal/sample.nas

##
## A rockin' metaprogramming hack.  Generates and returns a deep copy
## of the object in valid Nasal syntax.  A warning to those who might
## want to use this: it ignores function objects (which cannot be
## inspected from Nasal) and replaces them with nil references.  It
## also makes no attempt to escape special characters in strings, which
## can break re-parsing in strange (and possibly insecure!) ways.
##
dump = func(o) {
    result = "";
    if(typeof(o) == "scalar") {
        n = num(o);
        if(n == nil) { result = result ~ '"' ~ o ~ '"'; }
        else { result = result ~ o; }
    } elsif(typeof(o) == "vector") {
        result = result ~ "[ ";
        if(size(o) > 0) { result = result ~ dump(o[0]); }
        for(i=1; i<size(o); i=i+1) {
            result = result ~ ", " ~ dump(o[i]);
        }
        result = result ~ " ]";
    } elsif(typeof(o) == "hash") {
        ks = keys(o);
        result = result ~ "{ ";
        if(size(o) > 0) {
            k = ks[0];
            result = result ~ k ~ ":" ~ dump(o[k]);
        }
        for(i=1; i<size(o); i=i+1) {
            k = ks[i];
            result = result ~ ", " ~ k ~ " : " ~ dump(o[k]);
        }
        result = result ~ " }";
    } else {
        result = result ~ "nil";
    }
    return result;
}

References

References