Value Reification

What is the difference between a: 4 and b: !int & >3 & <5? The former explicitly assigns 4 to a, while the latter limits b’s acceptable values to integers greater than three and less than five. But are they equivalent?

While it can help to think of constraint expressions as miniature validator functions when you’re first starting out, the truth is that Vada considers constraints to be no different than any other value or expression. Every expression in Vada is, fundamentally, a description of a set and all set descriptions are values.

The difference between 4 and !int & >3 & <5 isn’t that one is a value and the other is a constraint; they’re both expressions that describe sets and they’re even descriptions of the same set. The difference is their concreteness. Both describe 4, but where a is a concrete description of 4, b is an abstract description of 4.

If you wanted to compare a and b, you’d first have to extract a list of b’s possible values from its upper and lower bounds, and then collapse that list to a uniform in order to derive a value that has the same concreteness and container as a.

In order words, you’d have to reify b in order to compare it to a.

A core pillar of Vada’s design is the assertion that all valid expressions produce sets which are members of a partially ordered superset that forms the language’s type lattice.^00-2

Value reification is the process of converting an upper (abstract) member of the lattice into an equivalent lower (concrete) member of the lattice by explicitly enumerating upper’s inner members into an appropriate container that reflects its topology.

Reifying an abstract value is comparable to traversing the greater type lattice to find the abstract value’s concrete pair, affirming its position in a secondary lattice of concretely-representable types. Abstract values that can’t be reified (e.g. infinite expressions such as >0 and !=2) have no concrete pairs, and are therefore excluded from the secondary lattice.

Semantic Interfaces

Another pillar of Vada’s design is the use of semantic interfaces (Sections) to create parallel access patterns.

someRecord: {
    --- SectionA:
    fieldA: 123
    fieldB: "abc"
    fieldC: (0.2, -0.05, 1.3)
    --- SectionB:
    fieldD: true
    fieldE: "jimmy"
}

Sections group records together without nesting them inside a struct. fieldA in the struct above is accessible through both someRecord.fieldA and someRecord::SectionA::fieldA

In the above record, all of the fields can be accessed directly as members or through Sections

This is primarily accomplished through the use of Sections. A section is something like a cross between an interface and a soft namespace, augmented with a grab-bag of things like inherent topology, captures, and field constraints.

First-Class Spatial Data

If you’re a MATLAB fan, Vada’s syntax for multi-dimensional uniforms should feel fairly familiar, as it uses comma-separated rows and semi-colon separated columns, extended with a sensible adjacent circumfix syntax stacking the outermost axis.^00-3

Vada is zero-indexed, but its indexing syntax differs slightly from most other languages; rather than the typical someVar[] syntax, Vada uses someVar.[], treating indices as members.

This syntax applies to uniforms, lists, and some fancy stuff like struct filtering and record destructuring expressions. The grammar’s a bit odd on the face of it, but it lets you do some cool stuff once you get used to it.

It’s also worth noting that Vada uses axis indexing, where the dimensionality of the index expression matches the dimensionality of the record and slices are rotated.

AKA, this:

someCube: ( 1.1, 2.1, 3.1;
            4.1, 5.1, 6.1;
            7.1, 8.1, 9.1 )
          ( 1.2, 2.2, 3.2;
            4.2, 5.2, 6.2;
            7.2, 8.2, 9.2 )
          ( 1.3, 2.3, 3.3;
            4.3, 5.3, 6.3;
            7.3, 8.3, 9.3 )

// Rotated from XZ to XY
xPlane: someCube.[0][ ][ ] // 1.1, 2.1, 3.1;
                           // 1.2, 2.2, 3.2;
                           // 1.3, 2.3, 3.3

// Rotated from YZ to XY
yPlane: someCube.[ ][0][ ] // 1.1, 4.1, 7.1;
                           // 1.2, 4.2, 7.2;
                           // 1.3, 4.3, 7.3

// Inherently XY oriented
zPlane: someCube.[ ][ ][0] // 1.1, 2.1, 3.1;
                           // 4.1, 5.1, 6.1;
                           // 7.1, 8.1, 9.1

Combined with prefix inversion ~someVec, prefix transposition %someVec, infix wedge product someVec ^^ anotherVec, infix cross product someVec ** anotherVec, and piped swizzle-reshape someVec->(1,0,2), Vada’s approach to multi-dimensional data removes mountains of boilerplate code for game development.

hCube: 
  { 
    [ (1.0, 2.0, 3.0)(4.0, 5.0, 6.0)(7.0, 8.0, 9.0) ]
    [ (1.1, 2.1, 3.1)(4.1, 5.1, 6.1)(7.1, 8.1, 9.1) ]
    [ (1.2, 2.2, 3.2)(4.2, 5.2, 6.2)(7.2, 8.2, 9.2) ] }
  { 
    [ (1.3, 2.3, 3.3)(4.3, 5.3, 6.3)(7.3, 8.3, 9.3) ]
    [ (1.4, 2.4, 3.4)(4.4, 5.4, 6.4)(7.4, 8.4, 9.4) ]
    [ (1.5, 2.5, 3.5)(4.5, 5.5, 6.5)(7.5, 8.5, 9.5) ] }
  { 
    [ (1.6, 2.6, 3.6)(4.6, 5.6, 6.6)(7.6, 8.6, 9.6) ]
    [ (1.7, 2.7, 3.7)(4.7, 5.7, 6.7)(7.7, 8.7, 9.7) ]
    [ (1.8, 2.8, 3.8)(4.8, 5.8, 6.8)(7.8, 8.8, 9.8) ] }

// Strictly speaking, all single-index slices of multi-dimensional uniforms
// can be seen as k-blades (pseudovectors) of the uniform's vector space.
xMat: hCube.[0][ ][ ][ ]   // [ (1.0, 2.0, 3.0)(1.1, 2.1, 3.1)(1.2, 2.2, 3.2) ]
                           // [ (1.3, 2.3, 3.3)(1.4, 2.4, 3.4)(1.5, 2.5, 3.5) ]
                           // [ (1.6, 2.6, 2.6)(1.7, 2.7, 3.7)(1.8, 2.8, 3.8) ]

yMat: hCube.[ ][0][ ][ ]   // [ (1.0, 1.1, 1.2)(4.0, 4.1, 4.2)(7.0, 7.1, 7.2) ]
                           // [ (1.3, 1.4, 1.5)(4.3, 4.4, 4.5)(7.3, 7.4, 7.5) ]
                           // [ (1.6, 1.7, 1.8)(4.6, 4.7, 4.8)(7.6, 7.7, 7.8) ]

zMat: hCube.[ ][ ][0][ ]   // [ (1.0, 2.0, 3.0)(4.0, 5.0, 6.0)(7.0, 8.0, 9.0) ]
                           // [ (1.3, 2.3, 3.3)(4.3, 5.3, 6.3)(7.3, 8.3, 9.3) ]
                           // [ (1.6, 2.6, 3.6)(4.6, 5.6, 6.6)(7.6, 8.6, 9.6) ]

wMat: hCube.[ ][ ][ ][0]   // [ (1.0, 2.0, 3.0)(4.0, 5.0, 6.0)(7.0, 8.0, 9.0) ]
                           // [ (1.1, 2.1, 3.1)(4.1, 5.1, 6.1)(7.1, 8.1, 9.1) ]
                           // [ (1.2, 2.2, 3.2)(4.2, 5.2, 6.2)(7.2, 8.2, 9.2) ]

// The .[] grammar makes it easy to chain index expressions,
// such that you can extract sub-slices without having to
// rely on a mess of range expressions
tVec: hCube.[ ][ ][ ][0]
               .[ ][ ][0] 
               .[0][ ] // (1.0, 2.0, 3.0)

tVec: hCube.[0][ ][0][0]

TL;DR: All the data axes. Literally. All of them. Slice it all the ways.

Sensible Data Serialization

Where data is generally immutable during evaluation, the serialization process is more flexible, as the way you structure your data for evaluation is decoupled from how you structure it for export.

Data representations can be handled at the schema, section, and archive level, letting you build up your serialization model piece-by-piece, override it where necessary, and handle it as modularly (or non-modularly) as you want — all while maintaining topological guarantees.

Critically, Vada’s serialization process is designed facilitate data flattening, turning hierarchical data and lists of structs into space-efficient and access-efficient binary archives utilizing structs of arrays and common data tables.

Where common numeric representations fall short, Vada lets you define custom bit-packing schemas that (optionally) export memory-safe unpacking instructions as a part of the binary archive’s header table. This enables rapid iteration in development environments by alleviating the need to regenerate the consuming applications’ unpacking code, and the selective inclusion of dynamic deserialization in production^00-4.

And it can do JSON/YAML/CSV stuff. Because reasons.

And because Vada knows how headache-inducing data packing can be, it gives you a ton of ways to handle it.

Directional Data Propagation

While Vada is heavily inspired by CUE, it handles scoping and propagation in a very different way.

Where CUE uses pure downwards propagation with siloed directories (in which there is no cross-branch propagation of data or constraints), Vada uses directional interfaces to allow cross-propagation from siblings without creating unresolvable circular dependencies.

The TL;DR on directional interfaces is that collections can differentiate what they expose for export for each direction they export it in (e.g. to-parent, to-child, to-sibling).

This lets users establish cross-dependencies while preventing circular dependencies footguns by presenting separate silos of data and selectively flattening siblings through single-child upwards interfaces.

Spoilers: 99% of the time, these data propagation details will only matter if you’re deliberately doing weird stuff.

Downward interfaces propagate implicitly. Upwards and horizontal interfaces propagate explicitly. All interfaces let you keep things clean by re-arranging and filtering data.

It might sound complicated, but it’s simple in practice.

Full Toolchain Incremental Compilation

Vada’s evaluator is written in a way that lets the language server and CLI share an incremental compute cache — every time semantic tokens are updated in your IDE, the intermediate data representation used to generate output files is updated as well.

And since Vada is deterministically evaluated, compute caches can be remotely calculated and shared.

This shared cache setup can be leveraged in a variety of CI/CD pipelines. It’s deliberately left as an open-ended opportunity for hackery.

Operators

While you don’t need to understand all of the inner details of Vada’s expression parsing in order to get the most out of it, knowing the basics can help you understand the overall design.

Vada uses a modified Pratt parsing system for its entire expression system. Name-paths, function calls, lists — it’s Pratt parsed all the way down, which means we can express operator precedence and operator associativity through two tables of left-right binding power tuples.

High binding power operators are processed before low binding power operators. Operators with a (low, high) pattern are left-associative, while those with a (high, low) are right-associative.

With that in mind, here are the main infix operators:

The main prefix operators

And the postfix operators:

The operators with binding powers lower than the standard arithmetic operators can be thought of as ‘clause delineating’ operators, while the ones with higher binding powers can be seen as ‘operand modifying’ operators. It’s also worth noting that the different binding powers of the conjunction and disjunction operators allows for disjunctions of conjunctions (of arithmetic expressions, etc.), and that the high binding power of the bitwise operators allows for easy ad-hoc flag math — the there’s some nuance here that’s bloody painful to discuss without mentioning Pratt parsing.

$someEnum: !flag[Red; Green; Blue; Yellow]
someBool: @false

currState: (someBool == @true) ?> $someEnum::Green | $someEnum::Red

matchVar: 4 + currState ?> (
              _::Red    ?> 1;
              _::Green  ?> 2;
              _::Blue   ?> 3;
              _::Yellow ?> 4;
              @null     ?> 0)

eq: matchVar == 6 // @true

Sequences and Circumfixes

A sequence expression is composed of series of comma and/or semi-colon delineated sub-expressions. Commas delineate horizontal sequences, while semi-colons delineate vertical sequences.

A circumfix expression is a normal expression wrapped in a pair of enclosing characters that indicate the container primitive it evaluates to. The three primary circumfixes are (), [], and {}, which signify uniforms, lists, and structs, respectively.

An un-enclosed sequence is evaluated as a list by default, and superfluous circumfixes are ignored when the container type of the inner expression is unambiguous, such that {([[1,2,3]])} would evaluate the same as [1,2,3]^01-1. which means you can use all three primary circumfixes the way you’d use parentheses in other languages, so long as the innermost operands are properly enclosed.

A good way to think of this behavior is that circumfixes constrain, but they don’t coerce.

Literals

Vada has a lot of literals, because there are a lot of ways to represent data.

([0][2-9]|[1-5][0-9]|6[0-4])a_([a-zA-Z0-9]+[a-zA-Z0-9_]*)(\.[a-zA-Z0-9]+[a-zA-Z0-9_]*)?

Keywords

One of the first things folks notice about Vada is that it uses a ton of sigils (leading symbols on variable names), with all four primary sigils showing up in even simple examples:

someRecord: @Ext::std::unitTypes::$phys & {
    name: "Jimbo"
    classes:
        [...{name: !str level: !int & >0}]
        & {"Fighter", 4}, {"Mage", 2}
    level: classes->#reduce(cls; @sum cls[1]) / classes->#count
}

Vada uses sigils to limit namespace pollution, optimize code-completion, and ensure expression clarity. Sigils also help users extrapolate from the code in front of them without having to rely on IDE features and make it easier to search the documentation; @if and if are two very different things.

At first glance sigils can feel like clutter, and concepts like sigil hoisting — shifting a terminal sigil to the start of a section path to get some extra juice out of IDE auto-complete, e.g. $::std::unitTypes::phys — and sigil filtering — ending a section path with a bare sigil to import/access the matching subset, e.g. @import: std::unitTypes::# — are kinda funky.

Where the semantic value of sigils (and their related access patterns) really starts to shine, however, is when you’re working with large archives that have substantial dependency trees and complex schemas.

In those cases, the visual noise is out-weighted by the benefits that come from being able to instantly know reference types without having to rely on IDE features. Sigils add clarity (and let us pull of some neat parsing tricks).

And when you get right down to it, the number of sigils that Vada uses is fairly modest, and what they signify is pretty, well, significant, too.

If it has a leading @ it’s always a noun. If it has a leading ! it’s always a built-in type or container. If it has a leading $ it’s always a user-created schema. If it has a leading # it’s always a verb. If it doesn’t have a sigil, it’s just a plain old reference.

Common Transforms

Don’t worry if that last one, #wedgeArb, looks confusing; the RHS columnar list is topologically equivalent to the struct { basis: (1,0,0; 0,1,0; 0,0,1) operand: (4,-2, 6) }

#fork and #open are incredibly powerful transforms for “resetting” structs and schemas passed down from parent modules. The downsides to abusing them are obvious. Discipline is suggested.

#reduce is roughly comparable to an ad-hoc !sink, in that it offers all of the same options in terms of striding the data and shaping the output.

Transforms, Side-Effects, and You

You can technically use @Local and some schema polymorphism, combined with smartly designed sinks, to model side-effect-like behaviors. But doing that requires you to rebuild simple fields as sinks and feed them using factory schemas that produce intermediary schemas which function as algebraic data types.

You can even go so far as to expose a top-level state as a sink that’s propagated downwards via the Implicit and then use topological weighting to effectively scope write access. But at that point you’d have an distributed state tree so goddamned glorious that you could probably get grant funding for it.

And it technically wouldn’t produce side effects, just side-effect-like behavior, because you’re technically feeding that top sink with non-reified values and modeling behavior branches on partial collapse-reductions of the top sink’s contents via the n-ary sinks you recursively ascend through on eval.

So, yes, you can have side effects if you want to one-shot your way to a tenure track position at a university somewhere.

String Primitives

Vada’s strings are complicated, but in a good way.

Numeric Primitives

Flag Primitives

The !flag primitive is a generalization of C-style enums that can be used in both bitwise and set operations, with the additional enhancements for hierarchical sets.

Vada’s flags are specifically designed to fulfill the role that bitvec-like flag sets do in game engines.

$someFlagset: !flag <~ {
    red,
    // flags can be left open, allowing for downstream customization 
    green: {...},
    blue: {
        cyan,
        turquoise,
        royal
    }
}

Raw Primitives

Raw primitives are primarily used for two things:

1. Output encoding
2. Hackery

Uniform Containers

We explored some of the coolness of uniforms up in the Core Concepts section, but the focus there was primarily on uniform numeric literals rather than the uniform container itself.

The uniform container is Vada’s atomic container.

TODO: Expand

List Containers

A list is a struct with anonymous members. That’s the long and short of it.

Take a look at this struct:

someStruct: {
    name: "jim"
    age: 21
    hobbies: [
        { name: "stamp collecting", experience: 4 },
        { name: "kyaking", experience: 2 },
        ...
    ]
    stats: {
        strength: 4
        agility: 6
        intelligence: 3
        luck: 8
        charm: 1
    }
}

If we were to access it as a list with ::@values, recursively flattening the member values into rows, we’d get:

someList: [ 
    "jim";
    21;
    [ ["stamp collecting"; 4], ["kyaking"; 2], ... ];
    [4; 6; 3; 8; 1]
]

And if we were to extract a schema from that list with #asAbstract, we’d get:

$someTopology: [
    !string;
    !num::int;
    [ [!string; !num::int], ... ];
    [!num::int, !num::int, !num::int, !num::int, !num::int]
]

Which we could then use as an anonymous constraint for other structs, whereby otherStruct::@values & $someTopology would constrain otherStruct’s members purely by position.

This topological equivalence is both an inherent side-effect of Vada’s design and an intentional feature of lists. Lists are iterable, filterable, bulk-constrainable jagged arrays. They lack the any-axis slicing of uniforms, and they aren’t as ergonomic as structs for topologically complex records, but the position they occupy as the middle-point between those two containers makes them incredibly useful.

// Concrete Spreads
simpleSpread:      [ ...[1, 2, 3], ...[4, 5, 6] ]  // [1, 2, 3, 4, 5, 6]
repeatedSpread:    [ ...[1, 2, 3] * 2 ]            // [1, 2, 3, 1, 2, 3]
interleavedSpread: [ ...[1, 2, 3] + ...[4, 5, 6] ] // [1, 4, 2, 5, 3, 6]

// Abstract Spreads
abstractRepeat: ...!num * 5    // [!num, !num, !num, !num, !num]
abstractRange:  ...!num & 1..4 // [!num] | [!num, !num] | [!num, !num, !num]

// Fancy
CombinedSpreads: [...!num & 0..6, ...!string & 0..6] & 1..8      // 0-5 numbers, 0-5 strings, for a minimum of 1 and a maximum of 8 items.
DisjunctSpreads: [ ...(!num & 0..) * 5 | ...(!num & ..0) & 1..6] // Either 5 numbers greater than zero or 1-5 numbers less than zero.

Struct Containers

If you manage to create a substantial Vada archive that isn’t primarily composed of structs, you’re probably doing something weird.

$characterInfo: {
    // Fields in the @local inherent can't be externally referenced
    --- @local {
        isBoring: !bool & @default -> @true
    }

    // Simple sections don't need circumfixes. They simply enclose
    // the fields that follow them, up until the next section declaration. 
    --- info
    name: !string & != ""
    age: !num & 0..
    background: !string

    // if you want to nest sections, however, you should use a circumfix
    --- classInfo {
        --- combat
        classes: [...{name: !string & != "" level: !num::int & 0..21}]  
        powerLevel: !num::int
        combatTitles: [...!string & != ""]
        
        -- secondary
        jobs: classes: [...{name: !string & != "" level: !num::int & 0..51}]
        jobTitles: [...!string & != ""]
    }

    // You can bulk-constrain sections.
    // When you bulk-constrain, the right side of the outermost
    // pipe expression is treated as the section body.
    --- attributes: !num::int ~> [
        strength,
        endurance,
        agility,
        courage
    ]

    // You can further constrain the section body to control
    // the number of fields, create field-set disjunctions, etc.
    --- skills: {
        name: !string
        level: !num::int
        } ~> [...] & 0..11

    // $["meta_\w*"] is a regex-based field name capture, which
    // lets you do fancy organizational things with open structs.
    --- metaData: $["meta_\w*"] & !string ~> {
        ...
    }

    // Anonymous sections can be used to create semi-open structs,
    // applying constraints to any additional fields that are added.
    --- !string ~> [...]
}

What the Hell is a Sink?

A Sink is an algebraic container that encapsulates an accumulator (read: a write-only open list) which is transformed when the sink is evaluated (read: unwrapped).

The inputs into a sink’s accumulator and the output of a sink’s evaluation transform are constrained separately, allowing sinks to function as both enums (in the Rust-y sense) and as vaguely cursed not-monads.

Understanding all of the shenanigans you can pull with sinks takes time, but you can use them as algebraic data types and super-disjunctions without diving into the funky bits.

$someSink: !sink <~ {
    @accumulator: {
        fieldA: [!number, ...]
        fieldB: [!number, ...]
        fieldC: [!number, ...]
    }
    @out: fieldA->#sum + fieldB->#sum * fieldC->#mean
}

Structs

Structs are the bread and butter of most Vada archives.

someList: [ "jim"; 21; [["stamp collecting"; 4], ["kyaking"; 2]] ]

are topologically equivalent.

Schemas + Sinks = Dynamic Type Generation

A sink schema that produces sink schemas, using recursion to incrementally refine intermediate schemas until a concrete record is produced (which can then still consume more inputs through its accumulator and optionally de-reify back to a schema on eval) can be pretty damn close to Turing complete.

Can a combination of sinks and schemas be used to create type generation tools that let you scaffold out large archives based on a sample dataset and a handful of smart declarations? Yes. Does the utility of the above make the combo any less cursed? No.

With great power comes great responsibility.

Unit vs Non-Unit Numeric Schemas

Numeric constraints don’t prevent you from doing math with differently-constrained operands; as long as the primitives are compatible, it works. And since Vada doesn’t have floats at the eval stage — you can output floats, but the actual math is deterministic — it means that you can do math with just about any combination of numerics you can think of.

Which is where Units come in.

All numerics are technically [!num, !flagset] tuples

$metricLengthFlags: !flagset <~ [ mm, cm, dm, m, dam, km ]
$USCLengthFlags:    !flagset <~ [ in, ft, yd, mi ]

!num::unitGroup::metric <~ {
    // This pulls $metricLengthFlags into the local namespace for easy reference
    @flags: $metricLengthFlags

    // Sets the 1-scale unit
    @baseUnit: m

    // Determines the family's relation to the base unit.
    // Intermediate relations are automatically calculated and cached.
    @baseScale: (
        mm  ?> *0.001;
        cm  ?> *0.01;
        dm  ?> *0.1;
        m   ?> *1; 
        dam ?> *10;
        km  ?> *1000
    )

    // Conversions for other unit groups. 
    // These external equivalencies require explicit units.
    @equiv: 
        !num::unitGroup::USC ?> (
            in ?> 2.54::cm;
            ft ?> 0.3048::m;
            yd ?> 0.9144::m;
            mi ?> 1.609344::km | 1609.344::m
        )
}

!num::unitGroup::USC <~ {
    @flags: $USCLengthFlags
    @baseUnit: in

    @baseScale: (
        in ?> *1;
        ft ?> *12;
        yd ?> *3;
        mi ?> *5280
    )

    @equiv: 
        !num::unitGroup::metric ?> (
            mm  ?>  0.039370::in;
            cm  ?>  0.39370::in;
            dm  ?>  3.9370::in;
            m   ?> 39.370::in | 3.2808::ft;
            dam ?> 32.808::ft;
            km  ?>  0.62137::mi | 1093.6::yd | 3280.8::ft;
        )
}

External Archive Management

Vada doesn’t ship a full npm-like archive manager, because I don’t have the brain juice for that and it isn’t aligned with Vada’s primary use case.

Instead, Vada ships with tooling for using and maintaining local-network git-aware archive registries. Because, let’s face it, a package manager that can pull the right local branch from the studio’s intranet is more useful than stranger danger-ing yourself with mystery archives from the internet.

The CLI can be used to browse these registries, validate the availability of an archive’s external dependencies, and determine what would be tree shaken from those external archives (and, optionally, internal collections) to generate a standalone archive from the source archive’s manifest.

And it can validate environment variables and build variables that influence how external archive dependencies are handled. Because bloody hell, that’s the bare minimum that a CLI should do for you. The most pre pre-checking you can do. Please, people, have standards.

Output Artifact Management

In addition to running the actual output process, you can also use the CLI to preview, compare, and validate output artifacts.