[fud2] Serialize Plans into Assignment Lists

[jku20]

This PR introduces functionality to export and import plans to and from a serialized data format. There are two flavors of syntax, JSON and a custom syntax I'm calling flang (fud2 lang). Currently this data format supports explicit file names with inputs and outputs from stdio left for the future. Path names as exported directly from the planners. This means they are generally relative to the directory the fud2 command was invoked in. This is incorrect behavior, but is likely better fixed in a followup PR which changes the planner interface to include filename assignment as part of the planners.

This PR adds two new planners: json and flang. Both of these read a plan from stdin, one specified in JSON and the other specified in flang.

Additionally the PR adds two new modes: json-plan and flang-plan, taking the plan from the current planner and writing it to stdout in JSON and flang syntax respectively.

The bulk of the engineering effort here is defining the AST and parser for flang. The JSON parser is implemented decoratively in the obvious way with serde. A description of the implemented syntax follows. Defining the semantics is ongoing work.

Flang

Motivation

Plan files currently have two forms which are interchangeable: JSON and flang. Generally the
JSON notation should be preferred due to its easier interop with other tools and parsing.
The custom language is written as it is easier to read and reason about. It's nice to have the
fantasy syntax the JSON represents be real.

Syntax

Flang syntax is defined as a list of semicolon statements. Each statement is composed of the
assignment of a list of variables by a function which itself takes in a list of variables. For
example, the following is a simple flang program:

a, "1 file path \" \\ cool.txt (2)" = op-one("input 1", in2);c=op-two(a);                       
d, e = op-3("1 file path \" \\ cool.txt (2)");                                                  
f = op-3("d");                                                                                  

A lot about the above is similar to other imperative languages. Some particular things to note:

• flang is whitespace insensitive
• flang has two ways to describe variables, as literals such as in a and as quoted strings such as in "1 file path cool.txt (2)". This lax attitude towards variables lets things like file paths exist as variables.
• flang has very few operators, lacking even - as an operator, which instead can be used in identifiers.

A more formal specification of the syntax follows:

<statements> ::=                                                                                
    | <statement>;                                                                              
    | <statement>; <statements>                                                                 
                                                                                                
<statement> ::=                                                                                 
    | <vars> = <id>(<vars>)                                                                     
                                                                                                
<vars> ::=                                                                                      
    | <id>                                                                                      
    | <id>, <vars>                                                                              
                                                                                                
<id> ::=                                                                                        
    | <shorthand-id>                                                                            
    | <quoted-id>                                                                               
                                                                                                
<shorthand-id> ::=                                                                              
    | <nice-char>                                                                               
    | <nice-char><shorthand-id>                                                                 
                                                                                                
<nice-char> ::=                                                                                 
    | <alphanumeric>                                                                            
    | - | _ | / | \ | . | :                                                                     
                                                                                                
<quoted-id> ::=                                                                                 
    | "<string>"                                                                                
                                                                                                
<string> ::=                        
    | <utf8-without-\-or-">         
    | \"                            
    | \\                            
    | <utf8-without-\-or"><string> 
    | \"<string>                    
    | \\<string>                    

TODO:

• implement planner based on parsing the fancy new format
• implement the op-seq mode
• implement json format as both planner and output using serde
• fix AST to make variables file paths instead of strings
• add end to end tests for new format <-- I want to wait until path stuff is finalized out before adding these

Extra notes on the implementation:
The error handling on this parser is utterly atrocious, though it is written in such a way that it should not be terribly difficult to make the error handling much better.

Notes to the Reviewer on Implementation

This is a really big PR, bigger than it should be, however, splitting it up significantly is difficult. To be a single testable unit, the parser infrastructure should likely all be implemented in a single PR. For testing and making the changes externally visiable, and making the visitor framework for traversing the ASTs into nice printable forms is quite helpful.

A decent way to group the files into three roughly independent parts is:

• The flang module represents all to do with representing and parsing flang. It includes the AST and parser.
• The functions in ast_converter.rs use the visitor framework from flang to pretty print the AST as well as convert and from current fud2 plans.
• The rest of the changes use the parser and visitors to implement visible functionality for fud2.

For ease of reviewability, this PR will be split into a multiple other PRs. An incomplete list follows:

• [fud2] Json Parser for Serialized Plans  #2564

[jku20]

It might make sense to simply replace the current "show" mode with the funny new syntax, but I'm hesitant at current without writing up the syntax a bit more formally.

[jku20]

Some todos