I'm working (slowly) on a fairly flexible standard syntax and syntax tree. It's not going to be as simple as S-expressions or JSON, though. I need five kinds of lists to get a reasonable mainstream-ish syntax, and this seems a bit unwieldy if you're just using it for data.
I'm not sure what a standardized AST would really be. A standardized concrete syntax tree is doable, but each language is going to have its own statements and expressions (equivalent to special forms in Lisp). This is similar to how we can use JSON objects to represent all sorts of different types as key-value sequences with different sets of keys.
For an evolving language, the AST needs to change from one version to the next. When you add a new kind of statement or expression, the tree-walkers in the tools usually need to adapt. The tools aren't stable unless the language is stable.
- Blocks (curly brackets and separated by newlines or semicolons):
{ a b; c d }
- Dotted lists, where sometimes the dots can be omitted:
foo.bar.baz
In combination, you can write something like:
for x in [1, 2, 3] {
foo.bar(x + 1, x * 2)
}
Which can be interpreted as 7 lists. The top-level list has five items, the last two of which are lists. The last list is a block containing one item, which is a three-item dotted list. The argument list after "bar" is a two-item comma list containing two phrases.
I've implemented this, but I'm not satisfied with it; the corner cases are tricky to understand.
I'm not sure what a standardized AST would really be. A standardized concrete syntax tree is doable, but each language is going to have its own statements and expressions (equivalent to special forms in Lisp). This is similar to how we can use JSON objects to represent all sorts of different types as key-value sequences with different sets of keys.
For an evolving language, the AST needs to change from one version to the next. When you add a new kind of statement or expression, the tree-walkers in the tools usually need to adapt. The tools aren't stable unless the language is stable.