DOC: language specification

[fhs]

Julia needs a language specification detailing its grammar. For couple of reasons:

1. It'll serve as a documentation for users of the language. I don't think the manual documents everything.
2. People can write compatible parsers for the language.

[StefanKarpinski]

If you want to tackle this, that's ok, but it seems like busy work. Until the language settles down, having formal specifications is just one more thing to be out of sync. Python, Ruby, Perl, Matlab and R don't, afaik, have formal specifications or grammars. Personally, I hate when languages give a formal grammar in their documentation instead of just showing you how things are expressed. It might be good for machines, but it's a terrible way to communicate with humans.

[JeffBezanson]

The parser code itself does a pretty good job of this. For each component
of the grammar there is a "parse-x" function that shows what to do. I'm
sure we can make it even clearer. I see less value in hand-translating it
to some other format that is even less well specified.
On Aug 25, 2013 5:33 PM, "Stefan Karpinski" notifications@github.com
wrote:

If you want to tackle this, that's ok, but it seems like busy work. Until
the language settles down, having formal specifications is just one more
thing to be out of sync. Python, Ruby, Perl, Matlab and R don't, afaik,
have formal specifications or grammars. Personally, I hate when languages
give a formal grammar in their documentation instead of just showing you
how things are expressed. It might be good for machines, but it's a
terrible way to communicate with humans.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/4144#issuecomment-23235997
.

[fhs]

Fair enough. I'll try to figure things out from reading the source.

Just FYI, Python and R have language specs (I don't know about others):

• http://docs.python.org/2/reference/
• http://cran.r-project.org/doc/manuals/r-release/R-lang.html

I suspect Julia spec would be much smaller than those, since it's a simpler language.

[JeffBezanson]

True, but our grammar is actually quite fiddly --- for example some contexts are "space sensitive" and some are not, marked by the with-space-sensitive macro. So I think there are good reasons to try hard to make the spec and the implementation the same. We should make sure it is in standard R5RS scheme (right now it isn't quite). Then it can be packaged with libflisp or any other scheme implementation to get a C-callable version, or one of the JS scheme implementations to get a JS-callable version. There are plenty of schemes for the JVM as well. I actually think this is a good solution for getting a fairly universally-usable parser.

[fhs]

I was actually trying to parse julia while preserving comments, and positions of tokens. I don't think the current parser can do that.

If you have the comments and positions, the comments right before a function, etc. can serve as its doc string. You can also pretty print julia source code in a standard format, fixing indentations, etc.

Golang has a parser like that, and it's very useful!

[JeffBezanson]

I think our best bet will be to add those features to the official parser. Obviously others may want to write their own parsers anyway, but if we are going to seriously use these features (e.g. doc strings) we don't want multiple implementations.

[StefanKarpinski]

Producing standardly formatted Julia would be nice, although I think that going from our AST back to formatted code might be a bit tricky since, e.g., we don't distinguish between ternary operators and if-else blocks in AST. This is very handy for manipulating them programmatically but means that we'd need a heuristic for deciding when to use which, although that's probably not too hard – if and else clauses are both single expressions with no assignment.

[cossio]

Would it make sense to revisit the idea of having a more formal but readable specification of the language now that 1.0 is out? Sometimes reading a terse and precise description of an aspect of the language can be more enlightening than an informal description, which seems to be the tone of most of the documentation at the moment.

For example, I really like the organization of Python docs. You have a very accessible Tutorial. Then there is the Language Reference, which contains the formal grammar of every element of the language as well as some explanation in prose.

[StefanKarpinski]

Sure, now would be a good time for someone to work on it if they’re interested. It’s a bit hard to validate that a grammar is correct without it generating a parser (even a slow one) so it might make sense to figure out a way to write a grammar that can actually be used to parse Julia code and check that it works.

[cossio]

Reopen?