Re: [PATCH v3 2/2] D: Move classification of symbols from the grammar to the lexer

[Iain Buclaw <ibuclaw at gdcproject dot org>]

On 13 August 2015 at 06:05, Doug Evans <xdje42@gmail.com> wrote:
> Iain Buclaw <ibuclaw@gdcproject.org> writes:
>> This patch rewrites the lexer to be just like C++'s, where it will
>> attempt to classify and reduce any alternating '.' and symbol-like
>> tokens.
>>
>> There is a little added complexity as existing D compilers will only
>> write the full module name to debug.  For instance, given
>> 'package.and.another.foo', the first three parts will return as
>> unknown/non-existent symbols.   While it would be nice to fix this in
>> my compiler, there does not seem to be a nice way to chain package and
>> module names together that is understood by GDB (I think language_go
>> has a similar problem to this).
>>
>> However, now I'm happy.  As I can finally begin to start supporting
>> and implementing exp.property style syntax in the grammar without
>> running into shift reduce conflicts (starting with .sizeof and .typeof
>> :-)
>>
>> Regards
>> Iain.
>> ---
>>
>> 2015-08-03  Iain Buclaw  <ibuclaw@gdcproject.org>
>>
>>       * d-exp.y (%union): Add voidval.
>>       (%token): Add UNKNOWN_NAME as a token to represent an unclassified
>>       name in the lexing stage.
>>       (PostfixExpression): Move symbol completion handling in grammar here
>>       from PrimaryExpression.
>>       (PrimaryExpression): Move routines to handle resolving identifier
>>       tokens in the grammar here from push_expression_name.
>>       (IdentifierExp): Remove the handling of alternating '.' and identifier
>>       tokens.
>>       (TypeExp): Allow TypeExp to be wrapped in parenthesis in the grammar.
>>       (BasicType): Remove C-style typename rules.
>>       (d_type_from_name, d_module_from_name, push_variable)
>>       (push_fieldnames, push_type_name, push_module_name)
>>       (push_expression_name): Remove.
>>       (lex_one_token): Rename from yylex.  Replace pstate with par_state.
>>       (token_and_value): New type.
>>       (token_fifo, popping, name_obstack): New globals.
>>       (classify_name): New function.
>>       (classify_inner_name): Likewise.
>>       (yylex): Likewise.
>>       (d_parse): Initialize token_fifo, popping and name_obstack.
>
> Hi.
>
> I didn't have time to give this more than a glance.
> Since it's all just pure D stuff, I'd say go for it.

Thanks, just giving it one last build + test incase any recent changes
require some fix-ups.

Regards
Iain