Previous: , Up: C++ Scanner Interface   [Contents][Index]


10.1.6.2 Complete Symbols

With both %define api.value.type variant and %define api.token.constructor, the parser defines the type symbol_type, and expects yylex to have the following prototype.

Function: parser::symbol_type yylex ()
Function: parser::symbol_type yylex (type1 arg1, ...)

Return a complete symbol, aggregating its type (i.e., the traditional value returned by yylex), its semantic value, and possibly its location. Invocations of ‘%lex-param {type1 arg1}’ yield additional arguments.

For each token type, Bison generates named constructors as follows.

Constructor on parser::symbol_type: symbol_type (int token, const value_type& value, const location_type& location)
Constructor on parser::symbol_type: symbol_type (int token, const location_type& location)
Constructor on parser::symbol_type: symbol_type (int token, const value_type& value)
Constructor on parser::symbol_type: symbol_type (int token)

Build a complete terminal symbol for the token type token (including the api.token.prefix), whose semantic value, if it has one, is value of adequate value_type. Pass the location iff location tracking is enabled.

Consistency between token and value_type is checked via an assert.

For instance, given the following declarations:

%define api.token.prefix {TOK_}
%token <std::string> IDENTIFIER;
%token <int> INTEGER;
%token ':';

you may use these constructors:

symbol_type (int token, const std::string&, const location_type&);
symbol_type (int token, const int&, const location_type&);
symbol_type (int token, const location_type&);

which should be used in a Flex-scanner as follows.

%%
[a-z]+   return yy::parser::symbol_type (TOK_IDENTIFIER, yytext, loc);
[0-9]+   return yy::parser::symbol_type (TOK_INTEGER, text_to_int (yytext), loc);
":"      return yy::parser::symbol_type (':', loc);
<<EOF>>  return yy::parser::symbol_type (0, loc);

Note that it is possible to generate and compile type incorrect code (e.g. ‘symbol_type (':', yytext, loc)’). It will fail at run time, provided the assertions are enabled (i.e., -DNDEBUG was not passed to the compiler). Bison supports an alternative that guarantees that type incorrect code will not even compile. Indeed, it generates named constructors as follows.

Method on parser: symbol_type make_token (const value_type& value, const location_type& location)
Method on parser: symbol_type make_token (const location_type& location)
Method on parser: symbol_type make_token (const value_type& value)
Method on parser: symbol_type make_token ()

Build a complete terminal symbol for the token type token (not including the api.token.prefix), whose semantic value, if it has one, is value of adequate value_type. Pass the location iff location tracking is enabled.

For instance, given the following declarations:

%define api.token.prefix {TOK_}
%token <std::string> IDENTIFIER;
%token <int> INTEGER;
%token COLON;
%token EOF 0;

Bison generates:

symbol_type make_IDENTIFIER (const std::string&, const location_type&);
symbol_type make_INTEGER (const int&, const location_type&);
symbol_type make_COLON (const location_type&);
symbol_type make_EOF (const location_type&);

which should be used in a scanner as follows.

[a-z]+   return yy::parser::make_IDENTIFIER (yytext, loc);
[0-9]+   return yy::parser::make_INTEGER (text_to_int (yytext), loc);
":"      return yy::parser::make_COLON (loc);
<<EOF>>  return yy::parser::make_EOF (loc);

Tokens that do not have an identifier are not accessible: you cannot simply use characters such as ':', they must be declared with %token, including the end-of-file token.


Previous: , Up: C++ Scanner Interface   [Contents][Index]