The lexical analysis step of Semantic is performed by the general
semantic-lex. For more information, see Writing Lexers.
semantic-lex produces lexical tokens of the form:
The Wisent's parser doesn't depend on the nature of analyzed input stream (buffer, string, etc.), and requires that lexical tokens have a different form (see Writing a lexer):
wisent-lex is the default Wisent's lexer used in Semantic.
Return the next available lexical token in Wisent's form.
wisent-lex-istreamcontains the list of lexical tokens produced by
semantic-lex. Pop the next token available and convert it to a form suitable for the Wisent's parser.
Mapping of lexical tokens as produced by
equivalent Wisent lexical tokens is straightforward:
(token-class start . end) ⇒ (token-class value start . end)
value is the input
buffer-substring from start to