Next: , Previous: fnid invocation, Up: Top

9 xtokid: Testing Language Scanners

xtokid accepts the names of files and/or directories on the command line, then extracts and prints a stream of tokens from those files for which it has a valid, enabled scanner. This is useful primarily for debugging new mkid scanners (see Defining scanners).

xtokid extracts tokens from source files, therefore it accepts the ‘--lang-map’, ‘--include’, ‘--exclude’, and ‘--lang-option’ options, as well as the language-specific scanner options, all of which are described in Extraction options. xtokid walks file trees, therefore it handles file and directory names on its command line and the ‘--prune’ option as described in Walker options.

The name ‘xtokid’ indicates that it is the “eXtract TOKens ID utility”.