xtokid accepts the names of files and/or directories on the command line, then extracts and prints a stream of tokens from those files for which it has a valid, enabled scanner. This is useful primarily for debugging new mkid scanners (see Defining scanners).
xtokid extracts tokens from source files, therefore it accepts the ‘--lang-map’, ‘--include’, ‘--exclude’, and ‘--lang-option’ options, as well as the language-specific scanner options, all of which are described in Extraction options. xtokid walks file trees, therefore it handles file and directory names on its command line and the ‘--prune’ option as described in Walker options.
The name ‘xtokid’ indicates that it is the “eXtract TOKens ID utility”.