Source-highlight Library
srchilite::WordTokenizer Class Reference

Tokenizes a paragraph separating words from spaces. More...

#include <wordtokenizer.h>

List of all members.

Public Types

typedef std::list< std::pair
< std::string, std::string > > 
WordTokenizerResults
 Results of the tokenizer; each element is a pair where the first string represents a possible space and the second string a possible word.

Static Public Member Functions

static void tokenize (const std::string &s, WordTokenizerResults &results)
 Tokenizes the passed string and stores the results.

Detailed Description

Tokenizes a paragraph separating words from spaces.


Member Typedef Documentation

typedef std::list<std::pair<std::string, std::string> > srchilite::WordTokenizer::WordTokenizerResults

Results of the tokenizer; each element is a pair where the first string represents a possible space and the second string a possible word.

The two elements are mutually exclusive


Member Function Documentation

void srchilite::WordTokenizer::tokenize ( const std::string &  s,
WordTokenizerResults results 
) [static]

Tokenizes the passed string and stores the results.

Parameters:
sthe string to tokenize
resultswhere to store the results

The documentation for this class was generated from the following files: