Source-highlight Library
srchilite::WordTokenizer Class Reference

Tokenizes a paragraph separating words from spaces. More...

#include <wordtokenizer.h>

Public Types

typedef std::list< std::pair
< std::string, std::string > > 
WordTokenizerResults
 Results of the tokenizer; each element is a pair where the first string represents a possible space and the second string a possible word. More...
 

Static Public Member Functions

static void tokenize (const std::string &s, WordTokenizerResults &results)
 Tokenizes the passed string and stores the results. More...
 

Detailed Description

Tokenizes a paragraph separating words from spaces.

Member Typedef Documentation

typedef std::list<std::pair<std::string, std::string> > srchilite::WordTokenizer::WordTokenizerResults

Results of the tokenizer; each element is a pair where the first string represents a possible space and the second string a possible word.

The two elements are mutually exclusive

Member Function Documentation

void srchilite::WordTokenizer::tokenize ( const std::string &  s,
WordTokenizerResults results 
)
static

Tokenizes the passed string and stores the results.

Parameters
sthe string to tokenize
resultswhere to store the results

The documentation for this class was generated from the following files: