7 #ifndef LETTERTOKENIZER_H
8 #define LETTERTOKENIZER_H
#define LUCENE_CLASS(Name)
Definition: LuceneObject.h:24
An abstract base class for simple, character-oriented tokenizers.
Definition: CharTokenizer.h:15
A LetterTokenizer is a tokenizer that divides text at non-letters. That's to say, it defines tokens a...
Definition: LetterTokenizer.h:19
virtual bool isTokenChar(wchar_t c)
Collects only characters which satisfy UnicodeUtil::isAlpha(c).
LetterTokenizer(const ReaderPtr &input)
Construct a new LetterTokenizer.
virtual ~LetterTokenizer()
LetterTokenizer(const AttributeSourcePtr &source, const ReaderPtr &input)
Construct a new LetterTokenizer using a given AttributeSource.
LetterTokenizer(const AttributeFactoryPtr &factory, const ReaderPtr &input)
Construct a new LetterTokenizer using a given AttributeFactory.
Definition: AbstractAllTermDocs.h:12
boost::shared_ptr< AttributeSource > AttributeSourcePtr
Definition: LuceneTypes.h:520
boost::shared_ptr< Reader > ReaderPtr
Definition: LuceneTypes.h:547
boost::shared_ptr< AttributeFactory > AttributeFactoryPtr
Definition: LuceneTypes.h:519