[pylucene-dev] PyLucene custom charTokenizer?

[Andi Vajda]

On Fri, 20 Jul 2007, keekles keekles wrote:

> I'm not that familiar with lucene, but basically what im looking to
> accomplish is the equivalent of a whitespace tokenizer with my own list of
> delimiters, in Lucene docs it just looks like simple inheritance but I dont
> really see any examples in PyLucene on how to subclass a charTokenizer other
> then the class from the lia SimpleKeywordAnalyzer which does not appear to
> be used or work as far as i can see. I realize this is probably a bit out of
> place to be asking here, but could someone explain or show me a valid
> example of a custom analyzer using a custom charTokenizer in PyLucene?

Because PyLucene wraps a gcj-compiled Java Lucene you cannot simply extend a 
PyLucene python class and expect your extension to be known to the 
Java Lucene library.

Instead, there are a number of extension points that are pre-defined 
extensions to Java Lucene classes that can delegate (wrap in reverse) any 
corresponding python class that implements the protocol of the wrapper.

All APIs that accept an instance of a class whose protocol can be implemented 
in python will do the right thing in wrapping your python customization with 
an instance of the corresponding pre-defined extension point class. Some of 
these customizations need to be created via a constructor though because there 
is no API that accepts a parameter of the given type. This capability was 
missing for CharTokenizer until I added it just now.

I fixed and moved the SimpleKeywordAnalyzer class to the lia/analysis/keyword 
directory and I added a new unit test to the KeywordAnalyzerTest.py file that 
illustrates how to use it.

These changes are checked into the PyLucene trunk rev 334.

Andi..