Lucene.NET TokenStream.Next method disappeared

Question

I have to update a project using Lucene.NET. It first time I meet this library, and I should update the references to a new version. So I did with Lucene references via a NuGet.

But actually I should in a way update some methods that disappeared in the new versions.

public abstract class AnalyzerView
{
    public abstract string Name { get; }

    public virtual string GetView(TokenStream tokenStream,out int numberOfTokens)
    {
        StringBuilder sb = new StringBuilder();

        Token token = tokenStream.Next();

        numberOfTokens = 0;

        while (token != null)
        {
            numberOfTokens++;
            sb.Append(GetTokenView(token));
            token = tokenStream.Next();
        }

        return sb.ToString();
    }

    protected abstract string GetTokenView(Token token);
}

The documentation is not very clear for me. As I understood I should use something like

while (tokenStream.IncrementToken()) { ... }

but how to obtain that following token is a mystery for me.

I saw similar question on StackOverflow, but my problem is that I need to obtain a Token rather thatn a string, because I have some code that uses tokens, like this one:

public class TermWithOffsetsView : AnalyzerView {
    public override string Name {
        get { return "Terms With Offsets"; }
    }

    protected override string GetTokenView(Token token) {
        return token.TermText() + 
            "   Start: " + token.StartOffset().ToString().PadLeft(5) + 
            "  End: " + token.EndOffset().ToString().PadLeft(5) + 
            "\r\n";
    }
}

Answer 1

Finally, after some researches, implemented something like this:

public abstract class AnalyzerView
{
    public abstract string Name { get; }

    public virtual string GetView(TokenStream tokenStream,out int numberOfTokens)
    {
        StringBuilder sb = new StringBuilder();
        numberOfTokens = 0;
        while (tokenStream.IncrementToken())
        {
            numberOfTokens++;
            sb.Append(GetTokenView(tokenStream));
        }

        return sb.ToString();
    }

    protected abstract string GetTokenView(TokenStream tokenStream);
}

with the derived class like this:

public class TermWithOffsetsView : AnalyzerView {
    public override string Name {
        get { return "Terms With Offsets"; }
    }

    protected override string GetTokenView(TokenStream tokenStream) {
        ITermAttribute termAtt = tokenStream.AddAttribute<ITermAttribute>();
        IOffsetAttribute offsetAtt=tokenStream.AddAttribute<IOffsetAttribute>();

        return string.Format("{0}   Start: {1}  End: {2}{3}",
            termAtt.Term,
            offsetAtt.StartOffset.ToString().PadLeft(5),
            offsetAtt.EndOffset.ToString().PadLeft(5),
            Environment.NewLine);
    }
}