Package version:

Type Alias LexicalTokenizer

LexicalTokenizer:
    | ClassicTokenizer
    | EdgeNGramTokenizer
    | KeywordTokenizer
    | MicrosoftLanguageTokenizer
    | MicrosoftLanguageStemmingTokenizer
    | NGramTokenizer
    | PathHierarchyTokenizer
    | PatternTokenizer
    | LuceneStandardTokenizer
    | UaxUrlEmailTokenizer

Contains the possible cases for Tokenizer.