How to Configure an Elasticsearch Tokenizer to Split UpperCase Words into Separate Tokens

Learn how to configure an Elasticsearch tokenizer and analyzer to split camelCase or PascalCase words into separate tokens for improved search matching. --- This video is based on the question https://stackoverflow.com/q/79457368/ asked by the user 'dominikweber' ( https://stackoverflow.com/u/2849119/ ) and on the answer https://stackoverflow.com/a/79457666/ provided by the user 'Paulo' ( https://stackoverflow.com/u/4534923/ ) at 'Stack Overflow' website. Thanks to these great users and Stackexchange community for their contributions. Visit these links for original content and any more details, such as alternate solutions, latest updates/developments on topic, comments, revision history etc. For example, the original title of the Question was: Tokenizer that splits uppercase in two words Also, Content (except music) licensed under CC BY-SA https://meta.stackexchange.com/help/licensing The original Question post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/by-sa/4.0/ ) license, and the original Answer post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/by-sa/4.0/ ) license. If anything seems off to you, please feel free to drop me a comment under this video. --- Introduction When indexing and searching text like LoremIpsum in Elasticsearch, you might want to split this camelCase or PascalCase word into multiple tokens: lorem, ipsum, and also keep the original loremipsum token for broader matching. This improves search flexibility by matching partial words or the entire combined word. The Problem By default, Elasticsearch does not split based on case changes in single tokens. So, searching for Lorem or Ipsum might not yield expected matches if the text is indexed only as one token loremipsum. Solution Overview We can solve this using a custom analyzer with the: keyword tokenizer (to treat the whole input as a single token initially) word_delimiter token filter configured to split on case changes lowercase filter for case normalization The key is to use the word_delimiter filter with: split_on_case_change: true preserve_original: true (to keep the full token) generate_word_parts: true This setup produces all expected tokens. Example Configuration Analyzer and Filter: [[See Video to Reveal this Text or Code Snippet]] Testing the Analyzer: You can test this analyzer with the _analyze API: [[See Video to Reveal this Text or Code Snippet]] The output tokens will be: loremipsum (original full token) lorem (split on case change) ipsum (split on case change) Mapping Your Field To use this analyzer, specify it in your index mapping for the text fields: [[See Video to Reveal this Text or Code Snippet]] Using the Analyzer in Search By default, queries use the analyzer defined in the mapping. However, you can explicitly specify the analyzer during search: [[See Video to Reveal this Text or Code Snippet]] Use this approach when your search query needs specific tokenization behavior. Summary Use a keyword tokenizer with a word_delimiter filter to split camelCase. Set preserve_original: true and split_on_case_change: true for desired tokens. Define your text fields with this custom analyzer in mappings. Optionally specify the analyzer at search time for query consistency. This setup solves the common issue of indexing and searching camelCase or PascalCase words in Elasticsearch, giving flexible and precise search results.