Clinton Gormley 5da9e5dcbc Docs: Improved tokenizer docs (#18356) ...

* Docs: Improved tokenizer docs

Added descriptions and runnable examples

* Addressed Nik's comments

* Added TESTRESPONSEs for all tokenizer examples

* Added TESTRESPONSEs for all analyzer examples too

* Added docs, examples, and TESTRESPONSES for character filters

* Skipping two tests:

One interprets "$1" as a stack variable - same problem exists with the REST tests

The other because the "took" value is always different

* Fixed tests with "took"

* Fixed failing tests and removed preserve_original from fingerprint analyzer

2016-05-19 19:42:23 +02:00

..

classic-tokenizer.asciidoc

Docs: Improved tokenizer docs (#18356)

2016-05-19 19:42:23 +02:00

edgengram-tokenizer.asciidoc

Docs: Improved tokenizer docs (#18356)

2016-05-19 19:42:23 +02:00

keyword-tokenizer.asciidoc

Docs: Improved tokenizer docs (#18356)

2016-05-19 19:42:23 +02:00

letter-tokenizer.asciidoc

Docs: Improved tokenizer docs (#18356)

2016-05-19 19:42:23 +02:00

lowercase-tokenizer.asciidoc

Docs: Improved tokenizer docs (#18356)

2016-05-19 19:42:23 +02:00

ngram-tokenizer.asciidoc

Docs: Improved tokenizer docs (#18356)

2016-05-19 19:42:23 +02:00

pathhierarchy-tokenizer.asciidoc

Docs: Improved tokenizer docs (#18356)

2016-05-19 19:42:23 +02:00

pattern-tokenizer.asciidoc

Docs: Improved tokenizer docs (#18356)

2016-05-19 19:42:23 +02:00

standard-tokenizer.asciidoc

Docs: Improved tokenizer docs (#18356)

2016-05-19 19:42:23 +02:00

thai-tokenizer.asciidoc

Docs: Improved tokenizer docs (#18356)

2016-05-19 19:42:23 +02:00

uaxurlemail-tokenizer.asciidoc

Docs: Improved tokenizer docs (#18356)

2016-05-19 19:42:23 +02:00

whitespace-tokenizer.asciidoc

Docs: Improved tokenizer docs (#18356)

2016-05-19 19:42:23 +02:00