From e993c6a862aec1ba28371dd21788754bc5b4ea49 Mon Sep 17 00:00:00 2001 From: David Pilato Date: Mon, 9 Nov 2015 15:35:06 +0100 Subject: [PATCH] Migrate mapper attachements plugin to asciidoc Followup for #14605 --- .../plugins/mapper-attachments.asciidoc | 183 +++++++++++------- docs/plugins/mapper.asciidoc | 8 +- docs/plugins/plugin-script.asciidoc | 5 +- docs/reference/mapping/types.asciidoc | 4 +- 4 files changed, 123 insertions(+), 77 deletions(-) rename plugins/mapper-attachments/README.md => docs/plugins/mapper-attachments.asciidoc (70%) diff --git a/plugins/mapper-attachments/README.md b/docs/plugins/mapper-attachments.asciidoc similarity index 70% rename from plugins/mapper-attachments/README.md rename to docs/plugins/mapper-attachments.asciidoc index 7114ad7b2cf..c13d8ee0b4e 100644 --- a/plugins/mapper-attachments/README.md +++ b/docs/plugins/mapper-attachments.asciidoc @@ -1,25 +1,46 @@ -Mapper Attachments Type for Elasticsearch -========================================= +[[mapper-attachments]] +=== Mapper Attachments Plugin -The mapper attachments plugin lets Elasticsearch index file attachments in common formats (such as PPT, XLS, PDF) using the Apache text extraction library [Tika](http://lucene.apache.org/tika/). +The mapper attachments plugin lets Elasticsearch index file attachments in common formats (such as PPT, XLS, PDF) +using the Apache text extraction library http://lucene.apache.org/tika/[Tika]. -In practice, the plugin adds the `attachment` type when mapping properties so that documents can be populated with file attachment contents (encoded as `base64`). +In practice, the plugin adds the `attachment` type when mapping properties so that documents can be populated with +file attachment contents (encoded as `base64`). -Installation ------------- +[[mapper-attachments-install]] +[float] +==== Installation -In order to install the plugin, run: +This plugin can be installed using the plugin manager: -```sh -bin/plugin install mapper-attachments -``` +[source,sh] +---------------------------------------------------------------- +sudo bin/plugin install mapper-attachments +---------------------------------------------------------------- -Hello, world ------------- +The plugin must be installed on every node in the cluster, and each node must +be restarted after installation. + +[[mapper-attachments-remove]] +[float] +==== Removal + +The plugin can be removed with the following command: + +[source,sh] +---------------------------------------------------------------- +sudo bin/plugin remove mapper-attachments +---------------------------------------------------------------- + +The node must be stopped before removing the plugin. + +[[mapper-attachments-helloworld]] +==== Hello, world Create a property mapping using the new type `attachment`: -```javascript +[source,js] +-------------------------- POST /trying-out-mapper-attachments { "mappings": { @@ -27,36 +48,42 @@ POST /trying-out-mapper-attachments "properties": { "cv": { "type": "attachment" } }}}} -``` +-------------------------- +// AUTOSENSE Index a new document populated with a `base64`-encoded attachment: -```javascript +[source,js] +-------------------------- POST /trying-out-mapper-attachments/person/1 { "cv": "e1xydGYxXGFuc2kNCkxvcmVtIGlwc3VtIGRvbG9yIHNpdCBhbWV0DQpccGFyIH0=" } -``` +-------------------------- +// AUTOSENSE Search for the document using words in the attachment: -```javascript +[source,js] +-------------------------- POST /trying-out-mapper-attachments/person/_search { "query": { "query_string": { "query": "ipsum" }}} -``` +-------------------------- +// AUTOSENSE If you get a hit for your indexed document, the plugin should be installed and working. -Usage ------------------------- +[[mapper-attachments-usage]] +==== Usage Using the attachment type is simple, in your mapping JSON, simply set a certain JSON element as attachment, for example: -```javascript +[source,js] +-------------------------- PUT /test PUT /test/person/_mapping { @@ -66,20 +93,24 @@ PUT /test/person/_mapping } } } -``` +-------------------------- +// AUTOSENSE In this case, the JSON to index can be: -```javascript +[source,js] +-------------------------- PUT /test/person/1 { "my_attachment" : "... base64 encoded attachment ..." } -``` +-------------------------- +// AUTOSENSE Or it is possible to use more elaborated JSON if content type, resource name or language need to be set explicitly: -``` +[source,js] +-------------------------- PUT /test/person/1 { "my_attachment" : { @@ -89,9 +120,10 @@ PUT /test/person/1 "_content" : "... base64 encoded attachment ..." } } -``` +-------------------------- +// AUTOSENSE -The `attachment` type not only indexes the content of the doc in `content` sub field, but also automatically adds meta +The `attachment` type not only indexes the content of the doc in `content` sub field, but also automatically adds meta data on the attachment as well (when available). The metadata supported are: @@ -107,10 +139,11 @@ The metadata supported are: They can be queried using the "dot notation", for example: `my_attachment.author`. -Both the meta data and the actual content are simple core type mappers (string, date, ...), thus, they can be controlled +Both the meta data and the actual content are simple core type mappers (string, date, …), thus, they can be controlled in the mappings. For example: -```javascript +[source,js] +-------------------------- PUT /test/person/_mapping { "person" : { @@ -131,19 +164,21 @@ PUT /test/person/_mapping } } } -``` +-------------------------- +// AUTOSENSE In the above example, the actual content indexed is mapped under `fields` name `content`, and we decide not to index it, so it will only be available in the `_all` field. The other fields map to their respective metadata names, but there is no need to specify the `type` (like `string` or `date`) since it is already known. -Copy To feature ---------------- +[[mapper-attachments-copy-to]] +==== Copy To feature -If you want to use [copy_to](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html#copy-to) +If you want to use http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html#copy-to[copy_to] feature, you need to define it on each sub-field you want to copy to another field: -```javascript +[source,js] +-------------------------- PUT /test/person/_mapping { "person": { @@ -163,16 +198,18 @@ PUT /test/person/_mapping } } } -``` +-------------------------- +// AUTOSENSE In this example, the extracted content will be copy as well to `copy` field. -Querying or accessing metadata ------------------------------- +[[mapper-attachments-querying-metadata]] +==== Querying or accessing metadata If you need to query on metadata fields, use the attachment field name dot the metadata field. For example: -``` +[source,js] +-------------------------- DELETE /test PUT /test PUT /test/person/_mapping @@ -204,11 +241,13 @@ GET /test/person/_search } } } -``` +-------------------------- +// AUTOSENSE Will give you: -``` +[source,js] +-------------------------- { "took": 2, "timed_out": false, @@ -235,17 +274,18 @@ Will give you: ] } } -``` +-------------------------- -Indexed Characters ------------------- +[[mapper-attachments-indexed-characters]] +==== Indexed Characters By default, `100000` characters are extracted when indexing the content. This default value can be changed by setting the `index.mapping.attachment.indexed_chars` setting. It can also be provided on a per document indexed using the `_indexed_chars` parameter. `-1` can be set to extract all text, but note that all the text needs to be allowed to be represented in memory: -``` +[source,js] +-------------------------- PUT /test/person/1 { "my_attachment" : { @@ -253,18 +293,19 @@ PUT /test/person/1 "_content" : "... base64 encoded attachment ..." } } -``` +-------------------------- +// AUTOSENSE -Metadata parsing error handling -------------------------------- +[[mapper-attachments-error-handling]] +==== Metadata parsing error handling While extracting metadata content, errors could happen for example when parsing dates. Parsing errors are ignored so your document is indexed. You can disable this feature by setting the `index.mapping.attachment.ignore_errors` setting to `false`. -Language Detection ------------------- +[[mapper-attachments-language-detection]] +==== Language Detection By default, language detection is disabled (`false`) as it could come with a cost. This default value can be changed by setting the `index.mapping.attachment.detect_language` setting. @@ -272,22 +313,24 @@ It can also be provided on a per document indexed using the `_detect_language` p Note that you can force language using `_language` field when sending your actual document: -```javascript +[source,js] +-------------------------- { "my_attachment" : { "_language" : "en", "_content" : "... base64 encoded attachment ..." } } -``` +-------------------------- -Highlighting attachments ------------------------- +[[mapper-attachments-highlighting]] +==== Highlighting attachments -If you want to highlight your attachment content, you will need to set `"store": true` and `"term_vector":"with_positions_offsets"` -for your attachment field. Here is a full script which does it: +If you want to highlight your attachment content, you will need to set `"store": true` and +`"term_vector":"with_positions_offsets"` for your attachment field. Here is a full script which does it: -``` +[source,js] +-------------------------- DELETE /test PUT /test PUT /test/person/_mapping @@ -326,11 +369,13 @@ GET /test/person/_search } } } -``` +-------------------------- +// AUTOSENSE It gives back: -```js +[source,js] +-------------------------- { "took": 9, "timed_out": false, @@ -357,29 +402,31 @@ It gives back: ] } } -``` +-------------------------- -Stand alone runner ------------------- +[[mapper-attachments-standalone]] +==== Stand alone runner If you want to run some tests within your IDE, you can use `StandaloneRunner` class. It accepts arguments: -* `-u file://URL/TO/YOUR/DOC` -* `--size` set extracted size (default to mapper attachment size) -* `BASE64` encoded binary +* `-u file://URL/TO/YOUR/DOC` +* `--size` set extracted size (default to mapper attachment size) +* `BASE64` encoded binary Example: -```sh +[source,sh] +-------------------------- StandaloneRunner BASE64Text StandaloneRunner -u /tmp/mydoc.pdf StandaloneRunner -u /tmp/mydoc.pdf --size 1000000 -``` +-------------------------- It produces something like: -``` +[source,text] +-------------------------- ## Extracted text --------------------- BEGIN ----------------------- This is the extracted text @@ -393,4 +440,4 @@ This is the extracted text - language: null - name: null - title: null -``` +-------------------------- diff --git a/docs/plugins/mapper.asciidoc b/docs/plugins/mapper.asciidoc index c6a3a7b35aa..fcfc877f8f9 100644 --- a/docs/plugins/mapper.asciidoc +++ b/docs/plugins/mapper.asciidoc @@ -8,11 +8,10 @@ Mapper plugins allow new field datatypes to be added to Elasticsearch. The core mapper plugins are: -https://github.com/elasticsearch/elasticsearch-mapper-attachments[Mapper Attachments Type plugin]:: +<>:: -Integrates http://lucene.apache.org/tika/[Apache Tika] to provide a new field -type `attachment` to allow indexing of documents such as PDFs and Microsoft -Word. +The mapper-attachments integrates http://lucene.apache.org/tika/[Apache Tika] to provide a new field +type `attachment` to allow indexing of documents such as PDFs and Microsoft Word. <>:: @@ -25,5 +24,6 @@ indexes the size in bytes of the original The mapper-murmur3 plugin allows hashes to be computed at index-time and stored in the index for later use with the `cardinality` aggregation. +include::mapper-attachments.asciidoc[] include::mapper-size.asciidoc[] include::mapper-murmur3.asciidoc[] diff --git a/docs/plugins/plugin-script.asciidoc b/docs/plugins/plugin-script.asciidoc index 3f7a30556af..58ba1e53d71 100644 --- a/docs/plugins/plugin-script.asciidoc +++ b/docs/plugins/plugin-script.asciidoc @@ -75,13 +75,12 @@ sudo bin/plugin install lmenezes/elasticsearch-kopf/2.x <2> When installing from Maven Central/Sonatype, `[org]` should be replaced by the artifact `groupId`, and `[user|component]` by the `artifactId`. For -instance, to install the -https://github.com/elastic/elasticsearch-mapper-attachments[mapper attachment] +instance, to install the {plugins}/mapper-attachments.html[`mapper-attachments`] plugin from Sonatype, run: [source,shell] ----------------------------------- -sudo bin/plugin install org.elasticsearch/elasticsearch-mapper-attachments/2.6.0 <1> +sudo bin/plugin install org.elasticsearch.plugin/mapper-attachments/3.0.0 <1> ----------------------------------- <1> When installing from `download.elastic.co` or from Maven Central/Sonatype, the version is required. diff --git a/docs/reference/mapping/types.asciidoc b/docs/reference/mapping/types.asciidoc index 52cd41e37e5..60d96577a43 100644 --- a/docs/reference/mapping/types.asciidoc +++ b/docs/reference/mapping/types.asciidoc @@ -37,8 +37,8 @@ document: Attachment datatype:: - See the https://github.com/elastic/elasticsearch-mapper-attachments[mapper attachment plugin] - which supports indexing ``attachments'' like Microsoft Office formats, Open + See the {plugins}/mapper-attachments.html[`mapper-attachments`] plugin + which supports indexing `attachments` like Microsoft Office formats, Open Document formats, ePub, HTML, etc. into an `attachment` datatype. [float]