honeymoose
/
OpenSearch
mirror of https://github.com/honeymoose/OpenSearch.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'docs/mapper-attachments'

This commit is contained in:
David Pilato 2015-11-23 12:14:31 +01:00
parent 6e2e91cf2a e993c6a862
commit 5b0e2823b1
4 changed files with 123 additions and 77 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

183
plugins/mapper-attachments/README.md → docs/plugins/mapper-attachments.asciidoc
View File

@ -1,25 +1,46 @@
Mapper Attachments Type for Elasticsearch [[mapper-attachments]]
========================================= === Mapper Attachments Plugin
The mapper attachments plugin lets Elasticsearch index file attachments in common formats (such as PPT, XLS, PDF) using the Apache text extraction library [Tika](http://lucene.apache.org/tika/). The mapper attachments plugin lets Elasticsearch index file attachments in common formats (such as PPT, XLS, PDF)
using the Apache text extraction library http://lucene.apache.org/tika/[Tika].
In practice, the plugin adds the `attachment` type when mapping properties so that documents can be populated with file attachment contents (encoded as `base64`). In practice, the plugin adds the `attachment` type when mapping properties so that documents can be populated with
file attachment contents (encoded as `base64`).
Installation [[mapper-attachments-install]]
------------ [float]
==== Installation
In order to install the plugin, run: This plugin can be installed using the plugin manager:
```sh [source,sh]
bin/plugin install mapper-attachments ----------------------------------------------------------------
``` sudo bin/plugin install mapper-attachments
----------------------------------------------------------------
Hello, world The plugin must be installed on every node in the cluster, and each node must
------------ be restarted after installation.
[[mapper-attachments-remove]]
[float]
==== Removal
The plugin can be removed with the following command:
[source,sh]
----------------------------------------------------------------
sudo bin/plugin remove mapper-attachments
----------------------------------------------------------------
The node must be stopped before removing the plugin.
[[mapper-attachments-helloworld]]
==== Hello, world
Create a property mapping using the new type `attachment`: Create a property mapping using the new type `attachment`:
```javascript [source,js]
--------------------------
POST /trying-out-mapper-attachments POST /trying-out-mapper-attachments
{ {
"mappings": { "mappings": {
@ -27,36 +48,42 @@ POST /trying-out-mapper-attachments
"properties": { "properties": {
"cv": { "type": "attachment" } "cv": { "type": "attachment" }
}}}} }}}}
``` --------------------------
// AUTOSENSE
Index a new document populated with a `base64`-encoded attachment: Index a new document populated with a `base64`-encoded attachment:
```javascript [source,js]
--------------------------
POST /trying-out-mapper-attachments/person/1 POST /trying-out-mapper-attachments/person/1
{ {
"cv": "e1xydGYxXGFuc2kNCkxvcmVtIGlwc3VtIGRvbG9yIHNpdCBhbWV0DQpccGFyIH0=" "cv": "e1xydGYxXGFuc2kNCkxvcmVtIGlwc3VtIGRvbG9yIHNpdCBhbWV0DQpccGFyIH0="
} }
``` --------------------------
// AUTOSENSE
Search for the document using words in the attachment: Search for the document using words in the attachment:
```javascript [source,js]
--------------------------
POST /trying-out-mapper-attachments/person/_search POST /trying-out-mapper-attachments/person/_search
{ {
"query": { "query": {
"query_string": { "query_string": {
"query": "ipsum" "query": "ipsum"
}}} }}}
``` --------------------------
// AUTOSENSE
If you get a hit for your indexed document, the plugin should be installed and working. If you get a hit for your indexed document, the plugin should be installed and working.
Usage [[mapper-attachments-usage]]
------------------------ ==== Usage
Using the attachment type is simple, in your mapping JSON, simply set a certain JSON element as attachment, for example: Using the attachment type is simple, in your mapping JSON, simply set a certain JSON element as attachment, for example:
```javascript [source,js]
--------------------------
PUT /test PUT /test
PUT /test/person/_mapping PUT /test/person/_mapping
{ {
@ -66,20 +93,24 @@ PUT /test/person/_mapping
} }
} }
} }
``` --------------------------
// AUTOSENSE
In this case, the JSON to index can be: In this case, the JSON to index can be:
```javascript [source,js]
--------------------------
PUT /test/person/1 PUT /test/person/1
{ {
"my_attachment" : "... base64 encoded attachment ..." "my_attachment" : "... base64 encoded attachment ..."
} }
``` --------------------------
// AUTOSENSE
Or it is possible to use more elaborated JSON if content type, resource name or language need to be set explicitly: Or it is possible to use more elaborated JSON if content type, resource name or language need to be set explicitly:
``` [source,js]
--------------------------
PUT /test/person/1 PUT /test/person/1
{ {
"my_attachment" : { "my_attachment" : {
@ -89,9 +120,10 @@ PUT /test/person/1
"_content" : "... base64 encoded attachment ..." "_content" : "... base64 encoded attachment ..."
} }
} }
``` --------------------------
// AUTOSENSE
The `attachment` type not only indexes the content of the doc in `content` sub field, but also automatically adds meta The `attachment` type not only indexes the content of the doc in `content` sub field, but also automatically adds meta
data on the attachment as well (when available). data on the attachment as well (when available).
The metadata supported are: The metadata supported are:
@ -107,10 +139,11 @@ The metadata supported are:
They can be queried using the "dot notation", for example: `my_attachment.author`. They can be queried using the "dot notation", for example: `my_attachment.author`.
Both the meta data and the actual content are simple core type mappers (string, date, ...), thus, they can be controlled Both the meta data and the actual content are simple core type mappers (string, date, …), thus, they can be controlled
in the mappings. For example: in the mappings. For example:
```javascript [source,js]
--------------------------
PUT /test/person/_mapping PUT /test/person/_mapping
{ {
"person" : { "person" : {
@ -131,19 +164,21 @@ PUT /test/person/_mapping
} }
} }
} }
``` --------------------------
// AUTOSENSE
In the above example, the actual content indexed is mapped under `fields` name `content`, and we decide not to index it, so In the above example, the actual content indexed is mapped under `fields` name `content`, and we decide not to index it, so
it will only be available in the `_all` field. The other fields map to their respective metadata names, but there is no it will only be available in the `_all` field. The other fields map to their respective metadata names, but there is no
need to specify the `type` (like `string` or `date`) since it is already known. need to specify the `type` (like `string` or `date`) since it is already known.
Copy To feature [[mapper-attachments-copy-to]]
--------------- ==== Copy To feature
If you want to use [copy_to](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html#copy-to) If you want to use http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html#copy-to[copy_to]
feature, you need to define it on each sub-field you want to copy to another field: feature, you need to define it on each sub-field you want to copy to another field:
```javascript [source,js]
--------------------------
PUT /test/person/_mapping PUT /test/person/_mapping
{ {
"person": { "person": {
@ -163,16 +198,18 @@ PUT /test/person/_mapping
} }
} }
} }
``` --------------------------
// AUTOSENSE
In this example, the extracted content will be copy as well to `copy` field. In this example, the extracted content will be copy as well to `copy` field.
Querying or accessing metadata [[mapper-attachments-querying-metadata]]
------------------------------ ==== Querying or accessing metadata
If you need to query on metadata fields, use the attachment field name dot the metadata field. For example: If you need to query on metadata fields, use the attachment field name dot the metadata field. For example:
``` [source,js]
--------------------------
DELETE /test DELETE /test
PUT /test PUT /test
PUT /test/person/_mapping PUT /test/person/_mapping
@ -204,11 +241,13 @@ GET /test/person/_search
} }
} }
} }
``` --------------------------
// AUTOSENSE
Will give you: Will give you:
``` [source,js]
--------------------------
{ {
"took": 2, "took": 2,
"timed_out": false, "timed_out": false,
@ -235,17 +274,18 @@ Will give you:
] ]
} }
} }
``` --------------------------
Indexed Characters [[mapper-attachments-indexed-characters]]
------------------ ==== Indexed Characters
By default, `100000` characters are extracted when indexing the content. This default value can be changed by setting By default, `100000` characters are extracted when indexing the content. This default value can be changed by setting
the `index.mapping.attachment.indexed_chars` setting. It can also be provided on a per document indexed using the the `index.mapping.attachment.indexed_chars` setting. It can also be provided on a per document indexed using the
`_indexed_chars` parameter. `-1` can be set to extract all text, but note that all the text needs to be allowed to be `_indexed_chars` parameter. `-1` can be set to extract all text, but note that all the text needs to be allowed to be
represented in memory: represented in memory:
``` [source,js]
--------------------------
PUT /test/person/1 PUT /test/person/1
{ {
"my_attachment" : { "my_attachment" : {
@ -253,18 +293,19 @@ PUT /test/person/1
"_content" : "... base64 encoded attachment ..." "_content" : "... base64 encoded attachment ..."
} }
} }
``` --------------------------
// AUTOSENSE
Metadata parsing error handling [[mapper-attachments-error-handling]]
------------------------------- ==== Metadata parsing error handling
While extracting metadata content, errors could happen for example when parsing dates. While extracting metadata content, errors could happen for example when parsing dates.
Parsing errors are ignored so your document is indexed. Parsing errors are ignored so your document is indexed.
You can disable this feature by setting the `index.mapping.attachment.ignore_errors` setting to `false`. You can disable this feature by setting the `index.mapping.attachment.ignore_errors` setting to `false`.
Language Detection [[mapper-attachments-language-detection]]
------------------ ==== Language Detection
By default, language detection is disabled (`false`) as it could come with a cost. By default, language detection is disabled (`false`) as it could come with a cost.
This default value can be changed by setting the `index.mapping.attachment.detect_language` setting. This default value can be changed by setting the `index.mapping.attachment.detect_language` setting.
@ -272,22 +313,24 @@ It can also be provided on a per document indexed using the `_detect_language` p
Note that you can force language using `_language` field when sending your actual document: Note that you can force language using `_language` field when sending your actual document:
```javascript [source,js]
--------------------------
{ {
"my_attachment" : { "my_attachment" : {
"_language" : "en", "_language" : "en",
"_content" : "... base64 encoded attachment ..." "_content" : "... base64 encoded attachment ..."
} }
} }
``` --------------------------
Highlighting attachments [[mapper-attachments-highlighting]]
------------------------ ==== Highlighting attachments
If you want to highlight your attachment content, you will need to set `"store": true` and `"term_vector":"with_positions_offsets"` If you want to highlight your attachment content, you will need to set `"store": true` and
for your attachment field. Here is a full script which does it: `"term_vector":"with_positions_offsets"` for your attachment field. Here is a full script which does it:
``` [source,js]
--------------------------
DELETE /test DELETE /test
PUT /test PUT /test
PUT /test/person/_mapping PUT /test/person/_mapping
@ -326,11 +369,13 @@ GET /test/person/_search
} }
} }
} }
``` --------------------------
// AUTOSENSE
It gives back: It gives back:
```js [source,js]
--------------------------
{ {
"took": 9, "took": 9,
"timed_out": false, "timed_out": false,
@ -357,29 +402,31 @@ It gives back:
] ]
} }
} }
``` --------------------------
Stand alone runner [[mapper-attachments-standalone]]
------------------ ==== Stand alone runner
If you want to run some tests within your IDE, you can use `StandaloneRunner` class. If you want to run some tests within your IDE, you can use `StandaloneRunner` class.
It accepts arguments: It accepts arguments:
* `-u file://URL/TO/YOUR/DOC` * `-u file://URL/TO/YOUR/DOC`
* `--size` set extracted size (default to mapper attachment size) * `--size` set extracted size (default to mapper attachment size)
* `BASE64` encoded binary * `BASE64` encoded binary
Example: Example:
```sh [source,sh]
--------------------------
StandaloneRunner BASE64Text StandaloneRunner BASE64Text
StandaloneRunner -u /tmp/mydoc.pdf StandaloneRunner -u /tmp/mydoc.pdf
StandaloneRunner -u /tmp/mydoc.pdf --size 1000000 StandaloneRunner -u /tmp/mydoc.pdf --size 1000000
``` --------------------------
It produces something like: It produces something like:
``` [source,text]
--------------------------
## Extracted text ## Extracted text
--------------------- BEGIN ----------------------- --------------------- BEGIN -----------------------
This is the extracted text This is the extracted text
@ -393,4 +440,4 @@ This is the extracted text
- language: null - language: null
- name: null - name: null
- title: null - title: null
``` --------------------------

8
docs/plugins/mapper.asciidoc
View File

@ -8,11 +8,10 @@ Mapper plugins allow new field datatypes to be added to Elasticsearch.
The core mapper plugins are: The core mapper plugins are:
https://github.com/elasticsearch/elasticsearch-mapper-attachments[Mapper Attachments Type plugin]:: <<mapper-attachments>>::
Integrates http://lucene.apache.org/tika/[Apache Tika] to provide a new field The mapper-attachments integrates http://lucene.apache.org/tika/[Apache Tika] to provide a new field
type `attachment` to allow indexing of documents such as PDFs and Microsoft type `attachment` to allow indexing of documents such as PDFs and Microsoft Word.
Word.
<<mapper-size>>:: <<mapper-size>>::
@ -25,5 +24,6 @@ indexes the size in bytes of the original
The mapper-murmur3 plugin allows hashes to be computed at index-time and stored The mapper-murmur3 plugin allows hashes to be computed at index-time and stored
in the index for later use with the `cardinality` aggregation. in the index for later use with the `cardinality` aggregation.
include::mapper-attachments.asciidoc[]
include::mapper-size.asciidoc[] include::mapper-size.asciidoc[]
include::mapper-murmur3.asciidoc[] include::mapper-murmur3.asciidoc[]

5
docs/plugins/plugin-script.asciidoc
View File

@ -75,13 +75,12 @@ sudo bin/plugin install lmenezes/elasticsearch-kopf/2.x <2>
When installing from Maven Central/Sonatype, `[org]` should be replaced by When installing from Maven Central/Sonatype, `[org]` should be replaced by
the artifact `groupId`, and `[user|component]` by the `artifactId`. For the artifact `groupId`, and `[user|component]` by the `artifactId`. For
instance, to install the instance, to install the {plugins}/mapper-attachments.html[`mapper-attachments`]
https://github.com/elastic/elasticsearch-mapper-attachments[mapper attachment]
plugin from Sonatype, run: plugin from Sonatype, run:
[source,shell] [source,shell]
----------------------------------- -----------------------------------
sudo bin/plugin install org.elasticsearch/elasticsearch-mapper-attachments/2.6.0 <1> sudo bin/plugin install org.elasticsearch.plugin/mapper-attachments/3.0.0 <1>
----------------------------------- -----------------------------------
<1> When installing from `download.elastic.co` or from Maven Central/Sonatype, the <1> When installing from `download.elastic.co` or from Maven Central/Sonatype, the
version is required. version is required.

4
docs/reference/mapping/types.asciidoc
View File

@ -37,8 +37,8 @@ document:
Attachment datatype:: Attachment datatype::
See the https://github.com/elastic/elasticsearch-mapper-attachments[mapper attachment plugin] See the {plugins}/mapper-attachments.html[`mapper-attachments`] plugin
which supports indexing ``attachments'' like Microsoft Office formats, Open which supports indexing `attachments` like Microsoft Office formats, Open
Document formats, ePub, HTML, etc. into an `attachment` datatype. Document formats, ePub, HTML, etc. into an `attachment` datatype.
[float] [float]