mirror of
https://github.com/honeymoose/OpenSearch.git
synced 2025-03-01 00:19:11 +00:00
9c85569883
We weren't doing it before because we weren't starting the plugins. Now we are. The hardest part of this was handling the files the tests expect to be on the filesystem. extraConfigFiles was broken.
417 lines
10 KiB
Plaintext
417 lines
10 KiB
Plaintext
[[mapper-attachments]]
|
|
=== Mapper Attachments Plugin
|
|
|
|
deprecated[5.0.0,The `mapper-attachments` plugin has been replaced by the <<ingest-attachment, `ingest-attachment`>> plugin]
|
|
|
|
The mapper attachments plugin lets Elasticsearch index file attachments in common formats (such as PPT, XLS, PDF)
|
|
using the Apache text extraction library http://lucene.apache.org/tika/[Tika].
|
|
|
|
In practice, the plugin adds the `attachment` type when mapping properties so that documents can be populated with
|
|
file attachment contents (encoded as `base64`).
|
|
|
|
[[mapper-attachments-install]]
|
|
[float]
|
|
==== Installation
|
|
|
|
This plugin can be installed using the plugin manager:
|
|
|
|
[source,sh]
|
|
----------------------------------------------------------------
|
|
sudo bin/elasticsearch-plugin install mapper-attachments
|
|
----------------------------------------------------------------
|
|
|
|
The plugin must be installed on every node in the cluster, and each node must
|
|
be restarted after installation.
|
|
|
|
[[mapper-attachments-remove]]
|
|
[float]
|
|
==== Removal
|
|
|
|
The plugin can be removed with the following command:
|
|
|
|
[source,sh]
|
|
----------------------------------------------------------------
|
|
sudo bin/elasticsearch-plugin remove mapper-attachments
|
|
----------------------------------------------------------------
|
|
|
|
The node must be stopped before removing the plugin.
|
|
|
|
[[mapper-attachments-helloworld]]
|
|
==== Hello, world
|
|
|
|
Create a property mapping using the new type `attachment`:
|
|
|
|
[source,js]
|
|
--------------------------
|
|
POST /trying-out-mapper-attachments
|
|
{
|
|
"mappings": {
|
|
"person": {
|
|
"properties": {
|
|
"cv": { "type": "attachment" }
|
|
}}}}
|
|
--------------------------
|
|
// CONSOLE
|
|
|
|
Index a new document populated with a `base64`-encoded attachment:
|
|
|
|
[source,js]
|
|
--------------------------
|
|
POST /trying-out-mapper-attachments/person/1?refresh
|
|
{
|
|
"cv": "e1xydGYxXGFuc2kNCkxvcmVtIGlwc3VtIGRvbG9yIHNpdCBhbWV0DQpccGFyIH0="
|
|
}
|
|
--------------------------
|
|
// CONSOLE
|
|
// TEST[continued]
|
|
|
|
Search for the document using words in the attachment:
|
|
|
|
[source,js]
|
|
--------------------------
|
|
POST /trying-out-mapper-attachments/person/_search
|
|
{
|
|
"query": {
|
|
"query_string": {
|
|
"query": "ipsum"
|
|
}}}
|
|
--------------------------
|
|
// CONSOLE
|
|
// TEST[continued]
|
|
|
|
If you get a hit for your indexed document, the plugin should be installed and working. It'll look like:
|
|
|
|
[source,js]
|
|
--------------------------
|
|
{
|
|
"timed_out": false,
|
|
"took": 53,
|
|
"hits": {
|
|
"total": 1,
|
|
"max_score": 0.3125,
|
|
"hits": [
|
|
{
|
|
"_score": 0.3125,
|
|
"_index": "trying-out-mapper-attachments",
|
|
"_type": "person",
|
|
"_id": "1",
|
|
"_source": {
|
|
"cv": "e1xydGYxXGFuc2kNCkxvcmVtIGlwc3VtIGRvbG9yIHNpdCBhbWV0DQpccGFyIH0="
|
|
}
|
|
}
|
|
]
|
|
},
|
|
"_shards": ...
|
|
}
|
|
--------------------------
|
|
// TESTRESPONSE[s/"took": 53/"took": "$body.took"/]
|
|
// TESTRESPONSE[s/"_shards": \.\.\./"_shards": "$body._shards"/]
|
|
|
|
[[mapper-attachments-usage]]
|
|
==== Usage
|
|
|
|
Using the attachment type is simple, in your mapping JSON, simply set a certain JSON element as attachment, for example:
|
|
|
|
[source,js]
|
|
--------------------------
|
|
PUT /test
|
|
{
|
|
"mappings": {
|
|
"person" : {
|
|
"properties" : {
|
|
"my_attachment" : { "type" : "attachment" }
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------
|
|
// CONSOLE
|
|
|
|
In this case, the JSON to index can be:
|
|
|
|
[source,js]
|
|
--------------------------
|
|
PUT /test/person/1
|
|
{
|
|
"my_attachment" : "... base64 encoded attachment ..."
|
|
}
|
|
--------------------------
|
|
// CONSOLE
|
|
|
|
Or it is possible to use more elaborated JSON if content type, resource name or language need to be set explicitly:
|
|
|
|
[source,js]
|
|
--------------------------
|
|
PUT /test/person/1
|
|
{
|
|
"my_attachment" : {
|
|
"_content_type" : "application/pdf",
|
|
"_name" : "resource/name/of/my.pdf",
|
|
"_language" : "en",
|
|
"_content" : "... base64 encoded attachment ..."
|
|
}
|
|
}
|
|
--------------------------
|
|
// CONSOLE
|
|
|
|
The `attachment` type not only indexes the content of the doc in `content` sub field, but also automatically adds meta
|
|
data on the attachment as well (when available).
|
|
|
|
The metadata supported are:
|
|
|
|
* `date`
|
|
* `title`
|
|
* `name` only available if you set `_name` see above
|
|
* `author`
|
|
* `keywords`
|
|
* `content_type`
|
|
* `content_length` is the original content_length before text extraction (aka file size)
|
|
* `language`
|
|
|
|
They can be queried using the "dot notation", for example: `my_attachment.author`.
|
|
|
|
Both the meta data and the actual content are simple core type mappers (text, date, …), thus, they can be controlled
|
|
in the mappings. For example:
|
|
|
|
[source,js]
|
|
--------------------------
|
|
PUT /test
|
|
{
|
|
"settings": {
|
|
"index": {
|
|
"analysis": {
|
|
"analyzer": {
|
|
"my_analyzer": {
|
|
"type": "custom",
|
|
"tokenizer": "standard",
|
|
"filter": ["standard"]
|
|
}
|
|
}
|
|
}
|
|
}
|
|
},
|
|
"mappings": {
|
|
"person" : {
|
|
"properties" : {
|
|
"file" : {
|
|
"type" : "attachment",
|
|
"fields" : {
|
|
"content" : {"index" : "no"},
|
|
"title" : {"store" : "yes"},
|
|
"date" : {"store" : "yes"},
|
|
"author" : {"analyzer" : "my_analyzer"},
|
|
"keywords" : {"store" : "yes"},
|
|
"content_type" : {"store" : "yes"},
|
|
"content_length" : {"store" : "yes"},
|
|
"language" : {"store" : "yes"}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------
|
|
// CONSOLE
|
|
|
|
In the above example, the actual content indexed is mapped under `fields` name `content`, and we decide not to index it, so
|
|
it will only be available in the `_all` field. The other fields map to their respective metadata names, but there is no
|
|
need to specify the `type` (like `text` or `date`) since it is already known.
|
|
|
|
==== Querying or accessing metadata
|
|
|
|
If you need to query on metadata fields, use the attachment field name dot the metadata field. For example:
|
|
|
|
[source,js]
|
|
--------------------------
|
|
PUT /test
|
|
PUT /test/person/_mapping
|
|
{
|
|
"person": {
|
|
"properties": {
|
|
"file": {
|
|
"type": "attachment",
|
|
"fields": {
|
|
"content_type": {
|
|
"type": "text",
|
|
"store": true
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
PUT /test/person/1?refresh=true
|
|
{
|
|
"file": "IkdvZCBTYXZlIHRoZSBRdWVlbiIgKGFsdGVybmF0aXZlbHkgIkdvZCBTYXZlIHRoZSBLaW5nIg=="
|
|
}
|
|
GET /test/person/_search
|
|
{
|
|
"fields": [ "file.content_type" ],
|
|
"query": {
|
|
"match": {
|
|
"file.content_type": "text plain"
|
|
}
|
|
}
|
|
}
|
|
--------------------------
|
|
// CONSOLE
|
|
|
|
Will give you:
|
|
|
|
[source,js]
|
|
--------------------------
|
|
{
|
|
"took": 2,
|
|
"timed_out": false,
|
|
"_shards": {
|
|
"total": 5,
|
|
"successful": 5,
|
|
"failed": 0
|
|
},
|
|
"hits": {
|
|
"total": 1,
|
|
"max_score": 0.16273327,
|
|
"hits": [
|
|
{
|
|
"_index": "test",
|
|
"_type": "person",
|
|
"_id": "1",
|
|
"_score": 0.16273327,
|
|
"fields": {
|
|
"file.content_type": [
|
|
"text/plain; charset=ISO-8859-1"
|
|
]
|
|
}
|
|
}
|
|
]
|
|
}
|
|
}
|
|
--------------------------
|
|
|
|
[[mapper-attachments-indexed-characters]]
|
|
==== Indexed Characters
|
|
|
|
By default, `100000` characters are extracted when indexing the content. This default value can be changed by setting
|
|
the `index.mapping.attachment.indexed_chars` setting. It can also be provided on a per document indexed using the
|
|
`_indexed_chars` parameter. `-1` can be set to extract all text, but note that all the text needs to be allowed to be
|
|
represented in memory:
|
|
|
|
[source,js]
|
|
--------------------------
|
|
PUT /test/person/1
|
|
{
|
|
"my_attachment" : {
|
|
"_indexed_chars" : -1,
|
|
"_content" : "... base64 encoded attachment ..."
|
|
}
|
|
}
|
|
--------------------------
|
|
// CONSOLE
|
|
|
|
[[mapper-attachments-error-handling]]
|
|
==== Metadata parsing error handling
|
|
|
|
While extracting metadata content, errors could happen for example when parsing dates.
|
|
Parsing errors are ignored so your document is indexed.
|
|
|
|
You can disable this feature by setting the `index.mapping.attachment.ignore_errors` setting to `false`.
|
|
|
|
[[mapper-attachments-language-detection]]
|
|
==== Language Detection
|
|
|
|
By default, language detection is disabled (`false`) as it could come with a cost.
|
|
This default value can be changed by setting the `index.mapping.attachment.detect_language` setting.
|
|
It can also be provided on a per document indexed using the `_detect_language` parameter.
|
|
|
|
Note that you can force language using `_language` field when sending your actual document:
|
|
|
|
[source,js]
|
|
--------------------------
|
|
{
|
|
"my_attachment" : {
|
|
"_language" : "en",
|
|
"_content" : "... base64 encoded attachment ..."
|
|
}
|
|
}
|
|
--------------------------
|
|
|
|
[[mapper-attachments-highlighting]]
|
|
==== Highlighting attachments
|
|
|
|
If you want to highlight your attachment content, you will need to set `"store": true` and
|
|
`"term_vector":"with_positions_offsets"` for your attachment field. Here is a full script which does it:
|
|
|
|
[source,js]
|
|
--------------------------
|
|
PUT /test
|
|
PUT /test/person/_mapping
|
|
{
|
|
"person": {
|
|
"properties": {
|
|
"file": {
|
|
"type": "attachment",
|
|
"fields": {
|
|
"content": {
|
|
"type": "text",
|
|
"term_vector":"with_positions_offsets",
|
|
"store": true
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
PUT /test/person/1?refresh=true
|
|
{
|
|
"file": "IkdvZCBTYXZlIHRoZSBRdWVlbiIgKGFsdGVybmF0aXZlbHkgIkdvZCBTYXZlIHRoZSBLaW5nIg=="
|
|
}
|
|
GET /test/person/_search
|
|
{
|
|
"fields": [],
|
|
"query": {
|
|
"match": {
|
|
"file.content": "king queen"
|
|
}
|
|
},
|
|
"highlight": {
|
|
"fields": {
|
|
"file.content": {
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------
|
|
// CONSOLE
|
|
|
|
It gives back:
|
|
|
|
[source,js]
|
|
--------------------------
|
|
{
|
|
"took": 9,
|
|
"timed_out": false,
|
|
"_shards": {
|
|
"total": 1,
|
|
"successful": 1,
|
|
"failed": 0
|
|
},
|
|
"hits": {
|
|
"total": 1,
|
|
"max_score": 0.13561106,
|
|
"hits": [
|
|
{
|
|
"_index": "test",
|
|
"_type": "person",
|
|
"_id": "1",
|
|
"_score": 0.13561106,
|
|
"highlight": {
|
|
"file.content": [
|
|
"\"God Save the <em>Queen</em>\" (alternatively \"God Save the <em>King</em>\"\n"
|
|
]
|
|
}
|
|
}
|
|
]
|
|
}
|
|
}
|
|
--------------------------
|