honeymoose
/
OpenSearch
mirror of https://github.com/honeymoose/OpenSearch.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
OpenSearch/docs/plugins/ingest-attachment.asciidoc

106 lines
3.4 KiB
Plaintext
Raw Normal View History

Ingest: Add attachment processor This is a simple port of the mapper attachment plugin to the ingest functionality, no new features. The only option is to limit the number of chars to prevent indexing of huge documents. Fields can be selected in the processor as well. Close #16303
2016-02-09 08:57:05 -05:00
[[ingest-attachment]]
Add ingest plugins to Elasticsearch plugin docs
2016-02-11 20:40:32 -05:00
=== Ingest Attachment Processor Plugin
Ingest: Add attachment processor This is a simple port of the mapper attachment plugin to the ingest functionality, no new features. The only option is to limit the number of chars to prevent indexing of huge documents. Fields can be selected in the processor as well. Close #16303
2016-02-09 08:57:05 -05:00
Edits to ingest plugin docs
2016-03-04 01:49:31 -05:00
The ingest attachment plugin lets Elasticsearch extract file attachments in common formats (such as PPT, XLS, and PDF) by
Ingest: Add attachment processor This is a simple port of the mapper attachment plugin to the ingest functionality, no new features. The only option is to limit the number of chars to prevent indexing of huge documents. Fields can be selected in the processor as well. Close #16303
2016-02-09 08:57:05 -05:00
using the Apache text extraction library http://lucene.apache.org/tika/[Tika].
Edits to ingest plugin docs
2016-03-04 01:49:31 -05:00
You can use the ingest attachment plugin as a replacement for the mapper attachment plugin.
Ingest: Add attachment processor This is a simple port of the mapper attachment plugin to the ingest functionality, no new features. The only option is to limit the number of chars to prevent indexing of huge documents. Fields can be selected in the processor as well. Close #16303
2016-02-09 08:57:05 -05:00
Ingest Attachment: Allow to prevent base64 conversions by using raw bytes (#16601) CBOR is natively supported in Elasticsearch and allows for byte arrays. This means, that by using CBOR the user can prevent base64 conversions for the data being sent back and forth. This PR adds support to extract data from a byte array in addition to a string. This also required to add a ByteArrayValueSource class.
2016-04-11 08:14:56 -04:00
The source field must be a base64 encoded binary. If you do not want to incur
the overhead of converting back and forth between base64, you can use the CBOR
format instead of JSON and specify the field as a bytes array instead of a string
representation. The processor will skip the base64 decoding then.
Ingest: Add attachment processor This is a simple port of the mapper attachment plugin to the ingest functionality, no new features. The only option is to limit the number of chars to prevent indexing of huge documents. Fields can be selected in the processor as well. Close #16303
2016-02-09 08:57:05 -05:00
Installation and removal instructions for Ingest Plugins (#18769) * Add install and remove instructions to ingest-geoip and ingest-attachment docs
2016-06-08 15:55:59 -04:00
[[ingest-attachment-install]]
[float]
==== Installation
This plugin can be installed using the plugin manager:
[source,sh]
----------------------------------------------------------------
sudo bin/elasticsearch-plugin install ingest-attachment
----------------------------------------------------------------
Add `// NOTCONSOLE` to docs We have 1074 snippets that look like they should be converted to `// CONSOLE`. At least that is what `gradle docs:listConsoleCandidates` says. This adds `// NOTCONSOLE` to explicitly mark snippets that *shouldn't* be converted to `// CONSOLE`. After marking the blindingly obvious ones this cuts the remaining snippet count to 1032.
2016-08-10 18:00:58 -04:00
// NOTCONSOLE
Installation and removal instructions for Ingest Plugins (#18769) * Add install and remove instructions to ingest-geoip and ingest-attachment docs
2016-06-08 15:55:59 -04:00
The plugin must be installed on every node in the cluster, and each node must
be restarted after installation.
[[ingest-attachment-remove]]
[float]
==== Removal
The plugin can be removed with the following command:
[source,sh]
----------------------------------------------------------------
sudo bin/elasticsearch-plugin remove ingest-attachment
----------------------------------------------------------------
Add `// NOTCONSOLE` to docs We have 1074 snippets that look like they should be converted to `// CONSOLE`. At least that is what `gradle docs:listConsoleCandidates` says. This adds `// NOTCONSOLE` to explicitly mark snippets that *shouldn't* be converted to `// CONSOLE`. After marking the blindingly obvious ones this cuts the remaining snippet count to 1032.
2016-08-10 18:00:58 -04:00
// NOTCONSOLE
Installation and removal instructions for Ingest Plugins (#18769) * Add install and remove instructions to ingest-geoip and ingest-attachment docs
2016-06-08 15:55:59 -04:00
The node must be stopped before removing the plugin.
[[using-ingest-attachment]]
==== Using the Attachment Processor in a Pipeline
Ingest: Add attachment processor This is a simple port of the mapper attachment plugin to the ingest functionality, no new features. The only option is to limit the number of chars to prevent indexing of huge documents. Fields can be selected in the processor as well. Close #16303
2016-02-09 08:57:05 -05:00
[[ingest-attachment-options]]
.Attachment options
[options="header"]
|======
| Name | Required | Default | Description
ingest: Streamline option naming for several processors: * `rename` processor, renamed `to` to `target_field` * `date` processor, renamed `match_field` to `field` and renamed `match_formats` to `formats` * `geoip` processor, renamed `source_field` to `field` and renamed `fields` to `properties` * `attachment` processor, renamed `source_field` to `field` and renamed `fields` to `properties` Closes #17835
2016-04-20 12:00:11 -04:00
| `field` | yes | - | The field to get the base64 encoded field from
Ingest: Add attachment processor This is a simple port of the mapper attachment plugin to the ingest functionality, no new features. The only option is to limit the number of chars to prevent indexing of huge documents. Fields can be selected in the processor as well. Close #16303
2016-02-09 08:57:05 -05:00
| `target_field` | no | attachment | The field that will hold the attachment information
| `indexed_chars` | no | 100000 | The number of chars being used for extraction to prevent huge fields. Use `-1` for no limit.
ingest: Streamline option naming for several processors: * `rename` processor, renamed `to` to `target_field` * `date` processor, renamed `match_field` to `field` and renamed `match_formats` to `formats` * `geoip` processor, renamed `source_field` to `field` and renamed `fields` to `properties` * `attachment` processor, renamed `source_field` to `field` and renamed `fields` to `properties` Closes #17835
2016-04-20 12:00:11 -04:00
| `properties` | no | all | Properties to select to be stored. Can be `content`, `title`, `name`, `author`, `keywords`, `date`, `content_type`, `content_length`, `language`
Ingest: Add attachment processor This is a simple port of the mapper attachment plugin to the ingest functionality, no new features. The only option is to limit the number of chars to prevent indexing of huge documents. Fields can be selected in the processor as well. Close #16303
2016-02-09 08:57:05 -05:00
|======
Add complete examples to some ingest docs These examples should make it more clear what the plugins do and they test that the snippets actually work. Relates to #18160
2016-08-10 11:12:41 -04:00
For example, this:
Ingest: Add attachment processor This is a simple port of the mapper attachment plugin to the ingest functionality, no new features. The only option is to limit the number of chars to prevent indexing of huge documents. Fields can be selected in the processor as well. Close #16303
2016-02-09 08:57:05 -05:00
[source,js]
--------------------------------------------------
Add complete examples to some ingest docs These examples should make it more clear what the plugins do and they test that the snippets actually work. Relates to #18160
2016-08-10 11:12:41 -04:00
PUT _ingest/pipeline/attachment
Ingest: Add attachment processor This is a simple port of the mapper attachment plugin to the ingest functionality, no new features. The only option is to limit the number of chars to prevent indexing of huge documents. Fields can be selected in the processor as well. Close #16303
2016-02-09 08:57:05 -05:00
{
Add complete examples to some ingest docs These examples should make it more clear what the plugins do and they test that the snippets actually work. Relates to #18160
2016-08-10 11:12:41 -04:00
"description" : "Extract attachment information",
Ingest: Add attachment processor This is a simple port of the mapper attachment plugin to the ingest functionality, no new features. The only option is to limit the number of chars to prevent indexing of huge documents. Fields can be selected in the processor as well. Close #16303
2016-02-09 08:57:05 -05:00
"processors" : [
{
"attachment" : {
ingest: Streamline option naming for several processors: * `rename` processor, renamed `to` to `target_field` * `date` processor, renamed `match_field` to `field` and renamed `match_formats` to `formats` * `geoip` processor, renamed `source_field` to `field` and renamed `fields` to `properties` * `attachment` processor, renamed `source_field` to `field` and renamed `fields` to `properties` Closes #17835
2016-04-20 12:00:11 -04:00
"field" : "data"
Ingest: Add attachment processor This is a simple port of the mapper attachment plugin to the ingest functionality, no new features. The only option is to limit the number of chars to prevent indexing of huge documents. Fields can be selected in the processor as well. Close #16303
2016-02-09 08:57:05 -05:00
}
}
]
}
Add complete examples to some ingest docs These examples should make it more clear what the plugins do and they test that the snippets actually work. Relates to #18160
2016-08-10 11:12:41 -04:00
PUT my_index/my_type/my_id?pipeline=attachment
{
"data": "e1xydGYxXGFuc2kNCkxvcmVtIGlwc3VtIGRvbG9yIHNpdCBhbWV0DQpccGFyIH0="
}
GET my_index/my_type/my_id
--------------------------------------------------
// CONSOLE
Returns this:
[source,js]
--------------------------------------------------
{
"found": true,
"_index": "my_index",
"_type": "my_type",
"_id": "my_id",
"_version": 1,
"_source": {
"data": "e1xydGYxXGFuc2kNCkxvcmVtIGlwc3VtIGRvbG9yIHNpdCBhbWV0DQpccGFyIH0=",
"attachment": {
"content_type": "application/rtf",
"language": "ro",
"content": "Lorem ipsum dolor sit amet",
Update documentation after merge with master
2016-08-10 13:07:22 -04:00
"content_length": 28
Add complete examples to some ingest docs These examples should make it more clear what the plugins do and they test that the snippets actually work. Relates to #18160
2016-08-10 11:12:41 -04:00
}
}
}
Ingest: Add attachment processor This is a simple port of the mapper attachment plugin to the ingest functionality, no new features. The only option is to limit the number of chars to prevent indexing of huge documents. Fields can be selected in the processor as well. Close #16303
2016-02-09 08:57:05 -05:00
--------------------------------------------------
Add complete examples to some ingest docs These examples should make it more clear what the plugins do and they test that the snippets actually work. Relates to #18160
2016-08-10 11:12:41 -04:00
// TESTRESPONSE
Ingest: Add attachment processor This is a simple port of the mapper attachment plugin to the ingest functionality, no new features. The only option is to limit the number of chars to prevent indexing of huge documents. Fields can be selected in the processor as well. Close #16303
2016-02-09 08:57:05 -05:00
NOTE: Extracting contents from binary data is a resource intensive operation and
consumes a lot of resources. It is highly recommended to run pipelines
using this processor in a dedicated ingest node.