honeymoose
/
OpenSearch
mirror of https://github.com/honeymoose/OpenSearch.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
OpenSearch/docs/plugins/ingest-attachment.asciidoc

110 lines
3.8 KiB
Plaintext
Raw Normal View History

Ingest: Add attachment processor This is a simple port of the mapper attachment plugin to the ingest functionality, no new features. The only option is to limit the number of chars to prevent indexing of huge documents. Fields can be selected in the processor as well. Close #16303
2016-02-09 08:57:05 -05:00
[[ingest-attachment]]
Add ingest plugins to Elasticsearch plugin docs
2016-02-11 20:40:32 -05:00
=== Ingest Attachment Processor Plugin
Ingest: Add attachment processor This is a simple port of the mapper attachment plugin to the ingest functionality, no new features. The only option is to limit the number of chars to prevent indexing of huge documents. Fields can be selected in the processor as well. Close #16303
2016-02-09 08:57:05 -05:00
Edits to ingest plugin docs
2016-03-04 01:49:31 -05:00
The ingest attachment plugin lets Elasticsearch extract file attachments in common formats (such as PPT, XLS, and PDF) by
Ingest: Add attachment processor This is a simple port of the mapper attachment plugin to the ingest functionality, no new features. The only option is to limit the number of chars to prevent indexing of huge documents. Fields can be selected in the processor as well. Close #16303
2016-02-09 08:57:05 -05:00
using the Apache text extraction library http://lucene.apache.org/tika/[Tika].
Edits to ingest plugin docs
2016-03-04 01:49:31 -05:00
You can use the ingest attachment plugin as a replacement for the mapper attachment plugin.
Ingest: Add attachment processor This is a simple port of the mapper attachment plugin to the ingest functionality, no new features. The only option is to limit the number of chars to prevent indexing of huge documents. Fields can be selected in the processor as well. Close #16303
2016-02-09 08:57:05 -05:00
Ingest Attachment: Allow to prevent base64 conversions by using raw bytes (#16601) CBOR is natively supported in Elasticsearch and allows for byte arrays. This means, that by using CBOR the user can prevent base64 conversions for the data being sent back and forth. This PR adds support to extract data from a byte array in addition to a string. This also required to add a ByteArrayValueSource class.
2016-04-11 08:14:56 -04:00
The source field must be a base64 encoded binary. If you do not want to incur
the overhead of converting back and forth between base64, you can use the CBOR
format instead of JSON and specify the field as a bytes array instead of a string
representation. The processor will skip the base64 decoding then.
Ingest: Add attachment processor This is a simple port of the mapper attachment plugin to the ingest functionality, no new features. The only option is to limit the number of chars to prevent indexing of huge documents. Fields can be selected in the processor as well. Close #16303
2016-02-09 08:57:05 -05:00
Installation and removal instructions for Ingest Plugins (#18769) * Add install and remove instructions to ingest-geoip and ingest-attachment docs
2016-06-08 15:55:59 -04:00
[[ingest-attachment-install]]
[float]
==== Installation
This plugin can be installed using the plugin manager:
[source,sh]
----------------------------------------------------------------
sudo bin/elasticsearch-plugin install ingest-attachment
----------------------------------------------------------------
The plugin must be installed on every node in the cluster, and each node must
be restarted after installation.
Add link to "offline install" and show the full URL
2016-09-19 09:04:29 -04:00
This plugin can be downloaded for <<plugin-management-custom-url,offline install>> from
Fix URL for plugins Reported at https://discuss.elastic.co/t/ingest-geoip-for-beta1/62505
2016-10-07 13:17:10 -04:00
{plugin_url}/ingest-attachment/ingest-attachment-{version}.zip.
Add offline install instructions for plugins Follow up of https://github.com/elastic/elasticsearch/issues/15925#issuecomment-171250150 This commit adds offline install instructions for plugins.
2016-09-12 09:34:44 -04:00
Installation and removal instructions for Ingest Plugins (#18769) * Add install and remove instructions to ingest-geoip and ingest-attachment docs
2016-06-08 15:55:59 -04:00
[[ingest-attachment-remove]]
[float]
==== Removal
The plugin can be removed with the following command:
[source,sh]
----------------------------------------------------------------
sudo bin/elasticsearch-plugin remove ingest-attachment
----------------------------------------------------------------
The node must be stopped before removing the plugin.
[[using-ingest-attachment]]
==== Using the Attachment Processor in a Pipeline
Ingest: Add attachment processor This is a simple port of the mapper attachment plugin to the ingest functionality, no new features. The only option is to limit the number of chars to prevent indexing of huge documents. Fields can be selected in the processor as well. Close #16303
2016-02-09 08:57:05 -05:00
[[ingest-attachment-options]]
.Attachment options
[options="header"]
|======
| Name | Required | Default | Description
ingest: Streamline option naming for several processors: * `rename` processor, renamed `to` to `target_field` * `date` processor, renamed `match_field` to `field` and renamed `match_formats` to `formats` * `geoip` processor, renamed `source_field` to `field` and renamed `fields` to `properties` * `attachment` processor, renamed `source_field` to `field` and renamed `fields` to `properties` Closes #17835
2016-04-20 12:00:11 -04:00
| `field` | yes | - | The field to get the base64 encoded field from
Ingest: Add attachment processor This is a simple port of the mapper attachment plugin to the ingest functionality, no new features. The only option is to limit the number of chars to prevent indexing of huge documents. Fields can be selected in the processor as well. Close #16303
2016-02-09 08:57:05 -05:00
| `target_field` | no | attachment | The field that will hold the attachment information
| `indexed_chars` | no | 100000 | The number of chars being used for extraction to prevent huge fields. Use `-1` for no limit.
ingest: Streamline option naming for several processors: * `rename` processor, renamed `to` to `target_field` * `date` processor, renamed `match_field` to `field` and renamed `match_formats` to `formats` * `geoip` processor, renamed `source_field` to `field` and renamed `fields` to `properties` * `attachment` processor, renamed `source_field` to `field` and renamed `fields` to `properties` Closes #17835
2016-04-20 12:00:11 -04:00
| `properties` | no | all | Properties to select to be stored. Can be `content`, `title`, `name`, `author`, `keywords`, `date`, `content_type`, `content_length`, `language`
add `ignore_missing` flag to ingest plugins (#22273) added `ignore_missing` flag to: - Attachment Processor - GeoIP Processor - User-Agent Processor
2016-12-20 13:53:28 -05:00
| `ignore_missing` | no | `false` | If `true` and `field` does not exist, the processor quietly exits without modifying the document
Ingest: Add attachment processor This is a simple port of the mapper attachment plugin to the ingest functionality, no new features. The only option is to limit the number of chars to prevent indexing of huge documents. Fields can be selected in the processor as well. Close #16303
2016-02-09 08:57:05 -05:00
|======
Add complete examples to some ingest docs These examples should make it more clear what the plugins do and they test that the snippets actually work. Relates to #18160
2016-08-10 11:12:41 -04:00
For example, this:
Ingest: Add attachment processor This is a simple port of the mapper attachment plugin to the ingest functionality, no new features. The only option is to limit the number of chars to prevent indexing of huge documents. Fields can be selected in the processor as well. Close #16303
2016-02-09 08:57:05 -05:00
[source,js]
--------------------------------------------------
Add complete examples to some ingest docs These examples should make it more clear what the plugins do and they test that the snippets actually work. Relates to #18160
2016-08-10 11:12:41 -04:00
PUT _ingest/pipeline/attachment
Ingest: Add attachment processor This is a simple port of the mapper attachment plugin to the ingest functionality, no new features. The only option is to limit the number of chars to prevent indexing of huge documents. Fields can be selected in the processor as well. Close #16303
2016-02-09 08:57:05 -05:00
{
Add complete examples to some ingest docs These examples should make it more clear what the plugins do and they test that the snippets actually work. Relates to #18160
2016-08-10 11:12:41 -04:00
"description" : "Extract attachment information",
Ingest: Add attachment processor This is a simple port of the mapper attachment plugin to the ingest functionality, no new features. The only option is to limit the number of chars to prevent indexing of huge documents. Fields can be selected in the processor as well. Close #16303
2016-02-09 08:57:05 -05:00
"processors" : [
{
"attachment" : {
ingest: Streamline option naming for several processors: * `rename` processor, renamed `to` to `target_field` * `date` processor, renamed `match_field` to `field` and renamed `match_formats` to `formats` * `geoip` processor, renamed `source_field` to `field` and renamed `fields` to `properties` * `attachment` processor, renamed `source_field` to `field` and renamed `fields` to `properties` Closes #17835
2016-04-20 12:00:11 -04:00
"field" : "data"
Ingest: Add attachment processor This is a simple port of the mapper attachment plugin to the ingest functionality, no new features. The only option is to limit the number of chars to prevent indexing of huge documents. Fields can be selected in the processor as well. Close #16303
2016-02-09 08:57:05 -05:00
}
}
]
}
Add complete examples to some ingest docs These examples should make it more clear what the plugins do and they test that the snippets actually work. Relates to #18160
2016-08-10 11:12:41 -04:00
PUT my_index/my_type/my_id?pipeline=attachment
{
"data": "e1xydGYxXGFuc2kNCkxvcmVtIGlwc3VtIGRvbG9yIHNpdCBhbWV0DQpccGFyIH0="
}
GET my_index/my_type/my_id
--------------------------------------------------
// CONSOLE
Returns this:
[source,js]
--------------------------------------------------
{
"found": true,
"_index": "my_index",
"_type": "my_type",
"_id": "my_id",
"_version": 1,
"_source": {
"data": "e1xydGYxXGFuc2kNCkxvcmVtIGlwc3VtIGRvbG9yIHNpdCBhbWV0DQpccGFyIH0=",
"attachment": {
"content_type": "application/rtf",
"language": "ro",
"content": "Lorem ipsum dolor sit amet",
Update documentation after merge with master
2016-08-10 13:07:22 -04:00
"content_length": 28
Add complete examples to some ingest docs These examples should make it more clear what the plugins do and they test that the snippets actually work. Relates to #18160
2016-08-10 11:12:41 -04:00
}
}
}
Ingest: Add attachment processor This is a simple port of the mapper attachment plugin to the ingest functionality, no new features. The only option is to limit the number of chars to prevent indexing of huge documents. Fields can be selected in the processor as well. Close #16303
2016-02-09 08:57:05 -05:00
--------------------------------------------------
Add complete examples to some ingest docs These examples should make it more clear what the plugins do and they test that the snippets actually work. Relates to #18160
2016-08-10 11:12:41 -04:00
// TESTRESPONSE
Ingest: Add attachment processor This is a simple port of the mapper attachment plugin to the ingest functionality, no new features. The only option is to limit the number of chars to prevent indexing of huge documents. Fields can be selected in the processor as well. Close #16303
2016-02-09 08:57:05 -05:00
NOTE: Extracting contents from binary data is a resource intensive operation and
consumes a lot of resources. It is highly recommended to run pipelines
using this processor in a dedicated ingest node.
Add link to foreach processor to ingest-attachment.asciidoc Add a Note to the foreach processor for processing an array of attachments, as this is required to be able to process an array of attachments.
2016-12-21 02:15:19 -05:00
NOTE: To process an array of attachments the {ref}/foreach-processor.html[foreach processor] is required.