OpenSearch/docs/plugins/ingest-attachment.asciidoc

[[ingest-attachment]]
=== Ingest Attachment Processor Plugin

The ingest attachment plugin lets Elasticsearch extract file attachments in common formats (such as PPT, XLS, and PDF) by
using the Apache text extraction library http://lucene.apache.org/tika/[Tika].

You can use the ingest attachment plugin as a replacement for the mapper attachment plugin.

The source field must be a base64 encoded binary. If you do not want to incur
the overhead of converting back and forth between base64, you can use the CBOR
format instead of JSON and specify the field as a bytes array instead of a string
representation. The processor will skip the base64 decoding then.

[[ingest-attachment-install]]
[float]
==== Installation

This plugin can be installed using the plugin manager:

[source,sh]
----------------------------------------------------------------
sudo bin/elasticsearch-plugin install ingest-attachment
----------------------------------------------------------------

The plugin must be installed on every node in the cluster, and each node must
be restarted after installation.

This plugin can be downloaded for <<plugin-management-custom-url,offline install>> from
{plugin_url}/ingest-attachment/ingest-attachment-{version}.zip.

[[ingest-attachment-remove]]
[float]
==== Removal

The plugin can be removed with the following command:

[source,sh]
----------------------------------------------------------------
sudo bin/elasticsearch-plugin remove ingest-attachment
----------------------------------------------------------------

The node must be stopped before removing the plugin.

[[using-ingest-attachment]]
==== Using the Attachment Processor in a Pipeline

[[ingest-attachment-options]]
.Attachment options
[options="header"]
|======
| Name                   | Required  | Default          | Description
| `field`                | yes       | -                | The field to get the base64 encoded field from
| `target_field`         | no        | attachment       | The field that will hold the attachment information
| `indexed_chars`        | no        | 100000           | The number of chars being used for extraction to prevent huge fields. Use `-1` for no limit.
| `properties`           | no        | all properties   | Array of properties to select to be stored. Can be `content`, `title`, `name`, `author`, `keywords`, `date`, `content_type`, `content_length`, `language`
| `ignore_missing`       | no        | `false`          | If `true` and `field` does not exist, the processor quietly exits without modifying the document
|======

For example, this:

[source,js]
--------------------------------------------------
PUT _ingest/pipeline/attachment
{
  "description" : "Extract attachment information",
  "processors" : [
    {
      "attachment" : {
        "field" : "data"
      }
    }
  ]
}
PUT my_index/my_type/my_id?pipeline=attachment
{
  "data": "e1xydGYxXGFuc2kNCkxvcmVtIGlwc3VtIGRvbG9yIHNpdCBhbWV0DQpccGFyIH0="
}
GET my_index/my_type/my_id
--------------------------------------------------
// CONSOLE

Returns this:

[source,js]
--------------------------------------------------
{
  "found": true,
  "_index": "my_index",
  "_type": "my_type",
  "_id": "my_id",
  "_version": 1,
  "_source": {
    "data": "e1xydGYxXGFuc2kNCkxvcmVtIGlwc3VtIGRvbG9yIHNpdCBhbWV0DQpccGFyIH0=",
    "attachment": {
      "content_type": "application/rtf",
      "language": "ro",
      "content": "Lorem ipsum dolor sit amet",
      "content_length": 28
    }
  }
}
--------------------------------------------------
// TESTRESPONSE


To specify only some fields to be extracted:

[source,js]
--------------------------------------------------
PUT _ingest/pipeline/attachment
{
  "description" : "Extract attachment information",
  "processors" : [
    {
      "attachment" : {
        "field" : "data",
        "properties": [ "content", "title" ]
      }
    }
  ]
}
--------------------------------------------------
// CONSOLE

NOTE: Extracting contents from binary data is a resource intensive operation and
      consumes a lot of resources. It is highly recommended to run pipelines
      using this processor in a dedicated ingest node.
      
[[ingest-attachment-with-arrays]]
==== Using the Attachment Processor with arrays

To use the attachment processor within an array of attachments the
{ref}/foreach-processor.html[foreach processor] is required. This
enables the attachment processor to be run on the individual elements
of the array.

For example, given the following source:

[source,js]
--------------------------------------------------
{
  "attachments" : [
    {
      "filename" : "ipsum.txt",
      "data" : "dGhpcyBpcwpqdXN0IHNvbWUgdGV4dAo="
    },
    {
      "filename" : "test.txt",
      "data" : "VGhpcyBpcyBhIHRlc3QK"
    }
  ]
}
--------------------------------------------------
// NOTCONSOLE

In this case, we want to process the data field in each element
of the attachments field and insert
the properties into the document so the following `foreach`
processor is used:

[source,js]
--------------------------------------------------
PUT _ingest/pipeline/attachment
{
  "description" : "Extract attachment information from arrays",
  "processors" : [
    {
      "foreach": {
        "field": "attachments",
        "processor": {
          "attachment": {
            "target_field": "_ingest._value.attachment",
            "field": "_ingest._value.data"
          }
        }
      }
    }
  ]
}
PUT my_index/my_type/my_id?pipeline=attachment
{
  "attachments" : [
    {
      "filename" : "ipsum.txt",
      "data" : "dGhpcyBpcwpqdXN0IHNvbWUgdGV4dAo="
    },
    {
      "filename" : "test.txt",
      "data" : "VGhpcyBpcyBhIHRlc3QK"
    }
  ]
}
GET my_index/my_type/my_id
--------------------------------------------------
// CONSOLE

Returns this:
[source,js]
--------------------------------------------------
{
  "_index" : "my_index",
  "_type" : "my_type",
  "_id" : "my_id",
  "_version" : 1,
  "found" : true,
  "_source" : {
    "attachments" : [
      {
        "filename" : "ipsum.txt",
        "data" : "dGhpcyBpcwpqdXN0IHNvbWUgdGV4dAo=",
        "attachment" : {
          "content_type" : "text/plain; charset=ISO-8859-1",
          "language" : "en",
          "content" : "this is\njust some text",
          "content_length" : 24
        }
      },
      {
        "filename" : "test.txt",
        "data" : "VGhpcyBpcyBhIHRlc3QK",
        "attachment" : {
          "content_type" : "text/plain; charset=ISO-8859-1",
          "language" : "en",
          "content" : "This is a test",
          "content_length" : 16
        }
      }
    ]
  }
}
--------------------------------------------------
// TESTRESPONSE


Note that the `target_field` needs to be set, otherwise the
default value is used which is a top level field `attachment`. The
properties on this top level field will contain the value of the
first attachment only. However, by specifying the
`target_field` on to a value on `_ingest._value` it will correctly
associate the properties with the correct attachment.