From 6625a0695834846ed38bd60e726673087cd119ed Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 15 Nov 2023 13:13:00 -0700 Subject: [PATCH] [DOC] Add new documentation for date index name processor (#5398) * Add new processor documentation Signed-off-by: Melissa Vagi --------- Signed-off-by: Melissa Vagi Co-authored-by: Heemin Kim Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> --- .../processors/date-index-name.md | 154 ++++++++++++++++++ 1 file changed, 154 insertions(+) create mode 100644 _ingest-pipelines/processors/date-index-name.md diff --git a/_ingest-pipelines/processors/date-index-name.md b/_ingest-pipelines/processors/date-index-name.md new file mode 100644 index 00000000..55a180f4 --- /dev/null +++ b/_ingest-pipelines/processors/date-index-name.md @@ -0,0 +1,154 @@ +--- +layout: default +title: Date index name +parent: Ingest processors +nav_order: 55 +--- + +# Date index name + +The `date_index_name` processor is used to point documents to the correct time-based index based on the date or timestamp field within the document. The processor sets the `_index` metadata field to a [date math]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/date/#date-math) index name expression. Then the processor fetches the date or timestamp from the `field` field in the document being processed and formats it into a date math index name expression. The extracted date, `index_name_prefix` value, and `date_rounding` value are then combined to create the date math index expression. For example, if the `field` field contains the value `2023-10-30T12:43:29.000Z` and `index_name_prefix` is set to `week_index-` and `date_rounding` is set to `w`, then the date math index name expression is `week_index-2023-10-30`. You can use the `date_formats` field to specify how the date in the date math index expression should be formatted. + +The following is the syntax for the `date_index_name` processor: + +```json +{ + "date_index_name": { + "field": "your_date_field or your_timestamp_field", + "date_rounding": "rounding_value" + } +} +``` +{% include copy-curl.html %} + +## Configuration parameters + +The following table lists the required and optional parameters for the `date_index_name` processor. + +Parameter | Required/Optional | Description | +|-----------|-----------|-----------| +`field` | Required | The date or timestamp field in the incoming document. Supports [template snippets]({{site.url}}{{site.baseurl}}/ingest-pipelines/create-ingest/#template-snippets). | +`date_rounding` | Required | The rounded date format within the index name . Valid values are `y` (year), `M` (month), `w` (week), `d` (day), `h` (hour), `m` (minute), and `s` (second). | +`date_formats` | Optional | An array of date formats used to parse the date or timestamp field. Valid options include a java time pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N. Default is `yyyy-MM-dd'T'HH:mm:ss.SSSXX`. | +`index_name_format` | Optional | The date format. Default is `yyyy-MM-dd`. Supports [template snippets]({{site.url}}{{site.baseurl}}/ingest-pipelines/create-ingest/#template-snippets). | +`index_name_prefix` | Optional | The index name prefix to append before the date. Supports [template snippets]({{site.url}}{{site.baseurl}}/ingest-pipelines/create-ingest/#template-snippets). +`description` | Optional | A brief description of the processor. | +`if` | Optional | A condition for running this processor. | +`ignore_failure` | Optional | If set to `true`, failures are ignored. Default is `false`. | +`locale` | `locale` | Optional | The locale to use when parsing the month name and week day of the date. Default is `ENGLISH`. Supports [template snippets]({{site.url}}{{site.baseurl}}/ingest-pipelines/create-ingest/#template-snippets). | +`on_failure` | Optional | A list of processors to run if the processor fails. | +`tag` | Optional | An identifier tag for the processor. Useful for debugging to distinguish between processors of the same type. | +`timezone` | Optional | The time zone to use when parsing the date. Default is `UTC`. | + +## Using the processor + +Follow these steps to use the processor in a pipeline. + +**Step 1: Create a pipeline.** + +The following query creates a pipeline, named `date-index-name1`, that uses the `date_index_name` processor to index logs into monthly indexes: + +```json +PUT /_ingest/pipeline/date-index-name1 +{ + "description": "Create weekly index pipeline", + "processors": [ + { + "date_index_name": { + "field": "date_field", + "index_name_prefix": "week_index-", + "date_rounding": "w", + "date_formats": ["YYYY-MM-DD"] + } + } + ] +} +``` +{% include copy-curl.html %} + +**Step 2 (Optional): Test the pipeline.** + +It is recommended that you test your pipeline before you ingest documents. +{: .tip} + +To test the pipeline, run the following query: + +```json +POST _ingest/pipeline/date-index-name1/_simulate +{ + "docs": [ + { + "_index": "testindex1", + "_id": "1", + "_source": { + "date_field": "2023-10-30" + } + } + ] +} +``` +{% include copy-curl.html %} + +#### Response + +The following example response confirms that the pipeline is working as expected: + +```json +{ + "docs": [ + { + "doc": { + "_index": "", + "_id": "1", + "_source": { + "date_field": "2023-10-30" + }, + "_ingest": { + "timestamp": "2023-11-13T18:23:10.408593092Z" + } + } + } + ] +} +``` + +**Step 3: Ingest a document.** + +The following query ingests a document into an index named `testindex1`: + +```json +PUT testindex1/_doc/1?pipeline=date-index-name1 +{ + "date_field": "2023-10-30" +} +``` +{% include copy-curl.html %} + +#### Response + +The request indexes the document into the index `week_index-2023-10-23` and will index all documents with a timestamp within that week into the same index because the pipeline rounds by week. + +```json +{ + "_index": "week_index-2023-10-30", + "_id": "1", + "_version": 4, + "result": "updated", + "_shards": { + "total": 2, + "successful": 1, + "failed": 0 + }, + "_seq_no": 3, + "_primary_term": 1 +} +``` + +**Step 4 (Optional): Retrieve the document.** + +To retrieve the document, run the following query: + +```json +GET week_index-2023-10-30/_doc/1 +``` +{% include copy-curl.html %}