From c6f4410a9d2d38c1c46a8a29be451f355479cb12 Mon Sep 17 00:00:00 2001 From: alicejw Date: Thu, 21 Apr 2022 15:58:19 -0700 Subject: [PATCH 01/10] for issue https://github.com/opensearch-project/documentation-website/issues/478 Signed-off-by: alicejw --- _opensearch/bucket-agg.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/_opensearch/bucket-agg.md b/_opensearch/bucket-agg.md index 1ae352b0..b9148ef6 100644 --- a/_opensearch/bucket-agg.md +++ b/_opensearch/bucket-agg.md @@ -684,6 +684,19 @@ GET opensearch_dashboards_sample_data_logs/_search } } ``` +You can use the `ignore_malformed` parameter set to `true` to ignore any malformed IP ranges if your index mappings include the `ip_range` type. + +```json +... +"mappings": { + "properties": { + "ips": { + "type": "ip_range", + "ignore_malformed": true + } + } +} +``` #### Sample response From 125ddec27e4d75f83b71c4460be87f2f26d87591 Mon Sep 17 00:00:00 2001 From: alicejw Date: Thu, 21 Apr 2022 16:38:38 -0700 Subject: [PATCH 02/10] clarify how to use ignore_malformed parameter Signed-off-by: alicejw --- _opensearch/bucket-agg.md | 25 ++++++++++++------------- 1 file changed, 12 insertions(+), 13 deletions(-) diff --git a/_opensearch/bucket-agg.md b/_opensearch/bucket-agg.md index b9148ef6..acd3e683 100644 --- a/_opensearch/bucket-agg.md +++ b/_opensearch/bucket-agg.md @@ -684,19 +684,6 @@ GET opensearch_dashboards_sample_data_logs/_search } } ``` -You can use the `ignore_malformed` parameter set to `true` to ignore any malformed IP ranges if your index mappings include the `ip_range` type. - -```json -... -"mappings": { - "properties": { - "ips": { - "type": "ip_range", - "ignore_malformed": true - } - } -} -``` #### Sample response @@ -722,7 +709,19 @@ You can use the `ignore_malformed` parameter set to `true` to ignore any malform } } ``` +When you add a document to an index with `ip_range` mappings set, if there are any malformed fields in the document, the entire document will get rejected. You can use the `ignore_malformed` parameter set to `true` to ignore any malformed IP ranges. This allows you to add the document and ignore the malformed fields. +```json +... +"mappings": { + "properties": { + "ips": { + "type": "ip_range", + "ignore_malformed": true + } + } +} +``` ## filter, filters A `filter` aggregation is a query clause, exactly like a search query — `match` or `term` or `range`. You can use the `filter` aggregation to narrow down the entire set of documents to a specific set before creating buckets. From 6462aa4a1c1d27dcb6e17f9c1ec57447ccf4023f Mon Sep 17 00:00:00 2001 From: alicejw Date: Thu, 21 Apr 2022 16:45:50 -0700 Subject: [PATCH 03/10] replace instances of 'indices' with 'indexes' Signed-off-by: alicejw --- Gemfile | 2 ++ _opensearch/index-alias.md | 18 +++++++++--------- _opensearch/index-data.md | 6 +++--- _opensearch/index-templates.md | 6 +++--- 4 files changed, 17 insertions(+), 15 deletions(-) diff --git a/Gemfile b/Gemfile index 53d29eb7..e3ad8804 100644 --- a/Gemfile +++ b/Gemfile @@ -30,3 +30,5 @@ gem "tzinfo-data", platforms: [:mingw, :mswin, :x64_mingw, :jruby] # Performance-booster for watching directories on Windows gem "wdm", "~> 0.1.0" if Gem.win_platform? + +gem "webrick" \ No newline at end of file diff --git a/_opensearch/index-alias.md b/_opensearch/index-alias.md index 68e36bfe..37a71a19 100644 --- a/_opensearch/index-alias.md +++ b/_opensearch/index-alias.md @@ -6,13 +6,13 @@ nav_order: 12 # Index aliases -An alias is a virtual index name that can point to one or more indices. +An alias is a virtual index name that can point to one or more indexes. -If your data is spread across multiple indices, rather than keeping track of which indices to query, you can create an alias and query it instead. +If your data is spread across multiple indexes, rather than keeping track of which indexes to query, you can create an alias and query it instead. -For example, if you’re storing logs into indices based on the month and you frequently query the logs for the previous two months, you can create a `last_2_months` alias and update the indices it points to each month. +For example, if you’re storing logs into indexes based on the month and you frequently query the logs for the previous two months, you can create a `last_2_months` alias and update the indexes it points to each month. -Because you can change the indices an alias points to at any time, referring to indices using aliases in your applications allows you to reindex your data without any downtime. +Because you can change the indexes an alias points to at any time, referring to indexes using aliases in your applications allows you to reindex your data without any downtime. --- @@ -63,7 +63,7 @@ To check if `alias1` refers to `index-1`, run the following command: GET alias1 ``` -## Add or remove indices +## Add or remove indexes You can perform multiple actions in the same `_aliases` operation. For example, the following command removes `index-1` and adds `index-2` to `alias1`: @@ -90,7 +90,7 @@ POST _aliases The `add` and `remove` actions occur atomically, which means that at no point will `alias1` point to both `index-1` and `index-2`. -You can also add indices based on an index pattern: +You can also add indexes based on an index pattern: ```json POST _aliases @@ -108,7 +108,7 @@ POST _aliases ## Manage aliases -To list the mapping of aliases to indices, run the following command: +To list the mapping of aliases to indexes, run the following command: ```json GET _cat/aliases?v @@ -121,7 +121,7 @@ alias index filter routing.index routing.search alias1 index-1 * - - ``` -To check which indices an alias points to, run the following command: +To check which indexes an alias points to, run the following command: ```json GET _alias/alias1 @@ -166,7 +166,7 @@ PUT index-1 ## Create filtered aliases -You can create a filtered alias to access a subset of documents or fields from the underlying indices. +You can create a filtered alias to access a subset of documents or fields from the underlying indexes. This command adds only a specific timestamp field to `alias1`: diff --git a/_opensearch/index-data.md b/_opensearch/index-data.md index 2a4fcd82..ff975d32 100644 --- a/_opensearch/index-data.md +++ b/_opensearch/index-data.md @@ -68,16 +68,16 @@ PUT movies/_doc/1 Because you must specify an ID, if you run this command 10 times, you still have just one document indexed with the `_version` field incremented to 10. -Indices default to one primary shard and one replica. If you want to specify non-default settings, create the index before adding documents: +Indexes default to one primary shard and one replica. If you want to specify non-default settings, create the index before adding documents: ```json PUT more-movies { "settings": { "number_of_shards": 6, "number_of_replicas": 2 } } ``` -## Naming restrictions for indices +## Naming restrictions for indexes -OpenSearch indices have the following naming restrictions: +OpenSearch indexes have the following naming restrictions: - All letters must be lowercase. - Index names can't begin with underscores (`_`) or hyphens (`-`). diff --git a/_opensearch/index-templates.md b/_opensearch/index-templates.md index 04821f97..326ad059 100644 --- a/_opensearch/index-templates.md +++ b/_opensearch/index-templates.md @@ -6,7 +6,7 @@ nav_order: 15 # Index templates -Index templates let you initialize new indices with predefined mappings and settings. For example, if you continuously index log data, you can define an index template so that all of these indices have the same number of shards and replicas. +Index templates let you initialize new indexes with predefined mappings and settings. For example, if you continuously index log data, you can define an index template so that all of these indexes have the same number of shards and replicas. ### Create a template @@ -95,7 +95,7 @@ GET logs-2020-01-01 } ``` -Any additional indices that match this pattern---`logs-2020-01-02`, `logs-2020-01-03`, and so on---will inherit the same mappings and settings. +Any additional indexes that match this pattern---`logs-2020-01-02`, `logs-2020-01-03`, and so on---will inherit the same mappings and settings. Index patterns cannot contain any of the following characters: `:`, `"`, `+`, `/`, `\`, `|`, `?`, `#`, `>`, and `<`. @@ -127,7 +127,7 @@ HEAD _index_template/ ### Configure multiple templates -You can create multiple index templates for your indices. If the index name matches more than one template, OpenSearch merges all mappings and settings from all matching templates and applies them to the index. +You can create multiple index templates for your indexes. If the index name matches more than one template, OpenSearch merges all mappings and settings from all matching templates and applies them to the index. The settings from the more recently created index templates override the settings of older index templates. So, you can first define a few common settings in a generic template that can act as a catch-all and then add more specialized settings as required. From b74df04afc0a7939ebd097923df785b0339e53f7 Mon Sep 17 00:00:00 2001 From: alicejw Date: Mon, 25 Apr 2022 09:43:52 -0700 Subject: [PATCH 04/10] to remove mistake from gemfile added line to build site locally, then commited by mistake Signed-off-by: alicejw --- Gemfile | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/Gemfile b/Gemfile index e3ad8804..2dc06281 100644 --- a/Gemfile +++ b/Gemfile @@ -29,6 +29,4 @@ end gem "tzinfo-data", platforms: [:mingw, :mswin, :x64_mingw, :jruby] # Performance-booster for watching directories on Windows -gem "wdm", "~> 0.1.0" if Gem.win_platform? - -gem "webrick" \ No newline at end of file +gem "wdm", "~> 0.1.0" if Gem.win_platform? \ No newline at end of file From 93614fb2cd224d80e7b0e54769e84c997a70521b Mon Sep 17 00:00:00 2001 From: alicejw Date: Mon, 25 Apr 2022 09:55:25 -0700 Subject: [PATCH 05/10] for reviewer feedback updates Signed-off-by: alicejw --- _opensearch/bucket-agg.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/_opensearch/bucket-agg.md b/_opensearch/bucket-agg.md index acd3e683..43b2f97b 100644 --- a/_opensearch/bucket-agg.md +++ b/_opensearch/bucket-agg.md @@ -709,17 +709,17 @@ GET opensearch_dashboards_sample_data_logs/_search } } ``` -When you add a document to an index with `ip_range` mappings set, if there are any malformed fields in the document, the entire document will get rejected. You can use the `ignore_malformed` parameter set to `true` to ignore any malformed IP ranges. This allows you to add the document and ignore the malformed fields. +If you add a document with malformed fields to an index that has `ip_range` set to `false` in its mappings, OpenSearch rejects the entire document. You can set `ignore_malformed` to `true` to specify that OpenSearch should ignore malformed fields. The default is `false`. ```json ... "mappings": { - "properties": { - "ips": { - "type": "ip_range", - "ignore_malformed": true - } + "properties": { + "ips": { + "type": "ip_range", + "ignore_malformed": true } + } } ``` ## filter, filters From 150772a6e35b5e406845be557cae6071f50ea6f4 Mon Sep 17 00:00:00 2001 From: alicejw Date: Thu, 28 Apr 2022 14:28:36 -0700 Subject: [PATCH 06/10] to create a mapping section Signed-off-by: alicejw --- _opensearch/mapping-guide.md | 74 ++++++++++++++++++++++++++++++++++++ 1 file changed, 74 insertions(+) create mode 100644 _opensearch/mapping-guide.md diff --git a/_opensearch/mapping-guide.md b/_opensearch/mapping-guide.md new file mode 100644 index 00000000..9cd35436 --- /dev/null +++ b/_opensearch/mapping-guide.md @@ -0,0 +1,74 @@ +--- +layout: default +title: Mapping +nav_order: 14 +--- + +# About Mapping with OpenSearch + +You can define how documents and their fields are stored and indexed by creating a mapping. + +Each field has its own data type. The mapping definition defines metadata fields that specify how to + +This section provides an example for how to create an index mapping, and how to add a document to it that will get ip_range validated. + + +--- + +#### Table of contents +1. TOC +{:toc} + + +--- +## Dynamic mapping + +When you index a document, OpenSearch adds fields automatically. You can also explicitly add fields to an index mapping. The following example shows how to add the ip_range field and specify `ignore_malformed` parameter to prevent ip addresses that do not conform to your ip_range data type. + +### Create an index with ip_range mapping + +To create an index, use a PUT request: + +```json +PUT _index_ip +{ + "mappings": { + "dynamic_templates": [ + { + "ip_range": { + "match": "*ip_range", + "mapping": { + "type": "ip_range", + "ignore_malformed": true + } + } + } + ] + } +} +``` + +You can add a document to your index that has an ip_range specified: + +```json +PUT _index_ip/_doc/ +{ + "source_ip_range": "192.168.1.1/32" +} +``` + +This indexed ip_range does not throw an error because `ignore_malformed` is set to true. + + + +