honeymoose
/
OpenSearch
mirror of https://github.com/honeymoose/OpenSearch.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
OpenSearch/docs/reference/migration/migrate_5_0/aggregations.asciidoc

34 lines
1.5 KiB
Plaintext
Raw Normal View History

Use the new points API to index numeric fields. #17746 This makes all numeric fields including `date`, `ip` and `token_count` use points instead of the inverted index as a lookup structure. This is expected to perform worse for exact queries, but faster for range queries. It also requires less storage. Notes about how the change works: - Numeric mappers have been split into a legacy version that is essentially the current mapper, and a new version that uses points, eg. LegacyDateFieldMapper and DateFieldMapper. - Since new and old fields have the same names, the decision about which one to use is made based on the index creation version. - If you try to force using a legacy field on a new index or a field that uses points on an old index, you will get an exception. - IP addresses now support IPv6 via Lucene's InetAddressPoint and store them in SORTED_SET doc values using the same encoding (fixed length of 16 bytes and sortable). - The internal MappedFieldType that is stored by the new mappers does not have any of the points-related properties set. Instead, it keeps setting the index options when parsing the `index` property of mappings and does `if (fieldType.indexOptions() != IndexOptions.NONE) { // add point field }` when parsing documents. Known issues that won't fix: - You can't use numeric fields in significant terms aggregations anymore since this requires document frequencies, which points do not record. - Term queries on numeric fields will now return constant scores instead of giving better scores to the rare values. Known issues that we could work around (in follow-up PRs, this one is too large already): - Range queries on `ip` addresses only work if both the lower and upper bounds are inclusive (exclusive bounds are not exposed in Lucene). We could either decide to implement it, or drop range support entirely and tell users to query subnets using the CIDR notation instead. - Since IP addresses now use a different representation for doc values, aggregations will fail when running a terms aggregation on an ip field on a list of indices that contains both pre-5.0 and 5.0 indices. - The ip range aggregation does not work on the new ip field. We need to either implement range aggs for SORTED_SET doc values or drop support for ip ranges and tell users to use filters instead. #17700 Closes #16751 Closes #17007 Closes #11513
2016-04-01 05:07:35 -04:00
[[breaking_50_aggregations_changes]]
=== Aggregation changes
==== Significant terms on numeric fields
Numeric fields have been refactored to use a different data structure that
performs better for range queries. However, since this data structure does
Make significant terms work on fields that are indexed with points. #18031 It will keep using the caching terms enum for keyword/text fields and falls back to IndexSearcher.count for fields that do not use the inverted index for searching (such as numbers and ip addresses). Note that this probably means that significant terms aggregations on these fields will be less efficient than they used to be. It should be ok under a sampler aggregation though. This moves tests back to the state they were in before numbers started using points, and also adds a new test that significant terms aggs fail if a field is not indexed. In the long term, we might want to follow the approach that Robert initially proposed that consists in collecting all documents from the background filter in order to compute frequencies using doc values. This would also mean that significant terms aggregations do not require fields to be indexed anymore.
2016-04-27 14:59:28 -04:00
not record document frequencies, numeric fields need to fall back to running
queries in order to estimate the number of matching documents in the
background set, which may incur a performance degradation.
It is recommended to use <<keyword,`keyword`>> fields instead, either directly
or through a <<multi-fields,multi-field>> if the numeric representation is
still needed for sorting, range queries or numeric aggregations like
Use the new points API to index numeric fields. #17746 This makes all numeric fields including `date`, `ip` and `token_count` use points instead of the inverted index as a lookup structure. This is expected to perform worse for exact queries, but faster for range queries. It also requires less storage. Notes about how the change works: - Numeric mappers have been split into a legacy version that is essentially the current mapper, and a new version that uses points, eg. LegacyDateFieldMapper and DateFieldMapper. - Since new and old fields have the same names, the decision about which one to use is made based on the index creation version. - If you try to force using a legacy field on a new index or a field that uses points on an old index, you will get an exception. - IP addresses now support IPv6 via Lucene's InetAddressPoint and store them in SORTED_SET doc values using the same encoding (fixed length of 16 bytes and sortable). - The internal MappedFieldType that is stored by the new mappers does not have any of the points-related properties set. Instead, it keeps setting the index options when parsing the `index` property of mappings and does `if (fieldType.indexOptions() != IndexOptions.NONE) { // add point field }` when parsing documents. Known issues that won't fix: - You can't use numeric fields in significant terms aggregations anymore since this requires document frequencies, which points do not record. - Term queries on numeric fields will now return constant scores instead of giving better scores to the rare values. Known issues that we could work around (in follow-up PRs, this one is too large already): - Range queries on `ip` addresses only work if both the lower and upper bounds are inclusive (exclusive bounds are not exposed in Lucene). We could either decide to implement it, or drop range support entirely and tell users to query subnets using the CIDR notation instead. - Since IP addresses now use a different representation for doc values, aggregations will fail when running a terms aggregation on an ip field on a list of indices that contains both pre-5.0 and 5.0 indices. - The ip range aggregation does not work on the new ip field. We need to either implement range aggs for SORTED_SET doc values or drop support for ip ranges and tell users to use filters instead. #17700 Closes #16751 Closes #17007 Closes #11513
2016-04-01 05:07:35 -04:00
<<search-aggregations-metrics-stats-aggregation,`stats` aggregations>>.
Add back support for `ip` range aggregations. #17859 This commit adds support for range aggregations on `ip` fields. However it will only work on 5.x indices. Closes #17700
2016-04-18 10:11:11 -04:00
==== `ip_range` aggregations
Now that Elasticsearch supports `ipv6`, `ip` addresses are encoded in the index
using a binary representation rather than a numeric representation. As a
consequence, the output of `ip_range` aggregations does not give numeric values
for `from` and `to` anymore.
Remove size 0 options in aggregations This removes the ability to set `size: 0` in the `terms`, `significant_terms` and `geohash_grid` aggregations for the reasons described in https://github.com/elastic/elasticsearch/issues/18838 Closes #18838
2016-06-14 06:37:10 -04:00
==== `size: 0` on Terms, Significant Terms and Geohash Grid Aggregations
`size: 0` is no longer valid for the terms, significant terms and geohash grid
aggregations. Instead a size should be explicitly specified with a number greater
than zero.
Clarify time units usage in docs This commit clarifies the distinction between supported time units for durations and supported time units for durations in the docs. Relates #19159
2016-06-29 17:02:15 -04:00
==== Fractional time values
Fractional time values (e.g., 0.5s) are no longer supported. For example, this means when setting
date histogram intervals "1.5h" will be rejected and should instead be input as "90m".