mirror of
https://github.com/honeymoose/OpenSearch.git
synced 2025-03-09 14:34:43 +00:00
b957aa46ce
While investigating memory consumption of deeply nested aggregations for #43091 the memory used to keep track of the doc ids and buckets in the BestBucketsDeferringCollector showed up as one of the main contributor. In my tests half of the memory held in the BestBucketsDeferringCollector is associated to segments that don't have matching docs in the selected buckets. This is expected on fields that have a big cardinality since each bucket can appear in very few segments. By allocating the builders lazily this change reduces the memory consumption by a factor 2 (from 1GB to 512MB), hence reducing the impact on gcs for these volatile allocations. This commit also switches the PackedLongValues.Builder with a RoaringDocIdSet in order to handle very sparse buckets more efficiently. I ran all my tests on the `geoname` rally track with the following query: ```` { "size": 0, "aggs": { "country_population": { "terms": { "size": 100, "field": "country_code.raw" }, "aggs": { "admin1_code": { "terms": { "size": 100, "field": "admin1_code.raw" }, "aggs": { "admin2_code": { "terms": { "size": 100, "field": "admin2_code.raw" }, "aggs": { "sum_population": { "sum": { "field": "population" } } } } } } } } } } ````