Commit Graph

135 Commits

Author SHA1 Message Date
Colin Goodheart-Smithe d4e83df3b8 Aggregations: Adds ability to sort on multiple criteria
The terms aggregation can now support sorting on multiple criteria by replacing the sort object with an array or sort object whose order signifies the priority of the sort. The existing syntax for sorting on a single criteria also still works.

Contributes to #6917
Replaces #7588
2014-09-15 11:08:29 +01:00
markharwood 3c8f8cc090 Aggs enhancement - allow Include/Exclude clauses to use array of terms as alternative to a regex
Closes #6782
2014-09-12 15:28:03 +01:00
Lee Hinman 1dd26888f6 [DOCS] Additional documentation for _score accessing
Closes #7043
2014-09-11 12:53:25 +02:00
smayzak 65a0ca021d The description was incorrect
Looked like a copy and paste from another aggregation
2014-09-10 16:05:03 +02:00
smayzak 6416f5d3d0 Fixing some grammar 2014-09-10 16:05:03 +02:00
David Pilato 7fdd3651fa [docs] Fix typo: resonable - reasonable 2014-09-10 15:57:57 +02:00
Colin Goodheart-Smithe b127b52fd3 Revert "Aggregations: Adds ability to sort on multiple criteria"
This reverts commit bfedd11ffa.
2014-09-08 20:27:19 +01:00
Colin Goodheart-Smithe bfedd11ffa Aggregations: Adds ability to sort on multiple criteria
The terms aggregation can now support sorting on multiple criteria by replacing the sort object with an array or sort object whose order signifies the priority of the sort. The existing syntax for sorting on a single criteria also still works.

Contributes to #6917
2014-09-08 15:20:33 +01:00
Clinton Gormley 1bdf79e527 Docs: Added explanation of how to do multi-field terms agg
Closes #5100
2014-09-07 11:09:52 +02:00
shrinidhichaudhari 13e3a5e99c Docs: Update cardinality-aggregation.asciidoc
Closes #7516
2014-09-06 20:45:45 +02:00
Clinton Gormley a059a6574a Update reverse-nested-aggregation.asciidoc
Fixed reverse nested example

Closes #7463
2014-09-02 11:40:41 +01:00
Adrien Grand 8e1d3d56b3 Docs: Replace added[1.4.0] with coming[1.4.0] since 1.4 is not released yet. 2014-08-29 11:57:22 +02:00
londocr 1213eec834 Spelling error of aggregation 2014-08-28 08:57:12 +02:00
Colin Goodheart-Smithe 7f943f0296 Aggregations: Scriptable Metrics Aggregation
A metrics aggregation which runs specified scripts at the init, collect, combine, and reduce phases

Closes #5923
2014-08-20 18:17:27 +01:00
Martijn van Groningen 383e64bd5c Aggregations: Add `children` bucket aggregator that is able to map buckets between parent types and child types using the already builtin parent/child support.
Closes #6936
2014-08-19 12:40:51 +02:00
Konrad Feldmeier 3b3e2ed5e9 Docs: Remove the 'Factor' paragraph to reflect #6490
The current implementation of 'date_histogram' does not understand
the `factor` parameter. Since the docs shouldn't raise false hopes,
I removed the section.

Closes #7277
2014-08-18 13:02:15 +02:00
Clinton Gormley 6477e13c77 Typo 2014-08-18 12:30:49 +02:00
smayzak 8449128032 error in code
The top-tags and terms were reversed.
2014-08-18 12:28:53 +02:00
Colin Goodheart-Smithe e6632ec63e [DOCS] fixed title for filters aggregation documentation 2014-08-07 08:37:43 +01:00
Clinton Gormley 7b0b315b71 Tidied up the filters agg docs and added a coming[] tag 2014-08-07 09:03:23 +02:00
Britta Weber a3cefd919e significant terms: add google normalized distance, add chi square
closes #6858
2014-08-04 08:15:26 +02:00
uboness 3c9c9f33e2 Aggregations Added Filters aggregation
A multi-bucket aggregation where multiple filters can be defined (each filter defines a bucket). The buckets will collect all the documents that match their associated filter.

This aggregation can be very useful when one wants to compare analytics between different criterias. It can also be accomplished using multiple definitions of the single filter aggregation, but here, the user will only need to define the sub-aggregations only once.

Closes #6118
2014-08-01 16:01:08 +01:00
Colin Goodheart-Smithe 655157c83a Aggregations: Added an option to show the upper bound of the error for the terms aggregation.
This is only applicable when the order is set to _count.  The upper bound of the error in the doc count is calculated by summing the doc count of the last term on each shard which did not return the term.  The implementation calculates the error by summing the doc count for the last term on each shard for which the term IS returned and then subtracts this value from the sum of the doc counts for the last term from ALL shards.

Closes #6696
2014-07-25 14:24:24 +01:00
Simon Willnauer 5bfea56457 [DOCS] move all coming tags to added in master 2014-07-23 16:37:19 +02:00
Adrien Grand abeefbddea Docs: Update documentation about execution hints for the terms aggregation. 2014-07-21 11:55:57 +02:00
Britta Weber 74927adced significant terms: infrastructure for changing easily the significance heuristic
This commit adds the infrastructure to allow pluging in different
measures for computing the significance of a term.
Significance measures can be provided externally by overriding

- SignificanceHeuristic
- SignificanceHeuristicBuilder
- SignificanceHeuristicParser

closes #6561
2014-07-14 11:00:50 +02:00
Florian Hopf 3689f67a76 Docs: Fixed invalid word count in geodistance agg doc
Closes #6838
2014-07-11 18:35:36 +02:00
Andrii Gakhov 80321d89d9 Docs: Update histogram-aggregation.asciidoc
filter in a filtered query should be under "filter" key

Closes #6738
2014-07-07 10:44:11 +02:00
Chris 011e20678d [DOCS] Fixed json example in nested-aggregation.asciidoc 2014-06-18 19:38:02 +02:00
Colin Goodheart-Smithe 7423ce0560 Aggregations: Added percentile rank aggregation
Percentile Rank Aggregation is the reverse of the Percetiles aggregation.  It determines the percentile rank (the proportion of values less than a given value) of the provided array of values.

Closes #6386
2014-06-18 12:02:08 +01:00
Martijn van Groningen 5e408f3d40 Change the top_hits to be a metric aggregation instead of a bucket aggregation (which can't have an sub aggs)
Closes #6395
Closes #6434
2014-06-10 09:09:50 +02:00
markharwood 724129e6ce Aggregations optimisation for memory usage. Added changes to core Aggregator class to support a new mode of deferred collection.
A new "breadth_first" results collection mode allows upper branches of aggregation tree to be calculated and then pruned
to a smaller selection before advancing into executing collection on child branches.

Closes #6128
2014-06-06 15:59:51 +01:00
Colin Goodheart-Smithe b9f4d44b14 Aggregations: Adds GeoBounds Aggregation
The GeoBounds Aggregation is a new single bucket aggregation which outputs the coordinates of a bounding box containing all the points from all the documents passed to the aggregation as well as the doc count. Geobound Aggregation also use a wrap_logitude parameter which specifies whether the resulting bounding box is permitted to overlap the international date line.  This option defaults to true.

This aggregation introduces the idea of MetricsAggregation which do not return double values and cannot be used for sorting.  The existing MetricsAggregation has been renamed to NumericMetricsAggregation and is a subclass of MetricsAggregation.  MetricsAggregations do not store doc counts and do not support child aggregations.

Closes #5634
2014-06-03 15:59:56 +01:00
Martijn van Groningen aab38fb2e6 Aggregations: added pagination support to `top_hits` aggregation by adding `from` option.
Closes #6299
2014-05-30 11:45:31 +02:00
Martijn van Groningen 5fafd2451a Added `top_hits` aggregation that keeps track of the most relevant document being aggregated per bucket.
Closes #6124
2014-05-23 16:01:18 +02:00
Simon Willnauer 9d5507047f Update Documentation Feature Flags [1.2.0] 2014-05-22 15:06:42 +02:00
Britta Weber 08e57890f8 use shard_min_doc_count also in TermsAggregation
This was discussed in issue #6041 and #5998 .

closes #6143
2014-05-14 14:10:04 +02:00
David Pilato 1cb2c3bdd3 [DOCS] reverse-nested aggs are added in 1.2.0 2014-05-13 20:00:42 +02:00
Adrien Grand cc530b9037 Use t-digest as a dependency.
Our improvements to t-digest have been pushed upstream and t-digest also got
some additional nice improvements around memory usage and speedups of quantile
estimation. So it makes sense to use it as a dependency now.

This also allows to remove the test dependency on Apache Mahout.

Close #6142
2014-05-13 10:38:08 +02:00
markharwood 1e560b0d92 Significant_terms agg: added option for a background_filter to define background context for analysis of term frequencies
Closes #5944
2014-05-13 09:10:30 +01:00
Clinton Gormley 5b93255ec8 [DOCS] Added "Aggregation" to all aggs titles 2014-05-13 01:35:58 +02:00
Rashid Khan 233aaa63c9 Change key to keyed 2014-05-12 13:15:07 -07:00
uboness fc52db1209 Changed the respnose structure of the percentiles aggregation where now all the percentiles are placed under a `values` object (or `values` array in case the `keyed` flag is set to `false`
Closes #5870
2014-05-07 18:35:24 +02:00
Britta Weber 7944369fd1 Add `shard_min_doc_count` parameter for significant terms similar to `shard_size`
Significant terms internally maintain a priority queue per shard with a size potentially
lower than the number of terms. This queue uses the score as criterion to determine if
a bucket is kept or not. If many terms with low subsetDF score very high
but the `min_doc_count` is set high, this might result in no terms being
returned because the pq is filled with low frequent terms which are all sorted
out in the end.

This can be avoided by increasing the `shard_size` parameter to a higher value.
However, it is not immediately clear to which value this parameter must be set
because we can not know how many terms with low frequency are scored higher that
the high frequent terms that we are actually interested in.

On the other hand, if there is no routing of docs to shards involved, we can maybe
assume that the documents of classes and also the terms therein are distributed evenly
across shards. In that case it might be easier to not add documents to the pq that have
subsetDF <= `shard_min_doc_count` which can be set to something like
`min_doc_count`/number of shards  because we would assume that even when summing up
the subsetDF across shards `min_doc_count` will not be reached.

closes #5998
closes #6041
2014-05-07 18:02:56 +02:00
gabriel-tessier 7b0efcbd96 fix typo 2014-05-06 15:54:36 +02:00
Martijn van Groningen 013b319415 Added `reverse_nested` aggregation.
The `reverse_nested` aggregation allows to aggregate on properties outside of the nested scope of a `nested` aggregation.

Closes #5507
2014-05-01 00:23:05 +07:00
gabriel-tessier 000c33aac3 fix typo 2014-04-07 09:23:46 +02:00
Martijn van Groningen ade1d0ef57 Added global ordinals (unique incremental numbering for terms) to fielddata.
Added a terms aggregation implementations that work on global ordinals, which is also the default.

Closes #5672
2014-04-07 11:06:41 +07:00
bleskes 5d832374dd Update Documentation Feature Flags [1.1.0] 2014-03-25 17:51:30 +01:00
uboness 7d6ad8d91c Added extended_bounds support for date_/histogram aggs
By default the date_/histogram returns all the buckets within the range of the data itself, that is, the documents with the smallest values (on which with histogram) will determine the min bucket (the bucket with the smallest key) and the documents with the highest values will determine the max bucket (the bucket with the highest key). Often, when when requesting empty buckets (min_doc_count : 0), this causes a confusion, specifically, when the data is also filtered.

To understand why, let's look at an example:

Lets say the you're filtering your request to get all docs from the last month, and in the date_histogram aggs you'd like to slice the data per day. You also specify min_doc_count:0 so that you'd still get empty buckets for those days to which no document belongs. By default, if the first document that fall in this last month also happen to fall on the first day of the **second week** of the month, the date_histogram will **not** return empty buckets for all those days prior to that second week. The reason for that is that by default the histogram aggregations only start building buckets when they encounter documents (hence, missing on all the days of the first week in our example).

With extended_bounds, you now can "force" the histogram aggregations to start building buckets on a specific min values and also keep on building buckets up to a max value (even if there are no documents anymore). Using extended_bounds only makes sense when min_doc_count is 0 (the empty buckets will never be returned if the min_doc_count is greater than 0).

Note that (as the name suggest) extended_bounds is **not** filtering buckets. Meaning, if the min bounds is higher than the values extracted from the documents, the documents will still dictate what the min bucket will be (and the same goes to the extended_bounds.max and the max bucket). For filtering buckets, one should nest the histogram agg under a range filter agg with the appropriate min/max.

Closes #5224
2014-03-20 14:48:27 +01:00
markharwood 5f1d9af9fe Documentation fix for significant_terms heading levels 2014-03-17 12:17:54 +00:00
Boaz Leskes ee8743f3f2 [Docs] added a missing reference to significantterms-aggergations
Also fix header level mismatch issue reported by the build
2014-03-17 11:45:55 +01:00
rphadake 36a0cb99d7 [Doc] doc updates for date histogram interval
Close #5308
2014-03-14 18:55:32 +01:00
Adrien Grand eef71da650 [Doc] Add a chart about the relative error of the percentiles aggregation. 2014-03-14 12:23:23 +01:00
markharwood 767bef0596 Significant_terms aggregation identifies terms that are significant rather than merely popular in a set.
Significance is related to the changes in document frequency observed between everyday use in the corpus and
frequency observed in the result set. The asciidocs include extensive details on the applications of this feature.

Closes #5146
2014-03-14 10:34:24 +00:00
Adrien Grand 5821fa042c Cardinality aggregation.
This aggregation computes unique term counts using the hyperloglog++ algorithm
which uses linear counting to estimate low cardinalities and hyperloglog on
higher cardinalities.

Since this algorithm works on hashes, it is useful for high-cardinality fields
to store the hash of values directly in the index, which is the purpose of
the new `murmur3` field type. This is less necessary on low-cardinality
string fields because the aggregator is smart enough to only compute the hash
once per unique value per segment thanks to ordinals, or on numeric fields
since hashing them is very fast.

Close #5426
2014-03-13 19:19:56 +01:00
Benjamin Devèze 2affa5004f Fix small typo in percentiles doc 2014-03-07 10:10:19 +01:00
Adrien Grand f359b7f38b [DOC] The percentiles aggregation is coming in 1.1.0. 2014-03-07 10:03:15 +01:00
uboness 9d0fc76f54 Added support for sorting buckets based on sub aggregations
Supports sorting on sub-aggs down the current hierarchy. This is supported as long as the aggregation in the specified order path are of a single-bucket type, where the last aggregation in the path points to either a single-bucket aggregation or a metrics one. If it's a single-bucket aggregation, the sort will be applied on the document count in the bucket (i.e. doc_count), and if it is a metrics type, the sort will be applied on the pointed out metric (in case of a single-metric aggregations, such as avg, the sort will be applied on the single metric value)

 NOTE: this commit adds a constraint on what should be considered a valid aggregation name. Aggregations names must be alpha-numeric and may contain '-' and '_'.

 Closes #5253
2014-03-06 00:05:27 +01:00
Zachary Tong 7b16c5857d Percentiles aggregation.
A new metric aggregation that can compute approximate values of arbitrary
percentiles.

Close #5323
2014-03-03 18:06:14 +01:00
Binh Ly 7e49848697 Clarify range aggregations 2014-02-28 14:38:57 -05:00
Clinton Gormley 53ce0e8e27 [DOCS] Fixed added[] tag version number 2014-02-28 15:29:43 +01:00
uboness d335630e57 [docs] fixed errors in aggs docs
- error in nested aggs example
- error in terms aggs example
2014-02-13 20:36:02 +01:00
Adrien Grand 9cb17408cb Make size=0 return all buckets for the geohash_grid aggregation.
Close #4875
2014-02-07 09:55:10 +01:00
Boaz Leskes 9bf263c741 [DOCS] Fix terms agg value script example 2014-02-06 16:35:49 +01:00
Boaz Leskes ae4ed29f9b [Docs] value_count supports script per 1.1 2014-02-06 15:04:50 +01:00
Adrien Grand 6777be60ce Add script support to value_count aggregations.
Close #5001
2014-02-04 14:29:32 +01:00
Clinton Gormley 238b26a466 [DOC] Tidied up geohashgrid aggregations 2014-02-04 11:54:32 +01:00
Jun Ohtani ba415b8ad2 Does not support "script" in value_clunt aggregation. 2014-02-04 10:26:07 +01:00
Adrien Grand cc1ff560df Rename `geohashgrid` to `geohash_grid` in documentation.
It was renamed in fc6bc4c477.

Close #4997
2014-02-04 09:39:55 +01:00
uboness d3f2173ef9 fixed date_/histogram aggregation documentation - added documentation for the `min_doc_count` setting
Closes #4944
2014-01-29 20:55:26 +01:00
uboness 9f04e5fe38 fixed nested example response in docs
Closes #4935
2014-01-29 13:09:12 +01:00
uboness dd389d1cc5 Made all multi-bucket aggs return consistent response format
Closes #4926
2014-01-28 17:46:57 +01:00
Adrien Grand 9282ae4ffd Terms aggregations: make size=0 return all terms.
Terms aggregations return up to `size` terms, so up to now, the way to get all
matching terms back was to set `size` to an arbitrary high number that would be
larger than the number of unique terms.

Terms aggregators already made sure to not allocate memory based on the `size`
parameter so this commit mostly consists in making `0` an alias for the
maximum integer value in the TermsParser.

Close #4837
2014-01-22 11:05:10 +01:00
Florian Gilcher eed079aaac Reference docs fixes
* Make it clearer that `aggs` is an allowed synomym
  for the `aggregations` key
* Fix broken example in for datehistogram, `1.5M` is
  not an allowed interval
* Make use of colon before examples consistent
* Fix typos
2014-01-20 12:14:17 +01:00
Dawid Weiss ae71b25145 Documentation typo. 2014-01-20 11:51:08 +01:00
Adrien Grand 5c237fe834 Add new option `min_doc_count` to terms and histogram aggregations.
`min_doc_count` is the minimum number of hits that a term or histogram key
should match in order to appear in the response.

`min_doc_count=0` replaces `compute_empty_buckets` for histograms and will
behave exactly like facets' `all_terms=true` for terms aggregations.

Close #4662
2014-01-13 10:09:38 +01:00
markharwood 602de04692 A GeoHashGrid aggregation that buckets GeoPoints into cells whose dimensions are determined by a choice of GeoHash resolution.
Added a long-based representation of GeoHashes to GeoHashUtils for fast evaluation in aggregations.
The new BucketUtils provides a common heuristic for determining the number of results to obtain from each shard in "top N" type requests.
2014-01-07 18:03:33 +00:00
Florian Schilling bc452dff84 * setup accurate GeoDistance Function
* adapt tests
* introduced default GeoDistance function
* Updated docs

closes #4498
2013-12-27 19:15:19 +09:00
Chris Simpson 4f8c916eed [Docs] Fix Typo
Fixes small typo in the geo_distance aggregation docs.
2013-12-18 11:21:21 +01:00
Adrien Grand 36bd9cc432 Aggregations: Ordinals-based string bucketing support.
When the ValuesSource has ordinals, terms ordinals are used as a cache key to
bucket ordinals. This can make terms aggregations on String terms significantly
faster.

Close #4350
2013-12-13 15:34:02 +01:00
uboness cdc7dfbb2c Changed the "script_lang" parameter to "lang" in all value source based aggs - to be consistent with all other script based APIs. 2013-12-02 02:01:03 +01:00
uboness 0d6a35b9a7 - Added support for term filtering based on include/exclude regex on the terms agg
- Added javadoc to the TermsBuilder

Closes #4267
2013-11-29 13:46:48 +01:00
uboness afb0d119e4 - Added docs for the value_count aggregation
- Fixed typos in the terms facets docs
- Fixed aggregation docs layout
- Added docs for shard_size in term aggregation
2013-11-29 12:35:42 +01:00
uboness c7f6c5266d initial commit of the aggregations module
Closes #3300
2013-11-24 03:13:08 -08:00