diff --git a/docs/reference/search.asciidoc b/docs/reference/search.asciidoc index d911ffb600c..a12833707d4 100644 --- a/docs/reference/search.asciidoc +++ b/docs/reference/search.asciidoc @@ -83,6 +83,8 @@ include::search/request-body.asciidoc[] include::search/facets.asciidoc[] +include::search/aggregations.asciidoc[] + include::search/suggesters.asciidoc[] include::search/multi-search.asciidoc[] diff --git a/docs/reference/search/aggregations.asciidoc b/docs/reference/search/aggregations.asciidoc new file mode 100644 index 00000000000..e47175f3586 --- /dev/null +++ b/docs/reference/search/aggregations.asciidoc @@ -0,0 +1,70 @@ +[[search-aggregations]] +== Aggregations + +Aggregations grew out of the <> module and the long expirience of how users use it (and would like to use it) for real-time data analytics purposes. As such, it serves as the next generation replacement for the functionality we currently refer to as "faceting". + +<> provide a great way to aggregate data within a document set context. This context is defined by the executed query in combination with the different levels of filters that can be defined (filtered queries, top level filters, and facet level filters). While powerful, their implementation is not designed from ground up to support complex aggregations and thus limited. + +.Are facets deprecated? +********************************** +As the functionality facets offer is a subset of the the one offered by aggregations, over time, we would like to see users move to aggregations for all realtime data analytics. That said, we are well aware that such transitions/migrations take time, and for this reason we are keeping the facets around for the time being. Nonetheless, facets are and should be considered deprecated and will likely be removed in one of the future major releases. +********************************** + +The aggregations module breaks the barriers the current facet implementation put in place. The new name ("Aggregations") also indicate the intention here - a generic yet extremely powerful framework for building aggregations - any types of aggregations. + +An aggregation can be seen as a _unit-of-work_ that builds analytic information over a set of documents. The context of the execution defines what this document set is (e.g. a top level aggregation executes within the context of the executed query/filters of the search request). + +There are many different types of aggregations, each with its own purpose and output. To better understand these types, it is often easier to break them into two main families: + +_Bucketing_:: + A family of aggregations that build buckets, where each bucket is associated with a _key_ and a document criteria. When the aggregations is executed, the buckets criterias are evaluated on every document in the context and when matches, the document is considered to "fall in" the relevant bucket. By the end of the aggreagation process, we'll end up with a list of buckets - each one with a set of documents that "belong" to it. + +_Metric_:: + Aggregations that keep track and compute metrics over a set of documents + +The interesting part comes next, since each bucket effectively defines a document set (all documents belonging to the bucket), one can potentially associated aggregations on the bucket level, and those will execute within the context of that bucket. This is where the real power of aggregations kicks in: *aggregations can be nested!* + +NOTE: Bucketing aggregations can have sub-aggregations (bucketing or metric). The sub aggregations will be computed for + each of the buckets their parent aggregation generates. There is not hard limit on the level/depth of nested aggregations (one can nest an aggregation under a "parent" aggregation which is itself a sub-aggregation of another highter aggregations) + +=== Structuring Aggregations + +The following snippet captures the basic structure of aggregations: + +[source,js] +-------------------------------------------------- +"aggregations" : { + "" : { + "" : { + + } + [,"aggregations" : { []+ } ]? + } + [,"" : { ... } ]* +} +-------------------------------------------------- + +The `aggregations` object (a.k.a `aggs` for short) in the json holds the aggregations to be computed. Each aggregation is associated with a logical name that the user defines (e.g. if the aggregation computes the average price, then it'll make sense to name it `avg_price`). These logical names will also be used to uniquely identify the aggregations in the response. Each aggregation has a specific type (`` in the above snippet) and is typically the first key within the named aggregation body. Each type of aggregation define its own body, depending on the nature of the aggregation (eg. an `avg` aggregation on a specific field will define the field on which the avg will be calculated). At the same level of the aggregation type definition, one can optionally define a set of additional aggregations, though this only makes sense if the aggregation you defined is of a bucketing nature. In this scenario, the sub-aggregations you define on the bucketing aggregation level will be computed for all the buckets built by the bucketing aggregation. For example, if the you define a set of aggregations under the `range` aggregation, the sub-aggregations will be computed for each of the range buckets that are defined. + +==== Values Source + +Some aggregations work on values extracted from the aggregated documents. Typically, the values will be extracted from a sepcific document field which is set under the `field` settings for the aggrations. It is also possible to define a `<>` that will generate the values (per document). + +When both `field` and `script` settings are configured for the aggregation, the script will be treated as a `value script`. While normal scripts are evaluated on a document level (i.e. the script has access to all the data associated with the document), value scripts are evaluated on the *value* level. In this mode, the values are extracted from the configured `field` and the `script` is used to apply a "transformation" over these value/s + +["NOTE",id="metrics-script-note"] +=============================== +When working with scripts, the `script_lang` and `params` settings can also be defined. The former defines the scripting language that is used (assuming the proper language is available in es either by default or as a plugin). The latter enables defining all the "dynamic" expressions in the script as parameters, and by that keep the script itself static between calls (this will ensure the use of the cached compiled scripts in elasticsearch). +=============================== + +Scripts can generate a single value or multiple values per documents. When generating multiple values, once can use the `script_values_sorted` settings to indicate whether these values are sorted or not. Internally, elasticsearch can perform optimizations when dealing with sorted values (for example, with the `min` aggregations, knowing the values are sorted, elasticsearch will skip the iterations over all the values and rely on the first value in the list to be the minimum value among all other values associated with the same document). + +include::aggregations/metrics.asciidoc[] + +include::aggregations/bucket.asciidoc[] + + + + + + diff --git a/docs/reference/search/aggregations/bucket.asciidoc b/docs/reference/search/aggregations/bucket.asciidoc new file mode 100644 index 00000000000..058675eb01e --- /dev/null +++ b/docs/reference/search/aggregations/bucket.asciidoc @@ -0,0 +1,30 @@ +[[search-aggregations-bucket]] +=== Bucket Aggregations + +Bucket aggregations don't calculate metrics over fields like the metrics aggregations do, but instead, they create buckets of documents. Each bucket is associated with a criteria (depends on the aggregation type) that determines whether or not a document in the current context "falls" in it. In other words, the buckets effectively define document sets. In addition to the buckets themselves, the `bucket` aggregations also compute and return the number of documents that "fell in" each bucket. + +Bucket aggregations, as opposed to `metrics` aggregations, can hold sub-aggregations. These sub aggregations will be aggregated for each of the buckets created by their "parent" bucket aggregation. + +There are different bucket aggregators, each with a different "bucketing" strategy. Some define a single bucket, some define fixed number of multiple buckets, and others dynamically create the buckets during the aggregation process. + +include::bucket/global-aggregation.asciidoc[] + +include::bucket/filter-aggregation.asciidoc[] + +include::bucket/missing-aggregation.asciidoc[] + +include::bucket/nested-aggregation.asciidoc[] + +include::bucket/terms-aggregation.asciidoc[] + +include::bucket/range-aggregation.asciidoc[] + +include::bucket/daterange-aggregation.asciidoc[] + +include::bucket/iprange-aggregation.asciidoc[] + +include::bucket/histogram-aggregation.asciidoc[] + +include::bucket/datehistogram-aggregation.asciidoc[] + +include::bucket/geodistance-aggregation.asciidoc[] \ No newline at end of file diff --git a/docs/reference/search/aggregations/bucket/datehistogram-aggregation.asciidoc b/docs/reference/search/aggregations/bucket/datehistogram-aggregation.asciidoc new file mode 100644 index 00000000000..537207e936d --- /dev/null +++ b/docs/reference/search/aggregations/bucket/datehistogram-aggregation.asciidoc @@ -0,0 +1,106 @@ +[[search-aggregations-bucket-datehistogram-aggregation]] +=== Date Histogram + +A multi-bucket aggregation similar to the <> except it can only be applied on date values. Since dates are represented in elasticsearch internally as long values, it is possible to use the normal `histogram` on dates as well, though accuracy will be compromized. The reason for this is in the fact that time based intervals are not fixed (think of leap years and on the number of days in a month). For this reason, we need a spcial support for time based data. From functionality perspective, this historam supports the same features as the normal <>. The main difference though is that the interval can be specified by date/time expressions. + +Requesting a month length bucket intervals + +[source,js] +-------------------------------------------------- +{ + "aggs" : { + "articles_over_time" : { + "date_histogram" : { + "field" : "date", + "interval" : "month" + } + } + } +} +-------------------------------------------------- + +or based on 1.5 months + +[source,js] +-------------------------------------------------- +{ + "aggs" : { + "articles_over_time" : { + "date_histogram" : { + "field" : "date", + "interval" : "1.5M" + } + } + } +} +-------------------------------------------------- + +Other available expressions for interval: `year`, `quarter`, `week`, `day`, `hour`, `minute`, `second` + +==== Time Zone + +By default, times are stored as UTC milliseconds since the epoch. Thus, all computation and "bucketing" / "rounding" is done on UTC. It is possible to provide a time zone (both pre rounding, and post rounding) value, which will cause all computations to take the relevant zone into account. The time returned for each bucket/entry is milliseconds since the epoch of the provided time zone. + +The parameters are `pre_zone` (pre rounding based on interval) and `post_zone` (post rounding based on interval). The `time_zone` parameter simply sets the `pre_zone` parameter. By default, those are set to `UTC`. + +The zone value accepts either a numeric value for the hours offset, for example: `"time_zone" : -2`. It also accepts a format of hours and minutes, like `"time_zone" : "-02:30"`. Another option is to provide a time zone accepted as one of the values listed here. + +Lets take an example. For `2012-04-01T04:15:30Z`, with a `pre_zone` of `-08:00`. For day interval, the actual time by applying the time zone and rounding falls under `2012-03-31`, so the returned value will be (in millis) of `2012-03-31T00:00:00Z` (UTC). For hour interval, applying the time zone results in `2012-03-31T20:15:30`, rounding it results in `2012-03-31T20:00:00`, but, we want to return it in UTC (`post_zone` is not set), so we convert it back to UTC: `2012-04-01T04:00:00Z`. Note, we are consistent in the results, returning the rounded value in UTC. + +`post_zone` simply takes the result, and adds the relevant offset. + +Sometimes, we want to apply the same conversion to UTC we did above for hour also for day (and up) intervals. We can set `pre_zone_adjust_large_interval` to `true`, which will apply the same conversion done for hour interval in the example, to day and above intervals (it can be set regardless of the interval, but only kick in when using day and higher intervals). + +==== Factor + +The date histogram works on numeric values (since time is stored in milliseconds since the epoch in UTC). But, sometimes, systems will store a different resolution (like seconds since UTC) in a numeric field. The `factor` parameter can be used to change the value in the field to milliseconds to actual do the relevant rounding, and then be applied again to get to the original unit. For example, when storing in a numeric field seconds resolution, the factor can be set to 1000. + +==== Pre/Post Offset + +Specific offsets can be provided for pre rounding and post rounding. The `pre_offset` for pre rounding, and `post_offset` for post rounding. The format is the date time format (`1h`, `1d`, etc...). + +==== Keys + +Since internally, dates are represented as 64bit numbers, these numbers are returned as the bucket keys (each key representing a date - milliseconds since the epoch). It is also possible to define a date format, which will result in returning the dates as formatted strings next to the numeric key values: + +[source,js] +-------------------------------------------------- +{ + "aggs" : { + "articles_over_time" : { + "date_histogram" : { + "field" : "date", + "interval" : "1M", + "format" : "yyyy-MM-dd" <1> + } + } + } +} +-------------------------------------------------- + +<1> Supports expressive date <> + +Response: + +[source,js] +-------------------------------------------------- +{ + "aggregations": { + "articles_over_time": [ + { + "key_as_string": "2013-02-02", + "key": 1328140800000, + "doc_count": 1 + }, + { + "key_as_string": "2013-03-02", + "key": 1330646400000, + "doc_count": 2 + }, + ... + ] + } +} +-------------------------------------------------- + +Like with the normal <>, both document level scripts and value level scripts are supported. It is also possilbe to control the order of the returned buckets using the `order` settings and empty buckets can also be returned by setting the `empty_buckets` field to `true` (defaults to `false`). \ No newline at end of file diff --git a/docs/reference/search/aggregations/bucket/daterange-aggregation.asciidoc b/docs/reference/search/aggregations/bucket/daterange-aggregation.asciidoc new file mode 100644 index 00000000000..f9496108453 --- /dev/null +++ b/docs/reference/search/aggregations/bucket/daterange-aggregation.asciidoc @@ -0,0 +1,106 @@ +[[search-aggregations-bucket-daterange-aggregation]] +=== Date Range + +A range aggregation that is dedicated for date values. The main difference between this aggregation and the normal <> aggregation is that the `from` and `to` values can be expressed in Date Math expressions, and it is also possible to specify a date format by which the `from` and `to` response fields will be returned: + +Example: + +[source,js] +-------------------------------------------------- +{ + "aggs": { + "range": { + "date_range": { + "field": "date", + "format": "MM-yyy", + "ranges": [ + { "to": "now-10M/M" }, + { "from": "now-10M/M" } + ] + } + } + } +} +-------------------------------------------------- + +In the example above, we created two range buckets, the first will "bucket" all documents dated prior to 10 months ago and +the second will "bucket" all documents dated since 10 months ago + +Response: + +[source,js] +-------------------------------------------------- +{ + ... + + "aggregations": { + "range": [ + { + "to": 1.3437792E+12, + "to_as_string": "08-2012", + "doc_count": 7 + }, + { + "from": 1.3437792E+12, + "from_as_string": "08-2012", + "doc_count": 2 + } + ] + } +} +-------------------------------------------------- + +[[date-format-pattern]] +==== Date Format/Pattern (copied from http://joda-time.sourceforge.net/apidocs/org/joda/time/format/DateTimeFormat.html[JodaDate]) + +All ASCII letters are reserved as format pattern letters, which are defined as follows: + +[options="header"] +|======= +|Symbol |Meaning |Presentation |Examples +|G |era |text |AD +|C |century of era (>=0) |number |20 +|Y |year of era (>=0) |year |1996 + +|x |weekyear |year |1996 +|w |week of weekyear |number |27 +|e |day of week |number |2 +|E |day of week |text |Tuesday; Tue + +|y |year |year |1996 +|D |day of year |number |189 +|M |month of year |month |July; Jul; 07 +|d |day of month |number |10 + +|a |halfday of day |text |PM +|K |hour of halfday (0~11) |number |0 +|h |clockhour of halfday (1~12) |number |12 + +|H |hour of day (0~23) |number |0 +|k |clockhour of day (1~24) |number |24 +|m |minute of hour |number |30 +|s |second of minute |number |55 +|S |fraction of second |number |978 + +|z |time zone |text |Pacific Standard Time; PST +|Z |time zone offset/id |zone |-0800; -08:00; America/Los_Angeles + +|' |escape for text |delimiter +|'' |single quote |literal |' +|======= + +The count of pattern letters determine the format. + +Text:: If the number of pattern letters is 4 or more, the full form is used; otherwise a short or abbreviated form is used if available. + +Number:: The minimum number of digits. Shorter numbers are zero-padded to this amount. + +Year:: Numeric presentation for year and weekyear fields are handled specially. For example, if the count of 'y' is 2, the year will be displayed as the zero-based year of the century, which is two digits. + +Month:: 3 or over, use text, otherwise use number. + +Zone:: 'Z' outputs offset without a colon, 'ZZ' outputs the offset with a colon, 'ZZZ' or more outputs the zone id. + +Zone names:: Time zone names ('z') cannot be parsed. + +Any characters in the pattern that are not in the ranges of ['a'..'z'] and ['A'..'Z'] will be treated as quoted text. For instance, characters like ':', '.', ' ', '#' and '?' will appear in the resulting time text even they are not embraced within single quotes. \ No newline at end of file diff --git a/docs/reference/search/aggregations/bucket/filter-aggregation.asciidoc b/docs/reference/search/aggregations/bucket/filter-aggregation.asciidoc new file mode 100644 index 00000000000..5166002f9b5 --- /dev/null +++ b/docs/reference/search/aggregations/bucket/filter-aggregation.asciidoc @@ -0,0 +1,38 @@ +[[search-aggregations-bucket-filter-aggregation]] +=== Filter + +Defines a single bucket of all the documents in the current document set context that match a specified filter. Often this will be used to narrow down the current aggregation context to a specific set of documents. + +Example: + +[source,js] +-------------------------------------------------- +{ + "aggs" : { + "in_stock_products" : { + "filter" : { "range" : { "stock" : { "gt" : 0 } } }, + "aggs" : { + "avg_price" : { "avg" : { "field" : "price" } } + } + } + } +} +-------------------------------------------------- + +In the above example, we calculate the average price of all the products that are currently in-stock. + +Response: + +[source,js] +-------------------------------------------------- +{ + ... + + "aggs" : { + "in_stock_products" : { + "doc_count" : 100, + "avg_price" : { "value" : 56.3 } + } + } +} +-------------------------------------------------- \ No newline at end of file diff --git a/docs/reference/search/aggregations/bucket/geodistance-aggregation.asciidoc b/docs/reference/search/aggregations/bucket/geodistance-aggregation.asciidoc new file mode 100644 index 00000000000..86893974f96 --- /dev/null +++ b/docs/reference/search/aggregations/bucket/geodistance-aggregation.asciidoc @@ -0,0 +1,103 @@ +[[search-aggregations-bucket-geodistance-aggregation]] +=== Geo Distance + +A multi-bucket aggregation that works on `geo_point` fields and onceptually works very similar to the <> aggregation. The user can define a point of origin and a set of distance range buckets. The aggregation evaluate the distance of each document value from the origin point and determines the buckets it belongs to based on the ranges (a document belongs to a bucket if the distance between the document and the origin falls within the distance range of the bucket). + +[source,js] +-------------------------------------------------- +{ + "aggs" : { + "rings_around_amsterdam" : { + "geo_distance" : { + "field" : "location", + "origin" : "52.3760, 4.894", + "ranges" : [ + { "to" : 100 }, + { "from" : 100, "to" : 300 }, + { "from" : 300 } + ] + } + } + } +} +-------------------------------------------------- + +Response: + +[source,js] +-------------------------------------------------- +{ + "aggregations": { + "rings": [ + { + "unit": "km", + "to": 100.0, + "doc_count": 3 + }, + { + "unit": "km", + "from": 100.0, + "to": 300.0, + "doc_count": 1 + }, + { + "unit": "km", + "from": 300.0, + "doc_count": 7 + } + ] + } +} +-------------------------------------------------- + +The specified field must be of type `geo_point` (which can only be set explicitly in the mappings). And it can also hold an array of `geo_point` fields, in which case all will be taken into account during aggregation. The origin point can accept all formats supported by the `geo_point` <>: + +* Object format: `{ "lat" : 52.3760, "lon" : 4.894 }` - this is the safest format as it is the most explicit about the `lat` & `lon` values +* String format: `"52.3760, 4.894"` - where the first number is the `lat` and the second is the `lon` +* Array format: `[4.894, 52.3760]` - which is based on the `GeoJson` standard and where the first number is the `lon` and the second one is the `lat` + +By default, the distance unit is `km` but it can also accept: `mi` (miles), `in` (inch), `yd` (yards), `m` (meters), `cm` (centimeters), `mm` (millimeters). + +[source,js] +-------------------------------------------------- +{ + "aggs" : { + "rings" : { + "geo_distance" : { + "field" : "location", + "origin" : "52.3760, 4.894", + "unit" : "mi", <1> + "ranges" : [ + { "to" : 100 }, + { "from" : 100, "to" : 300 }, + { "from" : 300 } + ] + } + } + } +} +-------------------------------------------------- + +<1> The distances will be computed as miles + +There are two distance calculation modes: `arc` (the default) and `plane`. The `arc` calculation is the most accurate one but also the more expensive one in terms of performance. The `plane` is faster but less accurate. Consider using `plane` when your search context is "narrow" and spans smaller geographical areas (like cities or even countries). `plane` may return higher error mergins for searches across very large areas (e.g. cross continent search). The distance calculation type can be set using the `distance_type` parameter: + +[source,js] +-------------------------------------------------- +{ + "aggs" : { + "rings" : { + "geo_distance" : { + "field" : "location", + "origin" : "52.3760, 4.894", + "distance_type" : "plane", + "ranges" : [ + { "to" : 100 }, + { "from" : 100, "to" : 300 }, + { "from" : 300 } + ] + } + } + } +} +-------------------------------------------------- \ No newline at end of file diff --git a/docs/reference/search/aggregations/bucket/global-aggregation.asciidoc b/docs/reference/search/aggregations/bucket/global-aggregation.asciidoc new file mode 100644 index 00000000000..037a9b34313 --- /dev/null +++ b/docs/reference/search/aggregations/bucket/global-aggregation.asciidoc @@ -0,0 +1,51 @@ +[[search-aggregations-bucket-global-aggregation]] +=== Global + +Defines a single bucket of all the documents within the search execution context. This context is defined by the indices and the document types you're searching on, but is *not* influenced by the search query itself. + +NOTE: Global aggregators can only be placed as top level aggregators (it makes no sense to embed a global aggregator + within another bucket aggregator) + +Example: + +[source,js] +-------------------------------------------------- +{ + "query" : { + "match" : { "title" : "shirt" } + }, + "aggs" : { + "all_products" : { + "global" : {}, <1> + "aggs" : { <2> + "avg_price" : { "avg" : { "field" : "price" } } + } + } + } +} +-------------------------------------------------- + +<1> The `global` aggregation has an empty body +<2> The sub-aggregations that are registered for this `global` aggregation + +The above aggregation demonstrates how one would compute aggregations (`avg_price` in this example) on all the documents in the search context, regardless of the query (in our example, it will compute the the average price over all products in our catalog, not just on the "shirts"). + +The response for the above aggreation: + +[source,js] +-------------------------------------------------- +{ + ... + + "aggregations" : { + "all_products" : { + "doc_count" : 100, <1> + "avg_price" : { + "value" : 56.3 + } + } + } +} +-------------------------------------------------- + +<1> The number of documents that were aggregated (in our case, all documents within the search context) \ No newline at end of file diff --git a/docs/reference/search/aggregations/bucket/histogram-aggregation.asciidoc b/docs/reference/search/aggregations/bucket/histogram-aggregation.asciidoc new file mode 100644 index 00000000000..eb13117fee6 --- /dev/null +++ b/docs/reference/search/aggregations/bucket/histogram-aggregation.asciidoc @@ -0,0 +1,203 @@ +[[search-aggregations-bucket-histogram-aggregation]] +=== Histogram + +A multi-bucket values source based aggregation that can be applied on numeric values extracted from the documents. It dynamically builds fixed size (a.k.a. interval) buckets over the values. For example, if the documents have a field that holds a price (numeric), we can configure this aggregation to dynamically build buckets with interval `5` (in case of price it may represent $5). When the aggregation executes, the price field of every document will be evaluated and will be rounded down to its closes bucket - for example, if the price is `32` and the bucket size is `5` then the rounding will yield `30` and thus the document will "fall" into the bucket that is associated withe the key `30`. To make this more formal, here is the rounding function that is used: + +[source,java] +-------------------------------------------------- +rem = value % interval +if (rem < 0) { + rem += interval +} +bucket_key = value - rem +-------------------------------------------------- + +The following snippet "buckets" the products based on their `price` by interval of `50`: + +[source,js] +-------------------------------------------------- +{ + "aggs" : { + "prices" : { + "histogram" : { + "field" : "price", + "interval" : 50 + } + } + } +} +-------------------------------------------------- + +And the following may be the response: + +[source,js] +-------------------------------------------------- +{ + "aggregations": { + "prices": [ + { + "key": 0, + "doc_count": 2 + }, + { + "key": 50, + "doc_count": 4 + }, + { + "key": 150, + "doc_count": 3 + } + ] + } +} +-------------------------------------------------- + +The response above shows that non of the aggregated products has a price that falls within the range of `[100 - 150)`. By default, the response will only contain the non-empty buckets, though it is possible to also return those, by setting the `empty_buckets` flag to `true`: + +[source,js] +-------------------------------------------------- +{ + "aggs" : { + "prices" : { + "histogram" : { + "field" : "price", + "interval" : 50, + "empty_buckets" : true + } + } + } +} +-------------------------------------------------- + +Response: + +[source,js] +-------------------------------------------------- +{ + "aggregations": { + "prices": [ + { + "key": 0, + "doc_count": 2 + }, + { + "key": 50, + "doc_count": 4 + }, + { + "key" : 100, + "doc_count" : 0 + }, + { + "key": 150, + "doc_count": 3 + } + ] + } +} +-------------------------------------------------- + +==== Order + +By default the returned buckets are sorted by their `key` ascending, though the order behaviour can be controled using the `order` setting. + +Ordering the buckets by their key - descending: + +[source,js] +-------------------------------------------------- +{ + "aggs" : { + "prices" : { + "histogram" : { + "field" : "price", + "interval" : 50, + "order" : { "_key" : "desc" } + } + } + } +} +-------------------------------------------------- + +Ordering the buckets by their `doc_count` - ascending: + +[source,js] +-------------------------------------------------- +{ + "aggs" : { + "prices" : { + "histogram" : { + "field" : "price", + "interval" : 50, + "order" : { "_count" : "asc" } + } + } + } +} +-------------------------------------------------- + +If the histogram aggregation has a direct metrics sub-aggregation, the latter can determine the order of the buckets: + +[source,js] +-------------------------------------------------- +{ + "aggs" : { + "prices" : { + "histogram" : { + "field" : "price", + "interval" : 50, + "order" : { "price_stats.min" : "asc" } <1> + }, + "aggs" : { + "price_stats" : { "stats" : {} } <2> + } + } + } +} +-------------------------------------------------- + +<1> The `{ "price_stats.min" : asc" }` will sort the buckets based on `min` value of their their `price_stats` sub-aggregation. + +<2> There is no need to configure the `price` field for the `price_stats` aggregation as it will inherit it by default from its parent histogram aggregation. + +==== Response Format + +By default, the buckets are retuned as an ordered array. It is also possilbe to request the response as a hash instead keyed by the buckets keys: + +[source,js] +-------------------------------------------------- +{ + "aggs" : { + "prices" : { + "histogram" : { + "field" : "price", + "interval" : 50, + "keyed" : true + } + } + } +} +-------------------------------------------------- + +Response: + +[source,js] +-------------------------------------------------- +{ + "aggregations": { + "prices": { + "0": { + "key": 0, + "doc_count": 2 + }, + "50": { + "key": 50, + "doc_count": 4 + }, + "150": { + "key": 150, + "doc_count": 3 + } + } + } +} +-------------------------------------------------- diff --git a/docs/reference/search/aggregations/bucket/iprange-aggregation.asciidoc b/docs/reference/search/aggregations/bucket/iprange-aggregation.asciidoc new file mode 100644 index 00000000000..a76df06d78d --- /dev/null +++ b/docs/reference/search/aggregations/bucket/iprange-aggregation.asciidoc @@ -0,0 +1,94 @@ +[[search-aggregations-bucket-iprange-aggregation]] +=== IPv4 Range + +Just like the dedicated <> range aggregation, there is also a dedicated range aggregation for IPv4 typed fields: + +Example: + +[source,js] +-------------------------------------------------- +{ + "aggs" : { + "ip_ranges" : { + "ip_range" : { + "field" : "ip", + "ranges" : [ + { "to" : "10.0.0.5" }, + { "from" : "10.0.0.5" } + ] + } + } + } +} +-------------------------------------------------- + +Response: + +[source,js] +-------------------------------------------------- +{ + ... + + "aggregations": { + "ip_ranges": [ + { + "to": 167772165, + "to_as_string": "10.0.0.5", + "doc_count": 4 + }, + { + "from": 167772165, + "from_as_string": "10.0.0.5", + "doc_count": 6 + } + ] + } +} +-------------------------------------------------- + +IP ranges can also be defined as CIDR masks: + +[source,js] +-------------------------------------------------- +{ + "aggs" : { + "ip_ranges" : { + "ip_range" : { + "field" : "ip", + "ranges" : [ + { "mask" : "10.0.0.0/25" }, + { "mask" : "10.0.0.127/25" } + ] + } + } + } +} +-------------------------------------------------- + +Response: + +[source,js] +-------------------------------------------------- +{ + "aggregations": { + "ip_ranges": [ + { + "key": "10.0.0.0/25", + "from": 1.6777216E+8, + "from_as_string": "10.0.0.0", + "to": 167772287, + "to_as_string": "10.0.0.127", + "doc_count": 127 + }, + { + "key": "10.0.0.127/25", + "from": 1.6777216E+8, + "from_as_string": "10.0.0.0", + "to": 167772287, + "to_as_string": "10.0.0.127", + "doc_count": 127 + } + ] + } +} +-------------------------------------------------- \ No newline at end of file diff --git a/docs/reference/search/aggregations/bucket/missing-aggregation.asciidoc b/docs/reference/search/aggregations/bucket/missing-aggregation.asciidoc new file mode 100644 index 00000000000..c8579d312d1 --- /dev/null +++ b/docs/reference/search/aggregations/bucket/missing-aggregation.asciidoc @@ -0,0 +1,34 @@ +[[search-aggregations-bucket-missing-aggregation]] +=== Missing + +A field data based single bucket aggregation, that creates a bucket of all documents in the current document set context that are missing a field value (effectively, missing a field). This aggregator will often be used in conjunction with other field data bucket aggregators (such as ranges) to return information for all the documents that could not be placed in any of the other buckets due to missing field data values. + +Example: + +[source,js] +-------------------------------------------------- +{ + "aggs" : { + "products_without_a_price" : { + "missing" : { "field" : "price" } + } + } +} +-------------------------------------------------- + +In the above example, we calculate the average price of all the products that are currently in-stock. + +Response: + +[source,js] +-------------------------------------------------- +{ + ... + + "aggs" : { + "products_without_a_price" : { + "doc_count" : 10 + } + } +} +-------------------------------------------------- \ No newline at end of file diff --git a/docs/reference/search/aggregations/bucket/nested-aggregation.asciidoc b/docs/reference/search/aggregations/bucket/nested-aggregation.asciidoc new file mode 100644 index 00000000000..4528276c1b2 --- /dev/null +++ b/docs/reference/search/aggregations/bucket/nested-aggregation.asciidoc @@ -0,0 +1,63 @@ +[[search-aggregations-bucket-nested-aggregation]] +=== Nested + +A special single bucket aggregation that enables aggregating nested documents. + +For example, lets say we have a index of products, and each product holds the list of resellers - each having its own price for the product. The mapping could look like: + +[source,js] +-------------------------------------------------- +{ + ... + + "product" : { + "properties" : { + "resellers" : { <1> + "type" : "nested" + "properties" : { + "name" : { "type" : "string" }, + "price" : { "type" : "double" } + } + } + } + } +} +-------------------------------------------------- + +<1> The `resellers` is an array that holds nested documents under the `product` object. + +The following aggregations will return the minimum price products can be purchased in: + +[source,js] +-------------------------------------------------- +{ + "query" : { + "match" : { "name" : "led tv" } + } + "aggs" : { + "resellers" : { + "nested" : { + "path" : "resellers" + }, + "aggs" : { + "min_price" : { "min" : { "field" : "nested.value" } } + } + } + } +} +-------------------------------------------------- + +As you can see above, the nested aggregation requires the `path` of the nested documents within the top level documents. Then one can define any type of aggregation over these nested documents. + +Response: + +[source,js] +-------------------------------------------------- +{ + "aggregations": { + "resellers": { + "min_price": 350 + } + } +} +-------------------------------------------------- diff --git a/docs/reference/search/aggregations/bucket/range-aggregation.asciidoc b/docs/reference/search/aggregations/bucket/range-aggregation.asciidoc new file mode 100644 index 00000000000..fadb0f60289 --- /dev/null +++ b/docs/reference/search/aggregations/bucket/range-aggregation.asciidoc @@ -0,0 +1,268 @@ +[[search-aggregations-bucket-range-aggregation]] +=== Range + +A multi-bucket value source based aggregation that enables the user to define a set of ranges - each representing a bucket. During the aggregation process, the values extracted from each document will be checked against each bucket range and "bucket" the relevant/matching document. + +Example: + +[source,js] +-------------------------------------------------- +{ + "aggs" : { + "price_ranges" : { + "range" : { + "field" : "price", + "ranges" : [ + { "to" : 50 }, + { "from" : 50, "to" : 100 }, + { "from" : 100 } + ] + } + } + } +} +-------------------------------------------------- + +Response: + +[source,js] +-------------------------------------------------- +{ + ... + + "aggregations": { + "price_ranges": [ + { + "to": 50, + "doc_count": 2 + }, + { + "from": 50, + "to": 100, + "doc_count": 4 + }, + { + "from": 100, + "doc_count": 4 + } + ] + } +} +-------------------------------------------------- + +==== Keyed Response + +Setting the `key` flag to `true` will associate a unique string key with each bucket and return the ranges as a hash rather than an array: + +[source,js] +-------------------------------------------------- +{ + "aggs" : { + "price_ranges" : { + "range" : { + "field" : "price", + "keyed" : true, + "ranges" : [ + { "to" : 50 }, + { "from" : 50, "to" : 100 }, + { "from" : 100 } + ] + } + } + } +} +-------------------------------------------------- + +Response: + +[source,js] +-------------------------------------------------- +{ + ... + + "aggregations": { + "price_ranges": { + "*-50.0": { + "to": 50, + "doc_count": 2 + }, + "50.0-100.0": { + "from": 50, + "to": 100, + "doc_count": 4 + }, + "100.0-*": { + "from": 100, + "doc_count": 4 + } + } + } +} +-------------------------------------------------- + +It is also possible to customize the key for each range: + +[source,js] +-------------------------------------------------- +{ + "aggs" : { + "price_ranges" : { + "range" : { + "field" : "price", + "keyed" : true, + "ranges" : [ + { "key" : "cheap", "to" : 50 }, + { "key" : "average", "from" : 50, "to" : 100 }, + { "key" : "expensive", "from" : 100 } + ] + } + } + } +} +-------------------------------------------------- + +==== Script + +[source,js] +-------------------------------------------------- +{ + "aggs" : { + "price_ranges" : { + "range" : { + "script" : "doc['price'].value", + "ranges" : [ + { "to" : 50 }, + { "from" : 50, "to" : 100 }, + { "from" : 100 } + ] + } + } + } +} +-------------------------------------------------- + +==== Value Script + +Lets say the product prices are in USD but we would like to get the price ranges in EURO. We can use value script to convert the prices prior the aggregation (assuming conversion rate of 0.8) + +[source,js] +-------------------------------------------------- +{ + "aggs" : { + "price_ranges" : { + "range" : { + "field" : "price", + "script" : "_value * conversion_rate", + "params" : { + "conversion_rate" : 0.8 + }, + "ranges" : [ + { "to" : 35 }, + { "from" : 35, "to" : 70 }, + { "from" : 70 } + ] + } + } + } +} +-------------------------------------------------- + +==== Sub Aggregations + +The following example, not only "bucket" the documents to the different buckets but also computes statistics over the prices in each price range + +[source,js] +-------------------------------------------------- +{ + "aggs" : { + "price_ranges" : { + "range" : { + "field" : "price", + "ranges" : [ + { "to" : 50 }, + { "from" : 50, "to" : 100 }, + { "from" : 100 } + ] + }, + "aggs" : { + "price_stats" : { + "stats" : { "field" : "price" } + } + } + } + } +} +-------------------------------------------------- + +Response: + +[source,js] +-------------------------------------------------- +{ + "aggregations": { + "price_ranges": [ + { + "to": 50, + "doc_count": 2, + "price_stats": { + "count": 2, + "min": 20, + "max": 47, + "avg": 33.5, + "sum": 67 + } + }, + { + "from": 50, + "to": 100, + "doc_count": 4, + "price_stats": { + "count": 4, + "min": 60, + "max": 98, + "avg": 82.5, + "sum": 330 + } + }, + { + "from": 100, + "doc_count": 4, + "price_stats": { + "count": 4, + "min": 134, + "max": 367, + "avg": 216, + "sum": 864 + } + } + ] + } +} +-------------------------------------------------- + +If a sub aggregation is also based on the same value source as the range aggregation (like the `stats` aggregation in the example above) it is possible to leave out the value source definition for it. The following will return the same response as above: + +[source,js] +-------------------------------------------------- +{ + "aggs" : { + "price_ranges" : { + "range" : { + "field" : "price", + "ranges" : [ + { "to" : 50 }, + { "from" : 50, "to" : 100 }, + { "from" : 100 } + ] + }, + "aggs" : { + "price_stats" : { + "stats" : {} <1> + } + } + } + } +} +-------------------------------------------------- + +<1> We don't need to specify the `price` as we "inherit" it by default from the parent `range` aggregation \ No newline at end of file diff --git a/docs/reference/search/aggregations/bucket/terms-aggregation.asciidoc b/docs/reference/search/aggregations/bucket/terms-aggregation.asciidoc new file mode 100644 index 00000000000..59c8cd5887a --- /dev/null +++ b/docs/reference/search/aggregations/bucket/terms-aggregation.asciidoc @@ -0,0 +1,156 @@ +[[search-aggregations-bucket-terms-aggregation]] +=== Terms + +A multi-bucket value source based aggregation where buckets are dynamically built - one per unique value. + +Example: + +[source,js] +-------------------------------------------------- +{ + "aggs" : { + "genders" : { + "terms" : { "field" : "gender" } + } + } +} +-------------------------------------------------- + +Response: + +[source,js] +-------------------------------------------------- +{ + ... + + "aggregations" : { + "genders" : { + "buckets" : [ + { + "key" : "male", + "doc_count" : 10 + }, + { + "key" : "female", + "doc_count" : 10 + }, + ] + } + } +} +-------------------------------------------------- + +By default, the `terms` aggregation will return the buckets for the top ten terms ordered by the `doc_count`. One can change this default behaviour by setting the `size` parameter. + +==== Size + +The `size` parameter can be set to define how many term buckets should be returned out of the overall terms list. By default, the node coordinating the search process will request each shard to provide its own top `size` term buckets and once all shards respond, it will reduces the results to the final list that will then be returned to the client. This means that if the number of unique terms is greater than `size`, the returned list is slightly off and not accurate (it could be that the term counts are slightly off and it could even be that a term that should have been in the top size buckets was not returned). The higher the `size` is, the more accurate the response at the cost of aggregation performance. + +==== Order + +The order of the buckets can be customized by setting the `order` parameter. By default, the buckets are ordered by their `doc_count` descending. It is also possible to change this behaviour as follows: + +Ordering the buckets by their `doc_count` in an ascending manner: + +[source,js] +-------------------------------------------------- +{ + "aggs" : { + "genders" : { + "terms" : { + "field" : "gender", + "order" : { "_count" : "asc" } + } + } + } +} +-------------------------------------------------- + +Ordering the buckets alphabetically by their terms in an ascending manner: + +[source,js] +-------------------------------------------------- +{ + "aggs" : { + "genders" : { + "terms" : { + "field" : "gender", + "order" : { "_term" : "asc" } + } + } + } +} +-------------------------------------------------- + + +Ordering the buckets by single value metrics sub-aggregation (identified by the aggregation name): + +[source,js] +-------------------------------------------------- +{ + "aggs" : { + "genders" : { + "terms" : { + "field" : "gender", + "order" : { "avg_height" : "desc" } + }, + "aggs" : { + "avg_height" : { "avg" : { "field" : "height" } } + } + } + } +} +-------------------------------------------------- + +Ordering the buckets by multi value metrics sub-aggregation (identified by the aggregation name): + +[source,js] +-------------------------------------------------- +{ + "aggs" : { + "genders" : { + "terms" : { + "field" : "gender", + "order" : { "stats.avg" : "desc" } + }, + "aggs" : { + "height_stats" : { "stats" : { "field" : "height" } } + } + } + } +} +-------------------------------------------------- + +==== Script + +Generating the terms using a script: + +[source,js] +-------------------------------------------------- +{ + "aggs" : { + "genders" : { + "terms" : { + "script" : "doc['gender'].value" + } + } + } +} +-------------------------------------------------- + +==== Value Script + +[source,js] +-------------------------------------------------- +{ + "aggs" : { + "genders" : { + "terms" : { + "field" : "gender", + "script" : "doc['gender'].value" + } + } + } +} +-------------------------------------------------- + diff --git a/docs/reference/search/aggregations/metrics.asciidoc b/docs/reference/search/aggregations/metrics.asciidoc new file mode 100644 index 00000000000..2ea6f274129 --- /dev/null +++ b/docs/reference/search/aggregations/metrics.asciidoc @@ -0,0 +1,16 @@ +[[search-aggregations-metrics]] +=== Metrics Aggregations + +The aggregations in this family compute metrics based on values extracted in one way or another from the documents that are being aggregated. The values are typically extracted from the fields of the document (using the field data), but can also be generated using scripts. Some aggregations output a single metric (e.g. `avg`) and are called `single-value metrics aggregation`, others generate multiple metrics (e.g. `stats`) and are called `multi-value metrics aggregation`. The distinction between single-value and multi-value metrics aggregations plays a role when these aggregations serve as direct sub-aggregations of some bucket aggregations (some bucket aggregation enable you to sort the returned buckets based on the metrics in each bucket). + +include::metrics/min-aggregation.asciidoc[] + +include::metrics/max-aggregation.asciidoc[] + +include::metrics/sum-aggregation.asciidoc[] + +include::metrics/avg-aggregation.asciidoc[] + +include::metrics/stats-aggregation.asciidoc[] + +include::metrics/extendedstats-aggregation.asciidoc[] diff --git a/docs/reference/search/aggregations/metrics/avg-aggregation.asciidoc b/docs/reference/search/aggregations/metrics/avg-aggregation.asciidoc new file mode 100644 index 00000000000..2c2d7f67848 --- /dev/null +++ b/docs/reference/search/aggregations/metrics/avg-aggregation.asciidoc @@ -0,0 +1,73 @@ +[[search-aggregations-metrics-avg-aggregation]] +=== Avg + +A `single-value` metrics aggregation that computes the average of numeric values that are extracted from the aggregated documents. These values can be extracted either from specific numeric fields in the documents, or be generated by a provided script. + +Assuming the data consists of documents representing exams grades (between 0 and 100) of students + +[source,js] +-------------------------------------------------- +{ + "aggs" : { + "avg_grade" : { "avg" : { "field" : "grade" } } + } +} +-------------------------------------------------- + +The above aggregation computes the average grade over all documents. The aggregation type is `avg` and the `field` setting defines the numeric field of the documents the average will be computed on. The above will return the following: + + +[source,js] +-------------------------------------------------- +{ + ... + + "aggregations": { + "avg_grade": { + "value": 75 + } + } +} +-------------------------------------------------- + +The name of the aggregation (`avg_grade` above) also serves as the key by which the aggreagtion result can be retrieved from the returned response. + +==== Script + +Computing the average grade based on a script: + +[source,js] +-------------------------------------------------- +{ + ..., + + "aggs" : { + "avg_grade" : { "avg" : { "script" : "doc['grade'].value" } } + } +} +-------------------------------------------------- + +===== Value Script + +It turned out that the exam was way above the level of the students and a grade correction needs to be applied. We can use value script to get the new average + +[source,js] +-------------------------------------------------- +{ + "aggs" : { + ... + + "aggs" : { + "avg_corrected_grade" : { + "avg" : { + "field" : "grade", + "script" : "_value * correction", + "params" : { + "correction" : 1.2 + } + } + } + } + } +} +-------------------------------------------------- \ No newline at end of file diff --git a/docs/reference/search/aggregations/metrics/extendedstats-aggregation.asciidoc b/docs/reference/search/aggregations/metrics/extendedstats-aggregation.asciidoc new file mode 100644 index 00000000000..287094f0b0b --- /dev/null +++ b/docs/reference/search/aggregations/metrics/extendedstats-aggregation.asciidoc @@ -0,0 +1,82 @@ +[[search-aggregations-metrics-extendedstats-aggregation]] +=== Extended Stats + +A `multi-value` metrics aggregation that computes stats over numeric values extracted from the aggregated documents. These values can be extracted either from specific numeric fields in the documents, or be generated by a provided script. + +The `exteded_stats` aggregations is an extended version of the `<>` aggregation, where additional metrics are added such as `sum_of_squares`, `variance` and `std_deviation`. + +Assuming the data consists of documents representing exams grades (between 0 and 100) of students + +[source,js] +-------------------------------------------------- +{ + "aggs" : { + "grades_stats" : { "extended_stats" : { "field" : "grade" } } + } +} +-------------------------------------------------- + +The above aggregation computes the grades statistics over all documents. The aggregation type is `extended_stats` and the `field` setting defines the numeric field of the documents the stats will be computed on. The above will return the following: + + +[source,js] +-------------------------------------------------- +{ + ... + + "aggregations": { + "grades_stats": { + "count": 6, + "min": 72, + "max": 117.6, + "avg": 94.2, + "sum": 565.2, + "sum_of_squares": 54551.51999999999, + "variance": 218.2799999999976, + "std_deviation": 14.774302013969987 + } + } +} +-------------------------------------------------- + +The name of the aggregation (`grades_stats` above) also serves as the key by which the aggreagtion result can be retrieved from the returned response. + +==== Script + +Computing the grades stats based on a script: + +[source,js] +-------------------------------------------------- +{ + ..., + + "aggs" : { + "grades_stats" : { "extended_stats" : { "script" : "doc['grade'].value" } } + } +} +-------------------------------------------------- + +===== Value Script + +It turned out that the exam was way above the level of the students and a grade correction needs to be applied. We can use value script to get the new stats + +[source,js] +-------------------------------------------------- +{ + "aggs" : { + ... + + "aggs" : { + "grades_stats" : { + "extended_stats" : { + "field" : "grade", + "script" : "_value * correction", + "params" : { + "correction" : 1.2 + } + } + } + } + } +} +-------------------------------------------------- \ No newline at end of file diff --git a/docs/reference/search/aggregations/metrics/max-aggregation.asciidoc b/docs/reference/search/aggregations/metrics/max-aggregation.asciidoc new file mode 100644 index 00000000000..d10bac42bde --- /dev/null +++ b/docs/reference/search/aggregations/metrics/max-aggregation.asciidoc @@ -0,0 +1,68 @@ +[[search-aggregations-metrics-max-aggregation]] +=== Max + +A `single-value` metrics aggregation that keeps track and returns the maximum value among the numeric values extracted from the aggregated documents. These values can be extracted either from specific numeric fields in the documents, or be generated by a provided script. + +Computing the max price value across all documents + +[source,js] +-------------------------------------------------- +{ + "aggs" : { + "max_price" : { "max" : { "field" : "price" } } + } +} +-------------------------------------------------- + +Response: + +[source,js] +-------------------------------------------------- +{ + ... + + "aggregations": { + "max_price": { + "value": 35 + } + } +} +-------------------------------------------------- + +As can be seen, the name of the aggregation (`max_price` above) also serves as the key by which the aggreagtion result can be retrieved from the returned response. + +==== Script + +Computing the max price value across all document, this time using a script: + +[source,js] +-------------------------------------------------- +{ + "aggs" : { + "max_price" : { "max" : { "script" : "doc['price'].value" } } + } +} +-------------------------------------------------- + + +==== Value Script + +Let's say that the prices of the documents in our index are in USD, but we would like to compute the max in EURO (and for the sake of this example, lets say the conversion rate is 1.2). We can use a value script to apply the conversion rate to every value before it's aggregated: + +[source,js] +-------------------------------------------------- +{ + "aggs" : { + "max_price_in_euros" : { + "max" : { + "field" : "price", + "script" : "_value * conversion_rate", + "params" : { + "conversion_rate" : 1.2 + } + } + } + } +} +-------------------------------------------------- + diff --git a/docs/reference/search/aggregations/metrics/min-aggregation.asciidoc b/docs/reference/search/aggregations/metrics/min-aggregation.asciidoc new file mode 100644 index 00000000000..27e2b153f68 --- /dev/null +++ b/docs/reference/search/aggregations/metrics/min-aggregation.asciidoc @@ -0,0 +1,67 @@ +[[search-aggregations-metrics-min-aggregation]] +=== Min + +A `single-value` metrics aggregation that keeps track and returns the minimum value among numeric values extracted from the aggregated documents. These values can be extracted either from specific numeric fields in the documents, or be generated by a provided script. + +Computing the min price value across all documents + +[source,js] +-------------------------------------------------- +{ + "aggs" : { + "min_price" : { "min" : { "field" : "price" } } + } +} +-------------------------------------------------- + +Response: + +[source,js] +-------------------------------------------------- +{ + ... + + "aggregations": { + "min_price": { + "value": 10 + } + } +} +-------------------------------------------------- + +As can be seen, the name of the aggregation (`min_price` above) also serves as the key by which the aggreagtion result can be retrieved from the returned response. + +==== Script + +Computing the min price value across all document, this time using a script: + +[source,js] +-------------------------------------------------- +{ + "aggs" : { + "min_price" : { "min" : { "script" : "doc['price'].value" } } + } +} +-------------------------------------------------- + + +==== Value Script + +Let's say that the prices of the documents in our index are in USD, but we would like to compute the min in EURO (and for the sake of this example, lets say the conversion rate is 1.2). We can use a value script to apply the conversion rate to every value before it's aggregated: + +[source,js] +-------------------------------------------------- +{ + "aggs" : { + "min_price_in_euros" : { + "min" : { + "field" : "price", + "script" : "_value * conversion_rate", + "params" : { + "conversion_rate" : 1.2 + } + } + } + } +} +-------------------------------------------------- \ No newline at end of file diff --git a/docs/reference/search/aggregations/metrics/stats-aggregation.asciidoc b/docs/reference/search/aggregations/metrics/stats-aggregation.asciidoc new file mode 100644 index 00000000000..8a3b2f47ce3 --- /dev/null +++ b/docs/reference/search/aggregations/metrics/stats-aggregation.asciidoc @@ -0,0 +1,79 @@ +[[search-aggregations-metrics-stats-aggregation]] +=== Stats + +A `multi-value` metrics aggregation that computes stats over numeric values extracted from the aggregated documents. These values can be extracted either from specific numeric fields in the documents, or be generated by a provided script. + +The stats that are returned consist of: `min`, `max`, `sum`, `count` and `avg`. + +Assuming the data consists of documents representing exams grades (between 0 and 100) of students + +[source,js] +-------------------------------------------------- +{ + "aggs" : { + "grades_stats" : { "stats" : { "field" : "grade" } } + } +} +-------------------------------------------------- + +The above aggregation computes the grades statistics over all documents. The aggregation type is `stats` and the `field` setting defines the numeric field of the documents the stats will be computed on. The above will return the following: + + +[source,js] +-------------------------------------------------- +{ + ... + + "aggregations": { + "grades_stats": { + "count": 6, + "min": 60, + "max": 98, + "avg": 78.5, + "sum": 471 + } + } +} +-------------------------------------------------- + +The name of the aggregation (`grades_stats` above) also serves as the key by which the aggreagtion result can be retrieved from the returned response. + +==== Script + +Computing the grades stats based on a script: + +[source,js] +-------------------------------------------------- +{ + ..., + + "aggs" : { + "grades_stats" : { "stats" : { "script" : "doc['grade'].value" } } + } +} +-------------------------------------------------- + +===== Value Script + +It turned out that the exam was way above the level of the students and a grade correction needs to be applied. We can use value script to get the new stats + +[source,js] +-------------------------------------------------- +{ + "aggs" : { + ... + + "aggs" : { + "grades_stats" : { + "stats" : { + "field" : "grade", + "script" : "_value * correction", + "params" : { + "correction" : 1.2 + } + } + } + } + } +} +-------------------------------------------------- \ No newline at end of file diff --git a/docs/reference/search/aggregations/metrics/sum-aggregation.asciidoc b/docs/reference/search/aggregations/metrics/sum-aggregation.asciidoc new file mode 100644 index 00000000000..856d5816034 --- /dev/null +++ b/docs/reference/search/aggregations/metrics/sum-aggregation.asciidoc @@ -0,0 +1,77 @@ +[[search-aggregations-metrics-sum-aggregation]] +=== Sum + +A `single-value` metrics aggregation that sums up numeric values that are extracted from the aggregated documents. These values can be extracted either from specific numeric fields in the documents, or be generated by a provided script. + +Assuming the data consists of documents representing stock ticks, where each tick holds the change in the stock price from the previous tick. + +[source,js] +-------------------------------------------------- +{ + "query" : { + "filtered" : { + "query" : { "match_all" : {}}, + "filter" : { + "range" : { "timestamp" : { "from" : "now/1d+9.5h", "to" : "now/1d+16h" }} + } + } + }, + "aggs" : { + "intraday_return" : { "sum" : { "field" : "change" } } + } +} +-------------------------------------------------- + +The above aggregation sums up all changes in the today's trading stock ticks which accounts for the intraday return. The aggregation type is `sum` and the `field` setting defines the numeric field of the documents of which values will be summed up. The above will return the following: + + +[source,js] +-------------------------------------------------- +{ + ... + + "aggregations": { + "intraday_return": { + "value": 2.18 + } + } +} +-------------------------------------------------- + +The name of the aggregation (`intraday_return` above) also serves as the key by which the aggreagtion result can be retrieved from the returned response. + +==== Script + +Computing the intraday return based on a script: + +[source,js] +-------------------------------------------------- +{ + ..., + + "aggs" : { + "intraday_return" : { "sum" : { "script" : "doc['change'].value" } } + } +} +-------------------------------------------------- + +===== Value Script + +Computing the sum of squares over all stock tick changes: + +[source,js] +-------------------------------------------------- +{ + "aggs" : { + ... + + "aggs" : { + "daytime_return" : { + "sum" : { + "field" : "change", + "script" : "_value * _value" } + } + } + } +} +-------------------------------------------------- diff --git a/src/main/java/org/elasticsearch/action/search/SearchRequestBuilder.java b/src/main/java/org/elasticsearch/action/search/SearchRequestBuilder.java index a4d13d7fdcf..2157f42422b 100644 --- a/src/main/java/org/elasticsearch/action/search/SearchRequestBuilder.java +++ b/src/main/java/org/elasticsearch/action/search/SearchRequestBuilder.java @@ -32,6 +32,7 @@ import org.elasticsearch.common.xcontent.XContentBuilder; import org.elasticsearch.index.query.FilterBuilder; import org.elasticsearch.index.query.QueryBuilder; import org.elasticsearch.search.Scroll; +import org.elasticsearch.search.aggregations.AbstractAggregationBuilder; import org.elasticsearch.search.builder.SearchSourceBuilder; import org.elasticsearch.search.facet.FacetBuilder; import org.elasticsearch.search.highlight.HighlightBuilder; @@ -566,6 +567,54 @@ public class SearchRequestBuilder extends ActionRequestBuilder extends private int version; private long totalUncompressedLength; - private BigLongArray offsets; + private LongArray offsets; private boolean closed; @@ -58,7 +59,7 @@ public abstract class CompressedIndexInput extends in.seek(metaDataPosition); this.totalUncompressedLength = in.readVLong(); int size = in.readVInt(); - offsets = new BigLongArray(size); + offsets = BigArrays.newLongArray(size); for (int i = 0; i < size; i++) { offsets.set(i, in.readVLong()); } @@ -139,7 +140,7 @@ public abstract class CompressedIndexInput extends @Override public void seek(long pos) throws IOException { int idx = (int) (pos / uncompressedLength); - if (idx >= offsets.size) { + if (idx >= offsets.size()) { // set the next "readyBuffer" to EOF currentOffsetIdx = idx; position = 0; @@ -184,7 +185,7 @@ public abstract class CompressedIndexInput extends return false; } // we reached the end... - if (currentOffsetIdx + 1 >= offsets.size) { + if (currentOffsetIdx + 1 >= offsets.size()) { return false; } valid = uncompress(in, uncompressed); diff --git a/src/main/java/org/elasticsearch/common/geo/GeoPoint.java b/src/main/java/org/elasticsearch/common/geo/GeoPoint.java index 35a1fe14d96..017409bb1e3 100644 --- a/src/main/java/org/elasticsearch/common/geo/GeoPoint.java +++ b/src/main/java/org/elasticsearch/common/geo/GeoPoint.java @@ -135,6 +135,12 @@ public class GeoPoint { return "[" + lat + ", " + lon + "]"; } + public static GeoPoint parseFromLatLon(String latLon) { + GeoPoint point = new GeoPoint(); + point.resetFromString(latLon); + return point; + } + /** * Parse a {@link GeoPoint} with a {@link XContentParser}: * diff --git a/src/main/java/org/elasticsearch/common/joda/TimeZoneRounding.java b/src/main/java/org/elasticsearch/common/joda/TimeZoneRounding.java deleted file mode 100644 index 6170a1113bd..00000000000 --- a/src/main/java/org/elasticsearch/common/joda/TimeZoneRounding.java +++ /dev/null @@ -1,282 +0,0 @@ -/* - * Licensed to ElasticSearch and Shay Banon under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. ElasticSearch licenses this - * file to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ - -package org.elasticsearch.common.joda; - -import org.elasticsearch.common.unit.TimeValue; -import org.joda.time.DateTimeConstants; -import org.joda.time.DateTimeField; -import org.joda.time.DateTimeZone; - -/** - */ -public abstract class TimeZoneRounding { - - public abstract long calc(long utcMillis); - - public static Builder builder(DateTimeField field) { - return new Builder(field); - } - - public static Builder builder(TimeValue interval) { - return new Builder(interval); - } - - public static class Builder { - - private DateTimeField field; - private long interval = -1; - - private DateTimeZone preTz = DateTimeZone.UTC; - private DateTimeZone postTz = DateTimeZone.UTC; - - private float factor = 1.0f; - - private long preOffset; - private long postOffset; - - private boolean preZoneAdjustLargeInterval = false; - - public Builder(DateTimeField field) { - this.field = field; - this.interval = -1; - } - - public Builder(TimeValue interval) { - this.field = null; - this.interval = interval.millis(); - } - - public Builder preZone(DateTimeZone preTz) { - this.preTz = preTz; - return this; - } - - public Builder preZoneAdjustLargeInterval(boolean preZoneAdjustLargeInterval) { - this.preZoneAdjustLargeInterval = preZoneAdjustLargeInterval; - return this; - } - - public Builder postZone(DateTimeZone postTz) { - this.postTz = postTz; - return this; - } - - public Builder preOffset(long preOffset) { - this.preOffset = preOffset; - return this; - } - - public Builder postOffset(long postOffset) { - this.postOffset = postOffset; - return this; - } - - public Builder factor(float factor) { - this.factor = factor; - return this; - } - - public TimeZoneRounding build() { - TimeZoneRounding timeZoneRounding; - if (field != null) { - if (preTz.equals(DateTimeZone.UTC) && postTz.equals(DateTimeZone.UTC)) { - timeZoneRounding = new UTCTimeZoneRoundingFloor(field); - } else if (preZoneAdjustLargeInterval || field.getDurationField().getUnitMillis() < DateTimeConstants.MILLIS_PER_HOUR * 12) { - timeZoneRounding = new TimeTimeZoneRoundingFloor(field, preTz, postTz); - } else { - timeZoneRounding = new DayTimeZoneRoundingFloor(field, preTz, postTz); - } - } else { - if (preTz.equals(DateTimeZone.UTC) && postTz.equals(DateTimeZone.UTC)) { - timeZoneRounding = new UTCIntervalTimeZoneRounding(interval); - } else if (preZoneAdjustLargeInterval || interval < DateTimeConstants.MILLIS_PER_HOUR * 12) { - timeZoneRounding = new TimeIntervalTimeZoneRounding(interval, preTz, postTz); - } else { - timeZoneRounding = new DayIntervalTimeZoneRounding(interval, preTz, postTz); - } - } - if (preOffset != 0 || postOffset != 0) { - timeZoneRounding = new PrePostTimeZoneRounding(timeZoneRounding, preOffset, postOffset); - } - if (factor != 1.0f) { - timeZoneRounding = new FactorTimeZoneRounding(timeZoneRounding, factor); - } - return timeZoneRounding; - } - } - - static class TimeTimeZoneRoundingFloor extends TimeZoneRounding { - - private final DateTimeField field; - private final DateTimeZone preTz; - private final DateTimeZone postTz; - - TimeTimeZoneRoundingFloor(DateTimeField field, DateTimeZone preTz, DateTimeZone postTz) { - this.field = field; - this.preTz = preTz; - this.postTz = postTz; - } - - @Override - public long calc(long utcMillis) { - long time = utcMillis + preTz.getOffset(utcMillis); - time = field.roundFloor(time); - // now, time is still in local, move it to UTC (or the adjustLargeInterval flag is set) - time = time - preTz.getOffset(time); - // now apply post Tz - time = time + postTz.getOffset(time); - return time; - } - } - - static class UTCTimeZoneRoundingFloor extends TimeZoneRounding { - - private final DateTimeField field; - - UTCTimeZoneRoundingFloor(DateTimeField field) { - this.field = field; - } - - @Override - public long calc(long utcMillis) { - return field.roundFloor(utcMillis); - } - } - - static class DayTimeZoneRoundingFloor extends TimeZoneRounding { - private final DateTimeField field; - private final DateTimeZone preTz; - private final DateTimeZone postTz; - - DayTimeZoneRoundingFloor(DateTimeField field, DateTimeZone preTz, DateTimeZone postTz) { - this.field = field; - this.preTz = preTz; - this.postTz = postTz; - } - - @Override - public long calc(long utcMillis) { - long time = utcMillis + preTz.getOffset(utcMillis); - time = field.roundFloor(time); - // after rounding, since its day level (and above), its actually UTC! - // now apply post Tz - time = time + postTz.getOffset(time); - return time; - } - } - - static class UTCIntervalTimeZoneRounding extends TimeZoneRounding { - - private final long interval; - - UTCIntervalTimeZoneRounding(long interval) { - this.interval = interval; - } - - @Override - public long calc(long utcMillis) { - return ((utcMillis / interval) * interval); - } - } - - - static class TimeIntervalTimeZoneRounding extends TimeZoneRounding { - - private final long interval; - private final DateTimeZone preTz; - private final DateTimeZone postTz; - - TimeIntervalTimeZoneRounding(long interval, DateTimeZone preTz, DateTimeZone postTz) { - this.interval = interval; - this.preTz = preTz; - this.postTz = postTz; - } - - @Override - public long calc(long utcMillis) { - long time = utcMillis + preTz.getOffset(utcMillis); - time = ((time / interval) * interval); - // now, time is still in local, move it to UTC - time = time - preTz.getOffset(time); - // now apply post Tz - time = time + postTz.getOffset(time); - return time; - } - } - - static class DayIntervalTimeZoneRounding extends TimeZoneRounding { - - private final long interval; - private final DateTimeZone preTz; - private final DateTimeZone postTz; - - DayIntervalTimeZoneRounding(long interval, DateTimeZone preTz, DateTimeZone postTz) { - this.interval = interval; - this.preTz = preTz; - this.postTz = postTz; - } - - @Override - public long calc(long utcMillis) { - long time = utcMillis + preTz.getOffset(utcMillis); - time = ((time / interval) * interval); - // after rounding, since its day level (and above), its actually UTC! - // now apply post Tz - time = time + postTz.getOffset(time); - return time; - } - } - - static class FactorTimeZoneRounding extends TimeZoneRounding { - - private final TimeZoneRounding timeZoneRounding; - - private final float factor; - - FactorTimeZoneRounding(TimeZoneRounding timeZoneRounding, float factor) { - this.timeZoneRounding = timeZoneRounding; - this.factor = factor; - } - - @Override - public long calc(long utcMillis) { - return timeZoneRounding.calc((long) (factor * utcMillis)); - } - } - - static class PrePostTimeZoneRounding extends TimeZoneRounding { - - private final TimeZoneRounding timeZoneRounding; - - private final long preOffset; - private final long postOffset; - - PrePostTimeZoneRounding(TimeZoneRounding timeZoneRounding, long preOffset, long postOffset) { - this.timeZoneRounding = timeZoneRounding; - this.preOffset = preOffset; - this.postOffset = postOffset; - } - - @Override - public long calc(long utcMillis) { - return postOffset + timeZoneRounding.calc(utcMillis + preOffset); - } - } -} diff --git a/src/main/java/org/elasticsearch/common/lucene/ReaderContextAware.java b/src/main/java/org/elasticsearch/common/lucene/ReaderContextAware.java new file mode 100644 index 00000000000..c14d043a5bb --- /dev/null +++ b/src/main/java/org/elasticsearch/common/lucene/ReaderContextAware.java @@ -0,0 +1,30 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.common.lucene; + +import org.apache.lucene.index.AtomicReaderContext; + +/** + * + */ +public interface ReaderContextAware { + + public void setNextReader(AtomicReaderContext reader); +} diff --git a/src/main/java/org/elasticsearch/common/lucene/ScorerAware.java b/src/main/java/org/elasticsearch/common/lucene/ScorerAware.java new file mode 100644 index 00000000000..231ab89f034 --- /dev/null +++ b/src/main/java/org/elasticsearch/common/lucene/ScorerAware.java @@ -0,0 +1,31 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.common.lucene; + +import org.apache.lucene.search.Scorer; + +/** + * + */ +public interface ScorerAware { + + void setScorer(Scorer scorer); + +} diff --git a/src/main/java/org/elasticsearch/common/rounding/DateTimeUnit.java b/src/main/java/org/elasticsearch/common/rounding/DateTimeUnit.java new file mode 100644 index 00000000000..c171cdf0bce --- /dev/null +++ b/src/main/java/org/elasticsearch/common/rounding/DateTimeUnit.java @@ -0,0 +1,72 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.common.rounding; + +import org.elasticsearch.ElasticSearchException; +import org.elasticsearch.common.joda.Joda; +import org.joda.time.DateTimeField; +import org.joda.time.chrono.ISOChronology; + +import java.math.BigInteger; + +/** + * + */ +public enum DateTimeUnit { + + WEEK_OF_WEEKYEAR( (byte) 1, ISOChronology.getInstanceUTC().weekOfWeekyear()), + YEAR_OF_CENTURY( (byte) 2, ISOChronology.getInstanceUTC().yearOfCentury()), + QUARTER( (byte) 3, Joda.QuarterOfYear.getField(ISOChronology.getInstanceUTC())), + MONTH_OF_YEAR( (byte) 4, ISOChronology.getInstanceUTC().monthOfYear()), + DAY_OF_MONTH( (byte) 5, ISOChronology.getInstanceUTC().dayOfMonth()), + HOUR_OF_DAY( (byte) 6, ISOChronology.getInstanceUTC().hourOfDay()), + MINUTES_OF_HOUR( (byte) 7, ISOChronology.getInstanceUTC().minuteOfHour()), + SECOND_OF_MINUTE( (byte) 8, ISOChronology.getInstanceUTC().secondOfMinute()); + + private final byte id; + private final DateTimeField field; + + private DateTimeUnit(byte id, DateTimeField field) { + this.id = id; + this.field = field; + } + + public byte id() { + return id; + } + + public DateTimeField field() { + return field; + } + + public static DateTimeUnit resolve(byte id) { + switch (id) { + case 1: return WEEK_OF_WEEKYEAR; + case 2: return YEAR_OF_CENTURY; + case 3: return QUARTER; + case 4: return MONTH_OF_YEAR; + case 5: return DAY_OF_MONTH; + case 6: return HOUR_OF_DAY; + case 7: return MINUTES_OF_HOUR; + case 8: return SECOND_OF_MINUTE; + default: throw new ElasticSearchException("Unknown date time unit id [" + id + "]"); + } + } +} \ No newline at end of file diff --git a/src/main/java/org/elasticsearch/common/rounding/Rounding.java b/src/main/java/org/elasticsearch/common/rounding/Rounding.java new file mode 100644 index 00000000000..7ebc4e24682 --- /dev/null +++ b/src/main/java/org/elasticsearch/common/rounding/Rounding.java @@ -0,0 +1,143 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.common.rounding; + +import org.elasticsearch.ElasticSearchException; +import org.elasticsearch.common.io.stream.StreamInput; +import org.elasticsearch.common.io.stream.StreamOutput; +import org.elasticsearch.common.io.stream.Streamable; + +import java.io.IOException; + +/** + * A strategy for rounding long values. + */ +public interface Rounding extends Streamable { + + byte id(); + + /** + * Rounds the given value. + * + * @param value The value to round. + * @return The rounded value. + */ + long round(long value); + + /** + * Given the rounded value (which was potentially generated by {@link #round(long)}, returns the next rounding value. For example, with + * interval based rounding, if the interval is 3, {@code nextRoundValue(6) = 9 }. + * + * @param value The current rounding value + * @return The next rounding value; + */ + long nextRoundingValue(long value); + + /** + * Rounding strategy which is based on an interval + * + * {@code rounded = value - (value % interval) } + */ + public static class Interval implements Rounding { + + final static byte ID = 0; + + private long interval; + + public Interval() { // for serialization + } + + /** + * Creates a new interval rounding. + * + * @param interval The interval + */ + public Interval(long interval) { + this.interval = interval; + } + + @Override + public byte id() { + return ID; + } + + static long round(long value, long interval) { + long rem = value % interval; + // We need this condition because % may return a negative result on negative numbers + // According to Google caliper's IntModBenchmark, using a condition is faster than attempts to use tricks to avoid + // the condition. Moreover, in our case, the condition is very likely to be always true (dates, prices, distances), + // so easily predictable by the CPU + if (rem < 0) { + rem += interval; + } + return value - rem; + } + + @Override + public long round(long value) { + return round(value, interval); + } + + @Override + public long nextRoundingValue(long value) { + assert value == round(value); + return value + interval; + } + + @Override + public void readFrom(StreamInput in) throws IOException { + interval = in.readVLong(); + } + + @Override + public void writeTo(StreamOutput out) throws IOException { + out.writeVLong(interval); + } + } + + public static class Streams { + + public static void write(Rounding rounding, StreamOutput out) throws IOException { + out.writeByte(rounding.id()); + rounding.writeTo(out); + } + + public static Rounding read(StreamInput in) throws IOException { + Rounding rounding = null; + byte id = in.readByte(); + switch (id) { + case Interval.ID: rounding = new Interval(); break; + case TimeZoneRounding.TimeTimeZoneRoundingFloor.ID: rounding = new TimeZoneRounding.TimeTimeZoneRoundingFloor(); break; + case TimeZoneRounding.UTCTimeZoneRoundingFloor.ID: rounding = new TimeZoneRounding.UTCTimeZoneRoundingFloor(); break; + case TimeZoneRounding.DayTimeZoneRoundingFloor.ID: rounding = new TimeZoneRounding.DayTimeZoneRoundingFloor(); break; + case TimeZoneRounding.UTCIntervalTimeZoneRounding.ID: rounding = new TimeZoneRounding.UTCIntervalTimeZoneRounding(); break; + case TimeZoneRounding.TimeIntervalTimeZoneRounding.ID: rounding = new TimeZoneRounding.TimeIntervalTimeZoneRounding(); break; + case TimeZoneRounding.DayIntervalTimeZoneRounding.ID: rounding = new TimeZoneRounding.DayIntervalTimeZoneRounding(); break; + case TimeZoneRounding.FactorTimeZoneRounding.ID: rounding = new TimeZoneRounding.FactorTimeZoneRounding(); break; + case TimeZoneRounding.PrePostTimeZoneRounding.ID: rounding = new TimeZoneRounding.PrePostTimeZoneRounding(); break; + default: throw new ElasticSearchException("unknown rounding id [" + id + "]"); + } + rounding.readFrom(in); + return rounding; + } + + } + +} diff --git a/src/main/java/org/elasticsearch/common/rounding/TimeZoneRounding.java b/src/main/java/org/elasticsearch/common/rounding/TimeZoneRounding.java new file mode 100644 index 00000000000..35ed4981dec --- /dev/null +++ b/src/main/java/org/elasticsearch/common/rounding/TimeZoneRounding.java @@ -0,0 +1,509 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.common.rounding; + +import org.elasticsearch.common.io.stream.StreamInput; +import org.elasticsearch.common.io.stream.StreamOutput; +import org.elasticsearch.common.unit.TimeValue; +import org.joda.time.DateTimeConstants; +import org.joda.time.DateTimeZone; + +import java.io.IOException; + +/** + */ +public abstract class TimeZoneRounding implements Rounding { + + public abstract long round(long utcMillis); + + public static Builder builder(DateTimeUnit unit) { + return new Builder(unit); + } + + public static Builder builder(TimeValue interval) { + return new Builder(interval); + } + + public static class Builder { + + private DateTimeUnit unit; + private long interval = -1; + + private DateTimeZone preTz = DateTimeZone.UTC; + private DateTimeZone postTz = DateTimeZone.UTC; + + private float factor = 1.0f; + + private long preOffset; + private long postOffset; + + private boolean preZoneAdjustLargeInterval = false; + + public Builder(DateTimeUnit unit) { + this.unit = unit; + this.interval = -1; + } + + public Builder(TimeValue interval) { + this.unit = null; + this.interval = interval.millis(); + } + + public Builder preZone(DateTimeZone preTz) { + this.preTz = preTz; + return this; + } + + public Builder preZoneAdjustLargeInterval(boolean preZoneAdjustLargeInterval) { + this.preZoneAdjustLargeInterval = preZoneAdjustLargeInterval; + return this; + } + + public Builder postZone(DateTimeZone postTz) { + this.postTz = postTz; + return this; + } + + public Builder preOffset(long preOffset) { + this.preOffset = preOffset; + return this; + } + + public Builder postOffset(long postOffset) { + this.postOffset = postOffset; + return this; + } + + public Builder factor(float factor) { + this.factor = factor; + return this; + } + + public TimeZoneRounding build() { + TimeZoneRounding timeZoneRounding; + if (unit != null) { + if (preTz.equals(DateTimeZone.UTC) && postTz.equals(DateTimeZone.UTC)) { + timeZoneRounding = new UTCTimeZoneRoundingFloor(unit); + } else if (preZoneAdjustLargeInterval || unit.field().getDurationField().getUnitMillis() < DateTimeConstants.MILLIS_PER_HOUR * 12) { + timeZoneRounding = new TimeTimeZoneRoundingFloor(unit, preTz, postTz); + } else { + timeZoneRounding = new DayTimeZoneRoundingFloor(unit, preTz, postTz); + } + } else { + if (preTz.equals(DateTimeZone.UTC) && postTz.equals(DateTimeZone.UTC)) { + timeZoneRounding = new UTCIntervalTimeZoneRounding(interval); + } else if (preZoneAdjustLargeInterval || interval < DateTimeConstants.MILLIS_PER_HOUR * 12) { + timeZoneRounding = new TimeIntervalTimeZoneRounding(interval, preTz, postTz); + } else { + timeZoneRounding = new DayIntervalTimeZoneRounding(interval, preTz, postTz); + } + } + if (preOffset != 0 || postOffset != 0) { + timeZoneRounding = new PrePostTimeZoneRounding(timeZoneRounding, preOffset, postOffset); + } + if (factor != 1.0f) { + timeZoneRounding = new FactorTimeZoneRounding(timeZoneRounding, factor); + } + return timeZoneRounding; + } + } + + static class TimeTimeZoneRoundingFloor extends TimeZoneRounding { + + static final byte ID = 1; + + private DateTimeUnit unit; + private DateTimeZone preTz; + private DateTimeZone postTz; + + TimeTimeZoneRoundingFloor() { // for serialization + } + + TimeTimeZoneRoundingFloor(DateTimeUnit unit, DateTimeZone preTz, DateTimeZone postTz) { + this.unit = unit; + this.preTz = preTz; + this.postTz = postTz; + } + + @Override + public byte id() { + return ID; + } + + @Override + public long round(long utcMillis) { + long time = utcMillis + preTz.getOffset(utcMillis); + time = unit.field().roundFloor(time); + // now, time is still in local, move it to UTC (or the adjustLargeInterval flag is set) + time = time - preTz.getOffset(time); + // now apply post Tz + time = time + postTz.getOffset(time); + return time; + } + + @Override + public long nextRoundingValue(long value) { +// return value + unit.field().getDurationField().getUnitMillis(); + return unit.field().roundCeiling(value + 1); + } + + @Override + public void readFrom(StreamInput in) throws IOException { + unit = DateTimeUnit.resolve(in.readByte()); + preTz = DateTimeZone.forID(in.readSharedString()); + postTz = DateTimeZone.forID(in.readSharedString()); + } + + @Override + public void writeTo(StreamOutput out) throws IOException { + out.writeByte(unit.id()); + out.writeSharedString(preTz.getID()); + out.writeSharedString(postTz.getID()); + } + } + + static class UTCTimeZoneRoundingFloor extends TimeZoneRounding { + + final static byte ID = 2; + + private DateTimeUnit unit; + + UTCTimeZoneRoundingFloor() { // for serialization + } + + UTCTimeZoneRoundingFloor(DateTimeUnit unit) { + this.unit = unit; + } + + @Override + public byte id() { + return ID; + } + + @Override + public long round(long utcMillis) { + return unit.field().roundFloor(utcMillis); + } + + @Override + public long nextRoundingValue(long value) { + return unit.field().roundCeiling(value + 1); + } + + @Override + public void readFrom(StreamInput in) throws IOException { + unit = DateTimeUnit.resolve(in.readByte()); + } + + @Override + public void writeTo(StreamOutput out) throws IOException { + out.writeByte(unit.id()); + } + } + + static class DayTimeZoneRoundingFloor extends TimeZoneRounding { + + final static byte ID = 3; + + private DateTimeUnit unit; + private DateTimeZone preTz; + private DateTimeZone postTz; + + DayTimeZoneRoundingFloor() { // for serialization + } + + DayTimeZoneRoundingFloor(DateTimeUnit unit, DateTimeZone preTz, DateTimeZone postTz) { + this.unit = unit; + this.preTz = preTz; + this.postTz = postTz; + } + + @Override + public byte id() { + return ID; + } + + @Override + public long round(long utcMillis) { + long time = utcMillis + preTz.getOffset(utcMillis); + time = unit.field().roundFloor(time); + // after rounding, since its day level (and above), its actually UTC! + // now apply post Tz + time = time + postTz.getOffset(time); + return time; + } + + @Override + public long nextRoundingValue(long value) { + return unit.field().getDurationField().getUnitMillis() + value; + } + + @Override + public void readFrom(StreamInput in) throws IOException { + unit = DateTimeUnit.resolve(in.readByte()); + preTz = DateTimeZone.forID(in.readSharedString()); + postTz = DateTimeZone.forID(in.readSharedString()); + } + + @Override + public void writeTo(StreamOutput out) throws IOException { + out.writeByte(unit.id()); + out.writeSharedString(preTz.getID()); + out.writeSharedString(postTz.getID()); + } + } + + static class UTCIntervalTimeZoneRounding extends TimeZoneRounding { + + final static byte ID = 4; + + private long interval; + + UTCIntervalTimeZoneRounding() { // for serialization + } + + UTCIntervalTimeZoneRounding(long interval) { + this.interval = interval; + } + + @Override + public byte id() { + return ID; + } + + @Override + public long round(long utcMillis) { + return Rounding.Interval.round(utcMillis, interval); + } + + @Override + public long nextRoundingValue(long value) { + return value + interval; + } + + @Override + public void readFrom(StreamInput in) throws IOException { + interval = in.readVLong(); + } + + @Override + public void writeTo(StreamOutput out) throws IOException { + out.writeVLong(interval); + } + } + + + static class TimeIntervalTimeZoneRounding extends TimeZoneRounding { + + final static byte ID = 5; + + private long interval; + private DateTimeZone preTz; + private DateTimeZone postTz; + + TimeIntervalTimeZoneRounding() { // for serialization + } + + TimeIntervalTimeZoneRounding(long interval, DateTimeZone preTz, DateTimeZone postTz) { + this.interval = interval; + this.preTz = preTz; + this.postTz = postTz; + } + + @Override + public byte id() { + return ID; + } + + @Override + public long round(long utcMillis) { + long time = utcMillis + preTz.getOffset(utcMillis); + time = Rounding.Interval.round(time, interval); + // now, time is still in local, move it to UTC + time = time - preTz.getOffset(time); + // now apply post Tz + time = time + postTz.getOffset(time); + return time; + } + + @Override + public long nextRoundingValue(long value) { + return value + interval; + } + + @Override + public void readFrom(StreamInput in) throws IOException { + interval = in.readVLong(); + preTz = DateTimeZone.forID(in.readSharedString()); + postTz = DateTimeZone.forID(in.readSharedString()); + } + + @Override + public void writeTo(StreamOutput out) throws IOException { + out.writeVLong(interval); + out.writeSharedString(preTz.getID()); + out.writeSharedString(postTz.getID()); + } + } + + static class DayIntervalTimeZoneRounding extends TimeZoneRounding { + + final static byte ID = 6; + + private long interval; + private DateTimeZone preTz; + private DateTimeZone postTz; + + DayIntervalTimeZoneRounding() { // for serialization + } + + DayIntervalTimeZoneRounding(long interval, DateTimeZone preTz, DateTimeZone postTz) { + this.interval = interval; + this.preTz = preTz; + this.postTz = postTz; + } + + @Override + public byte id() { + return ID; + } + + @Override + public long round(long utcMillis) { + long time = utcMillis + preTz.getOffset(utcMillis); + time = Rounding.Interval.round(time, interval); + // after rounding, since its day level (and above), its actually UTC! + // now apply post Tz + time = time + postTz.getOffset(time); + return time; + } + + @Override + public long nextRoundingValue(long value) { + return value + interval; + } + + @Override + public void readFrom(StreamInput in) throws IOException { + interval = in.readVLong(); + preTz = DateTimeZone.forID(in.readSharedString()); + postTz = DateTimeZone.forID(in.readSharedString()); + } + + @Override + public void writeTo(StreamOutput out) throws IOException { + out.writeVLong(interval); + out.writeSharedString(preTz.getID()); + out.writeSharedString(postTz.getID()); + } + } + + static class FactorTimeZoneRounding extends TimeZoneRounding { + + final static byte ID = 7; + + private TimeZoneRounding timeZoneRounding; + + private float factor; + + FactorTimeZoneRounding() { // for serialization + } + + FactorTimeZoneRounding(TimeZoneRounding timeZoneRounding, float factor) { + this.timeZoneRounding = timeZoneRounding; + this.factor = factor; + } + + @Override + public byte id() { + return ID; + } + + @Override + public long round(long utcMillis) { + return timeZoneRounding.round((long) (factor * utcMillis)); + } + + @Override + public long nextRoundingValue(long value) { + return timeZoneRounding.nextRoundingValue(value); + } + + @Override + public void readFrom(StreamInput in) throws IOException { + timeZoneRounding = (TimeZoneRounding) Rounding.Streams.read(in); + factor = in.readFloat(); + } + + @Override + public void writeTo(StreamOutput out) throws IOException { + Rounding.Streams.write(timeZoneRounding, out); + out.writeFloat(factor); + } + } + + static class PrePostTimeZoneRounding extends TimeZoneRounding { + + final static byte ID = 8; + + private TimeZoneRounding timeZoneRounding; + + private long preOffset; + private long postOffset; + + PrePostTimeZoneRounding() { // for serialization + } + + PrePostTimeZoneRounding(TimeZoneRounding timeZoneRounding, long preOffset, long postOffset) { + this.timeZoneRounding = timeZoneRounding; + this.preOffset = preOffset; + this.postOffset = postOffset; + } + + @Override + public byte id() { + return ID; + } + + @Override + public long round(long utcMillis) { + return postOffset + timeZoneRounding.round(utcMillis + preOffset); + } + + @Override + public long nextRoundingValue(long value) { + return postOffset + timeZoneRounding.nextRoundingValue(value - postOffset); + } + + @Override + public void readFrom(StreamInput in) throws IOException { + timeZoneRounding = (TimeZoneRounding) Rounding.Streams.read(in); + preOffset = in.readVLong(); + postOffset = in.readVLong(); + } + + @Override + public void writeTo(StreamOutput out) throws IOException { + Rounding.Streams.write(timeZoneRounding, out); + out.writeVLong(preOffset); + out.writeVLong(postOffset); + } + } +} diff --git a/src/main/java/org/elasticsearch/common/unit/TimeValue.java b/src/main/java/org/elasticsearch/common/unit/TimeValue.java index ab6745e1372..4499ce2989b 100644 --- a/src/main/java/org/elasticsearch/common/unit/TimeValue.java +++ b/src/main/java/org/elasticsearch/common/unit/TimeValue.java @@ -234,7 +234,7 @@ public class TimeValue implements Serializable, Streamable { if (sValue.endsWith("S")) { millis = Long.parseLong(sValue.substring(0, sValue.length() - 1)); } else if (sValue.endsWith("ms")) { - millis = (long) (Double.parseDouble(sValue.substring(0, sValue.length() - "ms".length()))); + millis = (long) (Double.parseDouble(sValue.substring(0, sValue.length() - 2))); } else if (sValue.endsWith("s")) { millis = (long) (Double.parseDouble(sValue.substring(0, sValue.length() - 1)) * 1000); } else if (sValue.endsWith("m")) { diff --git a/src/main/java/org/elasticsearch/common/util/BigArray.java b/src/main/java/org/elasticsearch/common/util/BigArray.java new file mode 100644 index 00000000000..b7a0b9caaec --- /dev/null +++ b/src/main/java/org/elasticsearch/common/util/BigArray.java @@ -0,0 +1,28 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.common.util; + +/** Base abstraction of an array. */ +interface BigArray { + + /** Return the length of this array. */ + public long size(); + +} diff --git a/src/main/java/org/elasticsearch/common/util/BigArrays.java b/src/main/java/org/elasticsearch/common/util/BigArrays.java new file mode 100644 index 00000000000..9fef20bd15b --- /dev/null +++ b/src/main/java/org/elasticsearch/common/util/BigArrays.java @@ -0,0 +1,337 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.common.util; + +import com.google.common.base.Preconditions; +import org.apache.lucene.util.ArrayUtil; +import org.apache.lucene.util.RamUsageEstimator; + +import java.util.Arrays; + +/** Utility class to work with arrays. */ +public enum BigArrays { + ; + + /** Page size in bytes: 16KB */ + public static final int PAGE_SIZE_IN_BYTES = 1 << 14; + + /** Returns the next size to grow when working with parallel arrays that may have different page sizes or number of bytes per element. */ + public static long overSize(long minTargetSize) { + return overSize(minTargetSize, PAGE_SIZE_IN_BYTES / 8, 1); + } + + /** Return the next size to grow to that is >= minTargetSize. + * Inspired from {@link ArrayUtil#oversize(int, int)} and adapted to play nicely with paging. */ + public static long overSize(long minTargetSize, int pageSize, int bytesPerElement) { + Preconditions.checkArgument(minTargetSize >= 0, "minTargetSize must be >= 0"); + Preconditions.checkArgument(pageSize >= 0, "pageSize must be > 0"); + Preconditions.checkArgument(bytesPerElement > 0, "bytesPerElement must be > 0"); + + long newSize; + if (minTargetSize < pageSize) { + newSize = ArrayUtil.oversize((int)minTargetSize, bytesPerElement); + } else { + newSize = minTargetSize + (minTargetSize >>> 3); + } + + if (newSize > pageSize) { + // round to a multiple of pageSize + newSize = newSize - (newSize % pageSize) + pageSize; + assert newSize % pageSize == 0; + } + + return newSize; + } + + static boolean indexIsInt(long index) { + return index == (int) index; + } + + private static class IntArrayWrapper implements IntArray { + + private final int[] array; + + IntArrayWrapper(int[] array) { + this.array = array; + } + + @Override + public long size() { + return array.length; + } + + @Override + public int get(long index) { + assert indexIsInt(index); + return array[(int) index]; + } + + @Override + public int set(long index, int value) { + assert indexIsInt(index); + final int ret = array[(int) index]; + array[(int) index] = value; + return ret; + } + + @Override + public int increment(long index, int inc) { + assert indexIsInt(index); + return array[(int) index] += inc; + } + + } + + private static class LongArrayWrapper implements LongArray { + + private final long[] array; + + LongArrayWrapper(long[] array) { + this.array = array; + } + + @Override + public long size() { + return array.length; + } + + @Override + public long get(long index) { + assert indexIsInt(index); + return array[(int) index]; + } + + @Override + public long set(long index, long value) { + assert indexIsInt(index); + final long ret = array[(int) index]; + array[(int) index] = value; + return ret; + } + + @Override + public long increment(long index, long inc) { + assert indexIsInt(index); + return array[(int) index] += inc; + } + } + + private static class DoubleArrayWrapper implements DoubleArray { + + private final double[] array; + + DoubleArrayWrapper(double[] array) { + this.array = array; + } + + @Override + public long size() { + return array.length; + } + + @Override + public double get(long index) { + assert indexIsInt(index); + return array[(int) index]; + } + + @Override + public double set(long index, double value) { + assert indexIsInt(index); + double ret = array[(int) index]; + array[(int) index] = value; + return ret; + } + + @Override + public double increment(long index, double inc) { + assert indexIsInt(index); + return array[(int) index] += inc; + } + + @Override + public void fill(long fromIndex, long toIndex, double value) { + assert indexIsInt(fromIndex); + assert indexIsInt(toIndex); + Arrays.fill(array, (int) fromIndex, (int) toIndex, value); + } + + } + + private static class ObjectArrayWrapper implements ObjectArray { + + private final Object[] array; + + ObjectArrayWrapper(Object[] array) { + this.array = array; + } + + @Override + public long size() { + return array.length; + } + + @SuppressWarnings("unchecked") + @Override + public T get(long index) { + assert indexIsInt(index); + return (T) array[(int) index]; + } + + @Override + public T set(long index, T value) { + assert indexIsInt(index); + @SuppressWarnings("unchecked") + T ret = (T) array[(int) index]; + array[(int) index] = value; + return ret; + } + + } + + /** Allocate a new {@link IntArray} of the given capacity. */ + public static IntArray newIntArray(long size) { + if (size <= BigIntArray.PAGE_SIZE) { + return new IntArrayWrapper(new int[(int) size]); + } else { + return new BigIntArray(size); + } + } + + /** Resize the array to the exact provided size. */ + public static IntArray resize(IntArray array, long size) { + if (array instanceof BigIntArray) { + ((BigIntArray) array).resize(size); + return array; + } else { + final IntArray newArray = newIntArray(size); + for (long i = 0, end = Math.min(size, array.size()); i < end; ++i) { + newArray.set(i, array.get(i)); + } + return newArray; + } + } + + /** Grow an array to a size that is larger than minSize, preserving content, and potentially reusing part of the provided array. */ + public static IntArray grow(IntArray array, long minSize) { + if (minSize <= array.size()) { + return array; + } + final long newSize = overSize(minSize, BigIntArray.PAGE_SIZE, RamUsageEstimator.NUM_BYTES_INT); + return resize(array, newSize); + } + + /** Allocate a new {@link LongArray} of the given capacity. */ + public static LongArray newLongArray(long size) { + if (size <= BigLongArray.PAGE_SIZE) { + return new LongArrayWrapper(new long[(int) size]); + } else { + return new BigLongArray(size); + } + } + + /** Resize the array to the exact provided size. */ + public static LongArray resize(LongArray array, long size) { + if (array instanceof BigLongArray) { + ((BigLongArray) array).resize(size); + return array; + } else { + final LongArray newArray = newLongArray(size); + for (long i = 0, end = Math.min(size, array.size()); i < end; ++i) { + newArray.set(i, array.get(i)); + } + return newArray; + } + } + + /** Grow an array to a size that is larger than minSize, preserving content, and potentially reusing part of the provided array. */ + public static LongArray grow(LongArray array, long minSize) { + if (minSize <= array.size()) { + return array; + } + final long newSize = overSize(minSize, BigLongArray.PAGE_SIZE, RamUsageEstimator.NUM_BYTES_LONG); + return resize(array, newSize); + } + + /** Allocate a new {@link LongArray} of the given capacity. */ + public static DoubleArray newDoubleArray(long size) { + if (size <= BigLongArray.PAGE_SIZE) { + return new DoubleArrayWrapper(new double[(int) size]); + } else { + return new BigDoubleArray(size); + } + } + + /** Resize the array to the exact provided size. */ + public static DoubleArray resize(DoubleArray array, long size) { + if (array instanceof BigDoubleArray) { + ((BigDoubleArray) array).resize(size); + return array; + } else { + final DoubleArray newArray = newDoubleArray(size); + for (long i = 0, end = Math.min(size, array.size()); i < end; ++i) { + newArray.set(i, array.get(i)); + } + return newArray; + } + } + + /** Grow an array to a size that is larger than minSize, preserving content, and potentially reusing part of the provided array. */ + public static DoubleArray grow(DoubleArray array, long minSize) { + if (minSize <= array.size()) { + return array; + } + final long newSize = overSize(minSize, BigDoubleArray.PAGE_SIZE, RamUsageEstimator.NUM_BYTES_DOUBLE); + return resize(array, newSize); + } + + /** Allocate a new {@link LongArray} of the given capacity. */ + public static ObjectArray newObjectArray(long size) { + if (size <= BigLongArray.PAGE_SIZE) { + return new ObjectArrayWrapper(new Object[(int) size]); + } else { + return new BigObjectArray(size); + } + } + + /** Resize the array to the exact provided size. */ + public static ObjectArray resize(ObjectArray array, long size) { + if (array instanceof BigObjectArray) { + ((BigObjectArray) array).resize(size); + return array; + } else { + final ObjectArray newArray = newObjectArray(size); + for (long i = 0, end = Math.min(size, array.size()); i < end; ++i) { + newArray.set(i, array.get(i)); + } + return newArray; + } + } + + /** Grow an array to a size that is larger than minSize, preserving content, and potentially reusing part of the provided array. */ + public static ObjectArray grow(ObjectArray array, long minSize) { + if (minSize <= array.size()) { + return array; + } + final long newSize = overSize(minSize, BigObjectArray.PAGE_SIZE, RamUsageEstimator.NUM_BYTES_OBJECT_REF); + return resize(array, newSize); + } + +} diff --git a/src/main/java/org/elasticsearch/common/util/BigDoubleArray.java b/src/main/java/org/elasticsearch/common/util/BigDoubleArray.java new file mode 100644 index 00000000000..69efd234f59 --- /dev/null +++ b/src/main/java/org/elasticsearch/common/util/BigDoubleArray.java @@ -0,0 +1,112 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.common.util; + +import com.google.common.base.Preconditions; +import org.apache.lucene.util.ArrayUtil; +import org.apache.lucene.util.RamUsageEstimator; + +import java.util.Arrays; + +/** + * Double array abstraction able to support more than 2B values. This implementation slices data into fixed-sized blocks of + * configurable length. + */ +final class BigDoubleArray extends AbstractBigArray implements DoubleArray { + + /** + * Page size, 16KB of memory per page. + */ + public static final int PAGE_SIZE = BigArrays.PAGE_SIZE_IN_BYTES / RamUsageEstimator.NUM_BYTES_DOUBLE; + + + private double[][] pages; + + /** Constructor. */ + public BigDoubleArray(long size) { + super(PAGE_SIZE); + this.size = size; + pages = new double[numPages(size)][]; + for (int i = 0; i < pages.length; ++i) { + pages[i] = new double[pageSize()]; + } + } + + @Override + public double get(long index) { + final int pageIndex = pageIndex(index); + final int indexInPage = indexInPage(index); + return pages[pageIndex][indexInPage]; + } + + @Override + public double set(long index, double value) { + final int pageIndex = pageIndex(index); + final int indexInPage = indexInPage(index); + final double[] page = pages[pageIndex]; + final double ret = page[indexInPage]; + page[indexInPage] = value; + return ret; + } + + @Override + public double increment(long index, double inc) { + final int pageIndex = pageIndex(index); + final int indexInPage = indexInPage(index); + return pages[pageIndex][indexInPage] += inc; + } + + @Override + protected int numBytesPerElement() { + return RamUsageEstimator.NUM_BYTES_INT; + } + + /** Change the size of this array. Content between indexes 0 and min(size(), newSize) will be preserved. */ + public void resize(long newSize) { + final int numPages = numPages(newSize); + if (numPages > pages.length) { + pages = Arrays.copyOf(pages, ArrayUtil.oversize(numPages, RamUsageEstimator.NUM_BYTES_OBJECT_REF)); + } + for (int i = numPages - 1; i >= 0 && pages[i] == null; --i) { + pages[i] = new double[pageSize()]; + } + for (int i = numPages; i < pages.length && pages[i] != null; ++i) { + pages[i] = null; + } + this.size = newSize; + } + + @Override + public void fill(long fromIndex, long toIndex, double value) { + Preconditions.checkArgument(fromIndex <= toIndex); + final int fromPage = pageIndex(fromIndex); + final int toPage = pageIndex(toIndex - 1); + if (fromPage == toPage) { + Arrays.fill(pages[fromPage], indexInPage(fromIndex), indexInPage(toIndex - 1) + 1, value); + } else { + Arrays.fill(pages[fromPage], indexInPage(fromIndex), pages[fromPage].length, value); + for (int i = fromPage + 1; i < toPage; ++i) { + Arrays.fill(pages[i], value); + } + Arrays.fill(pages[toPage], 0, indexInPage(toIndex - 1) + 1, value); + } + } + +} diff --git a/src/main/java/org/elasticsearch/common/util/BigIntArray.java b/src/main/java/org/elasticsearch/common/util/BigIntArray.java index 4c1c7b4b12a..79c3159671a 100644 --- a/src/main/java/org/elasticsearch/common/util/BigIntArray.java +++ b/src/main/java/org/elasticsearch/common/util/BigIntArray.java @@ -19,6 +19,7 @@ package org.elasticsearch.common.util; +import org.apache.lucene.util.ArrayUtil; import org.apache.lucene.util.RamUsageEstimator; import java.util.Arrays; @@ -27,17 +28,19 @@ import java.util.Arrays; * Int array abstraction able to support more than 2B values. This implementation slices data into fixed-sized blocks of * configurable length. */ -public final class BigIntArray extends AbstractBigArray implements IntArray { +final class BigIntArray extends AbstractBigArray implements IntArray { /** - * Default page size, 16KB of memory per page. + * Page size, 16KB of memory per page. */ - public static final int DEFAULT_PAGE_SIZE = 1 << 12; + public static final int PAGE_SIZE = BigArrays.PAGE_SIZE_IN_BYTES / RamUsageEstimator.NUM_BYTES_INT; + private int[][] pages; - public BigIntArray(int pageSize, long size) { - super(pageSize); + /** Constructor. */ + public BigIntArray(long size) { + super(PAGE_SIZE); this.size = size; pages = new int[numPages(size)][]; for (int i = 0; i < pages.length; ++i) { @@ -45,22 +48,24 @@ public final class BigIntArray extends AbstractBigArray implements IntArray { } } - public BigIntArray(long size) { - this(DEFAULT_PAGE_SIZE, size); - } - + @Override public int get(long index) { final int pageIndex = pageIndex(index); final int indexInPage = indexInPage(index); return pages[pageIndex][indexInPage]; } - public void set(long index, int value) { + @Override + public int set(long index, int value) { final int pageIndex = pageIndex(index); final int indexInPage = indexInPage(index); - pages[pageIndex][indexInPage] = value; + final int[] page = pages[pageIndex]; + final int ret = page[indexInPage]; + page[indexInPage] = value; + return ret; } + @Override public int increment(long index, int inc) { final int pageIndex = pageIndex(index); final int indexInPage = indexInPage(index); @@ -72,10 +77,19 @@ public final class BigIntArray extends AbstractBigArray implements IntArray { return RamUsageEstimator.NUM_BYTES_INT; } - @Override - public void clear(int sentinal) { - for (int[] page : pages) { - Arrays.fill(page, sentinal); + /** Change the size of this array. Content between indexes 0 and min(size(), newSize) will be preserved. */ + public void resize(long newSize) { + final int numPages = numPages(newSize); + if (numPages > pages.length) { + pages = Arrays.copyOf(pages, ArrayUtil.oversize(numPages, RamUsageEstimator.NUM_BYTES_OBJECT_REF)); } + for (int i = numPages - 1; i >= 0 && pages[i] == null; --i) { + pages[i] = new int[pageSize()]; + } + for (int i = numPages; i < pages.length && pages[i] != null; ++i) { + pages[i] = null; + } + this.size = newSize; } + } diff --git a/src/main/java/org/elasticsearch/common/util/BigLongArray.java b/src/main/java/org/elasticsearch/common/util/BigLongArray.java index b02bf7d5e5f..3faf67e7a1a 100644 --- a/src/main/java/org/elasticsearch/common/util/BigLongArray.java +++ b/src/main/java/org/elasticsearch/common/util/BigLongArray.java @@ -19,59 +19,77 @@ package org.elasticsearch.common.util; -import java.util.Locale; +import org.apache.lucene.util.ArrayUtil; +import org.apache.lucene.util.RamUsageEstimator; + +import java.util.Arrays; /** - * A GC friendly long[]. - * Allocating large arrays (that are not short-lived) generate fragmentation - * in old-gen space. This breaks such large long array into fixed size pages - * to avoid that problem. + * Long array abstraction able to support more than 2B values. This implementation slices data into fixed-sized blocks of + * configurable length. */ -public class BigLongArray { +final class BigLongArray extends AbstractBigArray implements LongArray { - private static final int DEFAULT_PAGE_SIZE = 4096; + /** + * Page size, 16KB of memory per page. + */ + public static final int PAGE_SIZE = BigArrays.PAGE_SIZE_IN_BYTES / RamUsageEstimator.NUM_BYTES_LONG; + - private final long[][] pages; - public final int size; + private long[][] pages; - private final int pageSize; - private final int pageCount; - - public BigLongArray(int size) { - this(size, DEFAULT_PAGE_SIZE); - } - - public BigLongArray(int size, int pageSize) { + /** Constructor. */ + public BigLongArray(long size) { + super(PAGE_SIZE); this.size = size; - this.pageSize = pageSize; - - int lastPageSize = size % pageSize; - int fullPageCount = size / pageSize; - pageCount = fullPageCount + (lastPageSize == 0 ? 0 : 1); - pages = new long[pageCount][]; - - for (int i = 0; i < fullPageCount; ++i) - pages[i] = new long[pageSize]; - - if (lastPageSize != 0) - pages[pages.length - 1] = new long[lastPageSize]; + pages = new long[numPages(size)][]; + for (int i = 0; i < pages.length; ++i) { + pages[i] = new long[pageSize()]; + } } - public void set(int idx, long value) { - if (idx < 0 || idx > size) - throw new IndexOutOfBoundsException(String.format(Locale.ROOT, "%d is not whithin [0, %d)", idx, size)); - - int page = idx / pageSize; - int pageIdx = idx % pageSize; - pages[page][pageIdx] = value; + @Override + public long get(long index) { + final int pageIndex = pageIndex(index); + final int indexInPage = indexInPage(index); + return pages[pageIndex][indexInPage]; } - public long get(int idx) { - if (idx < 0 || idx > size) - throw new IndexOutOfBoundsException(String.format(Locale.ROOT, "%d is not whithin [0, %d)", idx, size)); - - int page = idx / pageSize; - int pageIdx = idx % pageSize; - return pages[page][pageIdx]; + @Override + public long set(long index, long value) { + final int pageIndex = pageIndex(index); + final int indexInPage = indexInPage(index); + final long[] page = pages[pageIndex]; + final long ret = page[indexInPage]; + page[indexInPage] = value; + return ret; } + + @Override + public long increment(long index, long inc) { + final int pageIndex = pageIndex(index); + final int indexInPage = indexInPage(index); + return pages[pageIndex][indexInPage] += inc; + } + + @Override + protected int numBytesPerElement() { + return RamUsageEstimator.NUM_BYTES_LONG; + } + + /** Change the size of this array. Content between indexes 0 and min(size(), newSize) will be preserved. */ + public void resize(long newSize) { + final int numPages = numPages(newSize); + if (numPages > pages.length) { + pages = Arrays.copyOf(pages, ArrayUtil.oversize(numPages, RamUsageEstimator.NUM_BYTES_OBJECT_REF)); + } + for (int i = numPages - 1; i >= 0 && pages[i] == null; --i) { + pages[i] = new long[pageSize()]; + } + for (int i = numPages; i < pages.length && pages[i] != null; ++i) { + pages[i] = null; + } + this.size = newSize; + } + } diff --git a/src/main/java/org/elasticsearch/common/util/BigObjectArray.java b/src/main/java/org/elasticsearch/common/util/BigObjectArray.java new file mode 100644 index 00000000000..b3bd15b26c3 --- /dev/null +++ b/src/main/java/org/elasticsearch/common/util/BigObjectArray.java @@ -0,0 +1,90 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.common.util; + +import org.apache.lucene.util.ArrayUtil; +import org.apache.lucene.util.RamUsageEstimator; + +import java.util.Arrays; + +/** + * Int array abstraction able to support more than 2B values. This implementation slices data into fixed-sized blocks of + * configurable length. + */ +final class BigObjectArray extends AbstractBigArray implements ObjectArray { + + /** + * Page size, 16KB of memory per page. + */ + public static final int PAGE_SIZE = BigArrays.PAGE_SIZE_IN_BYTES / RamUsageEstimator.NUM_BYTES_OBJECT_REF; + + + private Object[][] pages; + + /** Constructor. */ + public BigObjectArray(long size) { + super(PAGE_SIZE); + this.size = size; + pages = new Object[numPages(size)][]; + for (int i = 0; i < pages.length; ++i) { + pages[i] = new Object[pageSize()]; + } + } + + @SuppressWarnings("unchecked") + @Override + public T get(long index) { + final int pageIndex = pageIndex(index); + final int indexInPage = indexInPage(index); + return (T) pages[pageIndex][indexInPage]; + } + + @Override + public T set(long index, T value) { + final int pageIndex = pageIndex(index); + final int indexInPage = indexInPage(index); + final Object[] page = pages[pageIndex]; + @SuppressWarnings("unchecked") + final T ret = (T) page[indexInPage]; + page[indexInPage] = value; + return ret; + } + + @Override + protected int numBytesPerElement() { + return RamUsageEstimator.NUM_BYTES_INT; + } + + /** Change the size of this array. Content between indexes 0 and min(size(), newSize) will be preserved. */ + public void resize(long newSize) { + final int numPages = numPages(newSize); + if (numPages > pages.length) { + pages = Arrays.copyOf(pages, ArrayUtil.oversize(numPages, RamUsageEstimator.NUM_BYTES_OBJECT_REF)); + } + for (int i = numPages - 1; i >= 0 && pages[i] == null; --i) { + pages[i] = new Object[pageSize()]; + } + for (int i = numPages; i < pages.length && pages[i] != null; ++i) { + pages[i] = null; + } + this.size = newSize; + } + +} diff --git a/src/main/java/org/elasticsearch/common/util/DoubleArray.java b/src/main/java/org/elasticsearch/common/util/DoubleArray.java new file mode 100644 index 00000000000..312923b86c8 --- /dev/null +++ b/src/main/java/org/elasticsearch/common/util/DoubleArray.java @@ -0,0 +1,47 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.common.util; + +/** + * Abstraction of an array of double values. + */ +public interface DoubleArray extends BigArray { + + /** + * Get an element given its index. + */ + public abstract double get(long index); + + /** + * Set a value at the given index and return the previous value. + */ + public abstract double set(long index, double value); + + /** + * Increment value at the given index by inc and return the value. + */ + public abstract double increment(long index, double inc); + + /** + * Fill slots between fromIndex inclusive to toIndex exclusive with value. + */ + public abstract void fill(long fromIndex, long toIndex, double value); + +} diff --git a/src/main/java/org/elasticsearch/common/util/IntArray.java b/src/main/java/org/elasticsearch/common/util/IntArray.java index 20a3a3e4b0f..76a9e2681df 100644 --- a/src/main/java/org/elasticsearch/common/util/IntArray.java +++ b/src/main/java/org/elasticsearch/common/util/IntArray.java @@ -22,7 +22,7 @@ package org.elasticsearch.common.util; /** * Abstraction of an array of integer values. */ -public interface IntArray { +public interface IntArray extends BigArray { /** * Get an element given its index. @@ -30,14 +30,13 @@ public interface IntArray { public abstract int get(long index); /** - * Set a value at the given index. + * Set a value at the given index and return the previous value. */ - public abstract void set(long index, int value); + public abstract int set(long index, int value); /** * Increment value at the given index by inc and return the value. */ public abstract int increment(long index, int inc); - void clear(int sentinal); } diff --git a/src/main/java/org/elasticsearch/common/util/IntArrays.java b/src/main/java/org/elasticsearch/common/util/IntArrays.java deleted file mode 100644 index 6a7eaaeb039..00000000000 --- a/src/main/java/org/elasticsearch/common/util/IntArrays.java +++ /dev/null @@ -1,80 +0,0 @@ -/* - * Licensed to ElasticSearch and Shay Banon under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. ElasticSearch licenses this - * file to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ - -package org.elasticsearch.common.util; - -import java.util.Arrays; - -/** - * Utility methods to work with {@link IntArray}s. - */ -public class IntArrays { - - private IntArrays() { - } - - /** - * Return a {@link IntArray} view over the provided array. - */ - public static IntArray wrap(final int[] array) { - return new IntArray() { - - private void checkIndex(long index) { - if (index > Integer.MAX_VALUE) { - throw new IndexOutOfBoundsException(Long.toString(index)); - } - } - - @Override - public void set(long index, int value) { - checkIndex(index); - array[(int) index] = value; - } - - @Override - public int increment(long index, int inc) { - checkIndex(index); - return array[(int) index] += inc; - } - - @Override - public int get(long index) { - checkIndex(index); - return array[(int) index]; - } - - @Override - public void clear(int sentinal) { - Arrays.fill(array, sentinal); - } - }; - } - - /** - * Return a newly allocated {@link IntArray} of the given length or more. - */ - public static IntArray allocate(long length) { - if (length <= BigIntArray.DEFAULT_PAGE_SIZE) { - return wrap(new int[(int) length]); - } else { - return new BigIntArray(length); - } - } - -} diff --git a/src/main/java/org/elasticsearch/common/util/LongArray.java b/src/main/java/org/elasticsearch/common/util/LongArray.java new file mode 100644 index 00000000000..b00058ecf8c --- /dev/null +++ b/src/main/java/org/elasticsearch/common/util/LongArray.java @@ -0,0 +1,42 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.common.util; + +/** + * Abstraction of an array of long values. + */ +public interface LongArray extends BigArray { + + /** + * Get an element given its index. + */ + public abstract long get(long index); + + /** + * Set a value at the given index and return the previous value. + */ + public abstract long set(long index, long value); + + /** + * Increment value at the given index by inc and return the value. + */ + public abstract long increment(long index, long inc); + +} diff --git a/src/main/java/org/elasticsearch/common/util/ObjectArray.java b/src/main/java/org/elasticsearch/common/util/ObjectArray.java new file mode 100644 index 00000000000..f0521ead67b --- /dev/null +++ b/src/main/java/org/elasticsearch/common/util/ObjectArray.java @@ -0,0 +1,37 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.common.util; + +/** + * Abstraction of an array of object values. + */ +public interface ObjectArray extends BigArray { + + /** + * Get an element given its index. + */ + public abstract T get(long index); + + /** + * Set a value at the given index and return the previous value. + */ + public abstract T set(long index, T value); + +} diff --git a/src/main/java/org/elasticsearch/common/xcontent/XContentBuilder.java b/src/main/java/org/elasticsearch/common/xcontent/XContentBuilder.java index 2dcb1ecc5c4..52631f1e227 100644 --- a/src/main/java/org/elasticsearch/common/xcontent/XContentBuilder.java +++ b/src/main/java/org/elasticsearch/common/xcontent/XContentBuilder.java @@ -1148,6 +1148,8 @@ public final class XContentBuilder implements BytesStream { generator.writeString(XContentBuilder.defaultDatePrinter.print(((Date) value).getTime())); } else if (value instanceof Calendar) { generator.writeString(XContentBuilder.defaultDatePrinter.print((((Calendar) value)).getTimeInMillis())); + } else if (value instanceof ReadableInstant) { + generator.writeString(XContentBuilder.defaultDatePrinter.print((((ReadableInstant) value)).getMillis())); } else if (value instanceof BytesReference) { BytesReference bytes = (BytesReference) value; if (!bytes.hasArray()) { diff --git a/src/main/java/org/elasticsearch/index/fielddata/plain/FSTBytesAtomicFieldData.java b/src/main/java/org/elasticsearch/index/fielddata/plain/FSTBytesAtomicFieldData.java index ecbded35f78..3ff7d8767c6 100644 --- a/src/main/java/org/elasticsearch/index/fielddata/plain/FSTBytesAtomicFieldData.java +++ b/src/main/java/org/elasticsearch/index/fielddata/plain/FSTBytesAtomicFieldData.java @@ -24,7 +24,8 @@ import org.apache.lucene.util.IntsRef; import org.apache.lucene.util.fst.*; import org.apache.lucene.util.fst.FST.Arc; import org.apache.lucene.util.fst.FST.BytesReader; -import org.elasticsearch.common.util.BigIntArray; +import org.elasticsearch.common.util.BigArrays; +import org.elasticsearch.common.util.IntArray; import org.elasticsearch.index.fielddata.AtomicFieldData; import org.elasticsearch.index.fielddata.ScriptDocValues; import org.elasticsearch.index.fielddata.ordinals.EmptyOrdinals; @@ -44,7 +45,7 @@ public class FSTBytesAtomicFieldData implements AtomicFieldData.WithOrdinals fst; @@ -95,7 +96,7 @@ public class FSTBytesAtomicFieldData implements AtomicFieldData.WithOrdinals fstEnum = new BytesRefFSTEnum(fst); - BigIntArray hashes = new BigIntArray(ordinals.getMaxOrd()); + IntArray hashes = BigArrays.newIntArray(ordinals.getMaxOrd()); // we don't store an ord 0 in the FST since we could have an empty string in there and FST don't support // empty strings twice. ie. them merge fails for long output. hashes.set(0, new BytesRef().hashCode()); @@ -164,9 +165,9 @@ public class FSTBytesAtomicFieldData implements AtomicFieldData.WithOrdinals fst, Docs ordinals, BigIntArray hashes) { + HashedBytesValues(FST fst, Docs ordinals, IntArray hashes) { super(fst, ordinals); this.hashes = hashes; } diff --git a/src/main/java/org/elasticsearch/index/fielddata/plain/PagedBytesAtomicFieldData.java b/src/main/java/org/elasticsearch/index/fielddata/plain/PagedBytesAtomicFieldData.java index 6850f898467..bf4a60db41d 100644 --- a/src/main/java/org/elasticsearch/index/fielddata/plain/PagedBytesAtomicFieldData.java +++ b/src/main/java/org/elasticsearch/index/fielddata/plain/PagedBytesAtomicFieldData.java @@ -23,7 +23,8 @@ import org.apache.lucene.util.BytesRef; import org.apache.lucene.util.PagedBytes; import org.apache.lucene.util.PagedBytes.Reader; import org.apache.lucene.util.packed.MonotonicAppendingLongBuffer; -import org.elasticsearch.common.util.BigIntArray; +import org.elasticsearch.common.util.BigArrays; +import org.elasticsearch.common.util.IntArray; import org.elasticsearch.index.fielddata.AtomicFieldData; import org.elasticsearch.index.fielddata.ScriptDocValues; import org.elasticsearch.index.fielddata.ordinals.EmptyOrdinals; @@ -43,7 +44,7 @@ public class PagedBytesAtomicFieldData implements AtomicFieldData.WithOrdinals { this.dateMathParser = new DateMathParser(dateTimeFormatter, timeUnit); } + public FormatDateTimeFormatter dateTimeFormatter() { + return dateTimeFormatter; + } + + public DateMathParser dateMathParser() { + return dateMathParser; + } + @Override public FieldType defaultFieldType() { return Defaults.FIELD_TYPE; diff --git a/src/main/java/org/elasticsearch/percolator/PercolateContext.java b/src/main/java/org/elasticsearch/percolator/PercolateContext.java index b370c1f0747..4c29cfa7fe4 100644 --- a/src/main/java/org/elasticsearch/percolator/PercolateContext.java +++ b/src/main/java/org/elasticsearch/percolator/PercolateContext.java @@ -55,6 +55,7 @@ import org.elasticsearch.script.ScriptService; import org.elasticsearch.search.Scroll; import org.elasticsearch.search.SearchHitField; import org.elasticsearch.search.SearchShardTarget; +import org.elasticsearch.search.aggregations.SearchContextAggregations; import org.elasticsearch.search.dfs.DfsSearchResult; import org.elasticsearch.search.facet.SearchContextFacets; import org.elasticsearch.search.fetch.FetchSearchResult; @@ -283,6 +284,16 @@ public class PercolateContext extends SearchContext { return fieldDataService; } + @Override + public SearchContextAggregations aggregations() { + throw new UnsupportedOperationException(); + } + + @Override + public SearchContext aggregations(SearchContextAggregations aggregations) { + throw new UnsupportedOperationException(); + } + @Override public SearchContextFacets facets() { return facets; diff --git a/src/main/java/org/elasticsearch/script/SearchScript.java b/src/main/java/org/elasticsearch/script/SearchScript.java index 2c35a14857f..36138229221 100644 --- a/src/main/java/org/elasticsearch/script/SearchScript.java +++ b/src/main/java/org/elasticsearch/script/SearchScript.java @@ -19,10 +19,11 @@ package org.elasticsearch.script; -import org.apache.lucene.index.AtomicReader; -import org.apache.lucene.index.AtomicReaderContext; -import org.apache.lucene.index.IndexReader; -import org.apache.lucene.search.Scorer; +import org.elasticsearch.common.lucene.ReaderContextAware; +import org.elasticsearch.common.lucene.ScorerAware; +import org.elasticsearch.search.SearchService; +import org.elasticsearch.search.internal.SearchContext; +import org.elasticsearch.search.lookup.SearchLookup; import java.util.Map; @@ -31,11 +32,7 @@ import java.util.Map; * * @see ExplainableSearchScript for script which can explain a score */ -public interface SearchScript extends ExecutableScript { - - void setScorer(Scorer scorer); - - void setNextReader(AtomicReaderContext context); +public interface SearchScript extends ExecutableScript, ReaderContextAware, ScorerAware { void setNextDocId(int doc); @@ -48,4 +45,34 @@ public interface SearchScript extends ExecutableScript { long runAsLong(); double runAsDouble(); + + public static class Builder { + + private String script; + private String lang; + private Map params; + + public Builder script(String script) { + this.script = script; + return this; + } + + public Builder lang(String lang) { + this.lang = lang; + return this; + } + + public Builder params(Map params) { + this.params = params; + return this; + } + + public SearchScript build(SearchContext context) { + return build(context.scriptService(), context.lookup()); + } + + public SearchScript build(ScriptService service, SearchLookup lookup) { + return service.search(lookup, lang, script, params); + } + } } \ No newline at end of file diff --git a/src/main/java/org/elasticsearch/script/mvel/MvelScriptEngineService.java b/src/main/java/org/elasticsearch/script/mvel/MvelScriptEngineService.java index b92c7483fab..65656c9739a 100644 --- a/src/main/java/org/elasticsearch/script/mvel/MvelScriptEngineService.java +++ b/src/main/java/org/elasticsearch/script/mvel/MvelScriptEngineService.java @@ -54,7 +54,7 @@ public class MvelScriptEngineService extends AbstractComponent implements Script parserConfiguration = new ParserConfiguration(); parserConfiguration.addPackageImport("java.util"); - parserConfiguration.addPackageImport("org.joda"); + parserConfiguration.addPackageImport("org.joda.time"); parserConfiguration.addImport("time", MVEL.getStaticMethod(System.class, "currentTimeMillis", new Class[0])); // unboxed version of Math, better performance since conversion from boxed to unboxed my mvel is not needed for (Method m : UnboxedMathUtils.class.getMethods()) { diff --git a/src/main/java/org/elasticsearch/search/SearchModule.java b/src/main/java/org/elasticsearch/search/SearchModule.java index 730bf777c11..37c8ccec333 100644 --- a/src/main/java/org/elasticsearch/search/SearchModule.java +++ b/src/main/java/org/elasticsearch/search/SearchModule.java @@ -25,6 +25,7 @@ import org.elasticsearch.common.inject.Module; import org.elasticsearch.common.inject.SpawnModules; import org.elasticsearch.index.query.functionscore.FunctionScoreModule; import org.elasticsearch.search.action.SearchServiceTransportAction; +import org.elasticsearch.search.aggregations.AggregationModule; import org.elasticsearch.search.controller.SearchPhaseController; import org.elasticsearch.search.dfs.DfsPhase; import org.elasticsearch.search.facet.FacetModule; @@ -47,7 +48,7 @@ public class SearchModule extends AbstractModule implements SpawnModules { @Override public Iterable spawnModules() { - return ImmutableList.of(new TransportSearchModule(), new FacetModule(), new HighlightModule(), new SuggestModule(), new FunctionScoreModule()); + return ImmutableList.of(new TransportSearchModule(), new FacetModule(), new HighlightModule(), new SuggestModule(), new FunctionScoreModule(), new AggregationModule()); } @Override diff --git a/src/main/java/org/elasticsearch/search/TransportSearchModule.java b/src/main/java/org/elasticsearch/search/TransportSearchModule.java index 269b8e8b219..7fbc3d79cc2 100644 --- a/src/main/java/org/elasticsearch/search/TransportSearchModule.java +++ b/src/main/java/org/elasticsearch/search/TransportSearchModule.java @@ -23,6 +23,7 @@ import com.google.common.collect.ImmutableList; import org.elasticsearch.common.inject.AbstractModule; import org.elasticsearch.common.inject.Module; import org.elasticsearch.common.inject.SpawnModules; +import org.elasticsearch.search.aggregations.TransportAggregationModule; import org.elasticsearch.search.facet.TransportFacetModule; /** @@ -32,7 +33,7 @@ public class TransportSearchModule extends AbstractModule implements SpawnModule @Override public Iterable spawnModules() { - return ImmutableList.of(new TransportFacetModule()); + return ImmutableList.of(new TransportFacetModule(), new TransportAggregationModule()); } @Override diff --git a/src/main/java/org/elasticsearch/search/aggregations/AbstractAggregationBuilder.java b/src/main/java/org/elasticsearch/search/aggregations/AbstractAggregationBuilder.java new file mode 100644 index 00000000000..6b69496b656 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/AbstractAggregationBuilder.java @@ -0,0 +1,18 @@ +package org.elasticsearch.search.aggregations; + +import org.elasticsearch.common.xcontent.ToXContent; + +/** + * + */ +public abstract class AbstractAggregationBuilder implements ToXContent { + + protected final String name; + protected final String type; + + protected AbstractAggregationBuilder(String name, String type) { + this.name = name; + this.type = type; + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/Aggregation.java b/src/main/java/org/elasticsearch/search/aggregations/Aggregation.java new file mode 100644 index 00000000000..36a25e5d281 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/Aggregation.java @@ -0,0 +1,32 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations; + +/** + * An aggregation + */ +public interface Aggregation { + + /** + * @return The name of this aggregation. + */ + String getName(); + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/AggregationBinaryParseElement.java b/src/main/java/org/elasticsearch/search/aggregations/AggregationBinaryParseElement.java new file mode 100644 index 00000000000..7b4f6676e53 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/AggregationBinaryParseElement.java @@ -0,0 +1,48 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations; + +import org.elasticsearch.common.inject.Inject; +import org.elasticsearch.common.xcontent.XContentFactory; +import org.elasticsearch.common.xcontent.XContentParser; +import org.elasticsearch.search.internal.SearchContext; + +/** + * + */ +public class AggregationBinaryParseElement extends AggregationParseElement { + + @Inject + public AggregationBinaryParseElement(AggregatorParsers aggregatorParsers) { + super(aggregatorParsers); + } + + @Override + public void parse(XContentParser parser, SearchContext context) throws Exception { + byte[] facetSource = parser.binaryValue(); + XContentParser aSourceParser = XContentFactory.xContent(facetSource).createParser(facetSource); + try { + aSourceParser.nextToken(); // move past the first START_OBJECT + super.parse(aSourceParser, context); + } finally { + aSourceParser.close(); + } + } +} \ No newline at end of file diff --git a/src/main/java/org/elasticsearch/search/aggregations/AggregationBuilder.java b/src/main/java/org/elasticsearch/search/aggregations/AggregationBuilder.java new file mode 100644 index 00000000000..772638dc5a7 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/AggregationBuilder.java @@ -0,0 +1,113 @@ +package org.elasticsearch.search.aggregations; + +import com.google.common.collect.Lists; +import org.elasticsearch.ElasticSearchGenerationException; +import org.elasticsearch.client.Requests; +import org.elasticsearch.common.bytes.BytesArray; +import org.elasticsearch.common.bytes.BytesReference; +import org.elasticsearch.common.xcontent.XContentBuilder; +import org.elasticsearch.common.xcontent.XContentFactory; + +import java.io.IOException; +import java.util.List; +import java.util.Map; + +/** + * A base class for all bucket aggregation builders. + */ +public abstract class AggregationBuilder> extends AbstractAggregationBuilder { + + private List aggregations; + private BytesReference aggregationsBinary; + + protected AggregationBuilder(String name, String type) { + super(name, type); + } + + /** + * Add a sub get to this bucket get. + */ + @SuppressWarnings("unchecked") + public B subAggregation(AbstractAggregationBuilder aggregation) { + if (aggregations == null) { + aggregations = Lists.newArrayList(); + } + aggregations.add(aggregation); + return (B) this; + } + + /** + * Sets a raw (xcontent / json) sub addAggregation. + */ + public B subAggregation(byte[] aggregationsBinary) { + return subAggregation(aggregationsBinary, 0, aggregationsBinary.length); + } + + /** + * Sets a raw (xcontent / json) sub addAggregation. + */ + public B subAggregation(byte[] aggregationsBinary, int aggregationsBinaryOffset, int aggregationsBinaryLength) { + return subAggregation(new BytesArray(aggregationsBinary, aggregationsBinaryOffset, aggregationsBinaryLength)); + } + + /** + * Sets a raw (xcontent / json) sub addAggregation. + */ + @SuppressWarnings("unchecked") + public B subAggregation(BytesReference aggregationsBinary) { + this.aggregationsBinary = aggregationsBinary; + return (B) this; + } + + /** + * Sets a raw (xcontent / json) sub addAggregation. + */ + public B subAggregation(XContentBuilder facets) { + return subAggregation(facets.bytes()); + } + + /** + * Sets a raw (xcontent / json) sub addAggregation. + */ + public B subAggregation(Map facets) { + try { + XContentBuilder builder = XContentFactory.contentBuilder(Requests.CONTENT_TYPE); + builder.map(facets); + return subAggregation(builder); + } catch (IOException e) { + throw new ElasticSearchGenerationException("Failed to generate [" + facets + "]", e); + } + } + + @Override + public final XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException { + builder.startObject(name); + + builder.field(type); + internalXContent(builder, params); + + if (aggregations != null || aggregationsBinary != null) { + builder.startObject("aggregations"); + + if (aggregations != null) { + for (AbstractAggregationBuilder subAgg : aggregations) { + subAgg.toXContent(builder, params); + } + } + + if (aggregationsBinary != null) { + if (XContentFactory.xContentType(aggregationsBinary) == builder.contentType()) { + builder.rawField("aggregations", aggregationsBinary); + } else { + builder.field("aggregations_binary", aggregationsBinary); + } + } + + builder.endObject(); + } + + return builder.endObject(); + } + + protected abstract XContentBuilder internalXContent(XContentBuilder builder, Params params) throws IOException; +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/AggregationBuilders.java b/src/main/java/org/elasticsearch/search/aggregations/AggregationBuilders.java new file mode 100644 index 00000000000..0972b55c5ed --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/AggregationBuilders.java @@ -0,0 +1,101 @@ +package org.elasticsearch.search.aggregations; + +import org.elasticsearch.search.aggregations.bucket.histogram.DateHistogramBuilder; +import org.elasticsearch.search.aggregations.bucket.histogram.HistogramBuilder; +import org.elasticsearch.search.aggregations.bucket.range.RangeBuilder; +import org.elasticsearch.search.aggregations.bucket.range.date.DateRangeBuilder; +import org.elasticsearch.search.aggregations.bucket.range.geodistance.GeoDistanceBuilder; +import org.elasticsearch.search.aggregations.bucket.range.ipv4.IPv4RangeBuilder; +import org.elasticsearch.search.aggregations.bucket.terms.TermsBuilder; +import org.elasticsearch.search.aggregations.bucket.filter.FilterAggregationBuilder; +import org.elasticsearch.search.aggregations.bucket.global.GlobalBuilder; +import org.elasticsearch.search.aggregations.bucket.missing.MissingBuilder; +import org.elasticsearch.search.aggregations.bucket.nested.NestedBuilder; +import org.elasticsearch.search.aggregations.metrics.valuecount.ValueCountBuilder; +import org.elasticsearch.search.aggregations.metrics.avg.AvgBuilder; +import org.elasticsearch.search.aggregations.metrics.max.MaxBuilder; +import org.elasticsearch.search.aggregations.metrics.min.MinBuilder; +import org.elasticsearch.search.aggregations.metrics.stats.StatsBuilder; +import org.elasticsearch.search.aggregations.metrics.stats.extended.ExtendedStatsBuilder; +import org.elasticsearch.search.aggregations.metrics.sum.SumBuilder; + +/** + * + */ +public class AggregationBuilders { + + protected AggregationBuilders() { + } + + public static ValueCountBuilder count(String name) { + return new ValueCountBuilder(name); + } + + public static AvgBuilder avg(String name) { + return new AvgBuilder(name); + } + + public static MaxBuilder max(String name) { + return new MaxBuilder(name); + } + + public static MinBuilder min(String name) { + return new MinBuilder(name); + } + + public static SumBuilder sum(String name) { + return new SumBuilder(name); + } + + public static StatsBuilder stats(String name) { + return new StatsBuilder(name); + } + + public static ExtendedStatsBuilder extendedStats(String name) { + return new ExtendedStatsBuilder(name); + } + + public static FilterAggregationBuilder filter(String name) { + return new FilterAggregationBuilder(name); + } + + public static GlobalBuilder global(String name) { + return new GlobalBuilder(name); + } + + public static MissingBuilder missing(String name) { + return new MissingBuilder(name); + } + + public static NestedBuilder nested(String name) { + return new NestedBuilder(name); + } + + public static GeoDistanceBuilder geoDistance(String name) { + return new GeoDistanceBuilder(name); + } + + public static HistogramBuilder histogram(String name) { + return new HistogramBuilder(name); + } + + public static DateHistogramBuilder dateHistogram(String name) { + return new DateHistogramBuilder(name); + } + + public static RangeBuilder range(String name) { + return new RangeBuilder(name); + } + + public static DateRangeBuilder dateRange(String name) { + return new DateRangeBuilder(name); + } + + public static IPv4RangeBuilder ipRange(String name) { + return new IPv4RangeBuilder(name); + } + + public static TermsBuilder terms(String name) { + return new TermsBuilder(name); + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/AggregationExecutionException.java b/src/main/java/org/elasticsearch/search/aggregations/AggregationExecutionException.java new file mode 100644 index 00000000000..7221f75f754 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/AggregationExecutionException.java @@ -0,0 +1,36 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations; + +import org.elasticsearch.ElasticSearchException; + +/** + * Thrown when failing to execute an aggregation + */ +public class AggregationExecutionException extends ElasticSearchException { + + public AggregationExecutionException(String msg) { + super(msg); + } + + public AggregationExecutionException(String msg, Throwable cause) { + super(msg, cause); + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/AggregationInitializationException.java b/src/main/java/org/elasticsearch/search/aggregations/AggregationInitializationException.java new file mode 100644 index 00000000000..85bde91dee4 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/AggregationInitializationException.java @@ -0,0 +1,36 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations; + +import org.elasticsearch.ElasticSearchException; + +/** + * Thrown when failing to execute an aggregation + */ +public class AggregationInitializationException extends ElasticSearchException { + + public AggregationInitializationException(String msg) { + super(msg); + } + + public AggregationInitializationException(String msg, Throwable cause) { + super(msg, cause); + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/AggregationModule.java b/src/main/java/org/elasticsearch/search/aggregations/AggregationModule.java new file mode 100644 index 00000000000..e239faf7468 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/AggregationModule.java @@ -0,0 +1,95 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations; + +import com.google.common.collect.Lists; +import org.elasticsearch.common.inject.AbstractModule; +import org.elasticsearch.common.inject.multibindings.Multibinder; +import org.elasticsearch.search.aggregations.bucket.histogram.DateHistogramParser; +import org.elasticsearch.search.aggregations.bucket.histogram.HistogramParser; +import org.elasticsearch.search.aggregations.bucket.range.RangeParser; +import org.elasticsearch.search.aggregations.bucket.range.date.DateRangeParser; +import org.elasticsearch.search.aggregations.bucket.range.geodistance.GeoDistanceParser; +import org.elasticsearch.search.aggregations.bucket.range.ipv4.IpRangeParser; +import org.elasticsearch.search.aggregations.bucket.terms.TermsParser; +import org.elasticsearch.search.aggregations.bucket.filter.FilterParser; +import org.elasticsearch.search.aggregations.bucket.global.GlobalParser; +import org.elasticsearch.search.aggregations.bucket.missing.MissingParser; +import org.elasticsearch.search.aggregations.bucket.nested.NestedParser; +import org.elasticsearch.search.aggregations.metrics.valuecount.ValueCountParser; +import org.elasticsearch.search.aggregations.metrics.avg.AvgParser; +import org.elasticsearch.search.aggregations.metrics.max.MaxParser; +import org.elasticsearch.search.aggregations.metrics.min.MinParser; +import org.elasticsearch.search.aggregations.metrics.stats.StatsParser; +import org.elasticsearch.search.aggregations.metrics.stats.extended.ExtendedStatsParser; +import org.elasticsearch.search.aggregations.metrics.sum.SumParser; + +import java.util.List; + +/** + * The main module for the get (binding all get components together) + */ +public class AggregationModule extends AbstractModule { + + private List> parsers = Lists.newArrayList(); + + public AggregationModule() { + parsers.add(AvgParser.class); + parsers.add(SumParser.class); + parsers.add(MinParser.class); + parsers.add(MaxParser.class); + parsers.add(StatsParser.class); + parsers.add(ExtendedStatsParser.class); + parsers.add(ValueCountParser.class); + + parsers.add(GlobalParser.class); + parsers.add(MissingParser.class); + parsers.add(FilterParser.class); + parsers.add(TermsParser.class); + parsers.add(RangeParser.class); + parsers.add(DateRangeParser.class); + parsers.add(IpRangeParser.class); + parsers.add(HistogramParser.class); + parsers.add(DateHistogramParser.class); + parsers.add(GeoDistanceParser.class); + parsers.add(NestedParser.class); + } + + /** + * Enabling extending the get module by adding a custom aggregation parser. + * + * @param parser The parser for the custom aggregator. + */ + public void addAggregatorParser(Class parser) { + parsers.add(parser); + } + + @Override + protected void configure() { + Multibinder multibinder = Multibinder.newSetBinder(binder(), Aggregator.Parser.class); + for (Class parser : parsers) { + multibinder.addBinding().to(parser); + } + bind(AggregatorParsers.class).asEagerSingleton(); + bind(AggregationParseElement.class).asEagerSingleton(); + bind(AggregationPhase.class).asEagerSingleton(); + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/AggregationParseElement.java b/src/main/java/org/elasticsearch/search/aggregations/AggregationParseElement.java new file mode 100644 index 00000000000..7c90c344819 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/AggregationParseElement.java @@ -0,0 +1,64 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations; + +import org.elasticsearch.common.inject.Inject; +import org.elasticsearch.common.xcontent.XContentParser; +import org.elasticsearch.search.SearchParseElement; +import org.elasticsearch.search.internal.SearchContext; + +/** + * The search parse element that is responsible for parsing the get part of the request. + * + * For example (in bold): + * 
+ *      curl -XGET 'localhost:9200/_search?search_type=count' -d '{
+ *          query: {
+ *              match_all : {}
+ *          },
+ *          addAggregation : {
+ *              avg_price: {
+ *                  avg : { field : price }
+ *              },
+ *              categories: {
+ *                  terms : { field : category, size : 12 },
+ *                  addAggregation: {
+ *                      avg_price : { avg : { field : price }}
+ *                  }
+ *              }
+ *          }
+ *      }'
+ *
+ */ +public class AggregationParseElement implements SearchParseElement { + + private final AggregatorParsers aggregatorParsers; + + @Inject + public AggregationParseElement(AggregatorParsers aggregatorParsers) { + this.aggregatorParsers = aggregatorParsers; + } + + @Override + public void parse(XContentParser parser, SearchContext context) throws Exception { + AggregatorFactories factories = aggregatorParsers.parseAggregators(parser, context); + context.aggregations(new SearchContextAggregations(factories)); + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/AggregationPhase.java b/src/main/java/org/elasticsearch/search/aggregations/AggregationPhase.java new file mode 100644 index 00000000000..be95dc18c46 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/AggregationPhase.java @@ -0,0 +1,178 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations; + +import com.google.common.collect.ImmutableMap; +import org.apache.lucene.index.AtomicReaderContext; +import org.apache.lucene.search.Filter; +import org.apache.lucene.search.Query; +import org.apache.lucene.search.Scorer; +import org.elasticsearch.ElasticSearchException; +import org.elasticsearch.common.inject.Inject; +import org.elasticsearch.common.lucene.search.Queries; +import org.elasticsearch.common.lucene.search.XCollector; +import org.elasticsearch.common.lucene.search.XConstantScoreQuery; +import org.elasticsearch.common.lucene.search.XFilteredQuery; +import org.elasticsearch.search.SearchParseElement; +import org.elasticsearch.search.SearchPhase; +import org.elasticsearch.search.aggregations.bucket.global.GlobalAggregator; +import org.elasticsearch.search.aggregations.support.AggregationContext; +import org.elasticsearch.search.internal.SearchContext; +import org.elasticsearch.search.query.QueryPhaseExecutionException; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; + +/** + * + */ +public class AggregationPhase implements SearchPhase { + + private final AggregationParseElement parseElement; + + private final AggregationBinaryParseElement binaryParseElement; + + @Inject + public AggregationPhase(AggregationParseElement parseElement, AggregationBinaryParseElement binaryParseElement) { + this.parseElement = parseElement; + this.binaryParseElement = binaryParseElement; + } + + @Override + public Map parseElements() { + return ImmutableMap.builder() + .put("aggregations", parseElement) + .put("aggs", parseElement) + .put("aggregations_binary", binaryParseElement) + .put("aggregationsBinary", binaryParseElement) + .put("aggs_binary", binaryParseElement) + .put("aggsBinary", binaryParseElement) + .build(); + } + + @Override + public void preProcess(SearchContext context) { + if (context.aggregations() != null) { + AggregationContext aggregationContext = new AggregationContext(context); + context.aggregations().aggregationContext(aggregationContext); + + List collectors = new ArrayList(); + Aggregator[] aggregators = context.aggregations().factories().createTopLevelAggregators(aggregationContext); + for (int i = 0; i < aggregators.length; i++) { + if (!(aggregators[i] instanceof GlobalAggregator)) { + Aggregator aggregator = aggregators[i]; + if (aggregator.shouldCollect()) { + collectors.add(aggregator); + } + } + } + context.aggregations().aggregators(aggregators); + if (!collectors.isEmpty()) { + context.searcher().addMainQueryCollector(new AggregationsCollector(collectors, aggregationContext)); + } + } + } + + @Override + public void execute(SearchContext context) throws ElasticSearchException { + if (context.aggregations() == null) { + return; + } + + if (context.queryResult().aggregations() != null) { + // no need to compute the facets twice, they should be computed on a per context basis + return; + } + + Aggregator[] aggregators = context.aggregations().aggregators(); + List globals = new ArrayList(); + for (int i = 0; i < aggregators.length; i++) { + if (aggregators[i] instanceof GlobalAggregator) { + globals.add(aggregators[i]); + } + } + + // optimize the global collector based execution + if (!globals.isEmpty()) { + AggregationsCollector collector = new AggregationsCollector(globals, context.aggregations().aggregationContext()); + Query query = new XConstantScoreQuery(Queries.MATCH_ALL_FILTER); + Filter searchFilter = context.searchFilter(context.types()); + if (searchFilter != null) { + query = new XFilteredQuery(query, searchFilter); + } + try { + context.searcher().search(query, collector); + } catch (Exception e) { + throw new QueryPhaseExecutionException(context, "Failed to execute global aggregators", e); + } + collector.postCollection(); + } + + List aggregations = new ArrayList(aggregators.length); + for (Aggregator aggregator : context.aggregations().aggregators()) { + aggregations.add(aggregator.buildAggregation(0)); + } + context.queryResult().aggregations(new InternalAggregations(aggregations)); + + } + + + static class AggregationsCollector extends XCollector { + + private final AggregationContext aggregationContext; + private final List collectors; + + AggregationsCollector(List collectors, AggregationContext aggregationContext) { + this.collectors = collectors; + this.aggregationContext = aggregationContext; + } + + @Override + public void setScorer(Scorer scorer) throws IOException { + aggregationContext.setScorer(scorer); + } + + @Override + public void collect(int doc) throws IOException { + for (int i = 0; i < collectors.size(); i++) { + collectors.get(i).collect(doc, 0); + } + } + + @Override + public void setNextReader(AtomicReaderContext context) throws IOException { + aggregationContext.setNextReader(context); + } + + @Override + public boolean acceptsDocsOutOfOrder() { + return true; + } + + @Override + public void postCollection() { + for (int i = 0; i < collectors.size(); i++) { + collectors.get(i).postCollection(); + } + } + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/AggregationStreams.java b/src/main/java/org/elasticsearch/search/aggregations/AggregationStreams.java new file mode 100644 index 00000000000..0be9474b7d0 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/AggregationStreams.java @@ -0,0 +1,68 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations; + +import com.google.common.collect.ImmutableMap; +import org.elasticsearch.common.bytes.BytesReference; +import org.elasticsearch.common.collect.MapBuilder; +import org.elasticsearch.common.io.stream.StreamInput; + +import java.io.IOException; + +/** + * A registry for all the dedicated streams in the aggregation module. This is to support dynamic addAggregation that + * know how to stream themselves. + */ +public class AggregationStreams { + + private static ImmutableMap streams = ImmutableMap.of(); + + /** + * A stream that knows how to read an aggregation from the input. + */ + public static interface Stream { + InternalAggregation readResult(StreamInput in) throws IOException; + } + + /** + * Registers the given stream and associate it with the given types. + * + * @param stream The streams to register + * @param types The types associated with the streams + */ + public static synchronized void registerStream(Stream stream, BytesReference... types) { + MapBuilder uStreams = MapBuilder.newMapBuilder(streams); + for (BytesReference type : types) { + uStreams.put(type, stream); + } + streams = uStreams.immutableMap(); + } + + /** + * Returns the stream that is registered for the given type + * + * @param type The given type + * @return The associated stream + */ + public static Stream stream(BytesReference type) { + return streams.get(type); + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/Aggregations.java b/src/main/java/org/elasticsearch/search/aggregations/Aggregations.java new file mode 100644 index 00000000000..da544ae1ce2 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/Aggregations.java @@ -0,0 +1,50 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations; + +import java.util.List; +import java.util.Map; + +/** + * Represents a set of computed addAggregation. + */ +public interface Aggregations extends Iterable { + + /** + * The list of {@link Aggregation}s. + */ + List asList(); + + /** + * Returns the {@link Aggregation}s keyed by aggregation name. + */ + Map asMap(); + + /** + * Returns the {@link Aggregation}s keyed by aggregation name. + */ + Map getAsMap(); + + /** + * Returns the aggregation that is associated with the specified name. + */ + A get(String name); + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/Aggregator.java b/src/main/java/org/elasticsearch/search/aggregations/Aggregator.java new file mode 100644 index 00000000000..b82f1c8d4b5 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/Aggregator.java @@ -0,0 +1,198 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations; + +import org.elasticsearch.common.xcontent.XContentParser; +import org.elasticsearch.search.aggregations.support.AggregationContext; +import org.elasticsearch.search.internal.SearchContext; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; + +public abstract class Aggregator { + + /** + * Defines the nature of the aggregator's aggregation execution when nested in other aggregators and the buckets they create. + */ + public static enum BucketAggregationMode { + + /** + * In this mode, a new aggregator instance will be created per bucket (created by the parent aggregator) + */ + PER_BUCKET, + + /** + * In this mode, a single aggregator instance will be created per parent aggregator, that will handle the aggregations of all its buckets. + */ + MULTI_BUCKETS + } + + protected final String name; + protected final Aggregator parent; + protected final AggregationContext context; + protected final int depth; + protected final long estimatedBucketCount; + + protected final BucketAggregationMode bucketAggregationMode; + protected final AggregatorFactories factories; + protected final Aggregator[] subAggregators; + + /** + * Constructs a new Aggregator. + * + * @param name The name of the aggregation + * @param bucketAggregationMode The nature of execution as a sub-aggregator (see {@link BucketAggregationMode}) + * @param factories The factories for all the sub-aggregators under this aggregator + * @param estimatedBucketsCount When served as a sub-aggregator, indicate how many buckets the parent aggregator will generate. + * @param context The aggregation context + * @param parent The parent aggregator (may be {@code null} for top level aggregators) + */ + protected Aggregator(String name, BucketAggregationMode bucketAggregationMode, AggregatorFactories factories, long estimatedBucketsCount, AggregationContext context, Aggregator parent) { + this.name = name; + this.parent = parent; + this.estimatedBucketCount = estimatedBucketsCount; + this.context = context; + this.depth = parent == null ? 0 : 1 + parent.depth(); + this.bucketAggregationMode = bucketAggregationMode; + assert factories != null : "sub-factories provided to BucketAggregator must not be null, use AggragatorFactories.EMPTY instead"; + this.factories = factories; + this.subAggregators = factories.createSubAggregators(this, estimatedBucketsCount); + } + + /** + * @return The name of the aggregation. + */ + public String name() { + return name; + } + + /** Return the estimated number of buckets. */ + public final long estimatedBucketCount() { + return estimatedBucketCount; + } + + /** Return the depth of this aggregator in the aggregation tree. */ + public final int depth() { + return depth; + } + + /** + * @return The parent aggregator of this aggregator. The addAggregation are hierarchical in the sense that some can + * be composed out of others (more specifically, bucket addAggregation can define other addAggregation that will + * be aggregated per bucket). This method returns the direct parent aggregator that contains this aggregator, or + * {@code null} if there is none (meaning, this aggregator is a top level one) + */ + public Aggregator parent() { + return parent; + } + + /** + * @return The current aggregation context. + */ + public AggregationContext context() { + return context; + } + + /** + * @return The bucket aggregation mode of this aggregator. This mode defines the nature in which the aggregation is executed + * @see BucketAggregationMode + */ + public BucketAggregationMode bucketAggregationMode() { + return bucketAggregationMode; + } + + /** + * @return Whether this aggregator is in the state where it can collect documents. Some aggregators can do their aggregations without + * actually collecting documents, for example, an aggregator that computes stats over unmapped fields doesn't need to collect + * anything as it knows to just return "empty" stats as the aggregation result. + */ + public abstract boolean shouldCollect(); + + /** + * Called during the query phase, to collect & aggregate the given document. + * + * @param doc The document to be collected/aggregated + * @param owningBucketOrdinal The ordinal of the bucket this aggregator belongs to, assuming this aggregator is not a top level aggregator. + * Typically, aggregators with {@code #bucketAggregationMode} set to {@link BucketAggregationMode#MULTI_BUCKETS} + * will heavily depend on this ordinal. Other aggregators may or may not use it and can see this ordinal as just + * an extra information for the aggregation context. For top level aggregators, the ordinal will always be + * equal to 0. + * @throws IOException + */ + public abstract void collect(int doc, long owningBucketOrdinal) throws IOException; + + /** + * Called after collection of all document is done. + */ + public final void postCollection() { + for (int i = 0; i < subAggregators.length; i++) { + subAggregators[i].postCollection(); + } + doPostCollection(); + } + + /** + * Can be overriden by aggregator implementation to be called back when the collection phase ends. + */ + protected void doPostCollection() { + } + + /** + * @return The aggregated & built aggregation + */ + public abstract InternalAggregation buildAggregation(long owningBucketOrdinal); + + public abstract InternalAggregation buildEmptyAggregation(); + + protected final InternalAggregations buildEmptySubAggregations() { + List aggs = new ArrayList(); + for (Aggregator aggregator : subAggregators) { + aggs.add(aggregator.buildEmptyAggregation()); + } + return new InternalAggregations(aggs); + } + + /** + * Parses the aggregation request and creates the appropriate aggregator factory for it. + * + * @see {@link AggregatorFactory} + */ + public static interface Parser { + + /** + * @return The aggregation type this parser is associated with. + */ + String type(); + + /** + * Returns the aggregator factory with which this parser is associated, may return {@code null} indicating the + * aggregation should be skipped (e.g. when trying to aggregate on unmapped fields). + * + * @param aggregationName The name of the aggregation + * @param parser The xcontent parser + * @param context The search context + * @return The resolved aggregator factory or {@code null} in case the aggregation should be skipped + * @throws java.io.IOException When parsing fails + */ + AggregatorFactory parse(String aggregationName, XContentParser parser, SearchContext context) throws IOException; + + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/AggregatorFactories.java b/src/main/java/org/elasticsearch/search/aggregations/AggregatorFactories.java new file mode 100644 index 00000000000..32c16671911 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/AggregatorFactories.java @@ -0,0 +1,176 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations; + +import org.elasticsearch.common.util.BigArrays; +import org.elasticsearch.common.util.ObjectArray; +import org.elasticsearch.search.aggregations.Aggregator.BucketAggregationMode; +import org.elasticsearch.search.aggregations.support.AggregationContext; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; + +/** + * + */ +public class AggregatorFactories { + + public static final AggregatorFactories EMPTY = new Empty(); + + private final AggregatorFactory[] factories; + + public static Builder builder() { + return new Builder(); + } + + private AggregatorFactories(AggregatorFactory[] factories) { + this.factories = factories; + } + + /** + * Create all aggregators so that they can be consumed with multiple buckets. + */ + public Aggregator[] createSubAggregators(Aggregator parent, final long estimatedBucketsCount) { + Aggregator[] aggregators = new Aggregator[count()]; + for (int i = 0; i < factories.length; ++i) { + final AggregatorFactory factory = factories[i]; + final Aggregator first = factory.create(parent.context(), parent, estimatedBucketsCount); + if (first.bucketAggregationMode() == BucketAggregationMode.MULTI_BUCKETS) { + // This aggregator already supports multiple bucket ordinals, can be used directly + aggregators[i] = first; + continue; + } + // the aggregator doesn't support multiple ordinals, let's wrap it so that it does. + aggregators[i] = new Aggregator(first.name(), BucketAggregationMode.MULTI_BUCKETS, AggregatorFactories.EMPTY, 1, first.context(), first.parent()) { + + ObjectArray aggregators; + + { + aggregators = BigArrays.newObjectArray(estimatedBucketsCount); + aggregators.set(0, first); + for (long i = 1; i < estimatedBucketsCount; ++i) { + aggregators.set(i, factory.create(parent.context(), parent, estimatedBucketsCount)); + } + } + + @Override + public boolean shouldCollect() { + return first.shouldCollect(); + } + + @Override + protected void doPostCollection() { + for (long i = 0; i < aggregators.size(); ++i) { + final Aggregator aggregator = aggregators.get(i); + if (aggregator != null) { + aggregator.postCollection(); + } + } + } + + @Override + public void collect(int doc, long owningBucketOrdinal) throws IOException { + aggregators = BigArrays.grow(aggregators, owningBucketOrdinal + 1); + Aggregator aggregator = aggregators.get(owningBucketOrdinal); + if (aggregator == null) { + aggregator = factory.create(parent.context(), parent, estimatedBucketsCount); + aggregators.set(owningBucketOrdinal, aggregator); + } + aggregator.collect(doc, 0); + } + + @Override + public InternalAggregation buildAggregation(long owningBucketOrdinal) { + return aggregators.get(owningBucketOrdinal).buildAggregation(0); + } + + @Override + public InternalAggregation buildEmptyAggregation() { + return first.buildEmptyAggregation(); + } + }; + } + return aggregators; + } + + public Aggregator[] createTopLevelAggregators(AggregationContext ctx) { + // These aggregators are going to be used with a single bucket ordinal, no need to wrap the PER_BUCKET ones + Aggregator[] aggregators = new Aggregator[factories.length]; + for (int i = 0; i < factories.length; i++) { + aggregators[i] = factories[i].create(ctx, null, 0); + } + return aggregators; + } + + public int count() { + return factories.length; + } + + void setParent(AggregatorFactory parent) { + for (AggregatorFactory factory : factories) { + factory.parent = parent; + } + } + + public void validate() { + for (AggregatorFactory factory : factories) { + factory.validate(); + } + } + + private final static class Empty extends AggregatorFactories { + + private static final AggregatorFactory[] EMPTY_FACTORIES = new AggregatorFactory[0]; + private static final Aggregator[] EMPTY_AGGREGATORS = new Aggregator[0]; + + private Empty() { + super(EMPTY_FACTORIES); + } + + @Override + public Aggregator[] createSubAggregators(Aggregator parent, long estimatedBucketsCount) { + return EMPTY_AGGREGATORS; + } + + @Override + public Aggregator[] createTopLevelAggregators(AggregationContext ctx) { + return EMPTY_AGGREGATORS; + } + + } + + public static class Builder { + + private List factories = new ArrayList(); + + public Builder add(AggregatorFactory factory) { + factories.add(factory); + return this; + } + + public AggregatorFactories build() { + if (factories.isEmpty()) { + return EMPTY; + } + return new AggregatorFactories(factories.toArray(new AggregatorFactory[factories.size()])); + } + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/AggregatorFactory.java b/src/main/java/org/elasticsearch/search/aggregations/AggregatorFactory.java new file mode 100644 index 00000000000..51251d4b8f7 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/AggregatorFactory.java @@ -0,0 +1,88 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations; + +import org.elasticsearch.search.aggregations.support.AggregationContext; + +/** + * A factory that knows how to create an {@link Aggregator} of a specific type. + */ +public abstract class AggregatorFactory { + + protected String name; + protected String type; + protected AggregatorFactory parent; + protected AggregatorFactories factories = AggregatorFactories.EMPTY; + + /** + * Constructs a new aggregator factory. + * + * @param name The aggregation name + * @param type The aggregation type + */ + public AggregatorFactory(String name, String type) { + this.name = name; + this.type = type; + } + + /** + * Registers sub-factories with this factory. The sub-factory will be responsible for the creation of sub-aggregators under the + * aggregator created by this factory. + * + * @param subFactories The sub-factories + * @return this factory (fluent interface) + */ + public AggregatorFactory subFactories(AggregatorFactories subFactories) { + this.factories = subFactories; + this.factories.setParent(this); + return this; + } + + /** + * Validates the state of this factory (makes sure the factory is properly configured) + */ + public final void validate() { + doValidate(); + factories.validate(); + } + + /** + * @return The parent factory if one exists (will always return {@code null} for top level aggregator factories). + */ + public AggregatorFactory parent() { + return parent; + } + + /** + * Creates the aggregator + * + * @param context The aggregation context + * @param parent The parent aggregator (if this is a top level factory, the parent will be {@code null}) + * @param expectedBucketsCount If this is a sub-factory of another factory, this will indicate the number of bucket the parent aggregator + * may generate (this is an estimation only). For top level factories, this will always be 0 + * + * @return The created aggregator + */ + public abstract Aggregator create(AggregationContext context, Aggregator parent, long expectedBucketsCount); + + public void doValidate() { + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/AggregatorParsers.java b/src/main/java/org/elasticsearch/search/aggregations/AggregatorParsers.java new file mode 100644 index 00000000000..fabbe8b4e44 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/AggregatorParsers.java @@ -0,0 +1,129 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations; + +import com.google.common.collect.ImmutableMap; +import org.elasticsearch.common.collect.MapBuilder; +import org.elasticsearch.common.inject.Inject; +import org.elasticsearch.common.xcontent.XContentParser; +import org.elasticsearch.search.SearchParseException; +import org.elasticsearch.search.internal.SearchContext; + +import java.io.IOException; +import java.util.Set; + +/** + * A registry for all the aggregator parser, also servers as the main parser for the aggregations module + */ +public class AggregatorParsers { + + private final ImmutableMap parsers; + + /** + * Constructs the AggregatorParsers out of all the given parsers + * + * @param parsers The available aggregator parsers (dynamically injected by the {@link org.elasticsearch.search.aggregations.AggregationModule}). + */ + @Inject + public AggregatorParsers(Set parsers) { + MapBuilder builder = MapBuilder.newMapBuilder(); + for (Aggregator.Parser parser : parsers) { + builder.put(parser.type(), parser); + } + this.parsers = builder.immutableMap(); + } + + /** + * Returns the parser that is registered under the given aggregation type. + * + * @param type The aggregation type + * @return The parser associated with the given aggregation type. + */ + public Aggregator.Parser parser(String type) { + return parsers.get(type); + } + + /** + * Parses the aggregation request recursively generating aggregator factories in turn. + * + * @param parser The input xcontent that will be parsed. + * @param context The search context. + * + * @return The parsed aggregator factories. + * + * @throws IOException When parsing fails for unknown reasons. + */ + public AggregatorFactories parseAggregators(XContentParser parser, SearchContext context) throws IOException { + return parseAggregators(parser, context, 0); + } + + + private AggregatorFactories parseAggregators(XContentParser parser, SearchContext context, int level) throws IOException { + XContentParser.Token token = null; + String currentFieldName = null; + + AggregatorFactories.Builder factories = new AggregatorFactories.Builder(); + + String aggregationName = null; + while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) { + if (token == XContentParser.Token.FIELD_NAME) { + aggregationName = parser.currentName(); + } else if (token == XContentParser.Token.START_OBJECT) { + String aggregatorType = null; + AggregatorFactory factory = null; + AggregatorFactories subFactories = null; + while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) { + if (token == XContentParser.Token.FIELD_NAME) { + currentFieldName = parser.currentName(); + } else if (token == XContentParser.Token.START_OBJECT) { + if ("aggregations".equals(currentFieldName) || "aggs".equals(currentFieldName)) { + subFactories = parseAggregators(parser, context, level+1); + } else { + aggregatorType = currentFieldName; + Aggregator.Parser aggregatorParser = parser(aggregatorType); + if (aggregatorParser == null) { + throw new SearchParseException(context, "Could not find aggregator type [" + currentFieldName + "]"); + } + factory = aggregatorParser.parse(aggregationName, parser, context); + } + } + } + + if (factory == null) { + // skipping the aggregation + continue; + } + + if (subFactories != null) { + factory.subFactories(subFactories); + } + + if (level == 0) { + factory.validate(); + } + + factories.add(factory); + } + } + + return factories.build(); + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/InternalAggregation.java b/src/main/java/org/elasticsearch/search/aggregations/InternalAggregation.java new file mode 100644 index 00000000000..08a5d8c385c --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/InternalAggregation.java @@ -0,0 +1,146 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations; + +import org.elasticsearch.cache.recycler.CacheRecycler; +import org.elasticsearch.common.bytes.BytesArray; +import org.elasticsearch.common.bytes.BytesReference; +import org.elasticsearch.common.io.stream.Streamable; +import org.elasticsearch.common.xcontent.ToXContent; +import org.elasticsearch.common.xcontent.XContentBuilderString; + +import java.util.List; + +/** + * An internal implementation of {@link Aggregation}. Serves as a base class for all aggregation implementations. + */ +public abstract class InternalAggregation implements Aggregation, ToXContent, Streamable { + + /** + * The aggregation type that holds all the string types that are associated with an aggregation: + *
    + *
  • name - used as the parser type
    • + *
  • stream - used as the stream type
    • + *
+ */ + public static class Type { + + private String name; + private BytesReference stream; + + public Type(String name) { + this(name, new BytesArray(name)); + } + + public Type(String name, String stream) { + this(name, new BytesArray(stream)); + } + + public Type(String name, BytesReference stream) { + this.name = name; + this.stream = stream; + } + + /** + * @return The name of the type (mainly used for registering the parser for the aggregator (see {@link org.elasticsearch.search.aggregations.Aggregator.Parser#type()}). + */ + public String name() { + return name; + } + + /** + * @return The name of the stream type (used for registering the aggregation stream + * (see {@link AggregationStreams#registerStream(AggregationStreams.Stream, org.elasticsearch.common.bytes.BytesReference...)}). + */ + public BytesReference stream() { + return stream; + } + } + + protected static class ReduceContext { + + private final List aggregations; + private final CacheRecycler cacheRecycler; + + public ReduceContext(List aggregations, CacheRecycler cacheRecycler) { + this.aggregations = aggregations; + this.cacheRecycler = cacheRecycler; + } + + public List aggregations() { + return aggregations; + } + + public CacheRecycler cacheRecycler() { + return cacheRecycler; + } + } + + + protected String name; + + /** Constructs an un initialized addAggregation (used for serialization) **/ + protected InternalAggregation() {} + + /** + * Constructs an get with a given name. + * + * @param name The name of the get. + */ + protected InternalAggregation(String name) { + this.name = name; + } + + @Override + public String getName() { + return name; + } + + /** + * @return The {@link Type} of this aggregation + */ + public abstract Type type(); + + /** + * Reduces the given addAggregation to a single one and returns it. In most cases, the assumption will be the all given + * addAggregation are of the same type (the same type as this aggregation). For best efficiency, when implementing, + * try reusing an existing get instance (typically the first in the given list) to save on redundant object + * construction. + */ + public abstract InternalAggregation reduce(ReduceContext reduceContext); + + + /** + * Common xcontent fields that are shared among addAggregation + */ + public static final class CommonFields { + public static final XContentBuilderString BUCKETS = new XContentBuilderString("buckets"); + public static final XContentBuilderString VALUE = new XContentBuilderString("value"); + public static final XContentBuilderString VALUE_AS_STRING = new XContentBuilderString("value_as_string"); + public static final XContentBuilderString DOC_COUNT = new XContentBuilderString("doc_count"); + public static final XContentBuilderString KEY = new XContentBuilderString("key"); + public static final XContentBuilderString KEY_AS_STRING = new XContentBuilderString("key_as_string"); + public static final XContentBuilderString FROM = new XContentBuilderString("from"); + public static final XContentBuilderString FROM_AS_STRING = new XContentBuilderString("from_as_string"); + public static final XContentBuilderString TO = new XContentBuilderString("to"); + public static final XContentBuilderString TO_AS_STRING = new XContentBuilderString("to_as_string"); + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/InternalAggregations.java b/src/main/java/org/elasticsearch/search/aggregations/InternalAggregations.java new file mode 100644 index 00000000000..55ac287f343 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/InternalAggregations.java @@ -0,0 +1,211 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations; + +import com.google.common.base.Function; +import com.google.common.collect.*; +import org.elasticsearch.cache.recycler.CacheRecycler; +import org.elasticsearch.common.bytes.BytesReference; +import org.elasticsearch.common.io.stream.StreamInput; +import org.elasticsearch.common.io.stream.StreamOutput; +import org.elasticsearch.common.io.stream.Streamable; +import org.elasticsearch.common.xcontent.ToXContent; +import org.elasticsearch.common.xcontent.XContentBuilder; +import org.elasticsearch.common.xcontent.XContentBuilderString; + +import java.io.IOException; +import java.util.*; + +import static com.google.common.collect.Maps.newHashMap; + +/** + * An internal implementation of {@link Aggregations}. + */ +public class InternalAggregations implements Aggregations, ToXContent, Streamable { + + public final static InternalAggregations EMPTY = new InternalAggregations(); + private static final Function SUPERTYPE_CAST = new Function() { + @Override + public Aggregation apply(InternalAggregation input) { + return input; + } + }; + + private List aggregations = ImmutableList.of(); + + private Map aggregationsAsMap; + + private InternalAggregations() { + } + + /** + * Constructs a new addAggregation. + */ + public InternalAggregations(List aggregations) { + this.aggregations = aggregations; + } + + /** Resets the internal addAggregation */ + void reset(List aggregations) { + this.aggregations = aggregations; + this.aggregationsAsMap = null; + } + + /** + * Iterates over the {@link Aggregation}s. + */ + @Override + public Iterator iterator() { + return Iterators.transform(aggregations.iterator(), SUPERTYPE_CAST); + } + + /** + * The list of {@link Aggregation}s. + */ + public List asList() { + return Lists.transform(aggregations, SUPERTYPE_CAST); + } + + /** + * Returns the {@link Aggregation}s keyed by map. + */ + public Map asMap() { + return getAsMap(); + } + + /** + * Returns the {@link Aggregation}s keyed by map. + */ + public Map getAsMap() { + if (aggregationsAsMap == null) { + Map aggregationsAsMap = newHashMap(); + for (InternalAggregation aggregation : aggregations) { + aggregationsAsMap.put(aggregation.getName(), aggregation); + } + this.aggregationsAsMap = aggregationsAsMap; + } + return Maps.transformValues(aggregationsAsMap, SUPERTYPE_CAST); + } + + /** + * @return the aggregation of the specified name. + */ + @SuppressWarnings("unchecked") + @Override + public A get(String name) { + return (A) asMap().get(name); + } + + /** + * Reduces the given lists of addAggregation. + * + * @param aggregationsList A list of addAggregation to reduce + * @return The reduced addAggregation + */ + public static InternalAggregations reduce(List aggregationsList, CacheRecycler cacheRecycler) { + if (aggregationsList.isEmpty()) { + return null; + } + + // first we collect all addAggregation of the same type and list them together + + Map> aggByName = new HashMap>(); + for (InternalAggregations aggregations : aggregationsList) { + for (InternalAggregation aggregation : aggregations.aggregations) { + List aggs = aggByName.get(aggregation.getName()); + if (aggs == null) { + aggs = new ArrayList(aggregationsList.size()); + aggByName.put(aggregation.getName(), aggs); + } + aggs.add(aggregation); + } + } + + // now we can use the first aggregation of each list to handle the reduce of its list + + List reducedAggregations = new ArrayList(); + for (Map.Entry> entry : aggByName.entrySet()) { + List aggregations = entry.getValue(); + InternalAggregation first = aggregations.get(0); // the list can't be empty as it's created on demand + reducedAggregations.add(first.reduce(new InternalAggregation.ReduceContext(aggregations, cacheRecycler))); + } + InternalAggregations result = aggregationsList.get(0); + result.reset(reducedAggregations); + return result; + } + + /** The fields required to write this addAggregation to xcontent */ + static class Fields { + public static final XContentBuilderString AGGREGATIONS = new XContentBuilderString("aggregations"); + } + + @Override + public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException { + if (aggregations.isEmpty()) { + return builder; + } + builder.startObject(Fields.AGGREGATIONS); + toXContentInternal(builder, params); + return builder.endObject(); + } + + /** + * Directly write all the addAggregation without their bounding object. Used by sub-addAggregation (non top level addAggregation) + */ + public XContentBuilder toXContentInternal(XContentBuilder builder, Params params) throws IOException { + for (Aggregation aggregation : aggregations) { + ((InternalAggregation) aggregation).toXContent(builder, params); + } + return builder; + } + + public static InternalAggregations readAggregations(StreamInput in) throws IOException { + InternalAggregations result = new InternalAggregations(); + result.readFrom(in); + return result; + } + + @Override + public void readFrom(StreamInput in) throws IOException { + int size = in.readVInt(); + if (size == 0) { + aggregations = ImmutableList.of(); + aggregationsAsMap = ImmutableMap.of(); + } else { + aggregations = Lists.newArrayListWithCapacity(size); + for (int i = 0; i < size; i++) { + BytesReference type = in.readBytesReference(); + InternalAggregation aggregation = AggregationStreams.stream(type).readResult(in); + aggregations.add(aggregation); + } + } + } + + @Override + public void writeTo(StreamOutput out) throws IOException { + out.writeVInt(aggregations.size()); + for (Aggregation aggregation : aggregations) { + InternalAggregation internal = (InternalAggregation) aggregation; + out.writeBytesReference(internal.type().stream()); + internal.writeTo(out); + } + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/SearchContextAggregations.java b/src/main/java/org/elasticsearch/search/aggregations/SearchContextAggregations.java new file mode 100644 index 00000000000..637bab2a4db --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/SearchContextAggregations.java @@ -0,0 +1,65 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations; + +import org.elasticsearch.search.aggregations.support.AggregationContext; + +/** + * The aggregation context that is part of the search context. + */ +public class SearchContextAggregations { + + private final AggregatorFactories factories; + private Aggregator[] aggregators; + private AggregationContext aggregationContext; + + /** + * Creates a new aggregation context with the parsed aggregator factories + */ + public SearchContextAggregations(AggregatorFactories factories) { + this.factories = factories; + } + + public AggregatorFactories factories() { + return factories; + } + + public Aggregator[] aggregators() { + return aggregators; + } + + public AggregationContext aggregationContext() { + return aggregationContext; + } + + public void aggregationContext(AggregationContext aggregationContext) { + this.aggregationContext = aggregationContext; + } + + /** + * Registers all the created aggregators (top level aggregators) for the search execution context. + * + * @param aggregators The top level aggregators of the search execution. + */ + public void aggregators(Aggregator[] aggregators) { + this.aggregators = aggregators; + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/TransportAggregationModule.java b/src/main/java/org/elasticsearch/search/aggregations/TransportAggregationModule.java new file mode 100644 index 00000000000..e80c7d9b5fb --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/TransportAggregationModule.java @@ -0,0 +1,78 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations; + +import org.elasticsearch.common.inject.AbstractModule; +import org.elasticsearch.search.aggregations.bucket.histogram.InternalDateHistogram; +import org.elasticsearch.search.aggregations.bucket.histogram.InternalHistogram; +import org.elasticsearch.search.aggregations.bucket.range.InternalRange; +import org.elasticsearch.search.aggregations.bucket.range.date.InternalDateRange; +import org.elasticsearch.search.aggregations.bucket.range.geodistance.InternalGeoDistance; +import org.elasticsearch.search.aggregations.bucket.range.ipv4.InternalIPv4Range; +import org.elasticsearch.search.aggregations.bucket.terms.DoubleTerms; +import org.elasticsearch.search.aggregations.bucket.terms.LongTerms; +import org.elasticsearch.search.aggregations.bucket.terms.StringTerms; +import org.elasticsearch.search.aggregations.bucket.terms.UnmappedTerms; +import org.elasticsearch.search.aggregations.bucket.filter.InternalFilter; +import org.elasticsearch.search.aggregations.bucket.global.InternalGlobal; +import org.elasticsearch.search.aggregations.bucket.missing.InternalMissing; +import org.elasticsearch.search.aggregations.bucket.nested.InternalNested; +import org.elasticsearch.search.aggregations.metrics.valuecount.InternalValueCount; +import org.elasticsearch.search.aggregations.metrics.avg.InternalAvg; +import org.elasticsearch.search.aggregations.metrics.max.InternalMax; +import org.elasticsearch.search.aggregations.metrics.min.InternalMin; +import org.elasticsearch.search.aggregations.metrics.stats.InternalStats; +import org.elasticsearch.search.aggregations.metrics.stats.extended.InternalExtendedStats; +import org.elasticsearch.search.aggregations.metrics.sum.InternalSum; + +/** + * A module that registers all the transport streams for the addAggregation + */ +public class TransportAggregationModule extends AbstractModule { + + @Override + protected void configure() { + + // calcs + InternalAvg.registerStreams(); + InternalSum.registerStreams(); + InternalMin.registerStreams(); + InternalMax.registerStreams(); + InternalStats.registerStreams(); + InternalExtendedStats.registerStreams(); + InternalValueCount.registerStreams(); + + // buckets + InternalGlobal.registerStreams(); + InternalFilter.registerStreams(); + InternalMissing.registerStreams(); + StringTerms.registerStreams(); + LongTerms.registerStreams(); + DoubleTerms.registerStreams(); + UnmappedTerms.registerStreams(); + InternalRange.registerStream(); + InternalDateRange.registerStream(); + InternalIPv4Range.registerStream(); + InternalHistogram.registerStream(); + InternalDateHistogram.registerStream(); + InternalGeoDistance.registerStream(); + InternalNested.registerStream(); + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/ValuesSourceAggregationBuilder.java b/src/main/java/org/elasticsearch/search/aggregations/ValuesSourceAggregationBuilder.java new file mode 100644 index 00000000000..6bea5b0636c --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/ValuesSourceAggregationBuilder.java @@ -0,0 +1,122 @@ +package org.elasticsearch.search.aggregations; + +import com.google.common.collect.Maps; +import org.elasticsearch.common.xcontent.XContentBuilder; + +import java.io.IOException; +import java.util.Map; + +/** + * A base class for all bucket aggregation builders that are based on values (either script generated or field data values) + */ +public abstract class ValuesSourceAggregationBuilder> extends AggregationBuilder { + + private String field; + private String script; + private String scriptLang; + private Map params; + + /** + * Constructs a new builder. + * + * @param name The name of the aggregation. + * @param type The type of the aggregation. + */ + protected ValuesSourceAggregationBuilder(String name, String type) { + super(name, type); + } + + /** + * Sets the field from which the values will be extracted. + * + * @param field The name of the field + * @return This builder (fluent interface support) + */ + @SuppressWarnings("unchecked") + public B field(String field) { + this.field = field; + return (B) this; + } + + /** + * Sets the script which generates the values. If the script is configured along with the field (as in {@link #field(String)}), then + * this script will be treated as a {@code value script}. A value script will be applied on the values that are extracted from + * the field data (you can refer to that value in the script using the {@code _value} reserved variable). If only the script is configured + * (and the no field is configured next to it), then the script will be responsible to generate the values that will be aggregated. + * + * @param script The configured script. + * @return This builder (fluent interface support) + */ + @SuppressWarnings("unchecked") + public B script(String script) { + this.script = script; + return (B) this; + } + + /** + * Sets the language of the script (if one is defined). + *

+ * Also see {@link #script(String)}. + * + * @param scriptLang The language of the script. + * @return This builder (fluent interface support) + */ + @SuppressWarnings("unchecked") + public B scriptLang(String scriptLang) { + this.scriptLang = scriptLang; + return (B) this; + } + + /** + * Sets the value of a parameter that is used in the script (if one is configured). + * + * @param name The name of the parameter. + * @param value The value of the parameter. + * @return This builder (fluent interface support) + */ + @SuppressWarnings("unchecked") + public B param(String name, Object value) { + if (params == null) { + params = Maps.newHashMap(); + } + params.put(name, value); + return (B) this; + } + + /** + * Sets the values of a parameters that are used in the script (if one is configured). + * + * @param params The the parameters. + * @return This builder (fluent interface support) + */ + @SuppressWarnings("unchecked") + public B params(Map params) { + if (this.params == null) { + this.params = Maps.newHashMap(); + } + this.params.putAll(params); + return (B) this; + } + + @Override + protected final XContentBuilder internalXContent(XContentBuilder builder, Params params) throws IOException { + builder.startObject(); + if (field != null) { + builder.field("field", field); + } + if (script != null) { + builder.field("script", script); + } + if (scriptLang != null) { + builder.field("script_lang", scriptLang); + } + if (this.params != null) { + builder.field("params").map(this.params); + } + + doInternalXContent(builder, params); + return builder.endObject(); + } + + protected abstract XContentBuilder doInternalXContent(XContentBuilder builder, Params params) throws IOException; +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/Bucket.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/Bucket.java new file mode 100644 index 00000000000..b883a307ec1 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/Bucket.java @@ -0,0 +1,106 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket; + +import org.elasticsearch.ElasticSearchIllegalArgumentException; +import org.elasticsearch.search.aggregations.Aggregations; +import org.elasticsearch.search.aggregations.metrics.MetricsAggregation; + +/** + * + */ +public interface Bucket { + + /** + * @return The number of documents that fall within this bucket + */ + long getDocCount(); + + Aggregations getAggregations(); + + static class Comparator implements java.util.Comparator { + + private final String aggName; + private final String valueName; + private final boolean asc; + + public Comparator(String expression, boolean asc) { + this.asc = asc; + int i = expression.indexOf('.'); + if (i < 0) { + this.aggName = expression; + this.valueName = null; + } else { + this.aggName = expression.substring(0, i); + this.valueName = expression.substring(i+1); + } + } + + public Comparator(String aggName, String valueName, boolean asc) { + this.aggName = aggName; + this.valueName = valueName; + this.asc = asc; + } + + public boolean asc() { + return asc; + } + + public String aggName() { + return aggName; + } + + public String valueName() { + return valueName; + } + + @Override + public int compare(B b1, B b2) { + double v1 = value(b1); + double v2 = value(b2); + if (v1 > v2) { + return asc ? 1 : -1; + } else if (v1 < v2) { + return asc ? -1 : 1; + } + return 0; + } + + private double value(B bucket) { + MetricsAggregation aggregation = bucket.getAggregations().get(aggName); + if (aggregation == null) { + throw new ElasticSearchIllegalArgumentException("Unknown aggregation named [" + aggName + "]"); + } + if (aggregation instanceof MetricsAggregation.SingleValue) { + //TODO should we throw an exception if the value name is specified? + return ((MetricsAggregation.SingleValue) aggregation).value(); + } + if (aggregation instanceof MetricsAggregation.MultiValue) { + if (valueName == null) { + throw new ElasticSearchIllegalArgumentException("Cannot sort on multi valued aggregation [" + aggName + "]. A value name is required"); + } + return ((MetricsAggregation.MultiValue) aggregation).value(valueName); + } + + throw new ElasticSearchIllegalArgumentException("A mal attempt to sort terms by aggregation [" + aggregation.getName() + + "]. Terms can only be ordered by either standard order or direct calc aggregators of the terms"); + } + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/BucketsAggregator.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/BucketsAggregator.java new file mode 100644 index 00000000000..2f6c7c89d33 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/BucketsAggregator.java @@ -0,0 +1,93 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket; + +import org.elasticsearch.common.util.BigArrays; +import org.elasticsearch.common.util.LongArray; +import org.elasticsearch.search.aggregations.Aggregator; +import org.elasticsearch.search.aggregations.InternalAggregation; +import org.elasticsearch.search.aggregations.InternalAggregations; +import org.elasticsearch.search.aggregations.support.AggregationContext; +import org.elasticsearch.search.aggregations.AggregatorFactories; + +import java.io.IOException; +import java.util.Arrays; + +/** + * + */ +public abstract class BucketsAggregator extends Aggregator { + + protected LongArray docCounts; + + public BucketsAggregator(String name, BucketAggregationMode bucketAggregationMode, AggregatorFactories factories, + long estimatedBucketsCount, AggregationContext context, Aggregator parent) { + super(name, bucketAggregationMode, factories, estimatedBucketsCount, context, parent); + docCounts = BigArrays.newLongArray(estimatedBucketsCount); + } + + /** + * Utility method to collect the given doc in the given bucket (identified by the bucket ordinal) + */ + protected final void collectBucket(int doc, long bucketOrd) throws IOException { + docCounts = BigArrays.grow(docCounts, bucketOrd + 1); + docCounts.increment(bucketOrd, 1); + for (int i = 0; i < subAggregators.length; i++) { + subAggregators[i].collect(doc, bucketOrd); + } + } + + /** + * Utility method to collect the given doc in the given bucket but not to update the doc counts of the bucket + */ + protected final void collectBucketNoCounts(int doc, long bucketOrd) throws IOException { + for (int i = 0; i < subAggregators.length; i++) { + subAggregators[i].collect(doc, bucketOrd); + } + } + + /** + * Utility method to increment the doc counts of the given bucket (identified by the bucket ordinal) + */ + protected final void incrementBucketDocCount(int inc, long bucketOrd) throws IOException { + docCounts = BigArrays.grow(docCounts, bucketOrd + 1); + docCounts.increment(bucketOrd, inc); + } + + /** + * Utility method to return the number of documents that fell in the given bucket (identified by the bucket ordinal) + */ + protected final long bucketDocCount(long bucketOrd) { + assert bucketOrd < docCounts.size(); + return docCounts.get(bucketOrd); + } + + /** + * Utility method to build the aggregations of the given bucket (identified by the bucket ordinal) + */ + protected final InternalAggregations bucketAggregations(long bucketOrd) { + InternalAggregation[] aggregations = new InternalAggregation[subAggregators.length]; + for (int i = 0; i < subAggregators.length; i++) { + aggregations[i] = subAggregators[i].buildAggregation(bucketOrd); + } + return new InternalAggregations(Arrays.asList(aggregations)); + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/LongHash.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/LongHash.java new file mode 100644 index 00000000000..8b60bfacbe3 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/LongHash.java @@ -0,0 +1,194 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket; + +import com.carrotsearch.hppc.hash.MurmurHash3; +import com.google.common.base.Preconditions; +import org.elasticsearch.common.util.BigArrays; +import org.elasticsearch.common.util.LongArray; + +/** + * Specialized hash table implementation similar to BytesRefHash that maps + * long values to ids. Collisions are resolved with open addressing and linear + * probing, growth is smooth thanks to {@link BigArrays} and capacity is always + * a multiple of 2 for faster identification of buckets. + */ +// IDs are internally stored as id + 1 so that 0 encodes for an empty slot +public final class LongHash { + + // Open addressing typically requires having smaller load factors compared to linked lists because + // collisions may result into worse lookup performance. + private static final float DEFAULT_MAX_LOAD_FACTOR = 0.6f; + + private final float maxLoadFactor; + private long size, maxSize; + private LongArray keys; + private LongArray ids; + private long mask; + + // Constructor with configurable capacity and default maximum load factor. + public LongHash(long capacity) { + this(capacity, DEFAULT_MAX_LOAD_FACTOR); + } + + //Constructor with configurable capacity and load factor. + public LongHash(long capacity, float maxLoadFactor) { + Preconditions.checkArgument(capacity >= 0, "capacity must be >= 0"); + Preconditions.checkArgument(maxLoadFactor > 0 && maxLoadFactor < 1, "maxLoadFactor must be > 0 and < 1"); + this.maxLoadFactor = maxLoadFactor; + long buckets = 1L + (long) (capacity / maxLoadFactor); + buckets = Math.max(1, Long.highestOneBit(buckets - 1) << 1); // next power of two + assert buckets == Long.highestOneBit(buckets); + maxSize = (long) (buckets * maxLoadFactor); + assert maxSize >= capacity; + size = 0; + keys = BigArrays.newLongArray(buckets); + ids = BigArrays.newLongArray(buckets); + mask = buckets - 1; + } + + /** + * Return the number of allocated slots to store this hash table. + */ + public long capacity() { + return keys.size(); + } + + /** + * Return the number of longs in this hash table. + */ + public long size() { + return size; + } + + private static long hash(long value) { + // Don't use the value directly. Under some cases eg dates, it could be that the low bits don't carry much value and we would like + // all bits of the hash to carry as much value + return MurmurHash3.hash(value); + } + + private static long slot(long hash, long mask) { + return hash & mask; + } + + private static long nextSlot(long curSlot, long mask) { + return (curSlot + 1) & mask; // linear probing + } + + /** + * Get the id associated with key at 0 <e; index <e; capacity() or -1 if this slot is unused. + */ + public long id(long index) { + return ids.get(index) - 1; + } + + /** + * Return the key at 0 <e; index <e; capacity(). The result is undefined if the slot is unused. + */ + public long key(long index) { + return keys.get(index); + } + + /** + * Get the id associated with key + */ + public long get(long key) { + final long slot = slot(hash(key), mask); + for (long index = slot; ; index = nextSlot(index, mask)) { + final long id = ids.get(index); + if (id == 0L || keys.get(index) == key) { + return id - 1; + } + } + } + + private long set(long key, long id) { + assert size < maxSize; + final long slot = slot(hash(key), mask); + for (long index = slot; ; index = nextSlot(index, mask)) { + final long curId = ids.get(index); + if (curId == 0) { // means unset + ids.set(index, id + 1); + keys.set(index, key); + ++size; + return id; + } else if (keys.get(index) == key) { + return - curId; + } + } + } + + /** + * Try to add key. Return its newly allocated id if it wasn't in the hash table yet, or -1-id + * if it was already present in the hash table. + */ + public long add(long key) { + if (size >= maxSize) { + assert size == maxSize; + grow(); + } + assert size < maxSize; + return set(key, size); + } + + private void grow() { + // The difference of this implementation of grow() compared to standard hash tables is that we are growing in-place, which makes + // the re-mapping of keys to slots a bit more tricky. + assert size == maxSize; + final long prevSize = size; + final long buckets = keys.size(); + // Resize arrays + final long newBuckets = buckets << 1; + assert newBuckets == Long.highestOneBit(newBuckets) : newBuckets; // power of 2 + keys = BigArrays.resize(keys, newBuckets); + ids = BigArrays.resize(ids, newBuckets); + mask = newBuckets - 1; + size = 0; + // First let's remap in-place: most data will be put in its final position directly + for (long i = 0; i < buckets; ++i) { + final long id = ids.set(i, 0); + if (id > 0) { + final long key = keys.set(i, 0); + final long newId = set(key, id - 1); + assert newId == id - 1 : newId + " " + (id - 1); + } + } + // The only entries which have not been put in their final position in the previous loop are those that were stored in a slot that + // is < slot(key, mask). This only happens when slot(key, mask) returned a slot that was close to the end of the array and colision + // resolution has put it back in the first slots. This time, collision resolution will have put them at the beginning of the newly + // allocated slots. Let's re-add them to make sure they are in the right slot. This 2nd loop will typically exit very early. + for (long i = buckets; i < newBuckets; ++i) { + final long id = ids.set(i, 0); + if (id > 0) { + --size; // we just removed an entry + final long key = keys.set(i, 0); + final long newId = set(key, id - 1); // add it back + assert newId == id - 1 : newId + " " + (id - 1); + assert newId == get(key); + } else { + break; + } + } + assert size == prevSize; + maxSize = (long) (newBuckets * maxLoadFactor); + assert size < maxSize; + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/SingleBucketAggregation.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/SingleBucketAggregation.java new file mode 100644 index 00000000000..9e80d2838e0 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/SingleBucketAggregation.java @@ -0,0 +1,105 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket; + +import org.elasticsearch.common.io.stream.StreamInput; +import org.elasticsearch.common.io.stream.StreamOutput; +import org.elasticsearch.common.xcontent.XContentBuilder; +import org.elasticsearch.search.aggregations.InternalAggregation; +import org.elasticsearch.search.aggregations.InternalAggregations; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; + +/** + * A base class for all the single bucket aggregations. + */ +@SuppressWarnings("unchecked") +public abstract class SingleBucketAggregation> extends InternalAggregation { + + protected long docCount; + protected InternalAggregations aggregations; + + protected SingleBucketAggregation() {} // for serialization + + /** + * Creates a single bucket aggregation. + * + * @param name The aggregation name. + * @param docCount The document count in the single bucket. + * @param aggregations The already built sub-aggregations that are associated with the bucket. + */ + protected SingleBucketAggregation(String name, long docCount, InternalAggregations aggregations) { + super(name); + this.docCount = docCount; + this.aggregations = aggregations; + } + + public long getDocCount() { + return docCount; + } + + public InternalAggregations getAggregations() { + return aggregations; + } + + @Override + public InternalAggregation reduce(ReduceContext reduceContext) { + List aggregations = reduceContext.aggregations(); + if (aggregations.size() == 1) { + return aggregations.get(0); + } + B reduced = null; + List subAggregationsList = new ArrayList(aggregations.size()); + for (InternalAggregation aggregation : aggregations) { + if (reduced == null) { + reduced = (B) aggregation; + } else { + this.docCount += ((B) aggregation).docCount; + } + subAggregationsList.add(((B) aggregation).aggregations); + } + reduced.aggregations = InternalAggregations.reduce(subAggregationsList, reduceContext.cacheRecycler()); + return reduced; + } + + @Override + public void readFrom(StreamInput in) throws IOException { + name = in.readString(); + docCount = in.readVLong(); + aggregations = InternalAggregations.readAggregations(in); + } + + @Override + public void writeTo(StreamOutput out) throws IOException { + out.writeString(name); + out.writeVLong(docCount); + aggregations.writeTo(out); + } + + @Override + public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException { + builder.startObject(name); + builder.field(CommonFields.DOC_COUNT, docCount); + aggregations.toXContentInternal(builder, params); + return builder.endObject(); + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/SingleBucketAggregator.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/SingleBucketAggregator.java new file mode 100644 index 00000000000..f33b8184e7c --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/SingleBucketAggregator.java @@ -0,0 +1,41 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket; + +import org.elasticsearch.search.aggregations.Aggregator; +import org.elasticsearch.search.aggregations.support.AggregationContext; +import org.elasticsearch.search.aggregations.AggregatorFactories; + +/** + * A bucket aggregator that doesn't create new buckets. + */ +public abstract class SingleBucketAggregator extends BucketsAggregator { + + protected SingleBucketAggregator(String name, AggregatorFactories factories, + AggregationContext aggregationContext, Aggregator parent) { + super(name, BucketAggregationMode.MULTI_BUCKETS, factories, parent == null ? 1 : parent.estimatedBucketCount(), aggregationContext, parent); + } + + @Override + public boolean shouldCollect() { + return true; + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/filter/Filter.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/filter/Filter.java new file mode 100644 index 00000000000..3b1a3533e65 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/filter/Filter.java @@ -0,0 +1,34 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.filter; + +import org.elasticsearch.search.aggregations.Aggregation; +import org.elasticsearch.search.aggregations.Aggregations; + +/** + * + */ +public interface Filter extends Aggregation { + + long getDocCount(); + + Aggregations getAggregations(); + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/filter/FilterAggregationBuilder.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/filter/FilterAggregationBuilder.java new file mode 100644 index 00000000000..c7e307cad36 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/filter/FilterAggregationBuilder.java @@ -0,0 +1,34 @@ +package org.elasticsearch.search.aggregations.bucket.filter; + +import org.elasticsearch.common.xcontent.XContentBuilder; +import org.elasticsearch.index.query.FilterBuilder; +import org.elasticsearch.search.aggregations.AggregationBuilder; +import org.elasticsearch.search.builder.SearchSourceBuilderException; + +import java.io.IOException; + +/** + * + */ +public class FilterAggregationBuilder extends AggregationBuilder { + + private FilterBuilder filter; + + public FilterAggregationBuilder(String name) { + super(name, InternalFilter.TYPE.name()); + } + + public FilterAggregationBuilder filter(FilterBuilder filter) { + this.filter = filter; + return this; + } + + @Override + protected XContentBuilder internalXContent(XContentBuilder builder, Params params) throws IOException { + if (filter == null) { + throw new SearchSourceBuilderException("filter must be set on filter aggregation [" + name + "]"); + } + filter.toXContent(builder, params); + return builder; + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/filter/FilterAggregator.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/filter/FilterAggregator.java new file mode 100644 index 00000000000..9979f4e2ecb --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/filter/FilterAggregator.java @@ -0,0 +1,100 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.filter; + +import org.apache.lucene.index.AtomicReaderContext; +import org.apache.lucene.search.Filter; +import org.apache.lucene.util.Bits; +import org.elasticsearch.common.lucene.ReaderContextAware; +import org.elasticsearch.common.lucene.docset.DocIdSets; +import org.elasticsearch.search.aggregations.AggregationExecutionException; +import org.elasticsearch.search.aggregations.Aggregator; +import org.elasticsearch.search.aggregations.InternalAggregation; +import org.elasticsearch.search.aggregations.bucket.SingleBucketAggregator; +import org.elasticsearch.search.aggregations.support.AggregationContext; +import org.elasticsearch.search.aggregations.AggregatorFactories; +import org.elasticsearch.search.aggregations.AggregatorFactory; + +import java.io.IOException; + +/** + * Aggregate all docs that match a filter. + */ +public class FilterAggregator extends SingleBucketAggregator implements ReaderContextAware { + + private final Filter filter; + + private Bits bits; + + public FilterAggregator(String name, + org.apache.lucene.search.Filter filter, + AggregatorFactories factories, + AggregationContext aggregationContext, + Aggregator parent) { + super(name, factories, aggregationContext, parent); + this.filter = filter; + } + + @Override + public void setNextReader(AtomicReaderContext reader) { + try { + bits = DocIdSets.toSafeBits(reader.reader(), filter.getDocIdSet(reader, reader.reader().getLiveDocs())); + } catch (IOException ioe) { + throw new AggregationExecutionException("Failed to aggregate filter aggregator [" + name + "]", ioe); + } + } + + @Override + public void collect(int doc, long owningBucketOrdinal) throws IOException { + if (bits.get(doc)) { + collectBucket(doc, owningBucketOrdinal); + } + } + + @Override + public InternalAggregation buildAggregation(long owningBucketOrdinal) { + return new InternalFilter(name, bucketDocCount(owningBucketOrdinal), bucketAggregations(owningBucketOrdinal)); + } + + @Override + public InternalAggregation buildEmptyAggregation() { + return new InternalFilter(name, 0, buildEmptySubAggregations()); + } + + public static class Factory extends AggregatorFactory { + + private org.apache.lucene.search.Filter filter; + + public Factory(String name, Filter filter) { + super(name, InternalFilter.TYPE.name()); + this.filter = filter; + } + + @Override + public Aggregator create(AggregationContext context, Aggregator parent, long expectedBucketsCount) { + FilterAggregator aggregator = new FilterAggregator(name, filter, factories, context, parent); + context.registerReaderContextAware(aggregator); + return aggregator; + } + + } +} + + diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/filter/FilterParser.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/filter/FilterParser.java new file mode 100644 index 00000000000..d6a02a29efc --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/filter/FilterParser.java @@ -0,0 +1,46 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.filter; + +import org.elasticsearch.common.xcontent.XContentParser; +import org.elasticsearch.index.query.ParsedFilter; +import org.elasticsearch.search.aggregations.Aggregator; +import org.elasticsearch.search.aggregations.AggregatorFactory; +import org.elasticsearch.search.internal.SearchContext; + +import java.io.IOException; + +/** + * + */ +public class FilterParser implements Aggregator.Parser { + + @Override + public String type() { + return InternalFilter.TYPE.name(); + } + + @Override + public AggregatorFactory parse(String aggregationName, XContentParser parser, SearchContext context) throws IOException { + ParsedFilter filter = context.queryParserService().parseInnerFilter(parser); + return new FilterAggregator.Factory(aggregationName, filter.filter()); + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/filter/InternalFilter.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/filter/InternalFilter.java new file mode 100644 index 00000000000..0144319de08 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/filter/InternalFilter.java @@ -0,0 +1,60 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.filter; + +import org.elasticsearch.common.io.stream.StreamInput; +import org.elasticsearch.search.aggregations.AggregationStreams; +import org.elasticsearch.search.aggregations.InternalAggregations; +import org.elasticsearch.search.aggregations.bucket.SingleBucketAggregation; + +import java.io.IOException; + +/** +* +*/ +public class InternalFilter extends SingleBucketAggregation implements Filter { + + public final static Type TYPE = new Type("filter"); + + public final static AggregationStreams.Stream STREAM = new AggregationStreams.Stream() { + @Override + public InternalFilter readResult(StreamInput in) throws IOException { + InternalFilter result = new InternalFilter(); + result.readFrom(in); + return result; + } + }; + + public static void registerStreams() { + AggregationStreams.registerStream(STREAM, TYPE.stream()); + } + + InternalFilter() {} // for serialization + + InternalFilter(String name, long docCount, InternalAggregations subAggregations) { + super(name, docCount, subAggregations); + } + + @Override + public Type type() { + return TYPE; + } + +} \ No newline at end of file diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/global/Global.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/global/Global.java new file mode 100644 index 00000000000..cf2951dffaf --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/global/Global.java @@ -0,0 +1,34 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.global; + +import org.elasticsearch.search.aggregations.Aggregation; +import org.elasticsearch.search.aggregations.Aggregations; + +/** + * + */ +public interface Global extends Aggregation { + + long getDocCount(); + + Aggregations getAggregations(); + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/global/GlobalAggregator.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/global/GlobalAggregator.java new file mode 100644 index 00000000000..24ad2f6c894 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/global/GlobalAggregator.java @@ -0,0 +1,74 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.global; + +import org.elasticsearch.search.aggregations.AggregationExecutionException; +import org.elasticsearch.search.aggregations.Aggregator; +import org.elasticsearch.search.aggregations.InternalAggregation; +import org.elasticsearch.search.aggregations.bucket.SingleBucketAggregator; +import org.elasticsearch.search.aggregations.support.AggregationContext; +import org.elasticsearch.search.aggregations.AggregatorFactories; +import org.elasticsearch.search.aggregations.AggregatorFactory; + +import java.io.IOException; + +/** + * + */ +public class GlobalAggregator extends SingleBucketAggregator { + + public GlobalAggregator(String name, AggregatorFactories subFactories, AggregationContext aggregationContext) { + super(name, subFactories, aggregationContext, null); + } + + @Override + public void collect(int doc, long owningBucketOrdinal) throws IOException { + assert owningBucketOrdinal == 0 : "global aggregator can only be a top level aggregator"; + collectBucket(doc, owningBucketOrdinal); + } + + @Override + public InternalAggregation buildAggregation(long owningBucketOrdinal) { + assert owningBucketOrdinal == 0 : "global aggregator can only be a top level aggregator"; + return new InternalGlobal(name, bucketDocCount(owningBucketOrdinal), bucketAggregations(owningBucketOrdinal)); + } + + @Override + public InternalAggregation buildEmptyAggregation() { + throw new UnsupportedOperationException("global aggregations cannot serve as sub-aggregations, hence should never be called on #buildEmptyAggregations"); + } + + public static class Factory extends AggregatorFactory { + + public Factory(String name) { + super(name, InternalGlobal.TYPE.name()); + } + + @Override + public Aggregator create(AggregationContext context, Aggregator parent, long expectedBucketsCount) { + if (parent != null) { + throw new AggregationExecutionException("Aggregation [" + parent.name() + "] cannot have a global " + + "sub-aggregation [" + name + "]. Global aggregations can only be defined as top level aggregations"); + } + return new GlobalAggregator(name, factories, context); + } + + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/global/GlobalBuilder.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/global/GlobalBuilder.java new file mode 100644 index 00000000000..5bdebe3aab1 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/global/GlobalBuilder.java @@ -0,0 +1,21 @@ +package org.elasticsearch.search.aggregations.bucket.global; + +import org.elasticsearch.common.xcontent.XContentBuilder; +import org.elasticsearch.search.aggregations.AggregationBuilder; + +import java.io.IOException; + +/** + * + */ +public class GlobalBuilder extends AggregationBuilder { + + public GlobalBuilder(String name) { + super(name, InternalGlobal.TYPE.name()); + } + + @Override + protected XContentBuilder internalXContent(XContentBuilder builder, Params params) throws IOException { + return builder.startObject().endObject(); + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/global/GlobalParser.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/global/GlobalParser.java new file mode 100644 index 00000000000..ff0c0d996aa --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/global/GlobalParser.java @@ -0,0 +1,45 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.global; + +import org.elasticsearch.common.xcontent.XContentParser; +import org.elasticsearch.search.aggregations.Aggregator; +import org.elasticsearch.search.aggregations.AggregatorFactory; +import org.elasticsearch.search.internal.SearchContext; + +import java.io.IOException; + +/** + * + */ +public class GlobalParser implements Aggregator.Parser { + + @Override + public String type() { + return InternalGlobal.TYPE.name(); + } + + @Override + public AggregatorFactory parse(String aggregationName, XContentParser parser, SearchContext context) throws IOException { + parser.nextToken(); + return new GlobalAggregator.Factory(aggregationName); + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/global/InternalGlobal.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/global/InternalGlobal.java new file mode 100644 index 00000000000..28a1fcce8ff --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/global/InternalGlobal.java @@ -0,0 +1,60 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.global; + +import org.elasticsearch.common.io.stream.StreamInput; +import org.elasticsearch.search.aggregations.AggregationStreams; +import org.elasticsearch.search.aggregations.InternalAggregations; +import org.elasticsearch.search.aggregations.bucket.SingleBucketAggregation; + +import java.io.IOException; + +/** + * A global scope get (the document set on which we aggregate is all documents in the search context (ie. index + type) + * regardless the query. + */ +public class InternalGlobal extends SingleBucketAggregation implements Global { + + public final static Type TYPE = new Type("global"); + + public final static AggregationStreams.Stream STREAM = new AggregationStreams.Stream() { + @Override + public InternalGlobal readResult(StreamInput in) throws IOException { + InternalGlobal result = new InternalGlobal(); + result.readFrom(in); + return result; + } + }; + + public static void registerStreams() { + AggregationStreams.registerStream(STREAM, TYPE.stream()); + } + + public InternalGlobal() {} // for serialization + + public InternalGlobal(String name, long docCount, InternalAggregations aggregations) { + super(name, docCount, aggregations); + } + + @Override + public Type type() { + return TYPE; + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/AbstractHistogramBase.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/AbstractHistogramBase.java new file mode 100644 index 00000000000..c6b6689a85c --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/AbstractHistogramBase.java @@ -0,0 +1,343 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.histogram; + +import com.carrotsearch.hppc.LongObjectOpenHashMap; +import com.google.common.collect.Lists; +import org.apache.lucene.util.CollectionUtil; +import org.elasticsearch.cache.recycler.CacheRecycler; +import org.elasticsearch.common.io.stream.StreamInput; +import org.elasticsearch.common.io.stream.StreamOutput; +import org.elasticsearch.common.io.stream.Streamable; +import org.elasticsearch.common.recycler.Recycler; +import org.elasticsearch.common.rounding.Rounding; +import org.elasticsearch.common.text.StringText; +import org.elasticsearch.common.text.Text; +import org.elasticsearch.common.xcontent.ToXContent; +import org.elasticsearch.common.xcontent.XContentBuilder; +import org.elasticsearch.search.aggregations.Aggregations; +import org.elasticsearch.search.aggregations.InternalAggregation; +import org.elasticsearch.search.aggregations.InternalAggregations; +import org.elasticsearch.search.aggregations.support.numeric.ValueFormatter; +import org.elasticsearch.search.aggregations.support.numeric.ValueFormatterStreams; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Iterator; +import java.util.List; +import java.util.ListIterator; + +/** + * An internal implementation of {@link HistogramBase} + */ +abstract class AbstractHistogramBase extends InternalAggregation implements HistogramBase, ToXContent, Streamable { + + public static class Bucket implements HistogramBase.Bucket { + + private long key; + private long docCount; + private InternalAggregations aggregations; + + public Bucket(long key, long docCount, InternalAggregations aggregations) { + this.key = key; + this.docCount = docCount; + this.aggregations = aggregations; + } + + @Override + public long getKey() { + return key; + } + + @Override + public long getDocCount() { + return docCount; + } + + @Override + public Aggregations getAggregations() { + return aggregations; + } + + Bucket reduce(List buckets, CacheRecycler cacheRecycler) { + if (buckets.size() == 1) { + return buckets.get(0); + } + List aggregations = new ArrayList(buckets.size()); + Bucket reduced = null; + for (Bucket bucket : buckets) { + if (reduced == null) { + reduced = bucket; + } else { + reduced.docCount += bucket.docCount; + } + aggregations.add((InternalAggregations) bucket.getAggregations()); + } + reduced.aggregations = InternalAggregations.reduce(aggregations, cacheRecycler); + return reduced; + } + } + + static class EmptyBucketInfo { + final Rounding rounding; + final InternalAggregations subAggregations; + + EmptyBucketInfo(Rounding rounding, InternalAggregations subAggregations) { + this.rounding = rounding; + this.subAggregations = subAggregations; + } + + public static EmptyBucketInfo readFrom(StreamInput in) throws IOException { + return new EmptyBucketInfo(Rounding.Streams.read(in), InternalAggregations.readAggregations(in)); + } + + public static void writeTo(EmptyBucketInfo info, StreamOutput out) throws IOException { + Rounding.Streams.write(info.rounding, out); + info.subAggregations.writeTo(out); + } + } + + public static interface Factory { + + String type(); + + AbstractHistogramBase create(String name, List buckets, InternalOrder order, EmptyBucketInfo emptyBucketInfo, ValueFormatter formatter, boolean keyed); + + Bucket createBucket(long key, long docCount, InternalAggregations aggregations); + + } + + private List buckets; + private LongObjectOpenHashMap bucketsMap; + private InternalOrder order; + private ValueFormatter formatter; + private boolean keyed; + private EmptyBucketInfo emptyBucketInfo; + + protected AbstractHistogramBase() {} // for serialization + + protected AbstractHistogramBase(String name, List buckets, InternalOrder order, EmptyBucketInfo emptyBucketInfo, ValueFormatter formatter, boolean keyed) { + super(name); + this.buckets = buckets; + this.order = order; + this.emptyBucketInfo = emptyBucketInfo; + this.formatter = formatter; + this.keyed = keyed; + } + + @Override + public Iterator iterator() { + return buckets.iterator(); + } + + @Override + public List buckets() { + return buckets; + } + + @Override + public B getByKey(long key) { + if (bucketsMap == null) { + bucketsMap = new LongObjectOpenHashMap(buckets.size()); + for (HistogramBase.Bucket bucket : buckets) { + bucketsMap.put(bucket.getKey(), bucket); + } + } + return (B) bucketsMap.get(key); + } + + // TODO extract the reduce logic to a strategy class and have it configurable at request time (two possible strategies - total & delta) + + @Override + public InternalAggregation reduce(ReduceContext reduceContext) { + List aggregations = reduceContext.aggregations(); + if (aggregations.size() == 1) { + + if (emptyBucketInfo == null) { + return aggregations.get(0); + } + + // we need to fill the gaps with empty buckets + AbstractHistogramBase histo = (AbstractHistogramBase) aggregations.get(0); + CollectionUtil.introSort(histo.buckets, order.asc ? InternalOrder.KEY_ASC.comparator() : InternalOrder.KEY_DESC.comparator()); + List list = order.asc ? histo.buckets : Lists.reverse(histo.buckets); + HistogramBase.Bucket prevBucket = null; + ListIterator iter = list.listIterator(); + while (iter.hasNext()) { + // look ahead on the next bucket without advancing the iter + // so we'll be able to insert elements at the right position + HistogramBase.Bucket nextBucket = list.get(iter.nextIndex()); + if (prevBucket != null) { + long key = emptyBucketInfo.rounding.nextRoundingValue(prevBucket.getKey()); + while (key != nextBucket.getKey()) { + iter.add(createBucket(key, 0, emptyBucketInfo.subAggregations)); + key = emptyBucketInfo.rounding.nextRoundingValue(key); + } + } + prevBucket = iter.next(); + } + + if (order != InternalOrder.KEY_ASC && order != InternalOrder.KEY_DESC) { + CollectionUtil.introSort(histo.buckets, order.comparator()); + } + + return histo; + + } + + AbstractHistogramBase reduced = (AbstractHistogramBase) aggregations.get(0); + + Recycler.V>> bucketsByKey = reduceContext.cacheRecycler().longObjectMap(-1); + for (InternalAggregation aggregation : aggregations) { + AbstractHistogramBase histogram = (AbstractHistogramBase) aggregation; + for (B bucket : histogram.buckets) { + List bucketList = bucketsByKey.v().get(((Bucket) bucket).key); + if (bucketList == null) { + bucketList = new ArrayList(aggregations.size()); + bucketsByKey.v().put(((Bucket) bucket).key, bucketList); + } + bucketList.add((Bucket) bucket); + } + } + + List reducedBuckets = new ArrayList(bucketsByKey.v().size()); + Object[] buckets = bucketsByKey.v().values; + boolean[] allocated = bucketsByKey.v().allocated; + for (int i = 0; i < allocated.length; i++) { + if (allocated[i]) { + Bucket bucket = ((List) buckets[i]).get(0).reduce(((List) buckets[i]), reduceContext.cacheRecycler()); + reducedBuckets.add(bucket); + } + } + bucketsByKey.release(); + + + + // adding empty buckets in needed + if (emptyBucketInfo != null) { + CollectionUtil.introSort(reducedBuckets, order.asc ? InternalOrder.KEY_ASC.comparator() : InternalOrder.KEY_DESC.comparator()); + List list = order.asc ? reducedBuckets : Lists.reverse(reducedBuckets); + HistogramBase.Bucket prevBucket = null; + ListIterator iter = list.listIterator(); + while (iter.hasNext()) { + HistogramBase.Bucket nextBucket = list.get(iter.nextIndex()); + if (prevBucket != null) { + long key = emptyBucketInfo.rounding.nextRoundingValue(prevBucket.getKey()); + while (key != nextBucket.getKey()) { + iter.add(createBucket(key, 0, emptyBucketInfo.subAggregations)); + key = emptyBucketInfo.rounding.nextRoundingValue(key); + } + } + prevBucket = iter.next(); + } + + if (order != InternalOrder.KEY_ASC && order != InternalOrder.KEY_DESC) { + CollectionUtil.introSort(reducedBuckets, order.comparator()); + } + + } else { + CollectionUtil.introSort(reducedBuckets, order.comparator()); + } + + + reduced.buckets = reducedBuckets; + return reduced; + } + + protected B createBucket(long key, long docCount, InternalAggregations aggregations) { + return (B) new Bucket(key, docCount, aggregations); + } + + @Override + public void readFrom(StreamInput in) throws IOException { + name = in.readString(); + order = InternalOrder.Streams.readOrder(in); + if (in.readBoolean()) { + emptyBucketInfo = EmptyBucketInfo.readFrom(in); + } + formatter = ValueFormatterStreams.readOptional(in); + keyed = in.readBoolean(); + int size = in.readVInt(); + List buckets = new ArrayList(size); + for (int i = 0; i < size; i++) { + buckets.add(createBucket(in.readLong(), in.readVLong(), InternalAggregations.readAggregations(in))); + } + this.buckets = buckets; + this.bucketsMap = null; + } + + @Override + public void writeTo(StreamOutput out) throws IOException { + out.writeString(name); + InternalOrder.Streams.writeOrder(order, out); + if (emptyBucketInfo == null) { + out.writeBoolean(false); + } else { + out.writeBoolean(true); + EmptyBucketInfo.writeTo(emptyBucketInfo, out); + } + ValueFormatterStreams.writeOptional(formatter, out); + out.writeBoolean(keyed); + out.writeVInt(buckets.size()); + for (HistogramBase.Bucket bucket : buckets) { + out.writeLong(((Bucket) bucket).key); + out.writeVLong(((Bucket) bucket).docCount); + ((Bucket) bucket).aggregations.writeTo(out); + } + } + + @Override + public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException { + if (keyed) { + builder.startObject(name); + } else { + builder.startArray(name); + } + + for (HistogramBase.Bucket bucket : buckets) { + if (formatter != null) { + Text keyTxt = new StringText(formatter.format(bucket.getKey())); + if (keyed) { + builder.startObject(keyTxt.string()); + } else { + builder.startObject(); + } + builder.field(CommonFields.KEY_AS_STRING, keyTxt); + } else { + if (keyed) { + builder.startObject(String.valueOf(bucket.getKey())); + } else { + builder.startObject(); + } + } + builder.field(CommonFields.KEY, ((Bucket) bucket).key); + builder.field(CommonFields.DOC_COUNT, ((Bucket) bucket).docCount); + ((Bucket) bucket).aggregations.toXContentInternal(builder, params); + builder.endObject(); + } + + if (keyed) { + builder.endObject(); + } else { + builder.endArray(); + } + return builder; + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/DateHistogram.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/DateHistogram.java new file mode 100644 index 00000000000..f7e907de2b1 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/DateHistogram.java @@ -0,0 +1,77 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.histogram; + +import org.joda.time.DateTime; + +/** + * + */ +public interface DateHistogram extends HistogramBase { + + static interface Bucket extends HistogramBase.Bucket { + + DateTime getKeyAsDate(); + + } + + static class Interval { + + public static final Interval SECOND = new Interval("1s"); + public static final Interval MINUTE = new Interval("1m"); + public static final Interval HOUR = new Interval("1h"); + public static final Interval DAY = new Interval("1d"); + public static final Interval WEEK = new Interval("1w"); + public static final Interval MONTH = new Interval("1M"); + public static final Interval QUARTER = new Interval("1q"); + public static final Interval YEAR = new Interval("1y"); + + public static Interval seconds(int sec) { + return new Interval(sec + "s"); + } + + public static Interval minutes(int min) { + return new Interval(min + "m"); + } + + public static Interval hours(int hours) { + return new Interval(hours + "h"); + } + + public static Interval days(int days) { + return new Interval(days + "d"); + } + + public static Interval week(int weeks) { + return new Interval(weeks + "w"); + } + + private final String expression; + + public Interval(String expression) { + this.expression = expression; + } + + @Override + public String toString() { + return expression; + } + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/DateHistogramBuilder.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/DateHistogramBuilder.java new file mode 100644 index 00000000000..122e1a28230 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/DateHistogramBuilder.java @@ -0,0 +1,115 @@ +package org.elasticsearch.search.aggregations.bucket.histogram; + +import org.elasticsearch.common.unit.TimeValue; +import org.elasticsearch.common.xcontent.XContentBuilder; +import org.elasticsearch.search.aggregations.ValuesSourceAggregationBuilder; +import org.elasticsearch.search.builder.SearchSourceBuilderException; + +import java.io.IOException; + +/** + * + */ +public class DateHistogramBuilder extends ValuesSourceAggregationBuilder { + + private Object interval; + private HistogramBase.Order order; + private String preZone; + private String postZone; + private boolean preZoneAdjustLargeInterval; + long preOffset = 0; + long postOffset = 0; + float factor = 1.0f; + + public DateHistogramBuilder(String name) { + super(name, InternalDateHistogram.TYPE.name()); + } + + public DateHistogramBuilder interval(long interval) { + this.interval = interval; + return this; + } + + public DateHistogramBuilder interval(DateHistogram.Interval interval) { + this.interval = interval; + return this; + } + + public DateHistogramBuilder order(DateHistogram.Order order) { + this.order = order; + return this; + } + + public DateHistogramBuilder preZone(String preZone) { + this.preZone = preZone; + return this; + } + + public DateHistogramBuilder postZone(String postZone) { + this.postZone = postZone; + return this; + } + + public DateHistogramBuilder preZoneAdjustLargeInterval(boolean preZoneAdjustLargeInterval) { + this.preZoneAdjustLargeInterval = preZoneAdjustLargeInterval; + return this; + } + + public DateHistogramBuilder preOffset(long preOffset) { + this.preOffset = preOffset; + return this; + } + + public DateHistogramBuilder postOffset(long postOffset) { + this.postOffset = postOffset; + return this; + } + + public DateHistogramBuilder factor(float factor) { + this.factor = factor; + return this; + } + + @Override + protected XContentBuilder doInternalXContent(XContentBuilder builder, Params params) throws IOException { + if (interval == null) { + throw new SearchSourceBuilderException("[interval] must be defined for histogram aggregation [" + name + "]"); + } + if (interval instanceof Number) { + interval = TimeValue.timeValueMillis(((Number) interval).longValue()).toString(); + } + builder.field("interval", interval); + + if (order != null) { + builder.field("order"); + order.toXContent(builder, params); + } + + if (preZone != null) { + builder.field("pre_zone", preZone); + } + + if (postZone != null) { + builder.field("post_zone", postZone); + } + + if (preZoneAdjustLargeInterval) { + builder.field("pre_zone_adjust_large_interval", true); + } + + if (preOffset != 0) { + builder.field("pre_offset", preOffset); + } + + if (postOffset != 0) { + builder.field("post_offset", postOffset); + } + + if (factor != 1.0f) { + builder.field("factor", factor); + } + + return builder; + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/DateHistogramParser.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/DateHistogramParser.java new file mode 100644 index 00000000000..4131617902a --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/DateHistogramParser.java @@ -0,0 +1,259 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.histogram; + +import com.google.common.collect.ImmutableMap; +import org.elasticsearch.common.collect.MapBuilder; +import org.elasticsearch.common.joda.DateMathParser; +import org.elasticsearch.common.rounding.DateTimeUnit; +import org.elasticsearch.common.rounding.TimeZoneRounding; +import org.elasticsearch.common.unit.TimeValue; +import org.elasticsearch.common.xcontent.XContentParser; +import org.elasticsearch.index.fielddata.IndexFieldData; +import org.elasticsearch.index.mapper.FieldMapper; +import org.elasticsearch.index.mapper.core.DateFieldMapper; +import org.elasticsearch.script.SearchScript; +import org.elasticsearch.search.SearchParseException; +import org.elasticsearch.search.aggregations.Aggregator; +import org.elasticsearch.search.aggregations.support.FieldContext; +import org.elasticsearch.search.aggregations.support.ValuesSourceConfig; +import org.elasticsearch.search.aggregations.support.numeric.NumericValuesSource; +import org.elasticsearch.search.aggregations.support.numeric.ValueFormatter; +import org.elasticsearch.search.aggregations.support.numeric.ValueParser; +import org.elasticsearch.search.aggregations.AggregatorFactory; +import org.elasticsearch.search.internal.SearchContext; +import org.joda.time.DateTimeZone; + +import java.io.IOException; +import java.util.Map; + +/** + * + */ +public class DateHistogramParser implements Aggregator.Parser { + + private final ImmutableMap dateFieldUnits; + + public DateHistogramParser() { + dateFieldUnits = MapBuilder.newMapBuilder() + .put("year", DateTimeUnit.YEAR_OF_CENTURY) + .put("1y", DateTimeUnit.YEAR_OF_CENTURY) + .put("quarter", DateTimeUnit.QUARTER) + .put("1q", DateTimeUnit.QUARTER) + .put("month", DateTimeUnit.MONTH_OF_YEAR) + .put("1M", DateTimeUnit.MONTH_OF_YEAR) + .put("week", DateTimeUnit.WEEK_OF_WEEKYEAR) + .put("1w", DateTimeUnit.WEEK_OF_WEEKYEAR) + .put("day", DateTimeUnit.DAY_OF_MONTH) + .put("1d", DateTimeUnit.DAY_OF_MONTH) + .put("hour", DateTimeUnit.HOUR_OF_DAY) + .put("1h", DateTimeUnit.HOUR_OF_DAY) + .put("minute", DateTimeUnit.MINUTES_OF_HOUR) + .put("1m", DateTimeUnit.MINUTES_OF_HOUR) + .put("second", DateTimeUnit.SECOND_OF_MINUTE) + .put("1s", DateTimeUnit.SECOND_OF_MINUTE) + .immutableMap(); + } + + @Override + public String type() { + return InternalDateHistogram.TYPE.name(); + } + + @Override + public AggregatorFactory parse(String aggregationName, XContentParser parser, SearchContext context) throws IOException { + + ValuesSourceConfig config = new ValuesSourceConfig(NumericValuesSource.class); + + String field = null; + String script = null; + String scriptLang = null; + Map scriptParams = null; + boolean keyed = false; + boolean computeEmptyBuckets = false; + InternalOrder order = (InternalOrder) Histogram.Order.KEY_ASC; + String interval = null; + boolean preZoneAdjustLargeInterval = false; + DateTimeZone preZone = DateTimeZone.UTC; + DateTimeZone postZone = DateTimeZone.UTC; + String format = null; + long preOffset = 0; + long postOffset = 0; + boolean assumeSorted = false; + + XContentParser.Token token; + String currentFieldName = null; + while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) { + if (token == XContentParser.Token.FIELD_NAME) { + currentFieldName = parser.currentName(); + } else if (token == XContentParser.Token.VALUE_STRING) { + if ("field".equals(currentFieldName)) { + field = parser.text(); + } else if ("script".equals(currentFieldName)) { + script = parser.text(); + } else if ("script_lang".equals(currentFieldName) || "scriptLang".equals(currentFieldName)) { + scriptLang = parser.text(); + } else if ("time_zone".equals(currentFieldName) || "timeZone".equals(currentFieldName)) { + preZone = parseZone(parser, token); + } else if ("pre_zone".equals(currentFieldName) || "preZone".equals(currentFieldName)) { + preZone = parseZone(parser, token); + } else if ("pre_zone_adjust_large_interval".equals(currentFieldName) || "preZoneAdjustLargeInterval".equals(currentFieldName)) { + preZoneAdjustLargeInterval = parser.booleanValue(); + } else if ("post_zone".equals(currentFieldName) || "postZone".equals(currentFieldName)) { + postZone = parseZone(parser, token); + } else if ("pre_offset".equals(currentFieldName) || "preOffset".equals(currentFieldName)) { + preOffset = parseOffset(parser.text()); + } else if ("post_offset".equals(currentFieldName) || "postOffset".equals(currentFieldName)) { + postOffset = parseOffset(parser.text()); + } else if ("interval".equals(currentFieldName)) { + interval = parser.text(); + } else if ("format".equals(currentFieldName)) { + format = parser.text(); + } + } else if (token == XContentParser.Token.VALUE_BOOLEAN) { + if ("keyed".equals(currentFieldName)) { + keyed = parser.booleanValue(); + } else if ("compute_empty_buckets".equals(currentFieldName) || "computeEmptyBuckets".equals(currentFieldName)) { + computeEmptyBuckets = parser.booleanValue(); + } else if ("script_values_sorted".equals(currentFieldName)) { + assumeSorted = parser.booleanValue(); + } + } else if (token == XContentParser.Token.START_OBJECT) { + if ("params".equals(currentFieldName)) { + scriptParams = parser.map(); + } else if ("order".equals(currentFieldName)) { + while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) { + if (token == XContentParser.Token.FIELD_NAME) { + currentFieldName = parser.currentName(); + } else if (token == XContentParser.Token.VALUE_STRING) { + String dir = parser.text(); + boolean asc = "asc".equals(dir); + order = resolveOrder(currentFieldName, asc); + //TODO should we throw an error if the value is not "asc" or "desc"??? + } + } + } + } + } + + if (interval == null) { + throw new SearchParseException(context, "Missing required field [interval] for histogram aggregation [" + aggregationName + "]"); + } + + SearchScript searchScript = null; + if (script != null) { + searchScript = context.scriptService().search(context.lookup(), scriptLang, script, scriptParams); + config.script(searchScript); + } + + if (!assumeSorted) { + // we need values to be sorted and unique for efficiency + config.ensureSorted(true); + } + + TimeZoneRounding.Builder tzRoundingBuilder; + DateTimeUnit dateTimeUnit = dateFieldUnits.get(interval); + if (dateTimeUnit != null) { + tzRoundingBuilder = TimeZoneRounding.builder(dateTimeUnit); + } else { + // the interval is a time value? + tzRoundingBuilder = TimeZoneRounding.builder(TimeValue.parseTimeValue(interval, null)); + } + + TimeZoneRounding rounding = tzRoundingBuilder + .preZone(preZone).postZone(postZone) + .preZoneAdjustLargeInterval(preZoneAdjustLargeInterval) + .preOffset(preOffset).postOffset(postOffset) + .build(); + + if (format != null) { + config.formatter(new ValueFormatter.DateTime(format)); + } + + if (field == null) { + + if (searchScript != null) { + ValueParser valueParser = new ValueParser.DateMath(new DateMathParser(DateFieldMapper.Defaults.DATE_TIME_FORMATTER, DateFieldMapper.Defaults.TIME_UNIT)); + config.parser(valueParser); + return new HistogramAggregator.Factory(aggregationName, config, rounding, order, keyed, computeEmptyBuckets, InternalDateHistogram.FACTORY); + } + + // falling back on the get field data context + return new HistogramAggregator.Factory(aggregationName, config, rounding, order, keyed, computeEmptyBuckets, InternalDateHistogram.FACTORY); + } + + FieldMapper mapper = context.smartNameFieldMapper(field); + if (mapper == null) { + config.unmapped(true); + return new HistogramAggregator.Factory(aggregationName, config, rounding, order, keyed, computeEmptyBuckets, InternalDateHistogram.FACTORY); + } + + if (!(mapper instanceof DateFieldMapper)) { + throw new SearchParseException(context, "date histogram can only be aggregated on date fields but [" + field + "] is not a date field"); + } + + IndexFieldData indexFieldData = context.fieldData().getForField(mapper); + config.fieldContext(new FieldContext(field, indexFieldData)); + return new HistogramAggregator.Factory(aggregationName, config, rounding, order, keyed, computeEmptyBuckets, InternalDateHistogram.FACTORY); + } + + private static InternalOrder resolveOrder(String key, boolean asc) { + if ("_key".equals(key) || "_time".equals(key)) { + return (InternalOrder) (asc ? InternalOrder.KEY_ASC : InternalOrder.KEY_DESC); + } + if ("_count".equals(key)) { + return (InternalOrder) (asc ? InternalOrder.COUNT_ASC : InternalOrder.COUNT_DESC); + } + int i = key.indexOf('.'); + if (i < 0) { + return HistogramBase.Order.aggregation(key, asc); + } + return HistogramBase.Order.aggregation(key.substring(0, i), key.substring(i + 1), asc); + } + + private long parseOffset(String offset) throws IOException { + if (offset.charAt(0) == '-') { + return -TimeValue.parseTimeValue(offset.substring(1), null).millis(); + } + int beginIndex = offset.charAt(0) == '+' ? 1 : 0; + return TimeValue.parseTimeValue(offset.substring(beginIndex), null).millis(); + } + + private DateTimeZone parseZone(XContentParser parser, XContentParser.Token token) throws IOException { + if (token == XContentParser.Token.VALUE_NUMBER) { + return DateTimeZone.forOffsetHours(parser.intValue()); + } else { + String text = parser.text(); + int index = text.indexOf(':'); + if (index != -1) { + int beginIndex = text.charAt(0) == '+' ? 1 : 0; + // format like -02:30 + return DateTimeZone.forOffsetHoursMinutes( + Integer.parseInt(text.substring(beginIndex, index)), + Integer.parseInt(text.substring(index + 1)) + ); + } else { + // id, listed here: http://joda-time.sourceforge.net/timezones.html + return DateTimeZone.forID(text); + } + } + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/Histogram.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/Histogram.java new file mode 100644 index 00000000000..09e342c9cb2 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/Histogram.java @@ -0,0 +1,30 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.histogram; + +/** + * TODO should be renamed to NumericHistogram? This would also clean up the code and make it less confusing + */ +public interface Histogram extends HistogramBase { + + static interface Bucket extends HistogramBase.Bucket { + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/HistogramAggregator.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/HistogramAggregator.java new file mode 100644 index 00000000000..af9f063743f --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/HistogramAggregator.java @@ -0,0 +1,164 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.histogram; + +import org.apache.lucene.util.CollectionUtil; +import org.elasticsearch.common.inject.internal.Nullable; +import org.elasticsearch.common.rounding.Rounding; +import org.elasticsearch.index.fielddata.LongValues; +import org.elasticsearch.search.aggregations.Aggregator; +import org.elasticsearch.search.aggregations.InternalAggregation; +import org.elasticsearch.search.aggregations.bucket.BucketsAggregator; +import org.elasticsearch.search.aggregations.bucket.LongHash; +import org.elasticsearch.search.aggregations.support.AggregationContext; +import org.elasticsearch.search.aggregations.support.ValuesSourceConfig; +import org.elasticsearch.search.aggregations.support.numeric.NumericValuesSource; +import org.elasticsearch.search.aggregations.support.numeric.ValueFormatter; +import org.elasticsearch.search.aggregations.AggregatorFactories; +import org.elasticsearch.search.aggregations.support.ValueSourceAggregatorFactory; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; + +public class HistogramAggregator extends BucketsAggregator { + + private final static int INITIAL_CAPACITY = 50; // TODO sizing + + private final NumericValuesSource valuesSource; + private final Rounding rounding; + private final InternalOrder order; + private final boolean keyed; + private final boolean computeEmptyBuckets; + private final AbstractHistogramBase.Factory histogramFactory; + + private final LongHash bucketOrds; + + public HistogramAggregator(String name, + AggregatorFactories factories, + Rounding rounding, + InternalOrder order, + boolean keyed, + boolean computeEmptyBuckets, + @Nullable NumericValuesSource valuesSource, + AbstractHistogramBase.Factory histogramFactory, + AggregationContext aggregationContext, + Aggregator parent) { + + super(name, BucketAggregationMode.PER_BUCKET, factories, 50, aggregationContext, parent); + this.valuesSource = valuesSource; + this.rounding = rounding; + this.order = order; + this.keyed = keyed; + this.computeEmptyBuckets = computeEmptyBuckets; + this.histogramFactory = histogramFactory; + + bucketOrds = new LongHash(INITIAL_CAPACITY); + } + + @Override + public boolean shouldCollect() { + return valuesSource != null; + } + + @Override + public void collect(int doc, long owningBucketOrdinal) throws IOException { + assert owningBucketOrdinal == 0; + final LongValues values = valuesSource.longValues(); + final int valuesCount = values.setDocument(doc); + + long previousKey = Long.MIN_VALUE; + for (int i = 0; i < valuesCount; ++i) { + long value = values.nextValue(); + long key = rounding.round(value); + assert key >= previousKey; + if (key == previousKey) { + continue; + } + long bucketOrd = bucketOrds.add(key); + if (bucketOrd < 0) { // already seen + bucketOrd = -1 - bucketOrd; + } + collectBucket(doc, bucketOrd); + previousKey = key; + } + } + + @Override + public InternalAggregation buildAggregation(long owningBucketOrdinal) { + assert owningBucketOrdinal == 0; + List buckets = new ArrayList((int) bucketOrds.size()); + for (long i = 0; i < bucketOrds.capacity(); ++i) { + final long ord = bucketOrds.id(i); + if (ord < 0) { + continue; // slot is not allocated + } + buckets.add(histogramFactory.createBucket(bucketOrds.key(i), bucketDocCount(ord), bucketAggregations(ord))); + } + + + CollectionUtil.introSort(buckets, order.comparator()); + + // value source will be null for unmapped fields + ValueFormatter formatter = valuesSource != null ? valuesSource.formatter() : null; + AbstractHistogramBase.EmptyBucketInfo emptyBucketInfo = computeEmptyBuckets ? new AbstractHistogramBase.EmptyBucketInfo(rounding, buildEmptySubAggregations()) : null; + return histogramFactory.create(name, buckets, order, emptyBucketInfo, formatter, keyed); + } + + @Override + public InternalAggregation buildEmptyAggregation() { + ValueFormatter formatter = valuesSource != null ? valuesSource.formatter() : null; + AbstractHistogramBase.EmptyBucketInfo emptyBucketInfo = computeEmptyBuckets ? new AbstractHistogramBase.EmptyBucketInfo(rounding, buildEmptySubAggregations()) : null; + return histogramFactory.create(name, (List) Collections.EMPTY_LIST, order, emptyBucketInfo, formatter, keyed); + } + + + + public static class Factory extends ValueSourceAggregatorFactory { + + private final Rounding rounding; + private final InternalOrder order; + private final boolean keyed; + private final boolean computeEmptyBuckets; + private final AbstractHistogramBase.Factory histogramFactory; + + public Factory(String name, ValuesSourceConfig valueSourceConfig, + Rounding rounding, InternalOrder order, boolean keyed, boolean computeEmptyBuckets, AbstractHistogramBase.Factory histogramFactory) { + super(name, histogramFactory.type(), valueSourceConfig); + this.rounding = rounding; + this.order = order; + this.keyed = keyed; + this.computeEmptyBuckets = computeEmptyBuckets; + this.histogramFactory = histogramFactory; + } + + @Override + protected Aggregator createUnmapped(AggregationContext aggregationContext, Aggregator parent) { + return new HistogramAggregator(name, factories, rounding, order, keyed, computeEmptyBuckets, null, histogramFactory, aggregationContext, parent); + } + + @Override + protected Aggregator create(NumericValuesSource valuesSource, long expectedBucketsCount, AggregationContext aggregationContext, Aggregator parent) { + return new HistogramAggregator(name, factories, rounding, order, keyed, computeEmptyBuckets, valuesSource, histogramFactory, aggregationContext, parent); + } + + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/HistogramBase.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/HistogramBase.java new file mode 100644 index 00000000000..7d2cc9fb7d3 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/HistogramBase.java @@ -0,0 +1,129 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.histogram; + +import org.elasticsearch.common.xcontent.ToXContent; +import org.elasticsearch.search.aggregations.Aggregation; + +import java.util.Comparator; +import java.util.List; + +/** + * A histogram get result + */ +interface HistogramBase extends Aggregation, Iterable { + + /** + * A bucket in the histogram where documents fall in + */ + static interface Bucket extends org.elasticsearch.search.aggregations.bucket.Bucket { + + /** + * @return The key associated with the bucket (all documents that fall in this bucket were rounded to this key) + */ + long getKey(); + + } + + List buckets(); + + /** + * Returns a bucket by the key associated with it. + * + * @param key The key of the bucket. + * @return The bucket that is associated with the given key. + */ + B getByKey(long key); + + + /** + * A strategy defining the order in which the buckets in this histogram are ordered. + */ + static abstract class Order implements ToXContent { + + public static final Order KEY_ASC = new InternalOrder((byte) 1, "_key", true, new Comparator() { + @Override + public int compare(HistogramBase.Bucket b1, HistogramBase.Bucket b2) { + if (b1.getKey() > b2.getKey()) { + return 1; + } + if (b1.getKey() < b2.getKey()) { + return -1; + } + return 0; + } + }); + + public static final Order KEY_DESC = new InternalOrder((byte) 2, "_key", false, new Comparator() { + @Override + public int compare(HistogramBase.Bucket b1, HistogramBase.Bucket b2) { + return -KEY_ASC.comparator().compare(b1, b2); + } + }); + + public static final Order COUNT_ASC = new InternalOrder((byte) 3, "_count", true, new Comparator() { + @Override + public int compare(HistogramBase.Bucket b1, HistogramBase.Bucket b2) { + if (b1.getDocCount() > b2.getDocCount()) { + return 1; + } + if (b1.getDocCount() < b2.getDocCount()) { + return -1; + } + return 0; + } + }); + + + public static final Order COUNT_DESC = new InternalOrder((byte) 4, "_count", false, new Comparator() { + @Override + public int compare(HistogramBase.Bucket b1, HistogramBase.Bucket b2) { + return -COUNT_ASC.comparator().compare(b1, b2); + } + }); + + /** + * Creates a bucket ordering strategy which sorts buckets based on a single-valued calc get + * + * @param aggregationName the name of the get + * @param asc The direction of the order (ascending or descending) + */ + public static InternalOrder aggregation(String aggregationName, boolean asc) { + return new InternalOrder.Aggregation(aggregationName, null, asc); + } + + /** + * Creates a bucket ordering strategy which sorts buckets based on a multi-valued calc get + * + * @param aggregationName the name of the get + * @param valueName The name of the value of the multi-value get by which the sorting will be applied + * @param asc The direction of the order (ascending or descending) + */ + public static InternalOrder aggregation(String aggregationName, String valueName, boolean asc) { + return new InternalOrder.Aggregation(aggregationName, valueName, asc); + } + + /** + * @return The bucket comparator by which the order will be applied. + */ + abstract Comparator comparator(); + + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/HistogramBuilder.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/HistogramBuilder.java new file mode 100644 index 00000000000..8102e0c52a6 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/HistogramBuilder.java @@ -0,0 +1,56 @@ +package org.elasticsearch.search.aggregations.bucket.histogram; + +import org.elasticsearch.common.xcontent.XContentBuilder; +import org.elasticsearch.search.aggregations.ValuesSourceAggregationBuilder; +import org.elasticsearch.search.builder.SearchSourceBuilderException; + +import java.io.IOException; + +/** + * + */ +public class HistogramBuilder extends ValuesSourceAggregationBuilder { + + private Long interval; + private HistogramBase.Order order; + private Boolean computeEmptyBuckets; + + public HistogramBuilder(String name) { + super(name, InternalHistogram.TYPE.name()); + } + + public HistogramBuilder interval(long interval) { + this.interval = interval; + return this; + } + + public HistogramBuilder order(Histogram.Order order) { + this.order = order; + return this; + } + + public HistogramBuilder emptyBuckets(boolean computeEmptyBuckets) { + this.computeEmptyBuckets = computeEmptyBuckets; + return this; + } + + @Override + protected XContentBuilder doInternalXContent(XContentBuilder builder, Params params) throws IOException { + if (interval == null) { + throw new SearchSourceBuilderException("[interval] must be defined for histogram aggregation [" + name + "]"); + } + builder.field("interval", interval); + + if (order != null) { + builder.field("order"); + order.toXContent(builder, params); + } + + if (computeEmptyBuckets != null) { + builder.field("empty_buckets", computeEmptyBuckets); + } + + return builder; + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/HistogramParser.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/HistogramParser.java new file mode 100644 index 00000000000..958cad20cd0 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/HistogramParser.java @@ -0,0 +1,157 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.histogram; + +import org.elasticsearch.common.rounding.Rounding; +import org.elasticsearch.common.xcontent.XContentParser; +import org.elasticsearch.index.fielddata.IndexFieldData; +import org.elasticsearch.index.mapper.FieldMapper; +import org.elasticsearch.search.SearchParseException; +import org.elasticsearch.search.aggregations.Aggregator; +import org.elasticsearch.search.aggregations.support.FieldContext; +import org.elasticsearch.search.aggregations.support.ValuesSourceConfig; +import org.elasticsearch.search.aggregations.support.numeric.NumericValuesSource; +import org.elasticsearch.search.aggregations.AggregatorFactory; +import org.elasticsearch.search.aggregations.support.numeric.ValueFormatter; +import org.elasticsearch.search.internal.SearchContext; + +import java.io.IOException; +import java.util.Map; + +/** + * Parses the histogram request + */ +public class HistogramParser implements Aggregator.Parser { + + @Override + public String type() { + return InternalHistogram.TYPE.name(); + } + + @Override + public AggregatorFactory parse(String aggregationName, XContentParser parser, SearchContext context) throws IOException { + + ValuesSourceConfig config = new ValuesSourceConfig(NumericValuesSource.class); + + String field = null; + String script = null; + String scriptLang = null; + Map scriptParams = null; + boolean keyed = false; + boolean emptyBuckets = false; + InternalOrder order = (InternalOrder) InternalOrder.KEY_ASC; + long interval = -1; + boolean assumeSorted = false; + String format = null; + + XContentParser.Token token; + String currentFieldName = null; + while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) { + if (token == XContentParser.Token.FIELD_NAME) { + currentFieldName = parser.currentName(); + } else if (token == XContentParser.Token.VALUE_STRING) { + if ("field".equals(currentFieldName)) { + field = parser.text(); + } else if ("script".equals(currentFieldName)) { + script = parser.text(); + } else if ("script_lang".equals(currentFieldName) || "scriptLang".equals(currentFieldName)) { + scriptLang = parser.text(); + } else if ("format".equals(currentFieldName)) { + format = parser.text(); + } + } else if (token == XContentParser.Token.VALUE_NUMBER) { + if ("interval".equals(currentFieldName)) { + interval = parser.longValue(); + } + } else if (token == XContentParser.Token.VALUE_BOOLEAN) { + if ("keyed".equals(currentFieldName)) { + keyed = parser.booleanValue(); + } else if ("empty_buckets".equals(currentFieldName) || "emptyBuckets".equals(currentFieldName)) { + emptyBuckets = parser.booleanValue(); + } else if ("script_values_sorted".equals(currentFieldName)) { + assumeSorted = parser.booleanValue(); + } + } else if (token == XContentParser.Token.START_OBJECT) { + if ("params".equals(currentFieldName)) { + scriptParams = parser.map(); + } else if ("order".equals(currentFieldName)) { + while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) { + if (token == XContentParser.Token.FIELD_NAME) { + currentFieldName = parser.currentName(); + } else if (token == XContentParser.Token.VALUE_STRING) { + String dir = parser.text(); + boolean asc = "asc".equals(dir); + order = resolveOrder(currentFieldName, asc); + //TODO should we throw an error if the value is not "asc" or "desc"??? + } + } + } + } + } + + if (interval < 0) { + throw new SearchParseException(context, "Missing required field [interval] for histogram aggregation [" + aggregationName + "]"); + } + Rounding rounding = new Rounding.Interval(interval); + + if (script != null) { + config.script(context.scriptService().search(context.lookup(), scriptLang, script, scriptParams)); + } + + if (!assumeSorted) { + // we need values to be sorted and unique for efficiency + config.ensureSorted(true); + } + + if (field == null) { + return new HistogramAggregator.Factory(aggregationName, config, rounding, order, keyed, emptyBuckets, InternalHistogram.FACTORY); + } + + FieldMapper mapper = context.smartNameFieldMapper(field); + if (mapper == null) { + config.unmapped(true); + return new HistogramAggregator.Factory(aggregationName, config, rounding, order, keyed, emptyBuckets, InternalHistogram.FACTORY); + } + + IndexFieldData indexFieldData = context.fieldData().getForField(mapper); + config.fieldContext(new FieldContext(field, indexFieldData)); + + if (format != null) { + config.formatter(new ValueFormatter.Number.Pattern(format)); + } + + return new HistogramAggregator.Factory(aggregationName, config, rounding, order, keyed, emptyBuckets, InternalHistogram.FACTORY); + + } + + static InternalOrder resolveOrder(String key, boolean asc) { + if ("_key".equals(key)) { + return (InternalOrder) (asc ? InternalOrder.KEY_ASC : InternalOrder.KEY_DESC); + } + if ("_count".equals(key)) { + return (InternalOrder) (asc ? InternalOrder.COUNT_ASC : InternalOrder.COUNT_DESC); + } + int i = key.indexOf('.'); + if (i < 0) { + return HistogramBase.Order.aggregation(key, asc); + } + return HistogramBase.Order.aggregation(key.substring(0, i), key.substring(i + 1), asc); + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/InternalDateHistogram.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/InternalDateHistogram.java new file mode 100644 index 00000000000..1fcbbdbe4f3 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/InternalDateHistogram.java @@ -0,0 +1,105 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.histogram; + +import org.elasticsearch.common.io.stream.StreamInput; +import org.elasticsearch.search.aggregations.AggregationStreams; +import org.elasticsearch.search.aggregations.InternalAggregation; +import org.elasticsearch.search.aggregations.InternalAggregations; +import org.elasticsearch.search.aggregations.support.numeric.ValueFormatter; +import org.joda.time.DateTime; + +import java.io.IOException; +import java.util.List; + +/** + * + */ +public class InternalDateHistogram extends AbstractHistogramBase implements DateHistogram { + + public final static Type TYPE = new Type("date_histogram", "dhisto"); + public final static Factory FACTORY = new Factory(); + + private final static AggregationStreams.Stream STREAM = new AggregationStreams.Stream() { + @Override + public InternalDateHistogram readResult(StreamInput in) throws IOException { + InternalDateHistogram histogram = new InternalDateHistogram(); + histogram.readFrom(in); + return histogram; + } + }; + + public static void registerStream() { + AggregationStreams.registerStream(STREAM, TYPE.stream()); + } + + static class Bucket extends AbstractHistogramBase.Bucket implements DateHistogram.Bucket { + + Bucket(long key, long docCount, InternalAggregations aggregations) { + super(key, docCount, aggregations); + } + + Bucket(long key, long docCount, List aggregations) { + super(key, docCount, new InternalAggregations(aggregations)); + } + + @Override + public DateTime getKeyAsDate() { + return new DateTime(getKey()); + } + } + + static class Factory implements AbstractHistogramBase.Factory { + + private Factory() { + } + + @Override + public String type() { + return TYPE.name(); + } + + @Override + public AbstractHistogramBase create(String name, List buckets, InternalOrder order, EmptyBucketInfo emptyBucketInfo, ValueFormatter formatter, boolean keyed) { + return new InternalDateHistogram(name, buckets, order, emptyBucketInfo, formatter, keyed); + } + + @Override + public AbstractHistogramBase.Bucket createBucket(long key, long docCount, InternalAggregations aggregations) { + return new Bucket(key, docCount, aggregations); + } + } + + InternalDateHistogram() {} // for serialization + + InternalDateHistogram(String name, List buckets, InternalOrder order, EmptyBucketInfo emptyBucketInfo, ValueFormatter formatter, boolean keyed) { + super(name, buckets, order, emptyBucketInfo, formatter, keyed); + } + + @Override + public Type type() { + return TYPE; + } + + @Override + protected DateHistogram.Bucket createBucket(long key, long docCount, InternalAggregations aggregations) { + return new Bucket(key, docCount, aggregations); + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/InternalHistogram.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/InternalHistogram.java new file mode 100644 index 00000000000..0d3c689416e --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/InternalHistogram.java @@ -0,0 +1,93 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.histogram; + +import org.elasticsearch.common.io.stream.StreamInput; +import org.elasticsearch.search.aggregations.AggregationStreams; +import org.elasticsearch.search.aggregations.InternalAggregations; +import org.elasticsearch.search.aggregations.support.numeric.ValueFormatter; + +import java.io.IOException; +import java.util.List; + +/** + * TODO should be renamed to InternalNumericHistogram (see comment on {@link Histogram})? + */ +public class InternalHistogram extends AbstractHistogramBase implements Histogram { + + public final static Type TYPE = new Type("histogram", "histo"); + public final static Factory FACTORY = new Factory(); + + private final static AggregationStreams.Stream STREAM = new AggregationStreams.Stream() { + @Override + public InternalHistogram readResult(StreamInput in) throws IOException { + InternalHistogram histogram = new InternalHistogram(); + histogram.readFrom(in); + return histogram; + } + }; + + public static void registerStream() { + AggregationStreams.registerStream(STREAM, TYPE.stream()); + } + + static class Bucket extends AbstractHistogramBase.Bucket implements Histogram.Bucket { + + Bucket(long key, long docCount, InternalAggregations aggregations) { + super(key, docCount, aggregations); + } + + } + + static class Factory implements AbstractHistogramBase.Factory { + + private Factory() { + } + + @Override + public String type() { + return TYPE.name(); + } + + public AbstractHistogramBase create(String name, List buckets, InternalOrder order, EmptyBucketInfo emptyBucketInfo, ValueFormatter formatter, boolean keyed) { + return new InternalHistogram(name, buckets, order, emptyBucketInfo, formatter, keyed); + } + + public Bucket createBucket(long key, long docCount, InternalAggregations aggregations) { + return new Bucket(key, docCount, aggregations); + } + + } + + public InternalHistogram() {} // for serialization + + public InternalHistogram(String name, List buckets, InternalOrder order, EmptyBucketInfo emptyBucketInfo, ValueFormatter formatter, boolean keyed) { + super(name, buckets, order, emptyBucketInfo, formatter, keyed); + } + + @Override + public Type type() { + return TYPE; + } + + protected Bucket createBucket(long key, long docCount, InternalAggregations aggregations) { + return new Bucket(key, docCount, aggregations); + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/InternalOrder.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/InternalOrder.java new file mode 100644 index 00000000000..d9aa5ecfed2 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/histogram/InternalOrder.java @@ -0,0 +1,123 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.histogram; + +import org.elasticsearch.common.io.stream.StreamInput; +import org.elasticsearch.common.io.stream.StreamOutput; +import org.elasticsearch.common.xcontent.XContentBuilder; + +import java.io.IOException; +import java.util.Comparator; + +/** + * An internal {@link HistogramBase.Order} strategy which is identified by a unique id. + */ +class InternalOrder extends HistogramBase.Order { + + final byte id; + final String key; + final boolean asc; + final Comparator comparator; + + InternalOrder(byte id, String key, boolean asc, Comparator comparator) { + this.id = id; + this.key = key; + this.asc = asc; + this.comparator = comparator; + } + + byte id() { + return id; + } + + String key() { + return key; + } + + boolean asc() { + return asc; + } + + @Override + Comparator comparator() { + return comparator; + } + + @Override + public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException { + return builder.startObject().field(key, asc ? "asc" : "desc").endObject(); + } + + static class Aggregation extends InternalOrder { + + static final byte ID = 0; + + Aggregation(String key, boolean asc) { + super(ID, key, asc, new HistogramBase.Bucket.Comparator(key, asc)); + } + + Aggregation(String aggName, String valueName, boolean asc) { + super(ID, key(aggName, valueName), asc, new HistogramBase.Bucket.Comparator(aggName, valueName, asc)); + } + + private static String key(String aggName, String valueName) { + return (valueName == null) ? aggName : aggName + "." + valueName; + } + + } + + static class Streams { + + /** + * Writes the given order to the given output (based on the id of the order). + */ + public static void writeOrder(InternalOrder order, StreamOutput out) throws IOException { + out.writeByte(order.id()); + if (order instanceof InternalOrder.Aggregation) { + out.writeBoolean(order.asc()); + out.writeString(order.key()); + } + } + + /** + * Reads an order from the given input (based on the id of the order). + * + * @see Streams#writeOrder(InternalOrder, org.elasticsearch.common.io.stream.StreamOutput) + */ + public static InternalOrder readOrder(StreamInput in) throws IOException { + byte id = in.readByte(); + switch (id) { + case 1: return (InternalOrder) Histogram.Order.KEY_ASC; + case 2: return (InternalOrder) Histogram.Order.KEY_DESC; + case 3: return (InternalOrder) Histogram.Order.COUNT_ASC; + case 4: return (InternalOrder) Histogram.Order.COUNT_DESC; + case 0: + boolean asc = in.readBoolean(); + String key = in.readString(); + return new InternalOrder.Aggregation(key, asc); + default: + throw new RuntimeException("unknown histogram order"); + } + } + + } + + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/missing/InternalMissing.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/missing/InternalMissing.java new file mode 100644 index 00000000000..3267136248e --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/missing/InternalMissing.java @@ -0,0 +1,62 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.missing; + +import org.elasticsearch.common.io.stream.StreamInput; +import org.elasticsearch.search.aggregations.AggregationStreams; +import org.elasticsearch.search.aggregations.InternalAggregations; +import org.elasticsearch.search.aggregations.bucket.SingleBucketAggregation; + +import java.io.IOException; + +/** + * + */ +public class InternalMissing extends SingleBucketAggregation implements Missing { + + public final static Type TYPE = new Type("missing"); + + public final static AggregationStreams.Stream STREAM = new AggregationStreams.Stream() { + @Override + public InternalMissing readResult(StreamInput in) throws IOException { + InternalMissing missing = new InternalMissing(); + missing.readFrom(in); + return missing; + } + }; + + public static void registerStreams() { + AggregationStreams.registerStream(STREAM, TYPE.stream()); + } + + + InternalMissing() { + } + + InternalMissing(String name, long docCount, InternalAggregations aggregations) { + super(name, docCount, aggregations); + } + + @Override + public Type type() { + return TYPE; + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/missing/Missing.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/missing/Missing.java new file mode 100644 index 00000000000..767e06bff05 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/missing/Missing.java @@ -0,0 +1,34 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.missing; + +import org.elasticsearch.search.aggregations.Aggregation; +import org.elasticsearch.search.aggregations.Aggregations; + +/** + * + */ +public interface Missing extends Aggregation { + + long getDocCount(); + + Aggregations getAggregations(); + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/missing/MissingAggregator.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/missing/MissingAggregator.java new file mode 100644 index 00000000000..3d27fa28715 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/missing/MissingAggregator.java @@ -0,0 +1,82 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.missing; + +import org.elasticsearch.search.aggregations.Aggregator; +import org.elasticsearch.search.aggregations.InternalAggregation; +import org.elasticsearch.search.aggregations.bucket.SingleBucketAggregator; +import org.elasticsearch.search.aggregations.support.AggregationContext; +import org.elasticsearch.search.aggregations.support.ValuesSource; +import org.elasticsearch.search.aggregations.support.ValuesSourceConfig; +import org.elasticsearch.search.aggregations.AggregatorFactories; +import org.elasticsearch.search.aggregations.support.ValueSourceAggregatorFactory; + +import java.io.IOException; + +/** + * + */ +public class MissingAggregator extends SingleBucketAggregator { + + private ValuesSource valuesSource; + + public MissingAggregator(String name, AggregatorFactories factories, ValuesSource valuesSource, + AggregationContext aggregationContext, Aggregator parent) { + super(name, factories, aggregationContext, parent); + this.valuesSource = valuesSource; + } + + @Override + public void collect(int doc, long owningBucketOrdinal) throws IOException { + if (valuesSource == null || valuesSource.bytesValues().setDocument(doc) == 0) { + collectBucket(doc, owningBucketOrdinal); + } + } + + @Override + public InternalAggregation buildAggregation(long owningBucketOrdinal) { + return new InternalMissing(name, bucketDocCount(owningBucketOrdinal), bucketAggregations(owningBucketOrdinal)); + } + + @Override + public InternalAggregation buildEmptyAggregation() { + return new InternalMissing(name, 0, buildEmptySubAggregations()); + } + + public static class Factory extends ValueSourceAggregatorFactory { + + public Factory(String name, ValuesSourceConfig valueSourceConfig) { + super(name, InternalMissing.TYPE.name(), valueSourceConfig); + } + + @Override + protected MissingAggregator createUnmapped(AggregationContext aggregationContext, Aggregator parent) { + return new MissingAggregator(name, factories, null, aggregationContext, parent); + } + + @Override + protected MissingAggregator create(ValuesSource valuesSource, long expectedBucketsCount, AggregationContext aggregationContext, Aggregator parent) { + return new MissingAggregator(name, factories, valuesSource, aggregationContext, parent); + } + } + +} + + diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/missing/MissingBuilder.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/missing/MissingBuilder.java new file mode 100644 index 00000000000..13942013c25 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/missing/MissingBuilder.java @@ -0,0 +1,32 @@ +package org.elasticsearch.search.aggregations.bucket.missing; + +import org.elasticsearch.common.xcontent.XContentBuilder; +import org.elasticsearch.search.aggregations.AggregationBuilder; + +import java.io.IOException; + +/** + * + */ +public class MissingBuilder extends AggregationBuilder { + + private String field; + + public MissingBuilder(String name) { + super(name, InternalMissing.TYPE.name()); + } + + public MissingBuilder field(String field) { + this.field = field; + return this; + } + + @Override + protected XContentBuilder internalXContent(XContentBuilder builder, Params params) throws IOException { + builder.startObject(); + if (field != null) { + builder.field("field", field); + } + return builder.endObject(); + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/missing/MissingParser.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/missing/MissingParser.java new file mode 100644 index 00000000000..1317d41b3eb --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/missing/MissingParser.java @@ -0,0 +1,75 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.missing; + +import org.elasticsearch.common.xcontent.XContentParser; +import org.elasticsearch.index.mapper.FieldMapper; +import org.elasticsearch.search.aggregations.Aggregator; +import org.elasticsearch.search.aggregations.support.FieldContext; +import org.elasticsearch.search.aggregations.support.ValuesSource; +import org.elasticsearch.search.aggregations.support.ValuesSourceConfig; +import org.elasticsearch.search.aggregations.AggregatorFactory; +import org.elasticsearch.search.internal.SearchContext; + +import java.io.IOException; + +/** + * + */ +public class MissingParser implements Aggregator.Parser { + + @Override + public String type() { + return InternalMissing.TYPE.name(); + } + + @Override + public AggregatorFactory parse(String aggregationName, XContentParser parser, SearchContext context) throws IOException { + + ValuesSourceConfig config = new ValuesSourceConfig(ValuesSource.class); + + String field = null; + + XContentParser.Token token; + String currentFieldName = null; + while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) { + if (token == XContentParser.Token.FIELD_NAME) { + currentFieldName = parser.currentName(); + } else if (token == XContentParser.Token.VALUE_STRING) { + if ("field".equals(currentFieldName)) { + field = parser.text(); + } + } + } + + if (field == null) { + return new MissingAggregator.Factory(aggregationName, config); + } + + FieldMapper mapper = context.smartNameFieldMapper(field); + if (mapper == null) { + config.unmapped(true); + return new MissingAggregator.Factory(aggregationName, config); + } + + config.fieldContext(new FieldContext(field, context.fieldData().getForField(mapper))); + return new MissingAggregator.Factory(aggregationName, config); + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/nested/InternalNested.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/nested/InternalNested.java new file mode 100644 index 00000000000..94019158294 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/nested/InternalNested.java @@ -0,0 +1,62 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.nested; + +import org.elasticsearch.common.io.stream.StreamInput; +import org.elasticsearch.search.aggregations.AggregationStreams; +import org.elasticsearch.search.aggregations.InternalAggregations; +import org.elasticsearch.search.aggregations.bucket.SingleBucketAggregation; + +import java.io.IOException; + +/** + * + */ +public class InternalNested extends SingleBucketAggregation implements Nested { + + public static final Type TYPE = new Type("nested"); + + public final static AggregationStreams.Stream STREAM = new AggregationStreams.Stream() { + @Override + public InternalNested readResult(StreamInput in) throws IOException { + InternalNested result = new InternalNested(); + result.readFrom(in); + return result; + } + }; + + public static void registerStream() { + AggregationStreams.registerStream(STREAM, TYPE.stream()); + } + + public InternalNested() { + } + + public InternalNested(String name, long docCount, InternalAggregations aggregations) { + super(name, docCount, aggregations); + } + + @Override + public Type type() { + return TYPE; + } + + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/nested/Nested.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/nested/Nested.java new file mode 100644 index 00000000000..3ab6fe8fc7b --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/nested/Nested.java @@ -0,0 +1,34 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.nested; + +import org.elasticsearch.search.aggregations.Aggregation; +import org.elasticsearch.search.aggregations.Aggregations; + +/** + * + */ +public interface Nested extends Aggregation { + + long getDocCount(); + + Aggregations getAggregations(); + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/nested/NestedAggregator.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/nested/NestedAggregator.java new file mode 100644 index 00000000000..0decc45f05a --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/nested/NestedAggregator.java @@ -0,0 +1,132 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.nested; + +import org.apache.lucene.index.AtomicReaderContext; +import org.apache.lucene.search.DocIdSet; +import org.apache.lucene.search.Filter; +import org.apache.lucene.util.Bits; +import org.apache.lucene.util.FixedBitSet; +import org.elasticsearch.common.lucene.ReaderContextAware; +import org.elasticsearch.common.lucene.docset.DocIdSets; +import org.elasticsearch.index.mapper.MapperService; +import org.elasticsearch.index.mapper.object.ObjectMapper; +import org.elasticsearch.index.search.nested.NonNestedDocsFilter; +import org.elasticsearch.search.aggregations.AggregationExecutionException; +import org.elasticsearch.search.aggregations.Aggregator; +import org.elasticsearch.search.aggregations.InternalAggregation; +import org.elasticsearch.search.aggregations.bucket.SingleBucketAggregator; +import org.elasticsearch.search.aggregations.support.AggregationContext; +import org.elasticsearch.search.aggregations.AggregatorFactories; +import org.elasticsearch.search.aggregations.AggregatorFactory; + +import java.io.IOException; + +/** + * + */ +public class NestedAggregator extends SingleBucketAggregator implements ReaderContextAware { + + private final Filter parentFilter; + private final Filter childFilter; + + private Bits childDocs; + private FixedBitSet parentDocs; + + public NestedAggregator(String name, AggregatorFactories factories, String nestedPath, AggregationContext aggregationContext, Aggregator parent) { + super(name, factories, aggregationContext, parent); + MapperService.SmartNameObjectMapper mapper = aggregationContext.searchContext().smartNameObjectMapper(nestedPath); + if (mapper == null) { + throw new AggregationExecutionException("facet nested path [" + nestedPath + "] not found"); + } + ObjectMapper objectMapper = mapper.mapper(); + if (objectMapper == null) { + throw new AggregationExecutionException("facet nested path [" + nestedPath + "] not found"); + } + if (!objectMapper.nested().isNested()) { + throw new AggregationExecutionException("facet nested path [" + nestedPath + "] is not nested"); + } + parentFilter = aggregationContext.searchContext().filterCache().cache(NonNestedDocsFilter.INSTANCE); + childFilter = aggregationContext.searchContext().filterCache().cache(objectMapper.nestedTypeFilter()); + } + + @Override + public void setNextReader(AtomicReaderContext reader) { + try { + DocIdSet docIdSet = parentFilter.getDocIdSet(reader, null); + // In ES if parent is deleted, then also the children are deleted. Therefore acceptedDocs can also null here. + childDocs = DocIdSets.toSafeBits(reader.reader(), childFilter.getDocIdSet(reader, null)); + if (DocIdSets.isEmpty(docIdSet)) { + parentDocs = null; + } else { + parentDocs = (FixedBitSet) docIdSet; + } + } catch (IOException ioe) { + throw new AggregationExecutionException("Failed to aggregate [" + name + "]", ioe); + } + } + + @Override + public void collect(int parentDoc, long bucketOrd) throws IOException { + + // here we translate the parent doc to a list of its nested docs, and then call super.collect for evey one of them + // so they'll be collected + + if (parentDoc == 0 || parentDocs == null) { + return; + } + int prevParentDoc = parentDocs.prevSetBit(parentDoc - 1); + int numChildren = 0; + for (int i = (parentDoc - 1); i > prevParentDoc; i--) { + if (childDocs.get(i)) { + ++numChildren; + collectBucketNoCounts(i, bucketOrd); + } + } + incrementBucketDocCount(numChildren, bucketOrd); + } + + @Override + public InternalAggregation buildAggregation(long owningBucketOrdinal) { + return new InternalNested(name, bucketDocCount(owningBucketOrdinal), bucketAggregations(owningBucketOrdinal)); + } + + @Override + public InternalAggregation buildEmptyAggregation() { + return new InternalNested(name, 0, buildEmptySubAggregations()); + } + + public static class Factory extends AggregatorFactory { + + private final String path; + + public Factory(String name, String path) { + super(name, InternalNested.TYPE.name()); + this.path = path; + } + + @Override + public Aggregator create(AggregationContext context, Aggregator parent, long expectedBucketsCount) { + NestedAggregator aggregator = new NestedAggregator(name, factories, path, context, parent); + context.registerReaderContextAware(aggregator); + return aggregator; + } + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/nested/NestedBuilder.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/nested/NestedBuilder.java new file mode 100644 index 00000000000..723b667c6ac --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/nested/NestedBuilder.java @@ -0,0 +1,34 @@ +package org.elasticsearch.search.aggregations.bucket.nested; + +import org.elasticsearch.common.xcontent.XContentBuilder; +import org.elasticsearch.search.aggregations.AggregationBuilder; +import org.elasticsearch.search.builder.SearchSourceBuilderException; + +import java.io.IOException; + +/** + * + */ +public class NestedBuilder extends AggregationBuilder { + + private String path; + + public NestedBuilder(String name) { + super(name, InternalNested.TYPE.name()); + } + + public NestedBuilder path(String path) { + this.path = path; + return this; + } + + @Override + protected XContentBuilder internalXContent(XContentBuilder builder, Params params) throws IOException { + builder.startObject(); + if (path == null) { + throw new SearchSourceBuilderException("nested path must be set on nested aggregation [" + name + "]"); + } + builder.field("path", path); + return builder.endObject(); + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/nested/NestedParser.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/nested/NestedParser.java new file mode 100644 index 00000000000..3e05975c6f7 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/nested/NestedParser.java @@ -0,0 +1,63 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.nested; + +import org.elasticsearch.common.xcontent.XContentParser; +import org.elasticsearch.search.SearchParseException; +import org.elasticsearch.search.aggregations.Aggregator; +import org.elasticsearch.search.aggregations.AggregatorFactory; +import org.elasticsearch.search.internal.SearchContext; + +import java.io.IOException; + +/** + * + */ +public class NestedParser implements Aggregator.Parser { + + @Override + public String type() { + return InternalNested.TYPE.name(); + } + + @Override + public AggregatorFactory parse(String aggregationName, XContentParser parser, SearchContext context) throws IOException { + String path = null; + + XContentParser.Token token; + String currentFieldName = null; + while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) { + if (token == XContentParser.Token.FIELD_NAME) { + currentFieldName = parser.currentName(); + } else if (token == XContentParser.Token.VALUE_STRING) { + if ("path".equals(currentFieldName)) { + path = parser.text(); + } + } + } + + if (path == null) { + // "field" doesn't exist, so we fall back to the context of the ancestors + throw new SearchParseException(context, "Missing [path] field for nested aggregation [" + aggregationName + "]"); + } + + return new NestedAggregator.Factory(aggregationName, path); + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/range/AbstractRangeBase.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/AbstractRangeBase.java new file mode 100644 index 00000000000..5ec644d70c7 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/AbstractRangeBase.java @@ -0,0 +1,286 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.range; + +import com.google.common.collect.Lists; +import org.elasticsearch.cache.recycler.CacheRecycler; +import org.elasticsearch.common.io.stream.StreamInput; +import org.elasticsearch.common.io.stream.StreamOutput; +import org.elasticsearch.common.xcontent.XContentBuilder; +import org.elasticsearch.search.aggregations.Aggregations; +import org.elasticsearch.search.aggregations.InternalAggregation; +import org.elasticsearch.search.aggregations.InternalAggregations; +import org.elasticsearch.search.aggregations.support.numeric.ValueFormatter; +import org.elasticsearch.search.aggregations.support.numeric.ValueFormatterStreams; + +import java.io.IOException; +import java.util.*; + +/** + * + */ +public abstract class AbstractRangeBase extends InternalAggregation implements RangeBase { + + public abstract static class Bucket implements RangeBase.Bucket { + + private double from = Double.NEGATIVE_INFINITY; + private double to = Double.POSITIVE_INFINITY; + private long docCount; + private InternalAggregations aggregations; + private String key; + private boolean explicitKey; + + public Bucket(String key, double from, double to, long docCount, InternalAggregations aggregations, ValueFormatter formatter) { + if (key != null) { + this.key = key; + explicitKey = true; + } else { + this.key = key(from, to, formatter); + explicitKey = false; + } + this.from = from; + this.to = to; + this.docCount = docCount; + this.aggregations = aggregations; + } + + public String getKey() { + return key; + } + + @Override + public double getFrom() { + return from; + } + + @Override + public double getTo() { + return to; + } + + @Override + public long getDocCount() { + return docCount; + } + + @Override + public Aggregations getAggregations() { + return aggregations; + } + + Bucket reduce(List ranges, CacheRecycler cacheRecycler) { + if (ranges.size() == 1) { + return ranges.get(0); + } + Bucket reduced = null; + List aggregationsList = Lists.newArrayListWithCapacity(ranges.size()); + for (Bucket range : ranges) { + if (reduced == null) { + reduced = range; + } else { + reduced.docCount += range.docCount; + } + aggregationsList.add(range.aggregations); + } + reduced.aggregations = InternalAggregations.reduce(aggregationsList, cacheRecycler); + return reduced; + } + + void toXContent(XContentBuilder builder, Params params, ValueFormatter formatter, boolean keyed) throws IOException { + if (keyed) { + builder.startObject(key); + } else { + builder.startObject(); + if (explicitKey) { + builder.field(CommonFields.KEY, key); + } + } + if (!Double.isInfinite(from)) { + builder.field(CommonFields.FROM, from); + if (formatter != null) { + builder.field(CommonFields.FROM_AS_STRING, formatter.format(from)); + } + } + if (!Double.isInfinite(to)) { + builder.field(CommonFields.TO, to); + if (formatter != null) { + builder.field(CommonFields.TO_AS_STRING, formatter.format(to)); + } + } + builder.field(CommonFields.DOC_COUNT, docCount); + aggregations.toXContentInternal(builder, params); + builder.endObject(); + } + + private static String key(double from, double to, ValueFormatter formatter) { + StringBuilder sb = new StringBuilder(); + sb.append(Double.isInfinite(from) ? "*" : formatter != null ? formatter.format(from) : from); + sb.append("-"); + sb.append(Double.isInfinite(to) ? "*" : formatter != null ? formatter.format(to) : to); + return sb.toString(); + } + + } + + public static interface Factory { + + public String type(); + + public AbstractRangeBase create(String name, List buckets, ValueFormatter formatter, boolean keyed); + + public B createBucket(String key, double from, double to, long docCount, InternalAggregations aggregations, ValueFormatter formatter); + + } + + private List ranges; + private Map rangeMap; + private ValueFormatter formatter; + private boolean keyed; + + private boolean unmapped; + + public AbstractRangeBase() {} // for serialization + + public AbstractRangeBase(String name, List ranges, ValueFormatter formatter, boolean keyed) { + this(name, ranges, formatter, keyed, false); + } + + public AbstractRangeBase(String name, List ranges, ValueFormatter formatter, boolean keyed, boolean unmapped) { + super(name); + this.ranges = ranges; + this.formatter = formatter; + this.keyed = keyed; + this.unmapped = unmapped; + } + + @Override + @SuppressWarnings("unchecked") + public Iterator iterator() { + Object iter = ranges.iterator(); + return (Iterator) iter; + } + + @Override + public B getByKey(String key) { + if (rangeMap == null) { + rangeMap = new HashMap(); + for (RangeBase.Bucket bucket : ranges) { + rangeMap.put(bucket.getKey(), (B) bucket); + } + } + return (B) rangeMap.get(key); + } + + @Override + public List buckets() { + return ranges; + } + + @Override + public AbstractRangeBase reduce(ReduceContext reduceContext) { + List aggregations = reduceContext.aggregations(); + if (aggregations.size() == 1) { + return (AbstractRangeBase) aggregations.get(0); + } + List> rangesList = null; + for (InternalAggregation aggregation : aggregations) { + AbstractRangeBase ranges = (AbstractRangeBase) aggregation; + if (ranges.unmapped) { + continue; + } + if (rangesList == null) { + rangesList = new ArrayList>(ranges.ranges.size()); + for (Bucket bucket : ranges.ranges) { + List sameRangeList = new ArrayList(aggregations.size()); + sameRangeList.add(bucket); + rangesList.add(sameRangeList); + } + } else { + int i = 0; + for (Bucket range : ranges.ranges) { + rangesList.get(i++).add(range); + } + } + } + + if (rangesList == null) { + // unmapped, we can just take the first one + return (AbstractRangeBase) aggregations.get(0); + } + + AbstractRangeBase reduced = (AbstractRangeBase) aggregations.get(0); + int i = 0; + for (List sameRangeList : rangesList) { + reduced.ranges.set(i++, (sameRangeList.get(0)).reduce(sameRangeList, reduceContext.cacheRecycler())); + } + return reduced; + } + + protected abstract B createBucket(String key, double from, double to, long docCount, InternalAggregations aggregations, ValueFormatter formatter); + + @Override + public void readFrom(StreamInput in) throws IOException { + name = in.readString(); + formatter = ValueFormatterStreams.readOptional(in); + keyed = in.readBoolean(); + int size = in.readVInt(); + List ranges = Lists.newArrayListWithCapacity(size); + for (int i = 0; i < size; i++) { + String key = in.readOptionalString(); + ranges.add(createBucket(key, in.readDouble(), in.readDouble(), in.readVLong(), InternalAggregations.readAggregations(in), formatter)); + } + this.ranges = ranges; + this.rangeMap = null; + } + + @Override + public void writeTo(StreamOutput out) throws IOException { + out.writeString(name); + ValueFormatterStreams.writeOptional(formatter, out); + out.writeBoolean(keyed); + out.writeVInt(ranges.size()); + for (B bucket : ranges) { + out.writeOptionalString(((Bucket) bucket).key); + out.writeDouble(((Bucket) bucket).from); + out.writeDouble(((Bucket) bucket).to); + out.writeVLong(((Bucket) bucket).docCount); + ((Bucket) bucket).aggregations.writeTo(out); + } + } + + @Override + public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException { + if (keyed) { + builder.startObject(name); + } else { + builder.startArray(name); + } + for (B range : ranges) { + ((Bucket) range).toXContent(builder, params, formatter, keyed); + } + if (keyed) { + builder.endObject(); + } else { + builder.endArray(); + } + return builder; + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/range/InternalRange.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/InternalRange.java new file mode 100644 index 00000000000..f1e3a2c093d --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/InternalRange.java @@ -0,0 +1,93 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.range; + +import org.elasticsearch.common.io.stream.StreamInput; +import org.elasticsearch.search.aggregations.AggregationStreams; +import org.elasticsearch.search.aggregations.InternalAggregations; +import org.elasticsearch.search.aggregations.support.numeric.ValueFormatter; + +import java.io.IOException; +import java.util.List; + +/** + * + */ +public class InternalRange extends AbstractRangeBase implements Range { + + static final Factory FACTORY = new Factory(); + + public final static Type TYPE = new Type("range"); + + private final static AggregationStreams.Stream STREAM = new AggregationStreams.Stream() { + @Override + public AbstractRangeBase readResult(StreamInput in) throws IOException { + InternalRange ranges = new InternalRange(); + ranges.readFrom(in); + return ranges; + } + }; + + public static void registerStream() { + AggregationStreams.registerStream(STREAM, TYPE.stream()); + } + + public static class Bucket extends AbstractRangeBase.Bucket implements Range.Bucket { + + public Bucket(String key, double from, double to, long docCount, InternalAggregations aggregations, ValueFormatter formatter) { + super(key, from, to, docCount, aggregations, formatter); + } + } + + + public static class Factory implements AbstractRangeBase.Factory { + + @Override + public String type() { + return TYPE.name(); + } + + @Override + public AbstractRangeBase create(String name, List ranges, ValueFormatter formatter, boolean keyed) { + return new InternalRange(name, ranges, formatter, keyed); + } + + + public Range.Bucket createBucket(String key, double from, double to, long docCount, InternalAggregations aggregations, ValueFormatter formatter) { + return new Bucket(key, from, to, docCount, aggregations, formatter); + } + } + + protected Range.Bucket createBucket(String key, double from, double to, long docCount, InternalAggregations aggregations, ValueFormatter formatter) { + return new Bucket(key, from, to, docCount, aggregations, formatter); + } + + public InternalRange() {} // for serialization + + public InternalRange(String name, List ranges, ValueFormatter formatter, boolean keyed) { + super(name, ranges, formatter, keyed); + } + + @Override + public Type type() { + return TYPE; + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/range/Range.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/Range.java new file mode 100644 index 00000000000..b4fd1c88fd4 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/Range.java @@ -0,0 +1,29 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.range; + +/** + * + */ +public interface Range extends RangeBase { + + static interface Bucket extends RangeBase.Bucket { + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/range/RangeAggregator.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/RangeAggregator.java new file mode 100644 index 00000000000..00799d98c6c --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/RangeAggregator.java @@ -0,0 +1,309 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.range; + +import com.google.common.collect.Lists; +import org.apache.lucene.util.InPlaceMergeSorter; +import org.elasticsearch.index.fielddata.DoubleValues; +import org.elasticsearch.search.aggregations.Aggregator; +import org.elasticsearch.search.aggregations.InternalAggregation; +import org.elasticsearch.search.aggregations.InternalAggregations; +import org.elasticsearch.search.aggregations.bucket.BucketsAggregator; +import org.elasticsearch.search.aggregations.support.AggregationContext; +import org.elasticsearch.search.aggregations.support.ValuesSourceConfig; +import org.elasticsearch.search.aggregations.support.numeric.NumericValuesSource; +import org.elasticsearch.search.aggregations.support.numeric.ValueFormatter; +import org.elasticsearch.search.aggregations.support.numeric.ValueParser; +import org.elasticsearch.search.aggregations.AggregatorFactories; +import org.elasticsearch.search.aggregations.support.ValueSourceAggregatorFactory; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; + +/** + * + */ +public class RangeAggregator extends BucketsAggregator { + + public static class Range { + + public String key; + public double from = Double.NEGATIVE_INFINITY; + String fromAsStr; + public double to = Double.POSITIVE_INFINITY; + String toAsStr; + + public Range(String key, double from, String fromAsStr, double to, String toAsStr) { + this.key = key; + this.from = from; + this.fromAsStr = fromAsStr; + this.to = to; + this.toAsStr = toAsStr; + } + + boolean matches(double value) { + return value >= from && value < to; + } + + @Override + public String toString() { + return "[" + from + " to " + to + ")"; + } + + public void process(ValueParser parser, AggregationContext aggregationContext) { + if (fromAsStr != null) { + from = parser != null ? parser.parseDouble(fromAsStr, aggregationContext.searchContext()) : Double.valueOf(fromAsStr); + } + if (toAsStr != null) { + to = parser != null ? parser.parseDouble(toAsStr, aggregationContext.searchContext()) : Double.valueOf(toAsStr); + } + } + } + + private final NumericValuesSource valuesSource; + private final Range[] ranges; + private final boolean keyed; + private final AbstractRangeBase.Factory rangeFactory; + + final double[] maxTo; + + public RangeAggregator(String name, + AggregatorFactories factories, + NumericValuesSource valuesSource, + AbstractRangeBase.Factory rangeFactory, + List ranges, + boolean keyed, + AggregationContext aggregationContext, + Aggregator parent) { + + super(name, BucketAggregationMode.PER_BUCKET, factories, ranges.size(), aggregationContext, parent); + assert valuesSource != null; + this.valuesSource = valuesSource; + this.keyed = keyed; + this.rangeFactory = rangeFactory; + this.ranges = ranges.toArray(new Range[ranges.size()]); + for (int i = 0; i < this.ranges.length; i++) { + this.ranges[i].process(valuesSource.parser(), context); + } + sortRanges(this.ranges); + + maxTo = new double[this.ranges.length]; + maxTo[0] = this.ranges[0].to; + for (int i = 1; i < this.ranges.length; ++i) { + maxTo[i] = Math.max(this.ranges[i].to,maxTo[i-1]); + } + + } + + @Override + public boolean shouldCollect() { + return true; + } + + @Override + public void collect(int doc, long owningBucketOrdinal) throws IOException { + assert owningBucketOrdinal == 0; + + final DoubleValues values = valuesSource.doubleValues(); + final int valuesCount = values.setDocument(doc); + for (int i = 0, lo = 0; i < valuesCount; ++i) { + final double value = values.nextValue(); + lo = collect(doc, value, lo); + } + } + + private int collect(int doc, double value, int lowBound) throws IOException { + int lo = lowBound, hi = ranges.length - 1; // all candidates are between these indexes + int mid = (lo + hi) >>> 1; + while (lo <= hi) { + if (value < ranges[mid].from) { + hi = mid - 1; + } else if (value >= maxTo[mid]) { + lo = mid + 1; + } else { + break; + } + mid = (lo + hi) >>> 1; + } + if (lo > hi) return lo; // no potential candidate + + // binary search the lower bound + int startLo = lo, startHi = mid; + while (startLo <= startHi) { + final int startMid = (startLo + startHi) >>> 1; + if (value >= maxTo[startMid]) { + startLo = startMid + 1; + } else { + startHi = startMid - 1; + } + } + + // binary search the upper bound + int endLo = mid, endHi = hi; + while (endLo <= endHi) { + final int endMid = (endLo + endHi) >>> 1; + if (value < ranges[endMid].from) { + endHi = endMid - 1; + } else { + endLo = endMid + 1; + } + } + + assert startLo == lowBound || value >= maxTo[startLo - 1]; + assert endHi == ranges.length - 1 || value < ranges[endHi + 1].from; + + for (int i = startLo; i <= endHi; ++i) { + if (ranges[i].matches(value)) { + collectBucket(doc, i); + } + } + + return endHi + 1; + } + + @Override + public InternalAggregation buildAggregation(long owningBucketOrdinal) { + assert owningBucketOrdinal == 0; + List buckets = Lists.newArrayListWithCapacity(ranges.length); + for (int i = 0; i < ranges.length; i++) { + Range range = ranges[i]; + RangeBase.Bucket bucket = rangeFactory.createBucket(range.key, range.from, range.to, bucketDocCount(i), + bucketAggregations(i), valuesSource.formatter()); + buckets.add(bucket); + } + // value source can be null in the case of unmapped fields + ValueFormatter formatter = valuesSource != null ? valuesSource.formatter() : null; + return rangeFactory.create(name, buckets, formatter, keyed); + } + + @Override + public InternalAggregation buildEmptyAggregation() { + InternalAggregations subAggs = buildEmptySubAggregations(); + List buckets = Lists.newArrayListWithCapacity(ranges.length); + for (int i = 0; i < ranges.length; i++) { + Range range = ranges[i]; + RangeBase.Bucket bucket = rangeFactory.createBucket(range.key, range.from, range.to, 0, subAggs, valuesSource.formatter()); + buckets.add(bucket); + } + // value source can be null in the case of unmapped fields + ValueFormatter formatter = valuesSource != null ? valuesSource.formatter() : null; + return rangeFactory.create(name, buckets, formatter, keyed); + } + + private static final void sortRanges(final Range[] ranges) { + new InPlaceMergeSorter() { + + @Override + protected void swap(int i, int j) { + final Range tmp = ranges[i]; + ranges[i] = ranges[j]; + ranges[j] = tmp; + } + + @Override + protected int compare(int i, int j) { + int cmp = Double.compare(ranges[i].from, ranges[j].from); + if (cmp == 0) { + cmp = Double.compare(ranges[i].to, ranges[j].to); + } + return cmp; + } + }.sort(0, ranges.length); + } + + public static class Unmapped extends Aggregator { + + private final List ranges; + private final boolean keyed; + private final AbstractRangeBase.Factory factory; + private final ValueFormatter formatter; + private final ValueParser parser; + + public Unmapped(String name, + List ranges, + boolean keyed, + ValueFormatter formatter, + ValueParser parser, + AggregationContext aggregationContext, + Aggregator parent, + AbstractRangeBase.Factory factory) { + + super(name, BucketAggregationMode.PER_BUCKET, AggregatorFactories.EMPTY, 0, aggregationContext, parent); + this.ranges = ranges; + for (Range range : this.ranges) { + range.process(parser, context); + } + this.keyed = keyed; + this.formatter = formatter; + this.parser = parser; + this.factory = factory; + } + + @Override + public boolean shouldCollect() { + return false; + } + + @Override + public void collect(int doc, long owningBucketOrdinal) throws IOException { + } + + @Override + public AbstractRangeBase buildAggregation(long owningBucketOrdinal) { + return (AbstractRangeBase) buildEmptyAggregation(); + } + + @Override + public AbstractRangeBase buildEmptyAggregation() { + InternalAggregations subAggs = buildEmptySubAggregations(); + List buckets = new ArrayList(ranges.size()); + for (RangeAggregator.Range range : ranges) { + buckets.add(factory.createBucket(range.key, range.from, range.to, 0, subAggs, formatter)); + } + return factory.create(name, buckets, formatter, keyed); + } + } + + public static class Factory extends ValueSourceAggregatorFactory { + + private final AbstractRangeBase.Factory rangeFactory; + private final List ranges; + private final boolean keyed; + + public Factory(String name, ValuesSourceConfig valueSourceConfig, AbstractRangeBase.Factory rangeFactory, List ranges, boolean keyed) { + super(name, rangeFactory.type(), valueSourceConfig); + this.rangeFactory = rangeFactory; + this.ranges = ranges; + this.keyed = keyed; + } + + @Override + protected Aggregator createUnmapped(AggregationContext aggregationContext, Aggregator parent) { + return new Unmapped(name, ranges, keyed, valuesSourceConfig.formatter(), valuesSourceConfig.parser(), aggregationContext, parent, rangeFactory); + } + + @Override + protected Aggregator create(NumericValuesSource valuesSource, long expectedBucketsCount, AggregationContext aggregationContext, Aggregator parent) { + return new RangeAggregator(name, factories, valuesSource, rangeFactory, ranges, keyed, aggregationContext, parent); + } + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/range/RangeBase.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/RangeBase.java new file mode 100644 index 00000000000..c27d29bc9cc --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/RangeBase.java @@ -0,0 +1,44 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.range; + +import org.elasticsearch.search.aggregations.Aggregation; + +import java.util.List; + +/** + * + */ +public interface RangeBase extends Aggregation, Iterable { + + public static interface Bucket extends org.elasticsearch.search.aggregations.bucket.Bucket { + + String getKey(); + + double getFrom(); + + double getTo(); + } + + List buckets(); + + B getByKey(String key); + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/range/RangeBuilder.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/RangeBuilder.java new file mode 100644 index 00000000000..f192b4dec99 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/RangeBuilder.java @@ -0,0 +1,40 @@ +package org.elasticsearch.search.aggregations.bucket.range; + +/** + * + */ +public class RangeBuilder extends RangeBuilderBase { + + public RangeBuilder(String name) { + super(name, InternalRange.TYPE.name()); + } + + + public RangeBuilder addRange(String key, double from, double to) { + ranges.add(new Range(key, from, to)); + return this; + } + + public RangeBuilder addRange(double from, double to) { + return addRange(null, from, to); + } + + public RangeBuilder addUnboundedTo(String key, double to) { + ranges.add(new Range(key, null, to)); + return this; + } + + public RangeBuilder addUnboundedTo(double to) { + return addUnboundedTo(null, to); + } + + public RangeBuilder addUnboundedFrom(String key, double from) { + ranges.add(new Range(key, from, null)); + return this; + } + + public RangeBuilder addUnboundedFrom(double from) { + return addUnboundedFrom(null, from); + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/range/RangeBuilderBase.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/RangeBuilderBase.java new file mode 100644 index 00000000000..53800c78647 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/RangeBuilderBase.java @@ -0,0 +1,62 @@ +package org.elasticsearch.search.aggregations.bucket.range; + +import com.google.common.collect.Lists; +import org.elasticsearch.common.xcontent.ToXContent; +import org.elasticsearch.common.xcontent.XContentBuilder; +import org.elasticsearch.search.aggregations.ValuesSourceAggregationBuilder; +import org.elasticsearch.search.builder.SearchSourceBuilderException; + +import java.io.IOException; +import java.util.List; + +/** + * + */ +public abstract class RangeBuilderBase> extends ValuesSourceAggregationBuilder { + + protected static class Range implements ToXContent { + + private String key; + private Object from; + private Object to; + + public Range(String key, Object from, Object to) { + this.key = key; + this.from = from; + this.to = to; + } + + @Override + public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException { + builder.startObject(); + if (key != null) { + builder.field("key", key); + } + if (from != null) { + builder.field("from", from); + } + if (to != null) { + builder.field("to", to); + } + return builder.endObject(); + } + } + + protected List ranges = Lists.newArrayList(); + + protected RangeBuilderBase(String name, String type) { + super(name, type); + } + + @Override + protected XContentBuilder doInternalXContent(XContentBuilder builder, Params params) throws IOException { + if (ranges.isEmpty()) { + throw new SearchSourceBuilderException("at least one range must be defined for range aggregation [" + name + "]"); + } + builder.startArray("ranges"); + for (Range range : ranges) { + range.toXContent(builder, params); + } + return builder.endArray(); + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/range/RangeParser.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/RangeParser.java new file mode 100644 index 00000000000..9d567c4d378 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/RangeParser.java @@ -0,0 +1,147 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.range; + +import org.elasticsearch.common.xcontent.XContentParser; +import org.elasticsearch.index.fielddata.IndexFieldData; +import org.elasticsearch.index.mapper.FieldMapper; +import org.elasticsearch.search.SearchParseException; +import org.elasticsearch.search.aggregations.Aggregator; +import org.elasticsearch.search.aggregations.support.FieldContext; +import org.elasticsearch.search.aggregations.support.ValuesSourceConfig; +import org.elasticsearch.search.aggregations.support.numeric.NumericValuesSource; +import org.elasticsearch.search.aggregations.AggregatorFactory; +import org.elasticsearch.search.internal.SearchContext; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; + +/** + * + */ +public class RangeParser implements Aggregator.Parser { + + @Override + public String type() { + return InternalRange.TYPE.name(); + } + + @Override + public AggregatorFactory parse(String aggregationName, XContentParser parser, SearchContext context) throws IOException { + + ValuesSourceConfig config = new ValuesSourceConfig(NumericValuesSource.class); + + String field = null; + List ranges = null; + String script = null; + String scriptLang = null; + Map scriptParams = null; + boolean keyed = false; + boolean assumeSorted = false; + + XContentParser.Token token; + String currentFieldName = null; + while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) { + if (token == XContentParser.Token.FIELD_NAME) { + currentFieldName = parser.currentName(); + } else if (token == XContentParser.Token.VALUE_STRING) { + if ("field".equals(currentFieldName)) { + field = parser.text(); + } else if ("script".equals(currentFieldName)) { + script = parser.text(); + } else if ("script_lang".equals(currentFieldName) || "scriptLang".equals(currentFieldName)) { + scriptLang = parser.text(); + } + } else if (token == XContentParser.Token.START_ARRAY) { + if ("ranges".equals(currentFieldName)) { + ranges = new ArrayList(); + while ((token = parser.nextToken()) != XContentParser.Token.END_ARRAY) { + double from = Double.NEGATIVE_INFINITY; + String fromAsStr = null; + double to = Double.POSITIVE_INFINITY; + String toAsStr = null; + String key = null; + String toOrFromOrKey = null; + while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) { + if (token == XContentParser.Token.FIELD_NAME) { + toOrFromOrKey = parser.currentName(); + } else if (token == XContentParser.Token.VALUE_NUMBER) { + if ("from".equals(toOrFromOrKey)) { + from = parser.doubleValue(); + } else if ("to".equals(toOrFromOrKey)) { + to = parser.doubleValue(); + } + } else if (token == XContentParser.Token.VALUE_STRING) { + if ("from".equals(toOrFromOrKey)) { + fromAsStr = parser.text(); + } else if ("to".equals(toOrFromOrKey)) { + toAsStr = parser.text(); + } else if ("key".equals(toOrFromOrKey)) { + key = parser.text(); + } + } + } + ranges.add(new RangeAggregator.Range(key, from, fromAsStr, to, toAsStr)); + } + } + } else if (token == XContentParser.Token.START_OBJECT) { + if ("params".equals(currentFieldName)) { + scriptParams = parser.map(); + } + } else if (token == XContentParser.Token.VALUE_BOOLEAN) { + if ("keyed".equals(currentFieldName)) { + keyed = parser.booleanValue(); + } else if ("script_values_sorted".equals(currentFieldName)) { + assumeSorted = parser.booleanValue(); + } + } + } + + if (ranges == null) { + throw new SearchParseException(context, "Missing [ranges] in ranges aggregator [" + aggregationName + "]"); + } + + if (script != null) { + config.script(context.scriptService().search(context.lookup(), scriptLang, script, scriptParams)); + } + + if (!assumeSorted) { + // we need values to be sorted and unique for efficiency + config.ensureSorted(true); + } + + if (field == null) { + return new RangeAggregator.Factory(aggregationName, config, InternalRange.FACTORY, ranges, keyed); + } + + + FieldMapper mapper = context.smartNameFieldMapper(field); + if (mapper == null) { + config.unmapped(true); + return new RangeAggregator.Factory(aggregationName, config, InternalRange.FACTORY, ranges, keyed); + } + + IndexFieldData indexFieldData = context.fieldData().getForField(mapper); + config.fieldContext(new FieldContext(field, indexFieldData)); + return new RangeAggregator.Factory(aggregationName, config, InternalRange.FACTORY, ranges, keyed); + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/range/date/DateRange.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/date/DateRange.java new file mode 100644 index 00000000000..1d5c8bdb12b --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/date/DateRange.java @@ -0,0 +1,37 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.range.date; + +import org.elasticsearch.search.aggregations.bucket.range.RangeBase; +import org.joda.time.DateTime; + +/** + * + */ +public interface DateRange extends RangeBase { + + static interface Bucket extends RangeBase.Bucket { + + DateTime getFromAsDate(); + + DateTime getToAsDate(); + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/range/date/DateRangeBuilder.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/date/DateRangeBuilder.java new file mode 100644 index 00000000000..06bef46dfc0 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/date/DateRangeBuilder.java @@ -0,0 +1,57 @@ +package org.elasticsearch.search.aggregations.bucket.range.date; + +import org.elasticsearch.common.xcontent.XContentBuilder; +import org.elasticsearch.search.aggregations.bucket.range.RangeBuilderBase; + +import java.io.IOException; + +/** + * + */ +public class DateRangeBuilder extends RangeBuilderBase { + + private String format; + + public DateRangeBuilder(String name) { + super(name, InternalDateRange.TYPE.name()); + } + + public DateRangeBuilder addRange(String key, Object from, Object to) { + ranges.add(new Range(key, from, to)); + return this; + } + + public DateRangeBuilder addRange(Object from, Object to) { + return addRange(null, from, to); + } + + public DateRangeBuilder addUnboundedTo(String key, Object to) { + ranges.add(new Range(key, null, to)); + return this; + } + + public DateRangeBuilder addUnboundedTo(Object to) { + return addUnboundedTo(null, to); + } + + public DateRangeBuilder addUnboundedFrom(String key, Object from) { + ranges.add(new Range(key, from, null)); + return this; + } + + public DateRangeBuilder addUnboundedFrom(Object from) { + return addUnboundedFrom(null, from); + } + + public DateRangeBuilder format(String format) { + this.format = format; + return this; + } + + @Override + protected XContentBuilder doInternalXContent(XContentBuilder builder, Params params) throws IOException { + super.doInternalXContent(builder, params); + builder.field("format", format); + return builder; + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/range/date/DateRangeParser.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/date/DateRangeParser.java new file mode 100644 index 00000000000..906cf338443 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/date/DateRangeParser.java @@ -0,0 +1,170 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.range.date; + +import org.elasticsearch.common.xcontent.XContentParser; +import org.elasticsearch.index.fielddata.IndexFieldData; +import org.elasticsearch.index.mapper.FieldMapper; +import org.elasticsearch.index.mapper.core.DateFieldMapper; +import org.elasticsearch.search.SearchParseException; +import org.elasticsearch.search.aggregations.AggregationExecutionException; +import org.elasticsearch.search.aggregations.Aggregator; +import org.elasticsearch.search.aggregations.bucket.range.RangeAggregator; +import org.elasticsearch.search.aggregations.support.FieldContext; +import org.elasticsearch.search.aggregations.support.ValuesSourceConfig; +import org.elasticsearch.search.aggregations.support.numeric.NumericValuesSource; +import org.elasticsearch.search.aggregations.support.numeric.ValueFormatter; +import org.elasticsearch.search.aggregations.support.numeric.ValueParser; +import org.elasticsearch.search.aggregations.AggregatorFactory; +import org.elasticsearch.search.internal.SearchContext; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; + +/** + * + */ +public class DateRangeParser implements Aggregator.Parser { + + @Override + public String type() { + return InternalDateRange.TYPE.name(); + } + + @Override + public AggregatorFactory parse(String aggregationName, XContentParser parser, SearchContext context) throws IOException { + + ValuesSourceConfig config = new ValuesSourceConfig(NumericValuesSource.class); + + String field = null; + List ranges = null; + String script = null; + String scriptLang = null; + Map scriptParams = null; + boolean keyed = false; + String format = null; + boolean assumeSorted = false; + + XContentParser.Token token; + String currentFieldName = null; + while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) { + if (token == XContentParser.Token.FIELD_NAME) { + currentFieldName = parser.currentName(); + } else if (token == XContentParser.Token.VALUE_STRING) { + if ("field".equals(currentFieldName)) { + field = parser.text(); + } else if ("script".equals(currentFieldName)) { + script = parser.text(); + } else if ("script_lang".equals(currentFieldName) || "scriptLang".equals(currentFieldName)) { + scriptLang = parser.text(); + } else if ("format".equals(currentFieldName)) { + format = parser.text(); + } + } else if (token == XContentParser.Token.START_ARRAY) { + if ("ranges".equals(currentFieldName)) { + ranges = new ArrayList(); + while ((token = parser.nextToken()) != XContentParser.Token.END_ARRAY) { + double from = Double.NEGATIVE_INFINITY; + String fromAsStr = null; + double to = Double.POSITIVE_INFINITY; + String toAsStr = null; + String key = null; + String toOrFromOrKey = null; + while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) { + if (token == XContentParser.Token.FIELD_NAME) { + toOrFromOrKey = parser.currentName(); + } else if (token == XContentParser.Token.VALUE_NUMBER) { + if ("from".equals(toOrFromOrKey)) { + from = parser.doubleValue(); + } else if ("to".equals(toOrFromOrKey)) { + to = parser.doubleValue(); + } + } else if (token == XContentParser.Token.VALUE_STRING) { + if ("from".equals(toOrFromOrKey)) { + fromAsStr = parser.text(); + } else if ("to".equals(toOrFromOrKey)) { + toAsStr = parser.text(); + } else if ("key".equals(toOrFromOrKey)) { + key = parser.text(); + } + } + } + ranges.add(new RangeAggregator.Range(key, from, fromAsStr, to, toAsStr)); + } + } + } else if (token == XContentParser.Token.START_OBJECT) { + if ("params".equals(currentFieldName)) { + scriptParams = parser.map(); + } + } else if (token == XContentParser.Token.VALUE_BOOLEAN) { + if ("keyed".equals(currentFieldName)) { + keyed = parser.booleanValue(); + } else if ("script_values_sorted".equals(currentFieldName)) { + assumeSorted = parser.booleanValue(); + } + } + } + + if (ranges == null) { + throw new SearchParseException(context, "Missing [ranges] in ranges aggregator [" + aggregationName + "]"); + } + + if (script != null) { + config.script(context.scriptService().search(context.lookup(), scriptLang, script, scriptParams)); + } + + if (!assumeSorted) { + // we need values to be sorted and unique for efficiency + config.ensureSorted(true); + } + + if (format != null) { + config.formatter(new ValueFormatter.DateTime(format)); + } else { + config.formatter(ValueFormatter.DateTime.DEFAULT); + } + + config.parser(ValueParser.DateMath.DEFAULT); + + if (field == null) { + return new RangeAggregator.Factory(aggregationName, config, InternalDateRange.FACTORY, ranges, keyed); + } + + FieldMapper mapper = context.smartNameFieldMapper(field); + if (mapper == null) { + config.unmapped(true); + return new RangeAggregator.Factory(aggregationName, config, InternalDateRange.FACTORY, ranges, keyed); + } + + if (!(mapper instanceof DateFieldMapper)) { + throw new AggregationExecutionException("date_range aggregation can only be applied to date fields which is not the case with field [" + field + "]"); + } + + IndexFieldData indexFieldData = context.fieldData().getForField(mapper); + config.fieldContext(new FieldContext(field, indexFieldData)); + if (format == null) { + config.formatter(new ValueFormatter.DateTime(((DateFieldMapper) mapper).dateTimeFormatter())); + } + config.parser(new ValueParser.DateMath(((DateFieldMapper) mapper).dateMathParser())); + return new RangeAggregator.Factory(aggregationName, config, InternalDateRange.FACTORY, ranges, keyed); + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/range/date/InternalDateRange.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/date/InternalDateRange.java new file mode 100644 index 00000000000..01de1762a09 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/date/InternalDateRange.java @@ -0,0 +1,111 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.range.date; + +import org.elasticsearch.common.io.stream.StreamInput; +import org.elasticsearch.search.aggregations.AggregationStreams; +import org.elasticsearch.search.aggregations.InternalAggregation; +import org.elasticsearch.search.aggregations.InternalAggregations; +import org.elasticsearch.search.aggregations.bucket.range.AbstractRangeBase; +import org.elasticsearch.search.aggregations.support.numeric.ValueFormatter; +import org.joda.time.DateTime; +import org.joda.time.DateTimeZone; + +import java.io.IOException; +import java.util.List; + +/** + * + */ +public class InternalDateRange extends AbstractRangeBase implements DateRange { + + public final static Type TYPE = new Type("date_range", "drange"); + + private final static AggregationStreams.Stream STREAM = new AggregationStreams.Stream() { + @Override + public AbstractRangeBase readResult(StreamInput in) throws IOException { + InternalDateRange ranges = new InternalDateRange(); + ranges.readFrom(in); + return ranges; + } + }; + + public static void registerStream() { + AggregationStreams.registerStream(STREAM, TYPE.stream()); + } + + public static final Factory FACTORY = new Factory(); + + public static class Bucket extends AbstractRangeBase.Bucket implements DateRange.Bucket { + + public Bucket(String key, double from, double to, long docCount, List aggregations, ValueFormatter formatter) { + super(key, from, to, docCount, new InternalAggregations(aggregations), formatter); + } + + public Bucket(String key, double from, double to, long docCount, InternalAggregations aggregations, ValueFormatter formatter) { + super(key, from, to, docCount, aggregations, formatter); + } + + @Override + public DateTime getFromAsDate() { + return Double.isInfinite(getFrom()) ? null : new DateTime((long) getFrom(), DateTimeZone.UTC); + } + + @Override + public DateTime getToAsDate() { + return Double.isInfinite(getTo()) ? null : new DateTime((long) getTo(), DateTimeZone.UTC); + } + } + + private static class Factory implements AbstractRangeBase.Factory { + + @Override + public String type() { + return TYPE.name(); + } + + @Override + public AbstractRangeBase create(String name, List buckets, ValueFormatter formatter, boolean keyed) { + return new InternalDateRange(name, buckets, formatter, keyed); + } + + @Override + public Bucket createBucket(String key, double from, double to, long docCount, InternalAggregations aggregations, ValueFormatter formatter) { + return new Bucket(key, from, to, docCount, aggregations, formatter); + } + } + + public InternalDateRange() { + } + + public InternalDateRange(String name, List ranges, ValueFormatter formatter, boolean keyed) { + super(name, ranges, formatter, keyed); + } + + @Override + protected Bucket createBucket(String key, double from, double to, long docCount, InternalAggregations aggregations, ValueFormatter formatter) { + return new Bucket(key, from, to, docCount, aggregations, formatter); + } + + @Override + public Type type() { + return TYPE; + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/range/geodistance/GeoDistance.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/geodistance/GeoDistance.java new file mode 100644 index 00000000000..28847241d80 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/geodistance/GeoDistance.java @@ -0,0 +1,31 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.range.geodistance; + +import org.elasticsearch.search.aggregations.bucket.range.RangeBase; + +/** + * + */ +public interface GeoDistance extends RangeBase { + + public static interface Bucket extends RangeBase.Bucket {} + +} \ No newline at end of file diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/range/geodistance/GeoDistanceBuilder.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/geodistance/GeoDistanceBuilder.java new file mode 100644 index 00000000000..ca80f855e16 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/geodistance/GeoDistanceBuilder.java @@ -0,0 +1,172 @@ +package org.elasticsearch.search.aggregations.bucket.range.geodistance; + +import com.google.common.collect.Lists; +import org.elasticsearch.common.geo.GeoDistance; +import org.elasticsearch.common.geo.GeoPoint; +import org.elasticsearch.common.unit.DistanceUnit; +import org.elasticsearch.common.xcontent.ToXContent; +import org.elasticsearch.common.xcontent.XContentBuilder; +import org.elasticsearch.search.aggregations.AggregationBuilder; +import org.elasticsearch.search.builder.SearchSourceBuilderException; + +import java.io.IOException; +import java.util.List; +import java.util.Locale; + +/** + * + */ +public class GeoDistanceBuilder extends AggregationBuilder { + + public static class Range implements ToXContent { + + private String key; + private Double from; + private Double to; + + public Range(String key, Double from, Double to) { + this.key = key; + this.from = from; + this.to = to; + } + + @Override + public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException { + builder.startObject(); + if (from != null) { + builder.field("from", from.doubleValue()); + } + if (to != null) { + builder.field("to", to.doubleValue()); + } + if (key != null) { + builder.field("key", key); + } + return builder.endObject(); + } + + } + + private String field; + private DistanceUnit unit; + private GeoDistance distanceType; + private GeoPoint point; + + private List ranges = Lists.newArrayList(); + + public GeoDistanceBuilder(String name) { + super(name, InternalGeoDistance.TYPE.name()); + } + + public GeoDistanceBuilder field(String field) { + this.field = field; + return this; + } + + public GeoDistanceBuilder unit(DistanceUnit unit) { + this.unit = unit; + return this; + } + + public GeoDistanceBuilder distanceType(GeoDistance distanceType) { + this.distanceType = distanceType; + return this; + } + + public GeoDistanceBuilder point(String latLon) { + return point(GeoPoint.parseFromLatLon(latLon)); + } + + public GeoDistanceBuilder point(GeoPoint point) { + this.point = point; + return this; + } + + public GeoDistanceBuilder geohash(String geohash) { + if (this.point == null) { + this.point = new GeoPoint(); + } + this.point.resetFromGeoHash(geohash); + return this; + } + + public GeoDistanceBuilder lat(double lat) { + if (this.point == null) { + point = new GeoPoint(); + } + point.resetLat(lat); + return this; + } + + public GeoDistanceBuilder lon(double lon) { + if (this.point == null) { + point = new GeoPoint(); + } + point.resetLon(lon); + return this; + } + + public GeoDistanceBuilder addRange(String key, double from, double to) { + ranges.add(new Range(key, from, to)); + return this; + } + + public GeoDistanceBuilder addRange(double from, double to) { + return addRange(null, from, to); + } + + public GeoDistanceBuilder addUnboundedTo(String key, double to) { + ranges.add(new Range(key, null, to)); + return this; + } + + public GeoDistanceBuilder addUnboundedTo(double to) { + return addUnboundedTo(null, to); + } + + public GeoDistanceBuilder addUnboundedFrom(String key, double from) { + ranges.add(new Range(key, from, null)); + return this; + } + + public GeoDistanceBuilder addUnboundedFrom(double from) { + return addUnboundedFrom(null, from); + } + + @Override + protected XContentBuilder internalXContent(XContentBuilder builder, Params params) throws IOException { + builder.startObject(); + if (ranges.isEmpty()) { + throw new SearchSourceBuilderException("at least one range must be defined for geo_distance aggregation [" + name + "]"); + } + if (point == null) { + throw new SearchSourceBuilderException("center point must be defined for geo_distance aggregation [" + name + "]"); + } + + if (field != null) { + builder.field("field", field); + } + + if (unit != null) { + builder.field("unit", unit); + } + + if (distanceType != null) { + builder.field("distance_type", distanceType.name().toLowerCase(Locale.ROOT)); + } + + builder.startObject("center") + .field("lat", point.lat()) + .field("lon", point.lon()) + .endObject(); + + builder.startArray("ranges"); + for (Range range : ranges) { + range.toXContent(builder, params); + } + builder.endArray(); + + return builder.endObject(); + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/range/geodistance/GeoDistanceParser.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/geodistance/GeoDistanceParser.java new file mode 100644 index 00000000000..81205a37ee9 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/geodistance/GeoDistanceParser.java @@ -0,0 +1,290 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.range.geodistance; + +import org.elasticsearch.common.geo.GeoDistance; +import org.elasticsearch.common.geo.GeoPoint; +import org.elasticsearch.common.unit.DistanceUnit; +import org.elasticsearch.common.xcontent.XContentParser; +import org.elasticsearch.index.fielddata.*; +import org.elasticsearch.index.mapper.FieldMapper; +import org.elasticsearch.search.SearchParseException; +import org.elasticsearch.search.aggregations.Aggregator; +import org.elasticsearch.search.aggregations.bucket.range.AbstractRangeBase; +import org.elasticsearch.search.aggregations.bucket.range.RangeAggregator; +import org.elasticsearch.search.aggregations.bucket.range.RangeAggregator.Unmapped; +import org.elasticsearch.search.aggregations.support.AggregationContext; +import org.elasticsearch.search.aggregations.support.FieldContext; +import org.elasticsearch.search.aggregations.support.FieldDataSource; +import org.elasticsearch.search.aggregations.support.ValuesSourceConfig; +import org.elasticsearch.search.aggregations.support.geopoints.GeoPointValuesSource; +import org.elasticsearch.search.aggregations.support.numeric.NumericValuesSource; +import org.elasticsearch.search.aggregations.AggregatorFactory; +import org.elasticsearch.search.aggregations.support.ValueSourceAggregatorFactory; +import org.elasticsearch.search.internal.SearchContext; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; + +/** + * + */ +public class GeoDistanceParser implements Aggregator.Parser { + + @Override + public String type() { + return InternalGeoDistance.TYPE.name(); + } + + private static String key(String key, double from, double to) { + if (key != null) { + return key; + } + StringBuilder sb = new StringBuilder(); + sb.append(from == 0 ? "*" : from); + sb.append("-"); + sb.append(Double.isInfinite(to) ? "*" : to); + return sb.toString(); + } + + @Override + public AggregatorFactory parse(String aggregationName, XContentParser parser, SearchContext context) throws IOException { + + String field = null; + List ranges = null; + GeoPoint origin = null; + DistanceUnit unit = DistanceUnit.KILOMETERS; + GeoDistance distanceType = GeoDistance.ARC; + boolean keyed = false; + + XContentParser.Token token; + String currentFieldName = null; + while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) { + if (token == XContentParser.Token.FIELD_NAME) { + currentFieldName = parser.currentName(); + } else if (token == XContentParser.Token.VALUE_STRING) { + if ("field".equals(currentFieldName)) { + field = parser.text(); + } else if ("unit".equals(currentFieldName)) { + unit = DistanceUnit.fromString(parser.text()); + } else if ("distance_type".equals(currentFieldName) || "distanceType".equals(currentFieldName)) { + distanceType = GeoDistance.fromString(parser.text()); + } else if ("point".equals(currentFieldName) || "origin".equals(currentFieldName) || "center".equals(currentFieldName)) { + origin = new GeoPoint(); + origin.resetFromString(parser.text()); + } + } else if (token == XContentParser.Token.VALUE_BOOLEAN) { + if ("keyed".equals(currentFieldName)) { + keyed = parser.booleanValue(); + } + } else if (token == XContentParser.Token.START_ARRAY) { + if ("ranges".equals(currentFieldName)) { + ranges = new ArrayList(); + while ((token = parser.nextToken()) != XContentParser.Token.END_ARRAY) { + String fromAsStr = null; + String toAsStr = null; + double from = 0.0; + double to = Double.POSITIVE_INFINITY; + String key = null; + String toOrFromOrKey = null; + while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) { + if (token == XContentParser.Token.FIELD_NAME) { + toOrFromOrKey = parser.currentName(); + } else if (token == XContentParser.Token.VALUE_NUMBER) { + if ("from".equals(toOrFromOrKey)) { + from = parser.doubleValue(); + } else if ("to".equals(toOrFromOrKey)) { + to = parser.doubleValue(); + } + } else if (token == XContentParser.Token.VALUE_STRING) { + if ("key".equals(toOrFromOrKey)) { + key = parser.text(); + } else if ("from".equals(toOrFromOrKey)) { + fromAsStr = parser.text(); + } else if ("to".equals(toOrFromOrKey)) { + toAsStr = parser.text(); + } + } + } + ranges.add(new RangeAggregator.Range(key(key, from, to), from, fromAsStr, to, toAsStr)); + } + } else if ("point".equals(currentFieldName) || "origin".equals(currentFieldName) || "center".equals(currentFieldName)) { + double lat = Double.NaN; + double lon = Double.NaN; + while ((token = parser.nextToken()) != XContentParser.Token.END_ARRAY) { + if (Double.isNaN(lon)) { + lon = parser.doubleValue(); + } else if (Double.isNaN(lat)) { + lat = parser.doubleValue(); + } else { + throw new SearchParseException(context, "malformed [origin] geo point array in geo_distance aggregator [" + aggregationName + "]. " + + "a geo point array must be of the form [lon, lat]"); + } + } + origin = new GeoPoint(lat, lon); + } + } else if (token == XContentParser.Token.START_OBJECT) { + if ("point".equals(currentFieldName) || "origin".equals(currentFieldName) || "center".equals(currentFieldName)) { + double lat = Double.NaN; + double lon = Double.NaN; + while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) { + if (token == XContentParser.Token.FIELD_NAME) { + currentFieldName = parser.currentName(); + } else if (token == XContentParser.Token.VALUE_NUMBER) { + if ("lat".equals(currentFieldName)) { + lat = parser.doubleValue(); + } else if ("lon".equals(currentFieldName)) { + lon = parser.doubleValue(); + } + } + } + if (Double.isNaN(lat) || Double.isNaN(lon)) { + throw new SearchParseException(context, "malformed [origin] geo point object. either [lat] or [lon] (or both) are " + + "missing in geo_distance aggregator [" + aggregationName + "]"); + } + origin = new GeoPoint(lat, lon); + } + } + } + + if (ranges == null) { + throw new SearchParseException(context, "Missing [ranges] in geo_distance aggregator [" + aggregationName + "]"); + } + + if (origin == null) { + throw new SearchParseException(context, "Missing [origin] in geo_distance aggregator [" + aggregationName + "]"); + } + + ValuesSourceConfig config = new ValuesSourceConfig(GeoPointValuesSource.class); + + if (field == null) { + return new GeoDistanceFactory(aggregationName, config, InternalGeoDistance.FACTORY, origin, unit, distanceType, ranges, keyed); + } + + FieldMapper mapper = context.smartNameFieldMapper(field); + if (mapper == null) { + config.unmapped(true); + return new GeoDistanceFactory(aggregationName, config, InternalGeoDistance.FACTORY, origin, unit, distanceType, ranges, keyed); + } + + IndexFieldData indexFieldData = context.fieldData().getForField(mapper); + config.fieldContext(new FieldContext(field, indexFieldData)); + return new GeoDistanceFactory(aggregationName, config, InternalGeoDistance.FACTORY, origin, unit, distanceType, ranges, keyed); + } + + private static class GeoDistanceFactory extends ValueSourceAggregatorFactory { + + private final GeoPoint origin; + private final DistanceUnit unit; + private final GeoDistance distanceType; + private final AbstractRangeBase.Factory rangeFactory; + private final List ranges; + private final boolean keyed; + + public GeoDistanceFactory(String name, ValuesSourceConfig valueSourceConfig, + AbstractRangeBase.Factory rangeFactory, GeoPoint origin, DistanceUnit unit, GeoDistance distanceType, + List ranges, boolean keyed) { + super(name, rangeFactory.type(), valueSourceConfig); + this.origin = origin; + this.unit = unit; + this.distanceType = distanceType; + this.rangeFactory = rangeFactory; + this.ranges = ranges; + this.keyed = keyed; + } + + @Override + protected Aggregator createUnmapped(AggregationContext aggregationContext, Aggregator parent) { + return new Unmapped(name, ranges, keyed, valuesSourceConfig.formatter(), valuesSourceConfig.parser(), aggregationContext, parent, rangeFactory); + } + + @Override + protected Aggregator create(final GeoPointValuesSource valuesSource, long expectedBucketsCount, AggregationContext aggregationContext, Aggregator parent) { + final DistanceValues distanceValues = new DistanceValues(valuesSource, distanceType, origin, unit); + FieldDataSource.Numeric distanceSource = new DistanceSource(distanceValues); + distanceSource = new FieldDataSource.Numeric.SortedAndUnique(distanceSource); + final NumericValuesSource numericSource = new NumericValuesSource(distanceSource, null, null); + return new RangeAggregator(name, factories, numericSource, rangeFactory, ranges, keyed, aggregationContext, parent); + } + + private static class DistanceValues extends DoubleValues { + + private final GeoPointValuesSource geoPointValues; + private GeoPointValues geoValues; + private final GeoDistance distanceType; + private final GeoPoint origin; + private final DistanceUnit unit; + + protected DistanceValues(GeoPointValuesSource geoPointValues, GeoDistance distanceType, GeoPoint origin, DistanceUnit unit) { + super(true); + this.geoPointValues = geoPointValues; + this.distanceType = distanceType; + this.origin = origin; + this.unit = unit; + } + + @Override + public int setDocument(int docId) { + geoValues = geoPointValues.values(); + return geoValues.setDocument(docId); + } + + @Override + public double nextValue() { + final GeoPoint target = geoValues.nextValue(); + return distanceType.calculate(origin.getLat(), origin.getLon(), target.getLat(), target.getLon(), unit); + } + + } + + private static class DistanceSource extends FieldDataSource.Numeric { + + private final DoubleValues values; + + public DistanceSource(DoubleValues values) { + this.values = values; + } + + @Override + public boolean isFloatingPoint() { + return true; + } + + @Override + public LongValues longValues() { + throw new UnsupportedOperationException(); + } + + @Override + public DoubleValues doubleValues() { + return values; + } + + @Override + public BytesValues bytesValues() { + throw new UnsupportedOperationException(); + } + + } + + } + +} \ No newline at end of file diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/range/geodistance/InternalGeoDistance.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/geodistance/InternalGeoDistance.java new file mode 100644 index 00000000000..005ca13d78c --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/geodistance/InternalGeoDistance.java @@ -0,0 +1,99 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.range.geodistance; + +import org.elasticsearch.common.io.stream.StreamInput; +import org.elasticsearch.search.aggregations.AggregationStreams; +import org.elasticsearch.search.aggregations.InternalAggregation; +import org.elasticsearch.search.aggregations.InternalAggregations; +import org.elasticsearch.search.aggregations.bucket.range.AbstractRangeBase; +import org.elasticsearch.search.aggregations.support.numeric.ValueFormatter; + +import java.io.IOException; +import java.util.List; + +/** + * + */ +public class InternalGeoDistance extends AbstractRangeBase implements GeoDistance { + + public static final Type TYPE = new Type("geo_distance", "gdist"); + + public static final AggregationStreams.Stream STREAM = new AggregationStreams.Stream() { + @Override + public InternalGeoDistance readResult(StreamInput in) throws IOException { + InternalGeoDistance geoDistance = new InternalGeoDistance(); + geoDistance.readFrom(in); + return geoDistance; + } + }; + + public static void registerStream() { + AggregationStreams.registerStream(STREAM, TYPE.stream()); + } + + public static final Factory FACTORY = new Factory(); + + static class Bucket extends AbstractRangeBase.Bucket implements GeoDistance.Bucket { + + Bucket(String key, double from, double to, long docCount, List aggregations, ValueFormatter formatter) { + this(key, from, to, docCount, new InternalAggregations(aggregations), formatter); + } + + Bucket(String key, double from, double to, long docCount, InternalAggregations aggregations, ValueFormatter formatter) { + super(key, from, to, docCount, aggregations, formatter); + } + + } + + private static class Factory implements AbstractRangeBase.Factory { + + @Override + public String type() { + return TYPE.name(); + } + + @Override + public AbstractRangeBase create(String name, List buckets, ValueFormatter formatter, boolean keyed) { + return new InternalGeoDistance(name, buckets, formatter, keyed); + } + + @Override + public Bucket createBucket(String key, double from, double to, long docCount, InternalAggregations aggregations, ValueFormatter formatter) { + return new Bucket(key, from, to, docCount, aggregations, formatter); + } + } + + InternalGeoDistance() {} // for serialization + + public InternalGeoDistance(String name, List ranges, ValueFormatter formatter, boolean keyed) { + super(name, ranges, formatter, keyed); + } + + @Override + protected Bucket createBucket(String key, double from, double to, long docCount, InternalAggregations aggregations, ValueFormatter formatter) { + return new Bucket(key, from, to, docCount, aggregations, formatter); + } + + @Override + public Type type() { + return TYPE; + } +} \ No newline at end of file diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/range/ipv4/IPv4Range.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/ipv4/IPv4Range.java new file mode 100644 index 00000000000..b73448b401e --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/ipv4/IPv4Range.java @@ -0,0 +1,36 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.range.ipv4; + +import org.elasticsearch.search.aggregations.bucket.range.RangeBase; + +/** + * + */ +public interface IPv4Range extends RangeBase { + + static interface Bucket extends RangeBase.Bucket { + + String getFromAsString(); + + String getToAsString(); + + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/range/ipv4/IPv4RangeBuilder.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/ipv4/IPv4RangeBuilder.java new file mode 100644 index 00000000000..084a16b7437 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/ipv4/IPv4RangeBuilder.java @@ -0,0 +1,112 @@ +package org.elasticsearch.search.aggregations.bucket.range.ipv4; + +import org.elasticsearch.search.aggregations.bucket.range.RangeBuilderBase; +import org.elasticsearch.search.builder.SearchSourceBuilderException; + +import java.util.regex.Pattern; + +/** + * + */ +public class IPv4RangeBuilder extends RangeBuilderBase { + + public static final long MAX_IP = 4294967296l; + private static final Pattern MASK_PATTERN = Pattern.compile("[\\.|/]"); + + public IPv4RangeBuilder(String name) { + super(name, InternalIPv4Range.TYPE.name()); + } + + public IPv4RangeBuilder addRange(String key, String from, String to) { + ranges.add(new Range(key, from, to)); + return this; + } + + public IPv4RangeBuilder addMaskRange(String mask) { + return addMaskRange(mask, mask); + } + + public IPv4RangeBuilder addMaskRange(String key, String mask) { + long[] fromTo = cidrMaskToMinMax(mask); + if (fromTo == null) { + throw new SearchSourceBuilderException("invalid CIDR mask [" + mask + "] in ip_range aggregation [" + name + "]"); + } + ranges.add(new Range(key, fromTo[0] < 0 ? null : fromTo[0], fromTo[1] < 0 ? null : fromTo[1])); + return this; + } + + public IPv4RangeBuilder addRange(String from, String to) { + return addRange(null, from, to); + } + + public IPv4RangeBuilder addUnboundedTo(String key, String to) { + ranges.add(new Range(key, null, to)); + return this; + } + + public IPv4RangeBuilder addUnboundedTo(String to) { + return addUnboundedTo(null, to); + } + + public IPv4RangeBuilder addUnboundedFrom(String key, String from) { + ranges.add(new Range(key, from, null)); + return this; + } + + public IPv4RangeBuilder addUnboundedFrom(String from) { + return addUnboundedFrom(null, from); + } + + /** + * Computes the min & max ip addresses (represented as long values - same way as stored in index) represented by the given CIDR mask + * expression. The returned array has the length of 2, where the first entry represents the {@code min} address and the second the {@code max}. + * A {@code -1} value for either the {@code min} or the {@code max}, represents an unbounded end. In other words: + * + *

+ * {@code min == -1 == "0.0.0.0" } + *

+ * + * and + * + *

+ * {@code max == -1 == "255.255.255.255" } + *

+ * + * @param cidr + * @return + */ + static long[] cidrMaskToMinMax(String cidr) { + String[] parts = MASK_PATTERN.split(cidr); + if (parts.length != 5) { + return null; + } + int addr = (( Integer.parseInt(parts[0]) << 24 ) & 0xFF000000) + | (( Integer.parseInt(parts[1]) << 16 ) & 0xFF0000) + | (( Integer.parseInt(parts[2]) << 8 ) & 0xFF00) + | ( Integer.parseInt(parts[3]) & 0xFF); + + int mask = (-1) << (32 - Integer.parseInt(parts[4])); + + int from = addr & mask; + long longFrom = intIpToLongIp(from); + if (longFrom == 0) { + longFrom = -1; + } + + int to = from + (~mask); + long longTo = intIpToLongIp(to) + 1; // we have to +1 the here as the range is non-inclusive on the "to" side + if (longTo == MAX_IP) { + longTo = -1; + } + + return new long[] { longFrom, longTo }; + } + + public static long intIpToLongIp(int i) { + long p1 = ((long) ((i >> 24 ) & 0xFF)) << 24; + int p2 = ((i >> 16 ) & 0xFF) << 16; + int p3 = ((i >> 8 ) & 0xFF) << 8; + int p4 = i & 0xFF; + return p1 + p2 + p3 + p4; + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/range/ipv4/InternalIPv4Range.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/ipv4/InternalIPv4Range.java new file mode 100644 index 00000000000..efb83aacfbb --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/ipv4/InternalIPv4Range.java @@ -0,0 +1,112 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.range.ipv4; + +import org.elasticsearch.common.io.stream.StreamInput; +import org.elasticsearch.search.aggregations.AggregationStreams; +import org.elasticsearch.search.aggregations.InternalAggregation; +import org.elasticsearch.search.aggregations.InternalAggregations; +import org.elasticsearch.search.aggregations.bucket.range.AbstractRangeBase; +import org.elasticsearch.search.aggregations.support.numeric.ValueFormatter; + +import java.io.IOException; +import java.util.List; + +/** + * + */ +public class InternalIPv4Range extends AbstractRangeBase implements IPv4Range { + + public static final long MAX_IP = 4294967296l; + + public final static Type TYPE = new Type("ip_range", "iprange"); + + private final static AggregationStreams.Stream STREAM = new AggregationStreams.Stream() { + @Override + public AbstractRangeBase readResult(StreamInput in) throws IOException { + InternalIPv4Range range = new InternalIPv4Range(); + range.readFrom(in); + return range; + } + }; + + public static void registerStream() { + AggregationStreams.registerStream(STREAM, TYPE.stream()); + } + + public static final Factory FACTORY = new Factory(); + + public static class Bucket extends AbstractRangeBase.Bucket implements IPv4Range.Bucket { + + public Bucket(String key, double from, double to, long docCount, List aggregations, ValueFormatter formatter) { + super(key, from, to, docCount, new InternalAggregations(aggregations), formatter); + } + + public Bucket(String key, double from, double to, long docCount, InternalAggregations aggregations, ValueFormatter formatter) { + super(key, from, to, docCount, aggregations, formatter); + } + + @Override + public String getFromAsString() { + return Double.isInfinite(getFrom()) ? null : getFrom() == 0 ? null : ValueFormatter.IPv4.format(getFrom()); + } + + @Override + public String getToAsString() { + return Double.isInfinite(getTo()) ? null : MAX_IP == getTo() ? null : ValueFormatter.IPv4.format(getTo()); + } + } + + private static class Factory implements AbstractRangeBase.Factory { + + @Override + public String type() { + return TYPE.name(); + } + + @Override + public AbstractRangeBase create(String name, List buckets, ValueFormatter formatter, boolean keyed) { + return new InternalIPv4Range(name, buckets, keyed); + } + + @Override + public Bucket createBucket(String key, double from, double to, long docCount, InternalAggregations aggregations, ValueFormatter formatter) { + return new Bucket(key, from, to, docCount, aggregations, formatter); + } + } + + public InternalIPv4Range() { + } + + public InternalIPv4Range(String name, List ranges, boolean keyed) { + super(name, ranges, ValueFormatter.IPv4, keyed); + } + + @Override + protected Bucket createBucket(String key, double from, double to, long docCount, InternalAggregations aggregations, ValueFormatter formatter) { + return new Bucket(key, from, to, docCount, aggregations, formatter); + } + + @Override + public Type type() { + return TYPE; + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/range/ipv4/IpRangeParser.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/ipv4/IpRangeParser.java new file mode 100644 index 00000000000..9bd802d249a --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/range/ipv4/IpRangeParser.java @@ -0,0 +1,178 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.range.ipv4; + +import org.elasticsearch.common.xcontent.XContentParser; +import org.elasticsearch.index.fielddata.IndexFieldData; +import org.elasticsearch.index.mapper.FieldMapper; +import org.elasticsearch.index.mapper.ip.IpFieldMapper; +import org.elasticsearch.search.SearchParseException; +import org.elasticsearch.search.aggregations.AggregationExecutionException; +import org.elasticsearch.search.aggregations.Aggregator; +import org.elasticsearch.search.aggregations.bucket.range.RangeAggregator; +import org.elasticsearch.search.aggregations.support.FieldContext; +import org.elasticsearch.search.aggregations.support.ValuesSourceConfig; +import org.elasticsearch.search.aggregations.support.numeric.NumericValuesSource; +import org.elasticsearch.search.aggregations.support.numeric.ValueFormatter; +import org.elasticsearch.search.aggregations.support.numeric.ValueParser; +import org.elasticsearch.search.aggregations.AggregatorFactory; +import org.elasticsearch.search.internal.SearchContext; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; + +/** + * + */ +public class IpRangeParser implements Aggregator.Parser { + + @Override + public String type() { + return InternalIPv4Range.TYPE.name(); + } + + @Override + public AggregatorFactory parse(String aggregationName, XContentParser parser, SearchContext context) throws IOException { + + ValuesSourceConfig config = new ValuesSourceConfig(NumericValuesSource.class); + + String field = null; + List ranges = null; + String script = null; + String scriptLang = null; + Map scriptParams = null; + boolean keyed = false; + boolean assumeSorted = false; + + XContentParser.Token token; + String currentFieldName = null; + while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) { + if (token == XContentParser.Token.FIELD_NAME) { + currentFieldName = parser.currentName(); + } else if (token == XContentParser.Token.VALUE_STRING) { + if ("field".equals(currentFieldName)) { + field = parser.text(); + } else if ("script".equals(currentFieldName)) { + script = parser.text(); + } else if ("script_lang".equals(currentFieldName) || "scriptLang".equals(currentFieldName)) { + scriptLang = parser.text(); + } + } else if (token == XContentParser.Token.START_ARRAY) { + if ("ranges".equals(currentFieldName)) { + ranges = new ArrayList(); + while ((token = parser.nextToken()) != XContentParser.Token.END_ARRAY) { + double from = Double.NEGATIVE_INFINITY; + String fromAsStr = null; + double to = Double.POSITIVE_INFINITY; + String toAsStr = null; + String key = null; + String mask = null; + String toOrFromOrMaskOrKey = null; + while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) { + if (token == XContentParser.Token.FIELD_NAME) { + toOrFromOrMaskOrKey = parser.currentName(); + } else if (token == XContentParser.Token.VALUE_NUMBER) { + if ("from".equals(toOrFromOrMaskOrKey)) { + from = parser.doubleValue(); + } else if ("to".equals(toOrFromOrMaskOrKey)) { + to = parser.doubleValue(); + } + } else if (token == XContentParser.Token.VALUE_STRING) { + if ("from".equals(toOrFromOrMaskOrKey)) { + fromAsStr = parser.text(); + } else if ("to".equals(toOrFromOrMaskOrKey)) { + toAsStr = parser.text(); + } else if ("key".equals(toOrFromOrMaskOrKey)) { + key = parser.text(); + } else if ("mask".equals(toOrFromOrMaskOrKey)) { + mask = parser.text(); + } + } + } + RangeAggregator.Range range = new RangeAggregator.Range(key, from, fromAsStr, to, toAsStr); + if (mask != null) { + parseMaskRange(mask, range, aggregationName, context); + } + ranges.add(range); + } + } + } else if (token == XContentParser.Token.START_OBJECT) { + if ("params".equals(currentFieldName)) { + scriptParams = parser.map(); + } + } else if (token == XContentParser.Token.VALUE_BOOLEAN) { + if ("keyed".equals(currentFieldName)) { + keyed = parser.booleanValue(); + } else if ("script_values_sorted".equals(currentFieldName)) { + assumeSorted = parser.booleanValue(); + } + } + } + + if (ranges == null) { + throw new SearchParseException(context, "Missing [ranges] in ranges aggregator [" + aggregationName + "]"); + } + + if (script != null) { + config.script(context.scriptService().search(context.lookup(), scriptLang, script, scriptParams)); + } + + if (!assumeSorted) { + // we need values to be sorted and unique for efficiency + config.ensureSorted(true); + } + + config.formatter(ValueFormatter.IPv4); + config.parser(ValueParser.IPv4); + + if (field == null) { + return new RangeAggregator.Factory(aggregationName, config, InternalIPv4Range.FACTORY, ranges, keyed); + } + + FieldMapper mapper = context.smartNameFieldMapper(field); + if (mapper == null) { + config.unmapped(true); + return new RangeAggregator.Factory(aggregationName, config, InternalIPv4Range.FACTORY, ranges, keyed); + } + + if (!(mapper instanceof IpFieldMapper)) { + throw new AggregationExecutionException("ip_range aggregation can only be applied to ip fields which is not the case with field [" + field + "]"); + } + + IndexFieldData indexFieldData = context.fieldData().getForField(mapper); + config.fieldContext(new FieldContext(field, indexFieldData)); + return new RangeAggregator.Factory(aggregationName, config, InternalIPv4Range.FACTORY, ranges, keyed); + } + + private static void parseMaskRange(String cidr, RangeAggregator.Range range, String aggregationName, SearchContext ctx) { + long[] fromTo = IPv4RangeBuilder.cidrMaskToMinMax(cidr); + if (fromTo == null) { + throw new SearchParseException(ctx, "invalid CIDR mask [" + cidr + "] in aggregation [" + aggregationName + "]"); + } + range.from = fromTo[0] < 0 ? Double.NEGATIVE_INFINITY : fromTo[0]; + range.to = fromTo[1] < 0 ? Double.POSITIVE_INFINITY : fromTo[1]; + if (range.key == null) { + range.key = cidr; + } + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/BucketPriorityQueue.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/BucketPriorityQueue.java new file mode 100644 index 00000000000..daad8c7eec6 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/BucketPriorityQueue.java @@ -0,0 +1,39 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.terms; + +import org.apache.lucene.util.PriorityQueue; + +import java.util.Comparator; + +public class BucketPriorityQueue extends PriorityQueue { + + private final Comparator comparator; + + public BucketPriorityQueue(int size, Comparator comparator) { + super(size); + this.comparator = comparator; + } + + @Override + protected boolean lessThan(Terms.Bucket a, Terms.Bucket b) { + return comparator.compare(a, b) > 0; // reverse, since we reverse again when adding to a list + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/DoubleTerms.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/DoubleTerms.java new file mode 100644 index 00000000000..0ff9a98f7f9 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/DoubleTerms.java @@ -0,0 +1,211 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.terms; + +import com.carrotsearch.hppc.DoubleObjectOpenHashMap; +import org.elasticsearch.common.io.stream.StreamInput; +import org.elasticsearch.common.io.stream.StreamOutput; +import org.elasticsearch.common.recycler.Recycler; +import org.elasticsearch.common.text.StringText; +import org.elasticsearch.common.text.Text; +import org.elasticsearch.common.xcontent.XContentBuilder; +import org.elasticsearch.search.aggregations.AggregationStreams; +import org.elasticsearch.search.aggregations.InternalAggregation; +import org.elasticsearch.search.aggregations.InternalAggregations; +import org.elasticsearch.search.aggregations.support.numeric.ValueFormatter; +import org.elasticsearch.search.aggregations.support.numeric.ValueFormatterStreams; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collection; +import java.util.List; + +/** + * + */ +public class DoubleTerms extends InternalTerms { + + public static final Type TYPE = new Type("terms", "dterms"); + + public static AggregationStreams.Stream STREAM = new AggregationStreams.Stream() { + @Override + public DoubleTerms readResult(StreamInput in) throws IOException { + DoubleTerms buckets = new DoubleTerms(); + buckets.readFrom(in); + return buckets; + } + }; + + public static void registerStreams() { + AggregationStreams.registerStream(STREAM, TYPE.stream()); + } + + static class Bucket extends InternalTerms.Bucket { + + double term; + + public Bucket(double term, long docCount, InternalAggregations aggregations) { + super(docCount, aggregations); + this.term = term; + } + + @Override + public Text getKey() { + return new StringText(String.valueOf(term)); + } + + @Override + public Number getKeyAsNumber() { + return term; + } + + @Override + protected int compareTerm(Terms.Bucket other) { + if (term > other.getKeyAsNumber().doubleValue()) { + return 1; + } + if (term < other.getKeyAsNumber().doubleValue()) { + return -1; + } + return 0; + } + } + + private ValueFormatter valueFormatter; + + DoubleTerms() {} // for serialization + + public DoubleTerms(String name, InternalOrder order, int requiredSize, Collection buckets) { + this(name, order, null, requiredSize, buckets); + } + + public DoubleTerms(String name, InternalOrder order, ValueFormatter valueFormatter, int requiredSize, Collection buckets) { + super(name, order, requiredSize, buckets); + this.valueFormatter = valueFormatter; + } + + @Override + public Type type() { + return TYPE; + } + + @Override + public InternalTerms reduce(ReduceContext reduceContext) { + List aggregations = reduceContext.aggregations(); + if (aggregations.size() == 1) { + return (InternalTerms) aggregations.get(0); + } + InternalTerms reduced = null; + + Recycler.V>> buckets = reduceContext.cacheRecycler().doubleObjectMap(-1); + for (InternalAggregation aggregation : aggregations) { + InternalTerms terms = (InternalTerms) aggregation; + if (terms instanceof UnmappedTerms) { + continue; + } + if (reduced == null) { + reduced = terms; + } + for (Terms.Bucket bucket : terms.buckets) { + + List existingBuckets = buckets.v().get(((Bucket) bucket).term); + if (existingBuckets == null) { + existingBuckets = new ArrayList(aggregations.size()); + buckets.v().put(((Bucket) bucket).term, existingBuckets); + } + existingBuckets.add((Bucket) bucket); + } + } + + if (reduced == null) { + // there are only unmapped terms, so we just return the first one (no need to reduce) + return (UnmappedTerms) aggregations.get(0); + } + + // TODO: would it be better to sort the backing array buffer of hppc map directly instead of using a PQ? + final int size = Math.min(requiredSize, buckets.v().size()); + BucketPriorityQueue ordered = new BucketPriorityQueue(size, order.comparator()); + boolean[] states = buckets.v().allocated; + Object[] internalBuckets = buckets.v().values; + for (int i = 0; i < states.length; i++) { + if (states[i]) { + List sameTermBuckets = (List) internalBuckets[i]; + ordered.insertWithOverflow(sameTermBuckets.get(0).reduce(sameTermBuckets, reduceContext.cacheRecycler())); + } + } + buckets.release(); + InternalTerms.Bucket[] list = new InternalTerms.Bucket[ordered.size()]; + for (int i = ordered.size() - 1; i >= 0; i--) { + list[i] = (Bucket) ordered.pop(); + } + reduced.buckets = Arrays.asList(list); + return reduced; + } + + @Override + public void readFrom(StreamInput in) throws IOException { + this.name = in.readString(); + this.order = InternalOrder.Streams.readOrder(in); + this.valueFormatter = ValueFormatterStreams.readOptional(in); + this.requiredSize = in.readVInt(); + int size = in.readVInt(); + List buckets = new ArrayList(size); + for (int i = 0; i < size; i++) { + buckets.add(new Bucket(in.readDouble(), in.readVLong(), InternalAggregations.readAggregations(in))); + } + this.buckets = buckets; + this.bucketMap = null; + } + + @Override + public void writeTo(StreamOutput out) throws IOException { + out.writeString(name); + InternalOrder.Streams.writeOrder(order, out); + ValueFormatterStreams.writeOptional(valueFormatter, out); + out.writeVInt(requiredSize); + out.writeVInt(buckets.size()); + for (InternalTerms.Bucket bucket : buckets) { + out.writeDouble(((Bucket) bucket).term); + out.writeVLong(bucket.getDocCount()); + ((InternalAggregations) bucket.getAggregations()).writeTo(out); + } + } + + @Override + public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException { + builder.startObject(name); + builder.startArray(CommonFields.BUCKETS); + for (InternalTerms.Bucket bucket : buckets) { + builder.startObject(); + builder.field(CommonFields.KEY, ((Bucket) bucket).term); + if (valueFormatter != null) { + builder.field(CommonFields.KEY_AS_STRING, valueFormatter.format(((Bucket) bucket).term)); + } + builder.field(CommonFields.DOC_COUNT, bucket.getDocCount()); + ((InternalAggregations) bucket.getAggregations()).toXContentInternal(builder, params); + builder.endObject(); + } + builder.endArray(); + builder.endObject(); + return builder; + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/DoubleTermsAggregator.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/DoubleTermsAggregator.java new file mode 100644 index 00000000000..4bde5c74672 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/DoubleTermsAggregator.java @@ -0,0 +1,126 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.terms; + +import org.elasticsearch.index.fielddata.DoubleValues; +import org.elasticsearch.search.aggregations.Aggregator; +import org.elasticsearch.search.aggregations.InternalAggregations; +import org.elasticsearch.search.aggregations.bucket.BucketsAggregator; +import org.elasticsearch.search.aggregations.bucket.LongHash; +import org.elasticsearch.search.aggregations.support.AggregationContext; +import org.elasticsearch.search.aggregations.support.numeric.NumericValuesSource; +import org.elasticsearch.search.aggregations.AggregatorFactories; + +import java.io.IOException; +import java.util.Arrays; +import java.util.Collections; + +/** + * + */ +public class DoubleTermsAggregator extends BucketsAggregator { + + private static final int INITIAL_CAPACITY = 50; // TODO sizing + + private final InternalOrder order; + private final int requiredSize; + private final NumericValuesSource valuesSource; + private final LongHash bucketOrds; + + public DoubleTermsAggregator(String name, AggregatorFactories factories, NumericValuesSource valuesSource, + InternalOrder order, int requiredSize, AggregationContext aggregationContext, Aggregator parent) { + super(name, BucketAggregationMode.PER_BUCKET, factories, INITIAL_CAPACITY, aggregationContext, parent); + this.valuesSource = valuesSource; + this.order = order; + this.requiredSize = requiredSize; + bucketOrds = new LongHash(INITIAL_CAPACITY); + } + + @Override + public boolean shouldCollect() { + return true; + } + + @Override + public void collect(int doc, long owningBucketOrdinal) throws IOException { + assert owningBucketOrdinal == 0; + final DoubleValues values = valuesSource.doubleValues(); + final int valuesCount = values.setDocument(doc); + + for (int i = 0; i < valuesCount; ++i) { + final double val = values.nextValue(); + final long bits = Double.doubleToRawLongBits(val); + long bucketOrdinal = bucketOrds.add(bits); + if (bucketOrdinal < 0) { // already seen + bucketOrdinal = - 1 - bucketOrdinal; + } + collectBucket(doc, bucketOrdinal); + } + } + + // private impl that stores a bucket ord. This allows for computing the aggregations lazily. + static class OrdinalBucket extends DoubleTerms.Bucket { + + long bucketOrd; + + public OrdinalBucket() { + super(0, 0, (InternalAggregations) null); + } + + } + + @Override + public DoubleTerms buildAggregation(long owningBucketOrdinal) { + assert owningBucketOrdinal == 0; + final int size = (int) Math.min(bucketOrds.size(), requiredSize); + + BucketPriorityQueue ordered = new BucketPriorityQueue(size, order.comparator()); + OrdinalBucket spare = null; + for (long i = 0; i < bucketOrds.capacity(); ++i) { + final long ord = bucketOrds.id(i); + if (ord < 0) { + // slot is not allocated + continue; + } + + if (spare == null) { + spare = new OrdinalBucket(); + } + spare.term = Double.longBitsToDouble(bucketOrds.key(i)); + spare.docCount = bucketDocCount(ord); + spare.bucketOrd = ord; + spare = (OrdinalBucket) ordered.insertWithOverflow(spare); + } + + final InternalTerms.Bucket[] list = new InternalTerms.Bucket[ordered.size()]; + for (int i = ordered.size() - 1; i >= 0; --i) { + final OrdinalBucket bucket = (OrdinalBucket) ordered.pop(); + bucket.aggregations = bucketAggregations(bucket.bucketOrd); + list[i] = bucket; + } + return new DoubleTerms(name, order, valuesSource.formatter(), requiredSize, Arrays.asList(list)); + } + + @Override + public DoubleTerms buildEmptyAggregation() { + return new DoubleTerms(name, order, valuesSource.formatter(), requiredSize, Collections.emptyList()); + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/InternalOrder.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/InternalOrder.java new file mode 100644 index 00000000000..9edea8ced64 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/InternalOrder.java @@ -0,0 +1,117 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.terms; + +import org.elasticsearch.common.io.stream.StreamInput; +import org.elasticsearch.common.io.stream.StreamOutput; +import org.elasticsearch.common.xcontent.XContentBuilder; + +import java.io.IOException; +import java.util.Comparator; + +/** + * + */ +class InternalOrder extends Terms.Order { + + final byte id; + final String key; + final boolean asc; + final Comparator comparator; + + InternalOrder(byte id, String key, boolean asc, Comparator comparator) { + this.id = id; + this.key = key; + this.asc = asc; + this.comparator = comparator; + } + + byte id() { + return id; + } + + String key() { + return key; + } + + boolean asc() { + return asc; + } + + @Override + protected Comparator comparator() { + return comparator; + } + + @Override + public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException { + return builder.startObject().field(key, asc ? "asc" : "desc").endObject(); + } + + static class Aggregation extends InternalOrder { + + static final byte ID = 0; + + Aggregation(String key, boolean asc) { + super(ID, key, asc, new Terms.Bucket.Comparator(key, asc)); + } + + Aggregation(String aggName, String valueName, boolean asc) { + super(ID, key(aggName, valueName), asc, new Terms.Bucket.Comparator(aggName, valueName, asc)); + } + + private static String key(String aggName, String valueName) { + return (valueName == null) ? aggName : aggName + "." + valueName; + } + + } + + public static class Streams { + + public static void writeOrder(InternalOrder order, StreamOutput out) throws IOException { + out.writeByte(order.id()); + if (order instanceof Aggregation) { + out.writeBoolean(((Terms.Bucket.Comparator) order.comparator).asc()); + out.writeString(((Terms.Bucket.Comparator) order.comparator).aggName()); + boolean hasValueName = ((Terms.Bucket.Comparator) order.comparator).aggName() != null; + out.writeBoolean(hasValueName); + if (hasValueName) { + out.writeString(((Terms.Bucket.Comparator) order.comparator).valueName()); + } + } + } + + public static InternalOrder readOrder(StreamInput in) throws IOException { + byte id = in.readByte(); + switch (id) { + case 1: return (InternalOrder) Terms.Order.COUNT_DESC; + case 2: return (InternalOrder) Terms.Order.COUNT_ASC; + case 3: return (InternalOrder) Terms.Order.TERM_DESC; + case 4: return (InternalOrder) Terms.Order.TERM_ASC; + case 0: + boolean asc = in.readBoolean(); + String key = in.readString(); + return new InternalOrder.Aggregation(key, asc); + default: + throw new RuntimeException("unknown histogram order"); + } + } + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/InternalTerms.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/InternalTerms.java new file mode 100644 index 00000000000..91bcea548ee --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/InternalTerms.java @@ -0,0 +1,176 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.terms; + +import com.google.common.collect.Maps; +import org.elasticsearch.cache.recycler.CacheRecycler; +import org.elasticsearch.common.io.stream.Streamable; +import org.elasticsearch.common.text.Text; +import org.elasticsearch.common.xcontent.ToXContent; +import org.elasticsearch.search.aggregations.Aggregations; +import org.elasticsearch.search.aggregations.InternalAggregation; +import org.elasticsearch.search.aggregations.InternalAggregations; + +import java.util.*; + +/** + * + */ +public abstract class InternalTerms extends InternalAggregation implements Terms, ToXContent, Streamable { + + public static abstract class Bucket implements Terms.Bucket { + + protected long docCount; + protected InternalAggregations aggregations; + + protected Bucket(long docCount, InternalAggregations aggregations) { + this.docCount = docCount; + this.aggregations = aggregations; + } + + @Override + public long getDocCount() { + return docCount; + } + + @Override + public Aggregations getAggregations() { + return aggregations; + } + + @Override + public int compareTo(Terms.Bucket o) { + long i = compareTerm(o); + if (i == 0) { + i = docCount - o.getDocCount(); + if (i == 0) { + i = System.identityHashCode(this) - System.identityHashCode(o); + } + } + return i > 0 ? 1 : -1; + } + + protected abstract int compareTerm(Terms.Bucket other); + + public Bucket reduce(List buckets, CacheRecycler cacheRecycler) { + if (buckets.size() == 1) { + return buckets.get(0); + } + Bucket reduced = null; + List aggregationsList = new ArrayList(buckets.size()); + for (Bucket bucket : buckets) { + if (reduced == null) { + reduced = bucket; + } else { + reduced.docCount += bucket.docCount; + } + aggregationsList.add(bucket.aggregations); + } + reduced.aggregations = InternalAggregations.reduce(aggregationsList, cacheRecycler); + return reduced; + } + } + + protected InternalOrder order; + protected int requiredSize; + protected Collection buckets; + protected Map bucketMap; + + protected InternalTerms() {} // for serialization + + protected InternalTerms(String name, InternalOrder order, int requiredSize, Collection buckets) { + super(name); + this.order = order; + this.requiredSize = requiredSize; + this.buckets = buckets; + } + + @Override + public Iterator iterator() { + Object o = buckets.iterator(); + return (Iterator) o; + } + + @Override + public Collection buckets() { + Object o = buckets; + return (Collection) o; + } + + @Override + public Terms.Bucket getByTerm(String term) { + if (bucketMap == null) { + bucketMap = Maps.newHashMapWithExpectedSize(buckets.size()); + for (Bucket bucket : buckets) { + bucketMap.put(bucket.getKey().string(), bucket); + } + } + return bucketMap.get(term); + } + + @Override + public InternalTerms reduce(ReduceContext reduceContext) { + List aggregations = reduceContext.aggregations(); + if (aggregations.size() == 1) { + return (InternalTerms) aggregations.get(0); + } + InternalTerms reduced = null; + + // TODO: would it be better to use a hppc map and then directly work on the backing array instead of using a PQ? + + Map> buckets = new HashMap>(requiredSize); + for (InternalAggregation aggregation : aggregations) { + InternalTerms terms = (InternalTerms) aggregation; + if (terms instanceof UnmappedTerms) { + continue; + } + if (reduced == null) { + reduced = terms; + } + for (Bucket bucket : terms.buckets) { + List existingBuckets = buckets.get(bucket.getKey()); + if (existingBuckets == null) { + existingBuckets = new ArrayList(aggregations.size()); + buckets.put(bucket.getKey(), existingBuckets); + } + existingBuckets.add(bucket); + } + } + + if (reduced == null) { + // there are only unmapped terms, so we just return the first one (no need to reduce) + return (UnmappedTerms) aggregations.get(0); + } + + final int size = Math.min(requiredSize, buckets.size()); + BucketPriorityQueue ordered = new BucketPriorityQueue(size, order.comparator()); + for (Map.Entry> entry : buckets.entrySet()) { + List sameTermBuckets = entry.getValue(); + ordered.insertWithOverflow(sameTermBuckets.get(0).reduce(sameTermBuckets, reduceContext.cacheRecycler())); + } + Bucket[] list = new Bucket[ordered.size()]; + for (int i = ordered.size() - 1; i >= 0; i--) { + list[i] = (Bucket) ordered.pop(); + } + reduced.buckets = Arrays.asList(list); + return reduced; + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/LongTerms.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/LongTerms.java new file mode 100644 index 00000000000..2c701390527 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/LongTerms.java @@ -0,0 +1,208 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.terms; + +import com.carrotsearch.hppc.LongObjectOpenHashMap; +import org.elasticsearch.common.io.stream.StreamInput; +import org.elasticsearch.common.io.stream.StreamOutput; +import org.elasticsearch.common.recycler.Recycler; +import org.elasticsearch.common.text.StringText; +import org.elasticsearch.common.text.Text; +import org.elasticsearch.common.xcontent.XContentBuilder; +import org.elasticsearch.search.aggregations.AggregationStreams; +import org.elasticsearch.search.aggregations.InternalAggregation; +import org.elasticsearch.search.aggregations.InternalAggregations; +import org.elasticsearch.search.aggregations.support.numeric.ValueFormatter; +import org.elasticsearch.search.aggregations.support.numeric.ValueFormatterStreams; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collection; +import java.util.List; + +/** + * + */ +public class LongTerms extends InternalTerms { + + public static final Type TYPE = new Type("terms", "lterms"); + + public static AggregationStreams.Stream STREAM = new AggregationStreams.Stream() { + @Override + public LongTerms readResult(StreamInput in) throws IOException { + LongTerms buckets = new LongTerms(); + buckets.readFrom(in); + return buckets; + } + }; + + public static void registerStreams() { + AggregationStreams.registerStream(STREAM, TYPE.stream()); + } + + + static class Bucket extends InternalTerms.Bucket { + + long term; + + public Bucket(long term, long docCount, InternalAggregations aggregations) { + super(docCount, aggregations); + this.term = term; + } + + @Override + public Text getKey() { + return new StringText(String.valueOf(term)); + } + + @Override + public Number getKeyAsNumber() { + return term; + } + + @Override + protected int compareTerm(Terms.Bucket other) { + long otherTerm = other.getKeyAsNumber().longValue(); + if (this.term > otherTerm) { + return 1; + } + if (this.term < otherTerm) { + return -1; + } + return 0; + } + } + + private ValueFormatter valueFormatter; + + LongTerms() {} // for serialization + + public LongTerms(String name, InternalOrder order, ValueFormatter valueFormatter, int requiredSize, Collection buckets) { + super(name, order, requiredSize, buckets); + this.valueFormatter = valueFormatter; + } + + @Override + public Type type() { + return TYPE; + } + + @Override + public InternalTerms reduce(ReduceContext reduceContext) { + List aggregations = reduceContext.aggregations(); + if (aggregations.size() == 1) { + return (InternalTerms) aggregations.get(0); + } + InternalTerms reduced = null; + + Recycler.V>> buckets = reduceContext.cacheRecycler().longObjectMap(-1); + for (InternalAggregation aggregation : aggregations) { + InternalTerms terms = (InternalTerms) aggregation; + if (terms instanceof UnmappedTerms) { + continue; + } + if (reduced == null) { + reduced = terms; + } + for (Terms.Bucket bucket : terms.buckets) { + List existingBuckets = buckets.v().get(((Bucket) bucket).term); + if (existingBuckets == null) { + existingBuckets = new ArrayList(aggregations.size()); + buckets.v().put(((Bucket) bucket).term, existingBuckets); + } + existingBuckets.add((Bucket) bucket); + } + } + + if (reduced == null) { + // there are only unmapped terms, so we just return the first one (no need to reduce) + return (UnmappedTerms) aggregations.get(0); + } + + // TODO: would it be better to sort the backing array buffer of the hppc map directly instead of using a PQ? + final int size = Math.min(requiredSize, buckets.v().size()); + BucketPriorityQueue ordered = new BucketPriorityQueue(size, order.comparator()); + Object[] internalBuckets = buckets.v().values; + boolean[] states = buckets.v().allocated; + for (int i = 0; i < states.length; i++) { + if (states[i]) { + List sameTermBuckets = (List) internalBuckets[i]; + ordered.insertWithOverflow(sameTermBuckets.get(0).reduce(sameTermBuckets, reduceContext.cacheRecycler())); + } + } + buckets.release(); + InternalTerms.Bucket[] list = new InternalTerms.Bucket[ordered.size()]; + for (int i = ordered.size() - 1; i >= 0; i--) { + list[i] = (Bucket) ordered.pop(); + } + reduced.buckets = Arrays.asList(list); + return reduced; + } + + @Override + public void readFrom(StreamInput in) throws IOException { + this.name = in.readString(); + this.order = InternalOrder.Streams.readOrder(in); + this.valueFormatter = ValueFormatterStreams.readOptional(in); + this.requiredSize = in.readVInt(); + int size = in.readVInt(); + List buckets = new ArrayList(size); + for (int i = 0; i < size; i++) { + buckets.add(new Bucket(in.readLong(), in.readVLong(), InternalAggregations.readAggregations(in))); + } + this.buckets = buckets; + this.bucketMap = null; + } + + @Override + public void writeTo(StreamOutput out) throws IOException { + out.writeString(name); + InternalOrder.Streams.writeOrder(order, out); + ValueFormatterStreams.writeOptional(valueFormatter, out); + out.writeVInt(requiredSize); + out.writeVInt(buckets.size()); + for (InternalTerms.Bucket bucket : buckets) { + out.writeLong(((Bucket) bucket).term); + out.writeVLong(bucket.getDocCount()); + ((InternalAggregations) bucket.getAggregations()).writeTo(out); + } + } + + @Override + public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException { + builder.startObject(name); + builder.startArray(CommonFields.BUCKETS); + for (InternalTerms.Bucket bucket : buckets) { + builder.startObject(); + builder.field(CommonFields.KEY, ((Bucket) bucket).term); + if (valueFormatter != null) { + builder.field(CommonFields.KEY_AS_STRING, valueFormatter.format(((Bucket) bucket).term)); + } + builder.field(CommonFields.DOC_COUNT, bucket.getDocCount()); + ((InternalAggregations) bucket.getAggregations()).toXContentInternal(builder, params); + builder.endObject(); + } + builder.endArray(); + builder.endObject(); + return builder; + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/LongTermsAggregator.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/LongTermsAggregator.java new file mode 100644 index 00000000000..9536021e335 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/LongTermsAggregator.java @@ -0,0 +1,125 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.terms; + +import org.elasticsearch.index.fielddata.LongValues; +import org.elasticsearch.search.aggregations.Aggregator; +import org.elasticsearch.search.aggregations.InternalAggregations; +import org.elasticsearch.search.aggregations.bucket.BucketsAggregator; +import org.elasticsearch.search.aggregations.bucket.LongHash; +import org.elasticsearch.search.aggregations.support.AggregationContext; +import org.elasticsearch.search.aggregations.support.numeric.NumericValuesSource; +import org.elasticsearch.search.aggregations.AggregatorFactories; + +import java.io.IOException; +import java.util.Arrays; +import java.util.Collections; + +/** + * + */ +public class LongTermsAggregator extends BucketsAggregator { + + private static final int INITIAL_CAPACITY = 50; // TODO sizing + + private final InternalOrder order; + private final int requiredSize; + private final NumericValuesSource valuesSource; + private final LongHash bucketOrds; + + public LongTermsAggregator(String name, AggregatorFactories factories, NumericValuesSource valuesSource, + InternalOrder order, int requiredSize, AggregationContext aggregationContext, Aggregator parent) { + super(name, BucketAggregationMode.PER_BUCKET, factories, INITIAL_CAPACITY, aggregationContext, parent); + this.valuesSource = valuesSource; + this.order = order; + this.requiredSize = requiredSize; + bucketOrds = new LongHash(INITIAL_CAPACITY); + } + + @Override + public boolean shouldCollect() { + return true; + } + + @Override + public void collect(int doc, long owningBucketOrdinal) throws IOException { + assert owningBucketOrdinal == 0; + final LongValues values = valuesSource.longValues(); + final int valuesCount = values.setDocument(doc); + + for (int i = 0; i < valuesCount; ++i) { + final long val = values.nextValue(); + long bucketOrdinal = bucketOrds.add(val); + if (bucketOrdinal < 0) { // already seen + bucketOrdinal = - 1 - bucketOrdinal; + } + collectBucket(doc, bucketOrdinal); + } + } + + // private impl that stores a bucket ord. This allows for computing the aggregations lazily. + static class OrdinalBucket extends LongTerms.Bucket { + + long bucketOrd; + + public OrdinalBucket() { + super(0, 0, (InternalAggregations) null); + } + + } + + @Override + public LongTerms buildAggregation(long owningBucketOrdinal) { + assert owningBucketOrdinal == 0; + final int size = (int) Math.min(bucketOrds.size(), requiredSize); + + BucketPriorityQueue ordered = new BucketPriorityQueue(size, order.comparator()); + OrdinalBucket spare = null; + for (long i = 0; i < bucketOrds.capacity(); ++i) { + final long ord = bucketOrds.id(i); + if (ord < 0) { + // slot is not allocated + continue; + } + + if (spare == null) { + spare = new OrdinalBucket(); + } + spare.term = bucketOrds.key(i); + spare.docCount = bucketDocCount(ord); + spare.bucketOrd = ord; + spare = (OrdinalBucket) ordered.insertWithOverflow(spare); + } + + final InternalTerms.Bucket[] list = new InternalTerms.Bucket[ordered.size()]; + for (int i = ordered.size() - 1; i >= 0; --i) { + final OrdinalBucket bucket = (OrdinalBucket) ordered.pop(); + bucket.aggregations = bucketAggregations(bucket.bucketOrd); + list[i] = bucket; + } + return new LongTerms(name, order, valuesSource.formatter(), requiredSize, Arrays.asList(list)); + } + + @Override + public LongTerms buildEmptyAggregation() { + return new LongTerms(name, order, valuesSource.formatter(), requiredSize, Collections.emptyList()); + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/StringTerms.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/StringTerms.java new file mode 100644 index 00000000000..e71b5eda079 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/StringTerms.java @@ -0,0 +1,143 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.terms; + +import org.apache.lucene.util.BytesRef; +import org.elasticsearch.common.bytes.BytesArray; +import org.elasticsearch.common.io.stream.StreamInput; +import org.elasticsearch.common.io.stream.StreamOutput; +import org.elasticsearch.common.text.BytesText; +import org.elasticsearch.common.text.Text; +import org.elasticsearch.common.xcontent.XContentBuilder; +import org.elasticsearch.search.aggregations.AggregationStreams; +import org.elasticsearch.search.aggregations.InternalAggregation; +import org.elasticsearch.search.aggregations.InternalAggregations; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Collection; +import java.util.List; + +/** + * + */ +public class StringTerms extends InternalTerms { + + public static final InternalAggregation.Type TYPE = new Type("terms", "sterms"); + + public static AggregationStreams.Stream STREAM = new AggregationStreams.Stream() { + @Override + public StringTerms readResult(StreamInput in) throws IOException { + StringTerms buckets = new StringTerms(); + buckets.readFrom(in); + return buckets; + } + }; + + public static void registerStreams() { + AggregationStreams.registerStream(STREAM, TYPE.stream()); + } + + + public static class Bucket extends InternalTerms.Bucket { + + final BytesRef termBytes; + + public Bucket(BytesRef term, long docCount, InternalAggregations aggregations) { + super(docCount, aggregations); + this.termBytes = term; + } + + @Override + public Text getKey() { + return new BytesText(new BytesArray(termBytes)); + } + + public String getTermAsString() { + return termBytes.utf8ToString(); + } + + @Override + public Number getKeyAsNumber() { + // this method is needed for scripted numeric faceting + return Double.parseDouble(termBytes.utf8ToString()); + } + + @Override + protected int compareTerm(Terms.Bucket other) { + return BytesRef.getUTF8SortedAsUnicodeComparator().compare(termBytes, ((Bucket) other).termBytes); + } + } + + StringTerms() {} // for serialization + + public StringTerms(String name, InternalOrder order, int requiredSize, Collection buckets) { + super(name, order, requiredSize, buckets); + } + + @Override + public Type type() { + return TYPE; + } + + @Override + public void readFrom(StreamInput in) throws IOException { + this.name = in.readString(); + this.order = InternalOrder.Streams.readOrder(in); + this.requiredSize = in.readVInt(); + int size = in.readVInt(); + List buckets = new ArrayList(size); + for (int i = 0; i < size; i++) { + buckets.add(new Bucket(in.readBytesRef(), in.readVLong(), InternalAggregations.readAggregations(in))); + } + this.buckets = buckets; + this.bucketMap = null; + } + + @Override + public void writeTo(StreamOutput out) throws IOException { + out.writeString(name); + InternalOrder.Streams.writeOrder(order, out); + out.writeVInt(requiredSize); + out.writeVInt(buckets.size()); + for (InternalTerms.Bucket bucket : buckets) { + out.writeBytesRef(((Bucket) bucket).termBytes); + out.writeVLong(bucket.getDocCount()); + ((InternalAggregations) bucket.getAggregations()).writeTo(out); + } + } + + @Override + public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException { + builder.startObject(name); + builder.startArray(CommonFields.BUCKETS); + for (InternalTerms.Bucket bucket : buckets) { + builder.startObject(); + builder.field(CommonFields.KEY, ((Bucket) bucket).termBytes); + builder.field(CommonFields.DOC_COUNT, bucket.getDocCount()); + ((InternalAggregations) bucket.getAggregations()).toXContentInternal(builder, params); + builder.endObject(); + } + builder.endArray(); + builder.endObject(); + return builder; + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/StringTermsAggregator.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/StringTermsAggregator.java new file mode 100644 index 00000000000..b539cb24976 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/StringTermsAggregator.java @@ -0,0 +1,122 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.terms; + +import org.apache.lucene.util.BytesRef; +import org.apache.lucene.util.BytesRefHash; +import org.elasticsearch.index.fielddata.BytesValues; +import org.elasticsearch.search.aggregations.Aggregator; +import org.elasticsearch.search.aggregations.bucket.BucketsAggregator; +import org.elasticsearch.search.aggregations.support.AggregationContext; +import org.elasticsearch.search.aggregations.support.ValuesSource; +import org.elasticsearch.search.aggregations.AggregatorFactories; + +import java.io.IOException; +import java.util.Arrays; +import java.util.Collections; + +/** + * An aggregator of string values. + */ +// TODO we need a similar aggregator that would use ords, similarly to TermsStringOrdinalsFacetExecutor +public class StringTermsAggregator extends BucketsAggregator { + + private static final int INITIAL_CAPACITY = 50; // TODO sizing + + private final ValuesSource valuesSource; + private final InternalOrder order; + private final int requiredSize; + private final BytesRefHash bucketOrds; + + public StringTermsAggregator(String name, AggregatorFactories factories, ValuesSource valuesSource, + InternalOrder order, int requiredSize, AggregationContext aggregationContext, Aggregator parent) { + + super(name, BucketAggregationMode.PER_BUCKET, factories, INITIAL_CAPACITY, aggregationContext, parent); + this.valuesSource = valuesSource; + this.order = order; + this.requiredSize = requiredSize; + bucketOrds = new BytesRefHash(); + } + + @Override + public boolean shouldCollect() { + return true; + } + + @Override + public void collect(int doc, long owningBucketOrdinal) throws IOException { + assert owningBucketOrdinal == 0; + final BytesValues values = valuesSource.bytesValues(); + final int valuesCount = values.setDocument(doc); + + for (int i = 0; i < valuesCount; ++i) { + final BytesRef bytes = values.nextValue(); + final int hash = values.currentValueHash(); + int bucketOrdinal = bucketOrds.add(bytes, hash); + if (bucketOrdinal < 0) { // already seen + bucketOrdinal = - 1 - bucketOrdinal; + } + collectBucket(doc, bucketOrdinal); + } + } + + // private impl that stores a bucket ord. This allows for computing the aggregations lazily. + static class OrdinalBucket extends StringTerms.Bucket { + + int bucketOrd; + + public OrdinalBucket() { + super(new BytesRef(), 0, null); + } + + } + + @Override + public StringTerms buildAggregation(long owningBucketOrdinal) { + assert owningBucketOrdinal == 0; + final int size = Math.min(bucketOrds.size(), requiredSize); + + BucketPriorityQueue ordered = new BucketPriorityQueue(size, order.comparator()); + OrdinalBucket spare = null; + for (int i = 0; i < bucketOrds.size(); ++i) { + if (spare == null) { + spare = new OrdinalBucket(); + } + bucketOrds.get(i, spare.termBytes); + spare.docCount = bucketDocCount(i); + spare.bucketOrd = i; + spare = (OrdinalBucket) ordered.insertWithOverflow(spare); + } + + final InternalTerms.Bucket[] list = new InternalTerms.Bucket[ordered.size()]; + for (int i = ordered.size() - 1; i >= 0; --i) { + final OrdinalBucket bucket = (OrdinalBucket) ordered.pop(); + bucket.aggregations = bucketAggregations(bucket.bucketOrd); + list[i] = bucket; + } + return new StringTerms(name, order, requiredSize, Arrays.asList(list)); + } + + @Override + public StringTerms buildEmptyAggregation() { + return new StringTerms(name, order, requiredSize, Collections.emptyList()); + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/Terms.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/Terms.java new file mode 100644 index 00000000000..aeeca9d91c1 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/Terms.java @@ -0,0 +1,153 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.terms; + +import org.elasticsearch.common.text.Text; +import org.elasticsearch.common.xcontent.ToXContent; +import org.elasticsearch.search.aggregations.Aggregation; +import org.elasticsearch.search.aggregations.support.ScriptValueType; + +import java.util.Collection; +import java.util.Comparator; + +/** + * + */ +public interface Terms extends Aggregation, Iterable { + + static enum ValueType { + + STRING(ScriptValueType.STRING), + LONG(ScriptValueType.LONG), + DOUBLE(ScriptValueType.DOUBLE); + + final ScriptValueType scriptValueType; + + private ValueType(ScriptValueType scriptValueType) { + this.scriptValueType = scriptValueType; + } + + static ValueType resolveType(String type) { + if ("string".equals(type)) { + return STRING; + } + if ("double".equals(type) || "float".equals(type)) { + return DOUBLE; + } + if ("long".equals(type) || "integer".equals(type) || "short".equals(type) || "byte".equals(type)) { + return LONG; + } + return null; + } + } + + static interface Bucket extends Comparable, org.elasticsearch.search.aggregations.bucket.Bucket { + + Text getKey(); + + Number getKeyAsNumber(); + } + + Collection buckets(); + + Bucket getByTerm(String term); + + + /** + * + */ + static abstract class Order implements ToXContent { + + /** + * Order by the (higher) count of each term. + */ + public static final Order COUNT_DESC = new InternalOrder((byte) 1, "_count", false, new Comparator() { + @Override + public int compare(Terms.Bucket o1, Terms.Bucket o2) { + long i = o2.getDocCount() - o1.getDocCount(); + if (i == 0) { + i = o2.compareTo(o1); + if (i == 0) { + i = System.identityHashCode(o2) - System.identityHashCode(o1); + } + } + return i > 0 ? 1 : -1; + } + }); + + /** + * Order by the (lower) count of each term. + */ + public static final Order COUNT_ASC = new InternalOrder((byte) 2, "_count", true, new Comparator() { + + @Override + public int compare(Terms.Bucket o1, Terms.Bucket o2) { + return -COUNT_DESC.comparator().compare(o1, o2); + } + }); + + /** + * Order by the terms. + */ + public static final Order TERM_DESC = new InternalOrder((byte) 3, "_term", false, new Comparator() { + + @Override + public int compare(Terms.Bucket o1, Terms.Bucket o2) { + return o2.compareTo(o1); + } + }); + + /** + * Order by the terms. + */ + public static final Order TERM_ASC = new InternalOrder((byte) 4, "_term", true, new Comparator() { + + @Override + public int compare(Terms.Bucket o1, Terms.Bucket o2) { + return -TERM_DESC.comparator().compare(o1, o2); + } + }); + + /** + * Creates a bucket ordering strategy which sorts buckets based on a single-valued calc get + * + * @param aggregationName the name of the get + * @param asc The direction of the order (ascending or descending) + */ + public static InternalOrder aggregation(String aggregationName, boolean asc) { + return new InternalOrder.Aggregation(aggregationName, null, asc); + } + + /** + * Creates a bucket ordering strategy which sorts buckets based on a multi-valued calc get + * + * @param aggregationName the name of the get + * @param valueName The name of the value of the multi-value get by which the sorting will be applied + * @param asc The direction of the order (ascending or descending) + */ + public static InternalOrder aggregation(String aggregationName, String valueName, boolean asc) { + return new InternalOrder.Aggregation(aggregationName, valueName, asc); + } + + + protected abstract Comparator comparator(); + + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/TermsAggregatorFactory.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/TermsAggregatorFactory.java new file mode 100644 index 00000000000..db89aa0714f --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/TermsAggregatorFactory.java @@ -0,0 +1,67 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.terms; + +import org.elasticsearch.search.aggregations.AggregationExecutionException; +import org.elasticsearch.search.aggregations.Aggregator; +import org.elasticsearch.search.aggregations.support.AggregationContext; +import org.elasticsearch.search.aggregations.support.ValuesSource; +import org.elasticsearch.search.aggregations.support.ValuesSourceConfig; +import org.elasticsearch.search.aggregations.support.bytes.BytesValuesSource; +import org.elasticsearch.search.aggregations.support.numeric.NumericValuesSource; +import org.elasticsearch.search.aggregations.support.ValueSourceAggregatorFactory; + +/** + * + */ +public class TermsAggregatorFactory extends ValueSourceAggregatorFactory { + + private final InternalOrder order; + private final int requiredSize; + + public TermsAggregatorFactory(String name, ValuesSourceConfig valueSourceConfig, InternalOrder order, int requiredSize) { + super(name, StringTerms.TYPE.name(), valueSourceConfig); + this.order = order; + this.requiredSize = requiredSize; + } + + @Override + protected Aggregator createUnmapped(AggregationContext aggregationContext, Aggregator parent) { + return new UnmappedTermsAggregator(name, order, requiredSize, aggregationContext, parent); + } + + @Override + protected Aggregator create(ValuesSource valuesSource, long expectedBucketsCount, AggregationContext aggregationContext, Aggregator parent) { + if (valuesSource instanceof BytesValuesSource) { + return new StringTermsAggregator(name, factories, valuesSource, order, requiredSize, aggregationContext, parent); + } + + if (valuesSource instanceof NumericValuesSource) { + if (((NumericValuesSource) valuesSource).isFloatingPoint()) { + return new DoubleTermsAggregator(name, factories, (NumericValuesSource) valuesSource, order, requiredSize, aggregationContext, parent); + } + return new LongTermsAggregator(name, factories, (NumericValuesSource) valuesSource, order, requiredSize, aggregationContext, parent); + } + + throw new AggregationExecutionException("terms aggregation cannot be applied to field [" + valuesSourceConfig.fieldContext().field() + + "]. It can only be applied to numeric or string fields."); + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/TermsBuilder.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/TermsBuilder.java new file mode 100644 index 00000000000..0d500fb3d3a --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/TermsBuilder.java @@ -0,0 +1,51 @@ +package org.elasticsearch.search.aggregations.bucket.terms; + +import org.elasticsearch.common.xcontent.XContentBuilder; +import org.elasticsearch.search.aggregations.ValuesSourceAggregationBuilder; + +import java.io.IOException; +import java.util.Locale; + +/** + * + */ +public class TermsBuilder extends ValuesSourceAggregationBuilder { + + private int size = -1; + private Terms.ValueType valueType; + private Terms.Order order; + + public TermsBuilder(String name) { + super(name, "terms"); + } + + public TermsBuilder size(int size) { + this.size = size; + return this; + } + + public TermsBuilder valueType(Terms.ValueType valueType) { + this.valueType = valueType; + return this; + } + + public TermsBuilder order(Terms.Order order) { + this.order = order; + return this; + } + + @Override + protected XContentBuilder doInternalXContent(XContentBuilder builder, Params params) throws IOException { + if (size >=0) { + builder.field("size", size); + } + if (valueType != null) { + builder.field("value_type", valueType.name().toLowerCase(Locale.ROOT)); + } + if (order != null) { + builder.field("order"); + order.toXContent(builder, params); + } + return builder; + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/TermsParser.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/TermsParser.java new file mode 100644 index 00000000000..5f446b9514c --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/TermsParser.java @@ -0,0 +1,200 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.terms; + +import org.elasticsearch.common.xcontent.XContentParser; +import org.elasticsearch.index.fielddata.IndexFieldData; +import org.elasticsearch.index.fielddata.IndexNumericFieldData; +import org.elasticsearch.index.mapper.FieldMapper; +import org.elasticsearch.index.mapper.core.DateFieldMapper; +import org.elasticsearch.index.mapper.ip.IpFieldMapper; +import org.elasticsearch.script.SearchScript; +import org.elasticsearch.search.aggregations.Aggregator; +import org.elasticsearch.search.aggregations.support.FieldContext; +import org.elasticsearch.search.aggregations.support.ValuesSource; +import org.elasticsearch.search.aggregations.support.ValuesSourceConfig; +import org.elasticsearch.search.aggregations.support.bytes.BytesValuesSource; +import org.elasticsearch.search.aggregations.support.numeric.NumericValuesSource; +import org.elasticsearch.search.aggregations.support.numeric.ValueFormatter; +import org.elasticsearch.search.aggregations.support.numeric.ValueParser; +import org.elasticsearch.search.aggregations.AggregatorFactory; +import org.elasticsearch.search.internal.SearchContext; + +import java.io.IOException; +import java.util.Map; + +/** + * + */ +public class TermsParser implements Aggregator.Parser { + + @Override + public String type() { + return StringTerms.TYPE.name(); + } + + // TODO add support for shard_size (vs. size) a la terms facets + // TODO add support for term filtering (regexp/include/exclude) a la terms facets + + @Override + public AggregatorFactory parse(String aggregationName, XContentParser parser, SearchContext context) throws IOException { + + String field = null; + String script = null; + String scriptLang = null; + Map scriptParams = null; + Terms.ValueType valueType = null; + int requiredSize = 10; + String orderKey = "_count"; + boolean orderAsc = false; + String format = null; + boolean assumeUnique = false; + + + XContentParser.Token token; + String currentFieldName = null; + while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) { + if (token == XContentParser.Token.FIELD_NAME) { + currentFieldName = parser.currentName(); + } else if (token == XContentParser.Token.VALUE_STRING) { + if ("field".equals(currentFieldName)) { + field = parser.text(); + } else if ("script".equals(currentFieldName)) { + script = parser.text(); + } else if ("script_lang".equals(currentFieldName) || "scriptLang".equals(currentFieldName)) { + scriptLang = parser.text(); + } else if ("value_type".equals(currentFieldName) || "valueType".equals(currentFieldName)) { + valueType = Terms.ValueType.resolveType(parser.text()); + } else if ("format".equals(currentFieldName)) { + format = parser.text(); + } + } else if (token == XContentParser.Token.VALUE_BOOLEAN) { + if ("script_values_unique".equals(currentFieldName)) { + assumeUnique = parser.booleanValue(); + } + } else if (token == XContentParser.Token.VALUE_NUMBER) { + if ("size".equals(currentFieldName)) { + requiredSize = parser.intValue(); + } + } else if (token == XContentParser.Token.START_OBJECT) { + if ("params".equals(currentFieldName)) { + scriptParams = parser.map(); + } else if ("order".equals(currentFieldName)) { + while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) { + if (token == XContentParser.Token.FIELD_NAME) { + orderKey = parser.currentName(); + } else if (token == XContentParser.Token.VALUE_STRING) { + String dir = parser.text(); + orderAsc = "asc".equalsIgnoreCase(dir); + //TODO: do we want to throw a parse error if the alternative is not "desc"??? + } + } + } + } + } + + InternalOrder order = resolveOrder(orderKey, orderAsc); + SearchScript searchScript = null; + if (script != null) { + searchScript = context.scriptService().search(context.lookup(), scriptLang, script, scriptParams); + } + + if (field == null) { + + Class valueSourceType = script == null ? + ValuesSource.class : // unknown, will inherit whatever is in the context + valueType != null ? valueType.scriptValueType.getValuesSourceType() : // the user explicitly defined a value type + BytesValuesSource.class; // defaulting to bytes + + ValuesSourceConfig config = new ValuesSourceConfig(valueSourceType); + if (valueType != null) { + config.scriptValueType(valueType.scriptValueType); + } + config.script(searchScript); + if (!assumeUnique) { + config.ensureUnique(true); + } + return new TermsAggregatorFactory(aggregationName, config, order, requiredSize); + } + + FieldMapper mapper = context.smartNameFieldMapper(field); + if (mapper == null) { + ValuesSourceConfig config = new ValuesSourceConfig(BytesValuesSource.class); + config.unmapped(true); + return new TermsAggregatorFactory(aggregationName, config, order, requiredSize); + } + IndexFieldData indexFieldData = context.fieldData().getForField(mapper); + + ValuesSourceConfig config; + + if (mapper instanceof DateFieldMapper) { + DateFieldMapper dateMapper = (DateFieldMapper) mapper; + ValueFormatter formatter = format == null ? + new ValueFormatter.DateTime(dateMapper.dateTimeFormatter()) : + new ValueFormatter.DateTime(format); + config = new ValuesSourceConfig(NumericValuesSource.class) + .formatter(formatter) + .parser(new ValueParser.DateMath(dateMapper.dateMathParser())); + + } else if (mapper instanceof IpFieldMapper) { + config = new ValuesSourceConfig(NumericValuesSource.class) + .formatter(ValueFormatter.IPv4) + .parser(ValueParser.IPv4); + + } else if (indexFieldData instanceof IndexNumericFieldData) { + config = new ValuesSourceConfig(NumericValuesSource.class); + if (format != null) { + config.formatter(new ValueFormatter.Number.Pattern(format)); + } + + } else { + config = new ValuesSourceConfig(BytesValuesSource.class); + // TODO: it will make sense to set false instead here if the aggregator factory uses + // ordinals instead of hash tables + config.needsHashes(true); + } + + config.script(searchScript); + + config.fieldContext(new FieldContext(field, indexFieldData)); + + // We need values to be unique to be able to run terms aggs efficiently + if (!assumeUnique) { + config.ensureUnique(true); + } + + return new TermsAggregatorFactory(aggregationName, config, order, requiredSize); + } + + static InternalOrder resolveOrder(String key, boolean asc) { + if ("_term".equals(key)) { + return (InternalOrder) (asc ? InternalOrder.TERM_ASC : InternalOrder.TERM_DESC); + } + if ("_count".equals(key)) { + return (InternalOrder) (asc ? InternalOrder.COUNT_ASC : InternalOrder.COUNT_DESC); + } + int i = key.indexOf('.'); + if (i < 0) { + return Terms.Order.aggregation(key, asc); + } + return Terms.Order.aggregation(key.substring(0, i), key.substring(i+1), asc); + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/UnmappedTerms.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/UnmappedTerms.java new file mode 100644 index 00000000000..97aae8a4448 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/UnmappedTerms.java @@ -0,0 +1,90 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.terms; + +import org.elasticsearch.common.io.stream.StreamInput; +import org.elasticsearch.common.io.stream.StreamOutput; +import org.elasticsearch.common.xcontent.XContentBuilder; +import org.elasticsearch.search.aggregations.AggregationStreams; + +import java.io.IOException; +import java.util.Collection; +import java.util.Collections; +import java.util.Map; + +/** + * + */ +public class UnmappedTerms extends InternalTerms { + + public static final Type TYPE = new Type("terms", "umterms"); + + private static final Collection BUCKETS = Collections.emptyList(); + private static final Map BUCKETS_MAP = Collections.emptyMap(); + + public static AggregationStreams.Stream STREAM = new AggregationStreams.Stream() { + @Override + public UnmappedTerms readResult(StreamInput in) throws IOException { + UnmappedTerms buckets = new UnmappedTerms(); + buckets.readFrom(in); + return buckets; + } + }; + + public static void registerStreams() { + AggregationStreams.registerStream(STREAM, TYPE.stream()); + } + + UnmappedTerms() {} // for serialization + + public UnmappedTerms(String name, InternalOrder order, int requiredSize) { + super(name, order, requiredSize, BUCKETS); + } + + @Override + public Type type() { + return TYPE; + } + + @Override + public void readFrom(StreamInput in) throws IOException { + this.name = in.readString(); + this.order = InternalOrder.Streams.readOrder(in); + this.requiredSize = in.readVInt(); + this.buckets = BUCKETS; + this.bucketMap = BUCKETS_MAP; + } + + @Override + public void writeTo(StreamOutput out) throws IOException { + out.writeString(name); + InternalOrder.Streams.writeOrder(order, out); + out.writeVInt(requiredSize); + } + + @Override + public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException { + builder.startObject(name); + builder.startArray(CommonFields.BUCKETS).endArray(); + builder.endObject(); + return builder; + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/UnmappedTermsAggregator.java b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/UnmappedTermsAggregator.java new file mode 100644 index 00000000000..16257e913ad --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/UnmappedTermsAggregator.java @@ -0,0 +1,62 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket.terms; + +import org.elasticsearch.search.aggregations.Aggregator; +import org.elasticsearch.search.aggregations.InternalAggregation; +import org.elasticsearch.search.aggregations.support.AggregationContext; +import org.elasticsearch.search.aggregations.AggregatorFactories; + +import java.io.IOException; + +/** + * + */ +public class UnmappedTermsAggregator extends Aggregator { + + private final InternalOrder order; + private final int requiredSize; + + public UnmappedTermsAggregator(String name, InternalOrder order, int requiredSize, AggregationContext aggregationContext, Aggregator parent) { + super(name, BucketAggregationMode.PER_BUCKET, AggregatorFactories.EMPTY, 0, aggregationContext, parent); + this.order = order; + this.requiredSize = requiredSize; + } + + @Override + public boolean shouldCollect() { + return false; + } + + @Override + public void collect(int doc, long owningBucketOrdinal) throws IOException { + } + + @Override + public InternalAggregation buildAggregation(long owningBucketOrdinal) { + assert owningBucketOrdinal == 0; + return new UnmappedTerms(name, order, requiredSize); + } + + @Override + public InternalAggregation buildEmptyAggregation() { + return new UnmappedTerms(name, order, requiredSize); + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/MetricsAggregation.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/MetricsAggregation.java new file mode 100644 index 00000000000..43093aae0f0 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/MetricsAggregation.java @@ -0,0 +1,61 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.metrics; + +import org.elasticsearch.search.aggregations.InternalAggregation; +import org.elasticsearch.search.aggregations.support.numeric.ValueFormatter; + +/** + * + */ +public abstract class MetricsAggregation extends InternalAggregation { + + protected ValueFormatter valueFormatter; + + public static abstract class SingleValue extends MetricsAggregation { + + protected SingleValue() {} + + protected SingleValue(String name) { + super(name); + } + + public abstract double value(); + } + + public static abstract class MultiValue extends MetricsAggregation { + + protected MultiValue() {} + + protected MultiValue(String name) { + super(name); + } + + public abstract double value(String name); + + } + + protected MetricsAggregation() {} // for serialization + + protected MetricsAggregation(String name) { + super(name); + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/MetricsAggregationBuilder.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/MetricsAggregationBuilder.java new file mode 100644 index 00000000000..1c8a5d69c0e --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/MetricsAggregationBuilder.java @@ -0,0 +1,25 @@ +package org.elasticsearch.search.aggregations.metrics; + +import org.elasticsearch.common.xcontent.XContentBuilder; +import org.elasticsearch.search.aggregations.AbstractAggregationBuilder; + +import java.io.IOException; + +/** + * + */ +public abstract class MetricsAggregationBuilder> extends AbstractAggregationBuilder { + + public MetricsAggregationBuilder(String name, String type) { + super(name, type); + } + + @Override + public final XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException { + builder.startObject(name).startObject(type); + internalXContent(builder, params); + return builder.endObject().endObject(); + } + + protected abstract void internalXContent(XContentBuilder builder, Params params) throws IOException; +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/ValuesSourceMetricsAggregationBuilder.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/ValuesSourceMetricsAggregationBuilder.java new file mode 100644 index 00000000000..0ef53fde1de --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/ValuesSourceMetricsAggregationBuilder.java @@ -0,0 +1,78 @@ +package org.elasticsearch.search.aggregations.metrics; + +import com.google.common.collect.Maps; +import org.elasticsearch.common.xcontent.XContentBuilder; + +import java.io.IOException; +import java.util.Map; + +/** + * + */ +public abstract class ValuesSourceMetricsAggregationBuilder> extends MetricsAggregationBuilder { + + private String field; + private String script; + private String scriptLang; + private Map params; + + protected ValuesSourceMetricsAggregationBuilder(String name, String type) { + super(name, type); + } + + @SuppressWarnings("unchecked") + public B field(String field) { + this.field = field; + return (B) this; + } + + @SuppressWarnings("unchecked") + public B script(String script) { + this.script = script; + return (B) this; + } + + @SuppressWarnings("unchecked") + public B scriptLang(String scriptLang) { + this.scriptLang = scriptLang; + return (B) this; + } + + @SuppressWarnings("unchecked") + public B params(Map params) { + if (this.params == null) { + this.params = params; + } else { + this.params.putAll(params); + } + return (B) this; + } + + @SuppressWarnings("unchecked") + public B param(String name, Object value) { + if (this.params == null) { + this.params = Maps.newHashMap(); + } + this.params.put(name, value); + return (B) this; + } + + @Override + protected void internalXContent(XContentBuilder builder, Params params) throws IOException { + if (field != null) { + builder.field("field", field); + } + + if (script != null) { + builder.field("script", script); + } + + if (scriptLang != null) { + builder.field("script_lang", scriptLang); + } + + if (this.params != null && !this.params.isEmpty()) { + builder.field("params").map(this.params); + } + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/ValuesSourceMetricsAggregatorParser.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/ValuesSourceMetricsAggregatorParser.java new file mode 100644 index 00000000000..fa7d72126b1 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/ValuesSourceMetricsAggregatorParser.java @@ -0,0 +1,103 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.metrics; + +import org.elasticsearch.common.xcontent.XContentParser; +import org.elasticsearch.index.fielddata.IndexFieldData; +import org.elasticsearch.index.mapper.FieldMapper; +import org.elasticsearch.search.aggregations.Aggregator; +import org.elasticsearch.search.aggregations.support.FieldContext; +import org.elasticsearch.search.aggregations.support.ValuesSourceConfig; +import org.elasticsearch.search.aggregations.support.numeric.NumericValuesSource; +import org.elasticsearch.search.aggregations.AggregatorFactory; +import org.elasticsearch.search.internal.SearchContext; + +import java.io.IOException; +import java.util.Map; + +/** + * + */ +public abstract class ValuesSourceMetricsAggregatorParser implements Aggregator.Parser { + + protected boolean requiresSortedValues() { + return false; + } + + @Override + public AggregatorFactory parse(String aggregationName, XContentParser parser, SearchContext context) throws IOException { + + ValuesSourceConfig config = new ValuesSourceConfig(NumericValuesSource.class); + + String field = null; + String script = null; + String scriptLang = null; + Map scriptParams = null; + boolean assumeSorted = false; + + XContentParser.Token token; + String currentFieldName = null; + while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) { + if (token == XContentParser.Token.FIELD_NAME) { + currentFieldName = parser.currentName(); + } else if (token == XContentParser.Token.VALUE_STRING) { + if ("field".equals(currentFieldName)) { + field = parser.text(); + } else if ("script".equals(currentFieldName)) { + script = parser.text(); + } else if ("script_lang".equals(currentFieldName) || "scriptLang".equals(currentFieldName)) { + scriptLang = parser.text(); + } + } else if (token == XContentParser.Token.START_OBJECT) { + if ("params".equals(currentFieldName)) { + scriptParams = parser.map(); + } + } else if (token == XContentParser.Token.VALUE_BOOLEAN) { + if ("script_values_sorted".equals(currentFieldName)) { + assumeSorted = parser.booleanValue(); + } + } + } + + if (script != null) { + config.script(context.scriptService().search(context.lookup(), scriptLang, script, scriptParams)); + } + + if (!assumeSorted && requiresSortedValues()) { + config.ensureSorted(true); + } + + if (field == null) { + return createFactory(aggregationName, config); + } + + FieldMapper mapper = context.smartNameFieldMapper(field); + if (mapper == null) { + config.unmapped(true); + return createFactory(aggregationName, config); + } + + IndexFieldData indexFieldData = context.fieldData().getForField(mapper); + config.fieldContext(new FieldContext(field, indexFieldData)); + return createFactory(aggregationName, config); + } + + protected abstract AggregatorFactory createFactory(String aggregationName, ValuesSourceConfig config); +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/avg/Avg.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/avg/Avg.java new file mode 100644 index 00000000000..3dd3aefbdfd --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/avg/Avg.java @@ -0,0 +1,30 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.metrics.avg; + +import org.elasticsearch.search.aggregations.Aggregation; + +/** + * + */ +public interface Avg extends Aggregation { + + double getValue(); +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/avg/AvgAggregator.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/avg/AvgAggregator.java new file mode 100644 index 00000000000..3a0c1eabef6 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/avg/AvgAggregator.java @@ -0,0 +1,113 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.metrics.avg; + +import org.elasticsearch.common.util.BigArrays; +import org.elasticsearch.common.util.DoubleArray; +import org.elasticsearch.common.util.LongArray; +import org.elasticsearch.index.fielddata.DoubleValues; +import org.elasticsearch.search.aggregations.Aggregator; +import org.elasticsearch.search.aggregations.InternalAggregation; +import org.elasticsearch.search.aggregations.support.AggregationContext; +import org.elasticsearch.search.aggregations.support.ValuesSourceConfig; +import org.elasticsearch.search.aggregations.support.numeric.NumericValuesSource; +import org.elasticsearch.search.aggregations.AggregatorFactories; +import org.elasticsearch.search.aggregations.support.ValueSourceAggregatorFactory; + +import java.io.IOException; + +/** + * + */ +public class AvgAggregator extends Aggregator { + + private final NumericValuesSource valuesSource; + + private LongArray counts; + private DoubleArray sums; + + + public AvgAggregator(String name, long estimatedBucketsCount, NumericValuesSource valuesSource, AggregationContext context, Aggregator parent) { + super(name, BucketAggregationMode.MULTI_BUCKETS, AggregatorFactories.EMPTY, estimatedBucketsCount, context, parent); + this.valuesSource = valuesSource; + if (valuesSource != null) { + final long initialSize = estimatedBucketsCount < 2 ? 1 : estimatedBucketsCount; + counts = BigArrays.newLongArray(initialSize); + sums = BigArrays.newDoubleArray(initialSize); + } + } + + @Override + public boolean shouldCollect() { + return valuesSource != null; + } + + @Override + public void collect(int doc, long owningBucketOrdinal) throws IOException { + assert valuesSource != null : "if value source is null, collect should never be called"; + + DoubleValues values = valuesSource.doubleValues(); + if (values == null) { + return; + } + + counts = BigArrays.grow(counts, owningBucketOrdinal + 1); + sums = BigArrays.grow(sums, owningBucketOrdinal + 1); + + final int valueCount = values.setDocument(doc); + counts.increment(owningBucketOrdinal, valueCount); + double sum = 0; + for (int i = 0; i < valueCount; i++) { + sum += values.nextValue(); + } + sums.increment(owningBucketOrdinal, sum); + } + + @Override + public InternalAggregation buildAggregation(long owningBucketOrdinal) { + if (valuesSource == null || owningBucketOrdinal >= counts.size()) { + return new InternalAvg(name, 0l, 0); + } + return new InternalAvg(name, sums.get(owningBucketOrdinal), counts.get(owningBucketOrdinal)); + } + + @Override + public InternalAggregation buildEmptyAggregation() { + return new InternalAvg(name, 0.0, 0l); + } + + public static class Factory extends ValueSourceAggregatorFactory.LeafOnly { + + public Factory(String name, String type, ValuesSourceConfig valuesSourceConfig) { + super(name, type, valuesSourceConfig); + } + + @Override + protected Aggregator createUnmapped(AggregationContext aggregationContext, Aggregator parent) { + return new AvgAggregator(name, 0, null, aggregationContext, parent); + } + + @Override + protected Aggregator create(NumericValuesSource valuesSource, long expectedBucketsCount, AggregationContext aggregationContext, Aggregator parent) { + return new AvgAggregator(name, expectedBucketsCount, valuesSource, aggregationContext, parent); + } + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/avg/AvgBuilder.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/avg/AvgBuilder.java new file mode 100644 index 00000000000..248d2498ce8 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/avg/AvgBuilder.java @@ -0,0 +1,13 @@ +package org.elasticsearch.search.aggregations.metrics.avg; + +import org.elasticsearch.search.aggregations.metrics.ValuesSourceMetricsAggregationBuilder; + +/** + * + */ +public class AvgBuilder extends ValuesSourceMetricsAggregationBuilder { + + public AvgBuilder(String name) { + super(name, InternalAvg.TYPE.name()); + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/avg/AvgParser.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/avg/AvgParser.java new file mode 100644 index 00000000000..eeb2d31fb40 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/avg/AvgParser.java @@ -0,0 +1,42 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.metrics.avg; + +import org.elasticsearch.search.aggregations.metrics.ValuesSourceMetricsAggregatorParser; +import org.elasticsearch.search.aggregations.support.ValuesSourceConfig; +import org.elasticsearch.search.aggregations.support.numeric.NumericValuesSource; +import org.elasticsearch.search.aggregations.AggregatorFactory; + +/** + * + */ +public class AvgParser extends ValuesSourceMetricsAggregatorParser { + + @Override + public String type() { + return InternalAvg.TYPE.name(); + } + + @Override + protected AggregatorFactory createFactory(String aggregationName, ValuesSourceConfig config) { + return new AvgAggregator.Factory(aggregationName, type(), config); + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/avg/InternalAvg.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/avg/InternalAvg.java new file mode 100644 index 00000000000..44c42414eef --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/avg/InternalAvg.java @@ -0,0 +1,123 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.metrics.avg; + +import org.elasticsearch.common.io.stream.StreamInput; +import org.elasticsearch.common.io.stream.StreamOutput; +import org.elasticsearch.common.xcontent.XContentBuilder; +import org.elasticsearch.search.aggregations.AggregationStreams; +import org.elasticsearch.search.aggregations.InternalAggregation; +import org.elasticsearch.search.aggregations.metrics.MetricsAggregation; +import org.elasticsearch.search.aggregations.support.numeric.ValueFormatterStreams; + +import java.io.IOException; +import java.util.List; + +/** +* +*/ +public class InternalAvg extends MetricsAggregation.SingleValue implements Avg { + + public final static Type TYPE = new Type("avg"); + + public final static AggregationStreams.Stream STREAM = new AggregationStreams.Stream() { + @Override + public InternalAvg readResult(StreamInput in) throws IOException { + InternalAvg result = new InternalAvg(); + result.readFrom(in); + return result; + } + }; + + public static void registerStreams() { + AggregationStreams.registerStream(STREAM, TYPE.stream()); + } + + private double sum; + private long count; + + InternalAvg() {} // for serialization + + public InternalAvg(String name, double sum, long count) { + super(name); + this.sum = sum; + this.count = count; + } + + @Override + public double value() { + return getValue(); + } + + public double getValue() { + return sum / count; + } + + @Override + public Type type() { + return TYPE; + } + + @Override + public InternalAvg reduce(ReduceContext reduceContext) { + List aggregations = reduceContext.aggregations(); + if (aggregations.size() == 1) { + return (InternalAvg) aggregations.get(0); + } + InternalAvg reduced = null; + for (InternalAggregation aggregation : aggregations) { + if (reduced == null) { + reduced = (InternalAvg) aggregation; + } else { + reduced.count += ((InternalAvg) aggregation).count; + reduced.sum += ((InternalAvg) aggregation).sum; + } + } + return reduced; + } + + @Override + public void readFrom(StreamInput in) throws IOException { + name = in.readString(); + valueFormatter = ValueFormatterStreams.readOptional(in); + sum = in.readDouble(); + count = in.readVLong(); + } + + @Override + public void writeTo(StreamOutput out) throws IOException { + out.writeString(name); + ValueFormatterStreams.writeOptional(valueFormatter, out); + out.writeDouble(sum); + out.writeVLong(count); + } + + @Override + public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException { + builder.startObject(name); + builder.field(CommonFields.VALUE, count != 0 ? getValue() : null); + if (count != 0 && valueFormatter != null) { + builder.field(CommonFields.VALUE_AS_STRING, valueFormatter.format(getValue())); + } + builder.endObject(); + return builder; + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/max/InternalMax.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/max/InternalMax.java new file mode 100644 index 00000000000..567b6506677 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/max/InternalMax.java @@ -0,0 +1,121 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.metrics.max; + +import org.elasticsearch.common.io.stream.StreamInput; +import org.elasticsearch.common.io.stream.StreamOutput; +import org.elasticsearch.common.xcontent.XContentBuilder; +import org.elasticsearch.search.aggregations.AggregationStreams; +import org.elasticsearch.search.aggregations.InternalAggregation; +import org.elasticsearch.search.aggregations.metrics.MetricsAggregation; +import org.elasticsearch.search.aggregations.support.numeric.ValueFormatterStreams; + +import java.io.IOException; +import java.util.List; + +/** +* +*/ +public class InternalMax extends MetricsAggregation.SingleValue implements Max { + + public final static Type TYPE = new Type("max"); + + public final static AggregationStreams.Stream STREAM = new AggregationStreams.Stream() { + @Override + public InternalMax readResult(StreamInput in) throws IOException { + InternalMax result = new InternalMax(); + result.readFrom(in); + return result; + } + }; + + public static void registerStreams() { + AggregationStreams.registerStream(STREAM, TYPE.stream()); + } + + private double max; + + InternalMax() {} // for serialization + + public InternalMax(String name, double max) { + super(name); + this.max = max; + } + + @Override + public double value() { + return max; + } + + public double getValue() { + return max; + } + + @Override + public Type type() { + return TYPE; + } + + @Override + public InternalMax reduce(ReduceContext reduceContext) { + List aggregations = reduceContext.aggregations(); + if (aggregations.size() == 1) { + return (InternalMax) aggregations.get(0); + } + InternalMax reduced = null; + for (InternalAggregation aggregation : aggregations) { + if (reduced == null) { + reduced = (InternalMax) aggregation; + } else { + reduced.max = Math.max(reduced.max, ((InternalMax) aggregation).max); + } + } + if (reduced != null) { + return reduced; + } + return (InternalMax) aggregations.get(0); + } + + @Override + public void readFrom(StreamInput in) throws IOException { + name = in.readString(); + valueFormatter = ValueFormatterStreams.readOptional(in); + max = in.readDouble(); + } + + @Override + public void writeTo(StreamOutput out) throws IOException { + out.writeString(name); + ValueFormatterStreams.writeOptional(valueFormatter, out); + out.writeDouble(max); + } + + @Override + public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException { + builder.startObject(name); + boolean hasValue = !Double.isInfinite(max); + builder.field(CommonFields.VALUE, hasValue ? max : null); + if (hasValue && valueFormatter != null) { + builder.field(CommonFields.VALUE_AS_STRING, valueFormatter.format(max)); + } + builder.endObject(); + return builder; + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/max/Max.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/max/Max.java new file mode 100644 index 00000000000..f43d7ea94b5 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/max/Max.java @@ -0,0 +1,30 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.metrics.max; + +import org.elasticsearch.search.aggregations.Aggregation; + +/** + * + */ +public interface Max extends Aggregation { + + double getValue(); +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/max/MaxAggregator.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/max/MaxAggregator.java new file mode 100644 index 00000000000..e6f99a86c24 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/max/MaxAggregator.java @@ -0,0 +1,112 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.metrics.max; + +import org.elasticsearch.common.util.BigArrays; +import org.elasticsearch.common.util.DoubleArray; +import org.elasticsearch.index.fielddata.DoubleValues; +import org.elasticsearch.search.aggregations.Aggregator; +import org.elasticsearch.search.aggregations.InternalAggregation; +import org.elasticsearch.search.aggregations.support.AggregationContext; +import org.elasticsearch.search.aggregations.support.ValuesSourceConfig; +import org.elasticsearch.search.aggregations.support.numeric.NumericValuesSource; +import org.elasticsearch.search.aggregations.AggregatorFactories; +import org.elasticsearch.search.aggregations.support.ValueSourceAggregatorFactory; + +import java.io.IOException; + +/** + * + */ +public class MaxAggregator extends Aggregator { + + private final NumericValuesSource valuesSource; + + private DoubleArray maxes; + + public MaxAggregator(String name, long estimatedBucketsCount, NumericValuesSource valuesSource, AggregationContext context, Aggregator parent) { + super(name, BucketAggregationMode.MULTI_BUCKETS, AggregatorFactories.EMPTY, estimatedBucketsCount, context, parent); + this.valuesSource = valuesSource; + if (valuesSource != null) { + final long initialSize = estimatedBucketsCount < 2 ? 1 : estimatedBucketsCount; + maxes = BigArrays.newDoubleArray(initialSize); + maxes.fill(0, maxes.size(), Double.NEGATIVE_INFINITY); + } + } + + @Override + public boolean shouldCollect() { + return valuesSource != null; + } + + @Override + public void collect(int doc, long owningBucketOrdinal) throws IOException { + assert valuesSource != null : "collect should only be called when value source is not null"; + + DoubleValues values = valuesSource.doubleValues(); + if (values == null) { + return; + } + + if (owningBucketOrdinal >= maxes.size()) { + long from = maxes.size(); + maxes = BigArrays.grow(maxes, owningBucketOrdinal + 1); + maxes.fill(from, maxes.size(), Double.NEGATIVE_INFINITY); + } + + final int valueCount = values.setDocument(doc); + double max = maxes.get(owningBucketOrdinal); + for (int i = 0; i < valueCount; i++) { + max = Math.max(max, values.nextValue()); + } + maxes.set(owningBucketOrdinal, max); + } + + @Override + public InternalAggregation buildAggregation(long owningBucketOrdinal) { + if (valuesSource == null) { + return new InternalMax(name, Double.NEGATIVE_INFINITY); + } + assert owningBucketOrdinal < maxes.size(); + return new InternalMax(name, maxes.get(owningBucketOrdinal)); + } + + @Override + public InternalAggregation buildEmptyAggregation() { + return new InternalMax(name, Double.NEGATIVE_INFINITY); + } + + public static class Factory extends ValueSourceAggregatorFactory.LeafOnly { + + public Factory(String name, ValuesSourceConfig valuesSourceConfig) { + super(name, InternalMax.TYPE.name(), valuesSourceConfig); + } + + @Override + protected Aggregator createUnmapped(AggregationContext aggregationContext, Aggregator parent) { + return new MaxAggregator(name, 0, null, aggregationContext, parent); + } + + @Override + protected Aggregator create(NumericValuesSource valuesSource, long expectedBucketsCount, AggregationContext aggregationContext, Aggregator parent) { + return new MaxAggregator(name, expectedBucketsCount, valuesSource, aggregationContext, parent); + } + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/max/MaxBuilder.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/max/MaxBuilder.java new file mode 100644 index 00000000000..3acf3c830cf --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/max/MaxBuilder.java @@ -0,0 +1,13 @@ +package org.elasticsearch.search.aggregations.metrics.max; + +import org.elasticsearch.search.aggregations.metrics.ValuesSourceMetricsAggregationBuilder; + +/** + * + */ +public class MaxBuilder extends ValuesSourceMetricsAggregationBuilder { + + public MaxBuilder(String name) { + super(name, InternalMax.TYPE.name()); + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/max/MaxParser.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/max/MaxParser.java new file mode 100644 index 00000000000..62fe88f8aea --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/max/MaxParser.java @@ -0,0 +1,42 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.metrics.max; + +import org.elasticsearch.search.aggregations.metrics.ValuesSourceMetricsAggregatorParser; +import org.elasticsearch.search.aggregations.support.ValuesSourceConfig; +import org.elasticsearch.search.aggregations.support.numeric.NumericValuesSource; +import org.elasticsearch.search.aggregations.AggregatorFactory; + +/** + * + */ +public class MaxParser extends ValuesSourceMetricsAggregatorParser { + + @Override + public String type() { + return InternalMax.TYPE.name(); + } + + @Override + protected AggregatorFactory createFactory(String aggregationName, ValuesSourceConfig config) { + return new MaxAggregator.Factory(aggregationName, config); + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/min/InternalMin.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/min/InternalMin.java new file mode 100644 index 00000000000..e18b851c887 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/min/InternalMin.java @@ -0,0 +1,123 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.metrics.min; + +import org.elasticsearch.common.io.stream.StreamInput; +import org.elasticsearch.common.io.stream.StreamOutput; +import org.elasticsearch.common.xcontent.XContentBuilder; +import org.elasticsearch.search.aggregations.AggregationStreams; +import org.elasticsearch.search.aggregations.InternalAggregation; +import org.elasticsearch.search.aggregations.metrics.MetricsAggregation; +import org.elasticsearch.search.aggregations.support.numeric.ValueFormatterStreams; + +import java.io.IOException; +import java.util.List; + +/** +* +*/ +public class InternalMin extends MetricsAggregation.SingleValue implements Min { + + public final static Type TYPE = new Type("min"); + + public final static AggregationStreams.Stream STREAM = new AggregationStreams.Stream() { + @Override + public InternalMin readResult(StreamInput in) throws IOException { + InternalMin result = new InternalMin(); + result.readFrom(in); + return result; + } + }; + + public static void registerStreams() { + AggregationStreams.registerStream(STREAM, TYPE.stream()); + } + + + private double min; + + InternalMin() {} // for serialization + + public InternalMin(String name, double min) { + super(name); + this.min = min; + } + + @Override + public double value() { + return min; + } + + public double getValue() { + return min; + } + + @Override + public Type type() { + return TYPE; + } + + @Override + public InternalMin reduce(ReduceContext reduceContext) { + List aggregations = reduceContext.aggregations(); + if (aggregations.size() == 1) { + return (InternalMin) aggregations.get(0); + } + InternalMin reduced = null; + for (InternalAggregation aggregation : aggregations) { + if (reduced == null) { + reduced = (InternalMin) aggregation; + } else { + reduced.min = Math.min(reduced.min, ((InternalMin) aggregation).min); + } + } + if (reduced != null) { + return reduced; + } + return (InternalMin) aggregations.get(0); + } + + @Override + public void readFrom(StreamInput in) throws IOException { + name = in.readString(); + valueFormatter = ValueFormatterStreams.readOptional(in); + min = in.readDouble(); + } + + @Override + public void writeTo(StreamOutput out) throws IOException { + out.writeString(name); + ValueFormatterStreams.writeOptional(valueFormatter, out); + out.writeDouble(min); + } + + @Override + public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException { + builder.startObject(name); + boolean hasValue = !Double.isInfinite(min); + builder.field(CommonFields.VALUE, hasValue ? min : null); + if (hasValue && valueFormatter != null) { + builder.field(CommonFields.VALUE_AS_STRING, valueFormatter.format(min)); + } + builder.endObject(); + return builder; + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/min/Min.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/min/Min.java new file mode 100644 index 00000000000..c06d8f00951 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/min/Min.java @@ -0,0 +1,30 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.metrics.min; + +import org.elasticsearch.search.aggregations.Aggregation; + +/** + * + */ +public interface Min extends Aggregation { + + double getValue(); +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/min/MinAggregator.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/min/MinAggregator.java new file mode 100644 index 00000000000..b690d1b45eb --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/min/MinAggregator.java @@ -0,0 +1,109 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.metrics.min; + +import org.elasticsearch.common.util.BigArrays; +import org.elasticsearch.common.util.DoubleArray; +import org.elasticsearch.index.fielddata.DoubleValues; +import org.elasticsearch.search.aggregations.Aggregator; +import org.elasticsearch.search.aggregations.InternalAggregation; +import org.elasticsearch.search.aggregations.support.AggregationContext; +import org.elasticsearch.search.aggregations.support.ValuesSourceConfig; +import org.elasticsearch.search.aggregations.support.numeric.NumericValuesSource; +import org.elasticsearch.search.aggregations.AggregatorFactories; +import org.elasticsearch.search.aggregations.support.ValueSourceAggregatorFactory; + +import java.io.IOException; + +/** + * + */ +public class MinAggregator extends Aggregator { + + private final NumericValuesSource valuesSource; + + private DoubleArray mins; + + public MinAggregator(String name, long estimatedBucketsCount, NumericValuesSource valuesSource, AggregationContext context, Aggregator parent) { + super(name, BucketAggregationMode.MULTI_BUCKETS, AggregatorFactories.EMPTY, estimatedBucketsCount, context, parent); + this.valuesSource = valuesSource; + if (valuesSource != null) { + if (valuesSource != null) { + final long initialSize = estimatedBucketsCount < 2 ? 1 : estimatedBucketsCount; + mins = BigArrays.newDoubleArray(initialSize); + mins.fill(0, mins.size(), Double.POSITIVE_INFINITY); + } + } + } + + @Override + public boolean shouldCollect() { + return valuesSource != null; + } + + @Override + public void collect(int doc, long owningBucketOrdinal) throws IOException { + assert valuesSource != null : "collect must only be called if #shouldCollect returns true"; + + DoubleValues values = valuesSource.doubleValues(); + if (values == null || values.setDocument(doc) == 0) { + return; + } + + if (owningBucketOrdinal >= mins.size()) { + long from = mins.size(); + mins = BigArrays.grow(mins, owningBucketOrdinal + 1); + mins.fill(from, mins.size(), Double.POSITIVE_INFINITY); + } + + mins.set(owningBucketOrdinal, Math.min(values.nextValue(), mins.get(owningBucketOrdinal))); + } + + @Override + public InternalAggregation buildAggregation(long owningBucketOrdinal) { + if (valuesSource == null) { + return new InternalMin(name, Double.POSITIVE_INFINITY); + } + assert owningBucketOrdinal < mins.size(); + return new InternalMin(name, mins.get(owningBucketOrdinal)); + } + + @Override + public InternalAggregation buildEmptyAggregation() { + return new InternalMin(name, Double.POSITIVE_INFINITY); + } + + public static class Factory extends ValueSourceAggregatorFactory.LeafOnly { + + public Factory(String name, ValuesSourceConfig valuesSourceConfig) { + super(name, InternalMin.TYPE.name(), valuesSourceConfig); + } + + @Override + protected Aggregator createUnmapped(AggregationContext aggregationContext, Aggregator parent) { + return new MinAggregator(name, 0, null, aggregationContext, parent); + } + + @Override + protected Aggregator create(NumericValuesSource valuesSource, long expectedBucketsCount, AggregationContext aggregationContext, Aggregator parent) { + return new MinAggregator(name, expectedBucketsCount, valuesSource, aggregationContext, parent); + } + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/min/MinBuilder.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/min/MinBuilder.java new file mode 100644 index 00000000000..d806bd28c47 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/min/MinBuilder.java @@ -0,0 +1,13 @@ +package org.elasticsearch.search.aggregations.metrics.min; + +import org.elasticsearch.search.aggregations.metrics.ValuesSourceMetricsAggregationBuilder; + +/** + * + */ +public class MinBuilder extends ValuesSourceMetricsAggregationBuilder { + + public MinBuilder(String name) { + super(name, InternalMin.TYPE.name()); + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/min/MinParser.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/min/MinParser.java new file mode 100644 index 00000000000..19467d83693 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/min/MinParser.java @@ -0,0 +1,46 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.metrics.min; + +import org.elasticsearch.search.aggregations.metrics.ValuesSourceMetricsAggregatorParser; +import org.elasticsearch.search.aggregations.support.ValuesSourceConfig; +import org.elasticsearch.search.aggregations.support.numeric.NumericValuesSource; +import org.elasticsearch.search.aggregations.AggregatorFactory; + +/** + * + */ +public class MinParser extends ValuesSourceMetricsAggregatorParser { + + @Override + public String type() { + return InternalMin.TYPE.name(); + } + + @Override + protected boolean requiresSortedValues() { + return true; + } + + @Override + protected AggregatorFactory createFactory(String aggregationName, ValuesSourceConfig config) { + return new MinAggregator.Factory(aggregationName, config); + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/stats/InternalStats.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/stats/InternalStats.java new file mode 100644 index 00000000000..589e658b945 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/stats/InternalStats.java @@ -0,0 +1,209 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.metrics.stats; + +import org.elasticsearch.common.io.stream.StreamInput; +import org.elasticsearch.common.io.stream.StreamOutput; +import org.elasticsearch.common.xcontent.XContentBuilder; +import org.elasticsearch.common.xcontent.XContentBuilderString; +import org.elasticsearch.search.aggregations.AggregationStreams; +import org.elasticsearch.search.aggregations.InternalAggregation; +import org.elasticsearch.search.aggregations.metrics.MetricsAggregation; +import org.elasticsearch.search.aggregations.support.numeric.ValueFormatterStreams; + +import java.io.IOException; +import java.util.List; + +/** +* +*/ +public class InternalStats extends MetricsAggregation.MultiValue implements Stats { + + public final static Type TYPE = new Type("stats"); + + public final static AggregationStreams.Stream STREAM = new AggregationStreams.Stream() { + @Override + public InternalStats readResult(StreamInput in) throws IOException { + InternalStats result = new InternalStats(); + result.readFrom(in); + return result; + } + }; + + public static void registerStreams() { + AggregationStreams.registerStream(STREAM, TYPE.stream()); + } + + protected long count; + protected double min; + protected double max; + protected double sum; + + protected InternalStats() {} // for serialization + + public InternalStats(String name, long count, double sum, double min, double max) { + super(name); + this.count = count; + this.sum = sum; + this.min = min; + this.max = max; + } + + @Override + public long getCount() { + return count; + } + + @Override + public double getMin() { + return min; + } + + @Override + public double getMax() { + return max; + } + + @Override + public double getAvg() { + return sum / count; + } + + @Override + public double getSum() { + return sum; + } + + @Override + public Type type() { + return TYPE; + } + + @Override + public double value(String name) { + if ("min".equals(name)) { + return min; + } else if ("max".equals(name)) { + return max; + } else if ("avg".equals(name)) { + return getAvg(); + } else if ("count".equals(name)) { + return count; + } else if ("sum".equals(name)) { + return sum; + } else { + throw new IllegalArgumentException("Unknown value [" + name + "] in common stats aggregation"); + } + } + + @Override + public InternalStats reduce(ReduceContext reduceContext) { + List aggregations = reduceContext.aggregations(); + if (aggregations.size() == 1) { + return (InternalStats) aggregations.get(0); + } + InternalStats reduced = null; + for (InternalAggregation aggregation : aggregations) { + if (reduced == null) { + if (((InternalStats) aggregation).count != 0) { + reduced = (InternalStats) aggregation; + } + } else { + if (((InternalStats) aggregation).count != 0) { + reduced.count += ((InternalStats) aggregation).count; + reduced.min = Math.min(reduced.min, ((InternalStats) aggregation).min); + reduced.max = Math.max(reduced.max, ((InternalStats) aggregation).max); + reduced.sum += ((InternalStats) aggregation).sum; + mergeOtherStats(reduced, aggregation); + } + } + } + if (reduced != null) { + return reduced; + } + return (InternalStats) aggregations.get(0); + } + + protected void mergeOtherStats(InternalStats to, InternalAggregation from) { + } + + @Override + public void readFrom(StreamInput in) throws IOException { + name = in.readString(); + valueFormatter = ValueFormatterStreams.readOptional(in); + count = in.readVLong(); + min = in.readDouble(); + max = in.readDouble(); + sum = in.readDouble(); + readOtherStatsFrom(in); + } + + public void readOtherStatsFrom(StreamInput in) throws IOException { + } + + @Override + public void writeTo(StreamOutput out) throws IOException { + out.writeString(name); + ValueFormatterStreams.writeOptional(valueFormatter, out); + out.writeVLong(count); + out.writeDouble(min); + out.writeDouble(max); + out.writeDouble(sum); + writeOtherStatsTo(out); + } + + protected void writeOtherStatsTo(StreamOutput out) throws IOException { + } + + static class Fields { + public static final XContentBuilderString COUNT = new XContentBuilderString("count"); + public static final XContentBuilderString MIN = new XContentBuilderString("min"); + public static final XContentBuilderString MIN_AS_STRING = new XContentBuilderString("min_as_string"); + public static final XContentBuilderString MAX = new XContentBuilderString("max"); + public static final XContentBuilderString MAX_AS_STRING = new XContentBuilderString("max_as_string"); + public static final XContentBuilderString AVG = new XContentBuilderString("avg"); + public static final XContentBuilderString AVG_AS_STRING = new XContentBuilderString("avg_as_string"); + public static final XContentBuilderString SUM = new XContentBuilderString("sum"); + public static final XContentBuilderString SUM_AS_STRING = new XContentBuilderString("sum_as_string"); + } + + @Override + public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException { + builder.startObject(name); + builder.field(Fields.COUNT, count); + builder.field(Fields.MIN, count != 0 ? min : null); + builder.field(Fields.MAX, count != 0 ? max : null); + builder.field(Fields.AVG, count != 0 ? getAvg() : null); + builder.field(Fields.SUM, count != 0 ? sum : null); + if (count != 0 && valueFormatter != null) { + builder.field(Fields.MIN_AS_STRING, valueFormatter.format(min)); + builder.field(Fields.MAX_AS_STRING, valueFormatter.format(max)); + builder.field(Fields.AVG_AS_STRING, valueFormatter.format(getAvg())); + builder.field(Fields.SUM_AS_STRING, valueFormatter.format(sum)); + } + otherStatsToXCotent(builder, params); + builder.endObject(); + return builder; + } + + protected XContentBuilder otherStatsToXCotent(XContentBuilder builder, Params params) throws IOException { + return builder; + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/stats/Stats.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/stats/Stats.java new file mode 100644 index 00000000000..7ccbb8a89d8 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/stats/Stats.java @@ -0,0 +1,54 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.metrics.stats; + +import org.elasticsearch.search.aggregations.Aggregation; + +/** + * Statistics over a set of values (either aggregated over field data or scripts) + */ +public interface Stats extends Aggregation { + + /** + * @return The number of values that were aggregated + */ + long getCount(); + + /** + * @return The minimum value of all aggregated values. + */ + double getMin(); + + /** + * @return The maximum value of all aggregated values. + */ + double getMax(); + + /** + * @return The avg value over all aggregated values. + */ + double getAvg(); + + /** + * @return The sum of aggregated values. + */ + double getSum(); + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/stats/StatsAggegator.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/stats/StatsAggegator.java new file mode 100644 index 00000000000..fe83bd1b8c3 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/stats/StatsAggegator.java @@ -0,0 +1,133 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.metrics.stats; + +import org.elasticsearch.common.util.BigArrays; +import org.elasticsearch.common.util.DoubleArray; +import org.elasticsearch.common.util.LongArray; +import org.elasticsearch.index.fielddata.DoubleValues; +import org.elasticsearch.search.aggregations.Aggregator; +import org.elasticsearch.search.aggregations.InternalAggregation; +import org.elasticsearch.search.aggregations.support.AggregationContext; +import org.elasticsearch.search.aggregations.support.ValuesSourceConfig; +import org.elasticsearch.search.aggregations.support.numeric.NumericValuesSource; +import org.elasticsearch.search.aggregations.AggregatorFactories; +import org.elasticsearch.search.aggregations.support.ValueSourceAggregatorFactory; + +import java.io.IOException; + +/** + * + */ +public class StatsAggegator extends Aggregator { + + private final NumericValuesSource valuesSource; + + private LongArray counts; + private DoubleArray sums; + private DoubleArray mins; + private DoubleArray maxes; + + public StatsAggegator(String name, long estimatedBucketsCount, NumericValuesSource valuesSource, AggregationContext context, Aggregator parent) { + super(name, BucketAggregationMode.MULTI_BUCKETS, AggregatorFactories.EMPTY, estimatedBucketsCount, context, parent); + this.valuesSource = valuesSource; + if (valuesSource != null) { + final long initialSize = estimatedBucketsCount < 2 ? 1 : estimatedBucketsCount; + counts = BigArrays.newLongArray(initialSize); + sums = BigArrays.newDoubleArray(initialSize); + mins = BigArrays.newDoubleArray(initialSize); + mins.fill(0, mins.size(), Double.POSITIVE_INFINITY); + maxes = BigArrays.newDoubleArray(initialSize); + maxes.fill(0, maxes.size(), Double.NEGATIVE_INFINITY); + } + } + + @Override + public boolean shouldCollect() { + return valuesSource != null; + } + + @Override + public void collect(int doc, long owningBucketOrdinal) throws IOException { + assert valuesSource != null : "collect must only be called if #shouldCollect returns true"; + + DoubleValues values = valuesSource.doubleValues(); + if (values == null) { + return; + } + + if (owningBucketOrdinal >= counts.size()) { + final long from = counts.size(); + final long overSize = BigArrays.overSize(owningBucketOrdinal + 1); + counts = BigArrays.resize(counts, overSize); + sums = BigArrays.resize(sums, overSize); + mins = BigArrays.resize(mins, overSize); + maxes = BigArrays.resize(maxes, overSize); + mins.fill(from, overSize, Double.POSITIVE_INFINITY); + maxes.fill(from, overSize, Double.NEGATIVE_INFINITY); + } + + final int valuesCount = values.setDocument(doc); + counts.increment(owningBucketOrdinal, valuesCount); + double sum = 0; + double min = mins.get(owningBucketOrdinal); + double max = maxes.get(owningBucketOrdinal); + for (int i = 0; i < valuesCount; i++) { + double value = values.nextValue(); + sum += value; + min = Math.min(min, value); + max = Math.max(max, value); + } + sums.increment(owningBucketOrdinal, sum); + mins.set(owningBucketOrdinal, min); + maxes.set(owningBucketOrdinal, max); + } + + @Override + public InternalAggregation buildAggregation(long owningBucketOrdinal) { + if (valuesSource == null) { + return new InternalStats(name, 0, 0, Double.POSITIVE_INFINITY, Double.NEGATIVE_INFINITY); + } + assert owningBucketOrdinal < counts.size(); + return new InternalStats(name, counts.get(owningBucketOrdinal), sums.get(owningBucketOrdinal), mins.get(owningBucketOrdinal), maxes.get(owningBucketOrdinal)); + } + + @Override + public InternalAggregation buildEmptyAggregation() { + return new InternalStats(name, 0, 0, Double.POSITIVE_INFINITY, Double.NEGATIVE_INFINITY); + } + + public static class Factory extends ValueSourceAggregatorFactory.LeafOnly { + + public Factory(String name, ValuesSourceConfig valuesSourceConfig) { + super(name, InternalStats.TYPE.name(), valuesSourceConfig); + } + + @Override + protected Aggregator createUnmapped(AggregationContext aggregationContext, Aggregator parent) { + return new StatsAggegator(name, 0, null, aggregationContext, parent); + } + + @Override + protected Aggregator create(NumericValuesSource valuesSource, long expectedBucketsCount, AggregationContext aggregationContext, Aggregator parent) { + return new StatsAggegator(name, expectedBucketsCount, valuesSource, aggregationContext, parent); + } + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/stats/StatsBuilder.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/stats/StatsBuilder.java new file mode 100644 index 00000000000..0d06c3e5aa6 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/stats/StatsBuilder.java @@ -0,0 +1,13 @@ +package org.elasticsearch.search.aggregations.metrics.stats; + +import org.elasticsearch.search.aggregations.metrics.ValuesSourceMetricsAggregationBuilder; + +/** + * + */ +public class StatsBuilder extends ValuesSourceMetricsAggregationBuilder { + + public StatsBuilder(String name) { + super(name, InternalStats.TYPE.name()); + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/stats/StatsParser.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/stats/StatsParser.java new file mode 100644 index 00000000000..d6de769613e --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/stats/StatsParser.java @@ -0,0 +1,41 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.metrics.stats; + +import org.elasticsearch.search.aggregations.metrics.ValuesSourceMetricsAggregatorParser; +import org.elasticsearch.search.aggregations.support.ValuesSourceConfig; +import org.elasticsearch.search.aggregations.support.numeric.NumericValuesSource; +import org.elasticsearch.search.aggregations.AggregatorFactory; + +/** + * + */ +public class StatsParser extends ValuesSourceMetricsAggregatorParser { + + @Override + public String type() { + return InternalStats.TYPE.name(); + } + + @Override + protected AggregatorFactory createFactory(String aggregationName, ValuesSourceConfig config) { + return new StatsAggegator.Factory(aggregationName, config); + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/stats/extended/ExtendedStats.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/stats/extended/ExtendedStats.java new file mode 100644 index 00000000000..1f6bb01d8b2 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/stats/extended/ExtendedStats.java @@ -0,0 +1,35 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.metrics.stats.extended; + +import org.elasticsearch.search.aggregations.metrics.stats.Stats; + +/** + * Statistics over a set of values (either aggregated over field data or scripts) + */ +public interface ExtendedStats extends Stats { + + double getSumOfSquares(); + + double getVariance(); + + double getStdDeviation(); + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/stats/extended/ExtendedStatsAggregator.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/stats/extended/ExtendedStatsAggregator.java new file mode 100644 index 00000000000..49e37de452e --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/stats/extended/ExtendedStatsAggregator.java @@ -0,0 +1,140 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.metrics.stats.extended; + +import org.elasticsearch.common.util.BigArrays; +import org.elasticsearch.common.util.DoubleArray; +import org.elasticsearch.common.util.LongArray; +import org.elasticsearch.index.fielddata.DoubleValues; +import org.elasticsearch.search.aggregations.Aggregator; +import org.elasticsearch.search.aggregations.InternalAggregation; +import org.elasticsearch.search.aggregations.support.AggregationContext; +import org.elasticsearch.search.aggregations.support.ValuesSourceConfig; +import org.elasticsearch.search.aggregations.support.numeric.NumericValuesSource; +import org.elasticsearch.search.aggregations.AggregatorFactories; +import org.elasticsearch.search.aggregations.support.ValueSourceAggregatorFactory; + +import java.io.IOException; + +/** + * + */ +public class ExtendedStatsAggregator extends Aggregator { + + private final NumericValuesSource valuesSource; + + private LongArray counts; + private DoubleArray sums; + private DoubleArray mins; + private DoubleArray maxes; + private DoubleArray sumOfSqrs; + + public ExtendedStatsAggregator(String name, long estimatedBucketsCount, NumericValuesSource valuesSource, AggregationContext context, Aggregator parent) { + super(name, BucketAggregationMode.MULTI_BUCKETS, AggregatorFactories.EMPTY, estimatedBucketsCount, context, parent); + this.valuesSource = valuesSource; + if (valuesSource != null) { + final long initialSize = estimatedBucketsCount < 2 ? 1 : estimatedBucketsCount; + counts = BigArrays.newLongArray(initialSize); + sums = BigArrays.newDoubleArray(initialSize); + mins = BigArrays.newDoubleArray(initialSize); + mins.fill(0, mins.size(), Double.POSITIVE_INFINITY); + maxes = BigArrays.newDoubleArray(initialSize); + maxes.fill(0, maxes.size(), Double.NEGATIVE_INFINITY); + sumOfSqrs = BigArrays.newDoubleArray(initialSize); + } + } + + @Override + public boolean shouldCollect() { + return valuesSource != null; + } + + @Override + public void collect(int doc, long owningBucketOrdinal) throws IOException { + assert valuesSource != null : "collect must only be called if #shouldCollect returns true"; + + DoubleValues values = valuesSource.doubleValues(); + if (values == null) { + return; + } + + if (owningBucketOrdinal >= counts.size()) { + final long from = counts.size(); + final long overSize = BigArrays.overSize(owningBucketOrdinal + 1); + counts = BigArrays.resize(counts, overSize); + sums = BigArrays.resize(sums, overSize); + mins = BigArrays.resize(mins, overSize); + maxes = BigArrays.resize(maxes, overSize); + sumOfSqrs = BigArrays.resize(sumOfSqrs, overSize); + mins.fill(from, overSize, Double.POSITIVE_INFINITY); + maxes.fill(from, overSize, Double.NEGATIVE_INFINITY); + } + + final int valuesCount = values.setDocument(doc); + counts.increment(owningBucketOrdinal, valuesCount); + double sum = 0; + double sumOfSqr = 0; + double min = mins.get(owningBucketOrdinal); + double max = maxes.get(owningBucketOrdinal); + for (int i = 0; i < valuesCount; i++) { + double value = values.nextValue(); + sum += value; + sumOfSqr += value * value; + min = Math.min(min, value); + max = Math.max(max, value); + } + sums.increment(owningBucketOrdinal, sum); + sumOfSqrs.increment(owningBucketOrdinal, sumOfSqr); + mins.set(owningBucketOrdinal, min); + maxes.set(owningBucketOrdinal, max); + } + + @Override + public InternalAggregation buildAggregation(long owningBucketOrdinal) { + if (valuesSource == null) { + return new InternalExtendedStats(name, 0, 0d, Double.POSITIVE_INFINITY, Double.NEGATIVE_INFINITY, 0d); + } + assert owningBucketOrdinal < counts.size(); + return new InternalExtendedStats(name, counts.get(owningBucketOrdinal), sums.get(owningBucketOrdinal), mins.get(owningBucketOrdinal), + maxes.get(owningBucketOrdinal), sumOfSqrs.get(owningBucketOrdinal)); + } + + @Override + public InternalAggregation buildEmptyAggregation() { + return new InternalExtendedStats(name, 0, 0d, Double.POSITIVE_INFINITY, Double.NEGATIVE_INFINITY, 0d); + } + + public static class Factory extends ValueSourceAggregatorFactory.LeafOnly { + + public Factory(String name, ValuesSourceConfig valuesSourceConfig) { + super(name, InternalExtendedStats.TYPE.name(), valuesSourceConfig); + } + + @Override + protected Aggregator createUnmapped(AggregationContext aggregationContext, Aggregator parent) { + return new ExtendedStatsAggregator(name, 0, null, aggregationContext, parent); + } + + @Override + protected Aggregator create(NumericValuesSource valuesSource, long expectedBucketsCount, AggregationContext aggregationContext, Aggregator parent) { + return new ExtendedStatsAggregator(name, expectedBucketsCount, valuesSource, aggregationContext, parent); + } + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/stats/extended/ExtendedStatsBuilder.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/stats/extended/ExtendedStatsBuilder.java new file mode 100644 index 00000000000..23d5a0e81e2 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/stats/extended/ExtendedStatsBuilder.java @@ -0,0 +1,13 @@ +package org.elasticsearch.search.aggregations.metrics.stats.extended; + +import org.elasticsearch.search.aggregations.metrics.ValuesSourceMetricsAggregationBuilder; + +/** + * + */ +public class ExtendedStatsBuilder extends ValuesSourceMetricsAggregationBuilder { + + public ExtendedStatsBuilder(String name) { + super(name, InternalExtendedStats.TYPE.name()); + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/stats/extended/ExtendedStatsParser.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/stats/extended/ExtendedStatsParser.java new file mode 100644 index 00000000000..7052e3519c1 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/stats/extended/ExtendedStatsParser.java @@ -0,0 +1,41 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.metrics.stats.extended; + +import org.elasticsearch.search.aggregations.metrics.ValuesSourceMetricsAggregatorParser; +import org.elasticsearch.search.aggregations.support.ValuesSourceConfig; +import org.elasticsearch.search.aggregations.support.numeric.NumericValuesSource; +import org.elasticsearch.search.aggregations.AggregatorFactory; + +/** + * + */ +public class ExtendedStatsParser extends ValuesSourceMetricsAggregatorParser { + + @Override + public String type() { + return InternalExtendedStats.TYPE.name(); + } + + @Override + protected AggregatorFactory createFactory(String aggregationName, ValuesSourceConfig config) { + return new ExtendedStatsAggregator.Factory(aggregationName, config); + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/stats/extended/InternalExtendedStats.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/stats/extended/InternalExtendedStats.java new file mode 100644 index 00000000000..4be77c61b06 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/stats/extended/InternalExtendedStats.java @@ -0,0 +1,132 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.metrics.stats.extended; + +import org.elasticsearch.common.io.stream.StreamInput; +import org.elasticsearch.common.io.stream.StreamOutput; +import org.elasticsearch.common.xcontent.XContentBuilder; +import org.elasticsearch.common.xcontent.XContentBuilderString; +import org.elasticsearch.search.aggregations.AggregationStreams; +import org.elasticsearch.search.aggregations.InternalAggregation; +import org.elasticsearch.search.aggregations.metrics.stats.InternalStats; + +import java.io.IOException; + +/** +* +*/ +public class InternalExtendedStats extends InternalStats implements ExtendedStats { + + public final static Type TYPE = new Type("extended_stats", "estats"); + + public final static AggregationStreams.Stream STREAM = new AggregationStreams.Stream() { + @Override + public InternalExtendedStats readResult(StreamInput in) throws IOException { + InternalExtendedStats result = new InternalExtendedStats(); + result.readFrom(in); + return result; + } + }; + + public static void registerStreams() { + AggregationStreams.registerStream(STREAM, TYPE.stream()); + } + + private double sumOfSqrs; + + InternalExtendedStats() {} // for serialization + + public InternalExtendedStats(String name, long count, double sum, double min, double max, double sumOfSqrs) { + super(name, count, sum, min, max); + this.sumOfSqrs = sumOfSqrs; + } + + @Override + public Type type() { + return TYPE; + } + + @Override + public double value(String name) { + if ("sum_of_squares".equals(name)) { + return sumOfSqrs; + } + if ("variance".equals(name)) { + return getVariance(); + } + if ("std_deviation".equals(name)) { + return getStdDeviation(); + } + return super.value(name); + } + + @Override + public double getSumOfSquares() { + return sumOfSqrs; + } + + @Override + public double getVariance() { + return (sumOfSqrs - ((sum * sum) / count)) / count; + } + + @Override + public double getStdDeviation() { + return Math.sqrt(getVariance()); + } + + @Override + protected void mergeOtherStats(InternalStats to, InternalAggregation from) { + ((InternalExtendedStats) to).sumOfSqrs += ((InternalExtendedStats) from).sumOfSqrs; + } + + @Override + public void readOtherStatsFrom(StreamInput in) throws IOException { + sumOfSqrs = in.readDouble(); + } + + @Override + protected void writeOtherStatsTo(StreamOutput out) throws IOException { + out.writeDouble(sumOfSqrs); + } + + static class Fields { + public static final XContentBuilderString SUM_OF_SQRS = new XContentBuilderString("sum_of_squares"); + public static final XContentBuilderString SUM_OF_SQRS_AS_STRING = new XContentBuilderString("sum_of_squares_as_string"); + public static final XContentBuilderString VARIANCE = new XContentBuilderString("variance"); + public static final XContentBuilderString VARIANCE_AS_STRING = new XContentBuilderString("variance_as_string"); + public static final XContentBuilderString STD_DEVIATION = new XContentBuilderString("std_deviation"); + public static final XContentBuilderString STD_DEVIATION_AS_STRING = new XContentBuilderString("std_deviation_as_string"); + } + + @Override + protected XContentBuilder otherStatsToXCotent(XContentBuilder builder, Params params) throws IOException { + builder.field(Fields.SUM_OF_SQRS, count != 0 ? sumOfSqrs : null); + builder.field(Fields.VARIANCE, count != 0 ? getVariance() : null); + builder.field(Fields.STD_DEVIATION, count != 0 ? getStdDeviation() : null); + if (count != 0 && valueFormatter != null) { + builder.field(Fields.SUM_OF_SQRS_AS_STRING, valueFormatter.format(sumOfSqrs)); + builder.field(Fields.VARIANCE_AS_STRING, valueFormatter.format(getVariance())); + builder.field(Fields.STD_DEVIATION_AS_STRING, valueFormatter.format(getStdDeviation())); + } + return builder; + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/sum/InternalSum.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/sum/InternalSum.java new file mode 100644 index 00000000000..59c4a3c6c8b --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/sum/InternalSum.java @@ -0,0 +1,121 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.metrics.sum; + +import org.elasticsearch.common.io.stream.StreamInput; +import org.elasticsearch.common.io.stream.StreamOutput; +import org.elasticsearch.common.xcontent.XContentBuilder; +import org.elasticsearch.search.aggregations.AggregationStreams; +import org.elasticsearch.search.aggregations.InternalAggregation; +import org.elasticsearch.search.aggregations.metrics.MetricsAggregation; +import org.elasticsearch.search.aggregations.support.numeric.ValueFormatterStreams; + +import java.io.IOException; +import java.util.List; + +/** +* +*/ +public class InternalSum extends MetricsAggregation.SingleValue implements Sum { + + public final static Type TYPE = new Type("sum"); + + public final static AggregationStreams.Stream STREAM = new AggregationStreams.Stream() { + @Override + public InternalSum readResult(StreamInput in) throws IOException { + InternalSum result = new InternalSum(); + result.readFrom(in); + return result; + } + }; + + public static void registerStreams() { + AggregationStreams.registerStream(STREAM, TYPE.stream()); + } + + private double sum; + + InternalSum() {} // for serialization + + InternalSum(String name, double sum) { + super(name); + this.sum = sum; + } + + @Override + public double value() { + return sum; + } + + public double getValue() { + return sum; + } + + @Override + public Type type() { + return TYPE; + } + + @Override + public InternalSum reduce(ReduceContext reduceContext) { + List aggregations = reduceContext.aggregations(); + if (aggregations.size() == 1) { + return (InternalSum) aggregations.get(0); + } + InternalSum reduced = null; + for (InternalAggregation aggregation : aggregations) { + if (reduced == null) { + reduced = (InternalSum) aggregation; + } else { + reduced.sum += ((InternalSum) aggregation).sum; + } + } + if (reduced != null) { + return reduced; + } + return (InternalSum) aggregations.get(0); + } + + @Override + public void readFrom(StreamInput in) throws IOException { + name = in.readString(); + valueFormatter = ValueFormatterStreams.readOptional(in); + sum = in.readDouble(); + } + + @Override + public void writeTo(StreamOutput out) throws IOException { + out.writeString(name); + ValueFormatterStreams.writeOptional(valueFormatter, out); + out.writeDouble(sum); + } + + @Override + public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException { + builder.startObject(name); + builder.field(CommonFields.VALUE, sum); + if (valueFormatter != null) { + builder.field(CommonFields.VALUE_AS_STRING, valueFormatter.format(sum)); + } + builder.endObject(); + return builder; + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/sum/Sum.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/sum/Sum.java new file mode 100644 index 00000000000..f5f75bc71d6 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/sum/Sum.java @@ -0,0 +1,30 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.metrics.sum; + +import org.elasticsearch.search.aggregations.Aggregation; + +/** + * + */ +public interface Sum extends Aggregation { + + double getValue(); +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/sum/SumAggregator.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/sum/SumAggregator.java new file mode 100644 index 00000000000..f268b5fad47 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/sum/SumAggregator.java @@ -0,0 +1,106 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.metrics.sum; + +import org.elasticsearch.common.util.BigArrays; +import org.elasticsearch.common.util.DoubleArray; +import org.elasticsearch.index.fielddata.DoubleValues; +import org.elasticsearch.search.aggregations.Aggregator; +import org.elasticsearch.search.aggregations.InternalAggregation; +import org.elasticsearch.search.aggregations.support.AggregationContext; +import org.elasticsearch.search.aggregations.support.ValuesSourceConfig; +import org.elasticsearch.search.aggregations.support.numeric.NumericValuesSource; +import org.elasticsearch.search.aggregations.AggregatorFactories; +import org.elasticsearch.search.aggregations.support.ValueSourceAggregatorFactory; + +import java.io.IOException; + +/** + * + */ +public class SumAggregator extends Aggregator { + + private final NumericValuesSource valuesSource; + + private DoubleArray sums; + + public SumAggregator(String name, long estimatedBucketsCount, NumericValuesSource valuesSource, AggregationContext context, Aggregator parent) { + super(name, BucketAggregationMode.MULTI_BUCKETS, AggregatorFactories.EMPTY, estimatedBucketsCount, context, parent); + this.valuesSource = valuesSource; + if (valuesSource != null) { + final long initialSize = estimatedBucketsCount < 2 ? 1 : estimatedBucketsCount; + sums = BigArrays.newDoubleArray(initialSize); + } + } + + @Override + public boolean shouldCollect() { + return valuesSource != null; + } + + @Override + public void collect(int doc, long owningBucketOrdinal) throws IOException { + assert valuesSource != null : "collect must only be called after #shouldCollect returns true"; + + DoubleValues values = valuesSource.doubleValues(); + if (values == null) { + return; + } + + sums = BigArrays.grow(sums, owningBucketOrdinal + 1); + + final int valuesCount = values.setDocument(doc); + double sum = 0; + for (int i = 0; i < valuesCount; i++) { + sum += values.nextValue(); + } + sums.increment(owningBucketOrdinal, sum); + } + + @Override + public InternalAggregation buildAggregation(long owningBucketOrdinal) { + if (valuesSource == null) { + return new InternalSum(name, 0); + } + return new InternalSum(name, sums.get(owningBucketOrdinal)); + } + + @Override + public InternalAggregation buildEmptyAggregation() { + return new InternalSum(name, 0.0); + } + + public static class Factory extends ValueSourceAggregatorFactory.LeafOnly { + + public Factory(String name, ValuesSourceConfig valuesSourceConfig) { + super(name, InternalSum.TYPE.name(), valuesSourceConfig); + } + + @Override + protected Aggregator createUnmapped(AggregationContext aggregationContext, Aggregator parent) { + return new SumAggregator(name, 0, null, aggregationContext, parent); + } + + @Override + protected Aggregator create(NumericValuesSource valuesSource, long expectedBucketsCount, AggregationContext aggregationContext, Aggregator parent) { + return new SumAggregator(name, expectedBucketsCount, valuesSource, aggregationContext, parent); + } + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/sum/SumBuilder.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/sum/SumBuilder.java new file mode 100644 index 00000000000..a524cc51e58 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/sum/SumBuilder.java @@ -0,0 +1,13 @@ +package org.elasticsearch.search.aggregations.metrics.sum; + +import org.elasticsearch.search.aggregations.metrics.ValuesSourceMetricsAggregationBuilder; + +/** + * + */ +public class SumBuilder extends ValuesSourceMetricsAggregationBuilder { + + public SumBuilder(String name) { + super(name, InternalSum.TYPE.name()); + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/sum/SumParser.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/sum/SumParser.java new file mode 100644 index 00000000000..9108fc94599 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/sum/SumParser.java @@ -0,0 +1,41 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.metrics.sum; + +import org.elasticsearch.search.aggregations.metrics.ValuesSourceMetricsAggregatorParser; +import org.elasticsearch.search.aggregations.support.ValuesSourceConfig; +import org.elasticsearch.search.aggregations.support.numeric.NumericValuesSource; +import org.elasticsearch.search.aggregations.AggregatorFactory; + +/** + * + */ +public class SumParser extends ValuesSourceMetricsAggregatorParser { + + @Override + public String type() { + return InternalSum.TYPE.name(); + } + + @Override + protected AggregatorFactory createFactory(String aggregationName, ValuesSourceConfig config) { + return new SumAggregator.Factory(aggregationName, config); + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/valuecount/InternalValueCount.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/valuecount/InternalValueCount.java new file mode 100644 index 00000000000..5324c3b36ca --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/valuecount/InternalValueCount.java @@ -0,0 +1,111 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.metrics.valuecount; + +import org.elasticsearch.common.io.stream.StreamInput; +import org.elasticsearch.common.io.stream.StreamOutput; +import org.elasticsearch.common.xcontent.XContentBuilder; +import org.elasticsearch.search.aggregations.AggregationStreams; +import org.elasticsearch.search.aggregations.InternalAggregation; +import org.elasticsearch.search.aggregations.metrics.MetricsAggregation; + +import java.io.IOException; +import java.util.List; + +/** + * An internal implementation of {@link ValueCount}. + */ +public class InternalValueCount extends MetricsAggregation implements ValueCount { + + public static final Type TYPE = new Type("value_count", "vcount"); + + private static final AggregationStreams.Stream STREAM = new AggregationStreams.Stream() { + @Override + public InternalValueCount readResult(StreamInput in) throws IOException { + InternalValueCount count = new InternalValueCount(); + count.readFrom(in); + return count; + } + }; + + public static void registerStreams() { + AggregationStreams.registerStream(STREAM, TYPE.stream()); + } + + private long value; + + InternalValueCount() {} // for serialization + + public InternalValueCount(String name, long value) { + super(name); + this.value = value; + } + + @Override + public long getValue() { + return value; + } + + @Override + public Type type() { + return TYPE; + } + + @Override + public InternalAggregation reduce(ReduceContext reduceContext) { + List aggregations = reduceContext.aggregations(); + if (aggregations.size() == 1) { + return aggregations.get(0); + } + InternalValueCount reduced = null; + for (InternalAggregation aggregation : aggregations) { + if (reduced == null) { + reduced = (InternalValueCount) aggregation; + } else { + reduced.value += ((InternalValueCount) aggregation).value; + } + } + return reduced; + } + + @Override + public void readFrom(StreamInput in) throws IOException { + name = in.readString(); + value = in.readVLong(); + } + + @Override + public void writeTo(StreamOutput out) throws IOException { + out.writeString(name); + out.writeVLong(value); + } + + @Override + public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException { + return builder.startObject(name) + .field(CommonFields.VALUE, value) + .endObject(); + } + + @Override + public String toString() { + return "count[" + value + "]"; + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/valuecount/ValueCount.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/valuecount/ValueCount.java new file mode 100644 index 00000000000..e4dd0afc21b --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/valuecount/ValueCount.java @@ -0,0 +1,35 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.metrics.valuecount; + +import org.elasticsearch.search.aggregations.Aggregation; + +/** + * An get that holds the number of values that the current document set has for a specific + * field. + */ +public interface ValueCount extends Aggregation { + + /** + * @return The count + */ + long getValue(); + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/valuecount/ValueCountAggregator.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/valuecount/ValueCountAggregator.java new file mode 100644 index 00000000000..972c0677f11 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/valuecount/ValueCountAggregator.java @@ -0,0 +1,105 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.metrics.valuecount; + +import org.elasticsearch.common.util.BigArrays; +import org.elasticsearch.common.util.LongArray; +import org.elasticsearch.index.fielddata.BytesValues; +import org.elasticsearch.search.aggregations.Aggregator; +import org.elasticsearch.search.aggregations.InternalAggregation; +import org.elasticsearch.search.aggregations.support.AggregationContext; +import org.elasticsearch.search.aggregations.support.ValuesSourceConfig; +import org.elasticsearch.search.aggregations.support.bytes.BytesValuesSource; +import org.elasticsearch.search.aggregations.AggregatorFactories; +import org.elasticsearch.search.aggregations.support.ValueSourceAggregatorFactory; + +import java.io.IOException; + +/** + * A field data based aggregator that counts the number of values a specific field has within the aggregation context. + * + * This aggregator works in a multi-bucket mode, that is, when serves as a sub-aggregator, a single aggregator instance aggregates the + * counts for all buckets owned by the parent aggregator) + */ +public class ValueCountAggregator extends Aggregator { + + private final BytesValuesSource valuesSource; + + // a count per bucket + LongArray counts; + + public ValueCountAggregator(String name, long expectedBucketsCount, BytesValuesSource valuesSource, AggregationContext aggregationContext, Aggregator parent) { + super(name, BucketAggregationMode.MULTI_BUCKETS, AggregatorFactories.EMPTY, 0, aggregationContext, parent); + this.valuesSource = valuesSource; + if (valuesSource != null) { + // expectedBucketsCount == 0 means it's a top level bucket + final long initialSize = expectedBucketsCount < 2 ? 1 : expectedBucketsCount; + counts = BigArrays.newLongArray(initialSize); + } + } + + @Override + public boolean shouldCollect() { + return valuesSource != null; + } + + @Override + public void collect(int doc, long owningBucketOrdinal) throws IOException { + BytesValues values = valuesSource.bytesValues(); + if (values == null) { + return; + } + counts = BigArrays.grow(counts, owningBucketOrdinal + 1); + counts.increment(owningBucketOrdinal, values.setDocument(doc)); + } + + @Override + public InternalAggregation buildAggregation(long owningBucketOrdinal) { + if (valuesSource == null) { + return new InternalValueCount(name, 0); + } + assert owningBucketOrdinal < counts.size(); + return new InternalValueCount(name, counts.get(owningBucketOrdinal)); + } + + @Override + public InternalAggregation buildEmptyAggregation() { + return new InternalValueCount(name, 0l); + } + + public static class Factory extends ValueSourceAggregatorFactory.LeafOnly { + + public Factory(String name, ValuesSourceConfig valuesSourceBuilder) { + super(name, InternalValueCount.TYPE.name(), valuesSourceBuilder); + } + + @Override + protected Aggregator createUnmapped(AggregationContext aggregationContext, Aggregator parent) { + return new ValueCountAggregator(name, 0, null, aggregationContext, parent); + } + + @Override + protected Aggregator create(BytesValuesSource valuesSource, long expectedBucketsCount, AggregationContext aggregationContext, Aggregator parent) { + return new ValueCountAggregator(name, expectedBucketsCount, valuesSource, aggregationContext, parent); + } + + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/valuecount/ValueCountBuilder.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/valuecount/ValueCountBuilder.java new file mode 100644 index 00000000000..1584c2ed72d --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/valuecount/ValueCountBuilder.java @@ -0,0 +1,30 @@ +package org.elasticsearch.search.aggregations.metrics.valuecount; + +import org.elasticsearch.common.xcontent.XContentBuilder; +import org.elasticsearch.search.aggregations.metrics.MetricsAggregationBuilder; + +import java.io.IOException; + +/** + * + */ +public class ValueCountBuilder extends MetricsAggregationBuilder { + + private String field; + + public ValueCountBuilder(String name) { + super(name, InternalValueCount.TYPE.name()); + } + + public ValueCountBuilder field(String field) { + this.field = field; + return this; + } + + @Override + protected void internalXContent(XContentBuilder builder, Params params) throws IOException { + if (field != null) { + builder.field("field", field); + } + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/metrics/valuecount/ValueCountParser.java b/src/main/java/org/elasticsearch/search/aggregations/metrics/valuecount/ValueCountParser.java new file mode 100644 index 00000000000..9dc0cf0fa58 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/metrics/valuecount/ValueCountParser.java @@ -0,0 +1,77 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.metrics.valuecount; + +import org.elasticsearch.common.xcontent.XContentParser; +import org.elasticsearch.index.fielddata.IndexFieldData; +import org.elasticsearch.index.mapper.FieldMapper; +import org.elasticsearch.search.aggregations.Aggregator; +import org.elasticsearch.search.aggregations.support.FieldContext; +import org.elasticsearch.search.aggregations.support.ValuesSourceConfig; +import org.elasticsearch.search.aggregations.support.bytes.BytesValuesSource; +import org.elasticsearch.search.aggregations.AggregatorFactory; +import org.elasticsearch.search.internal.SearchContext; + +import java.io.IOException; + +/** + * + */ +public class ValueCountParser implements Aggregator.Parser { + + @Override + public String type() { + return InternalValueCount.TYPE.name(); + } + + @Override + public AggregatorFactory parse(String aggregationName, XContentParser parser, SearchContext context) throws IOException { + + ValuesSourceConfig config = new ValuesSourceConfig(BytesValuesSource.class); + + String field = null; + + XContentParser.Token token; + String currentFieldName = null; + while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) { + if (token == XContentParser.Token.FIELD_NAME) { + currentFieldName = parser.currentName(); + } else if (token == XContentParser.Token.VALUE_STRING) { + if ("field".equals(currentFieldName)) { + field = parser.text(); + } + } + } + + if (field == null) { + return new ValueCountAggregator.Factory(aggregationName, config); + } + + FieldMapper mapper = context.smartNameFieldMapper(field); + if (mapper == null) { + config.unmapped(true); + return new ValueCountAggregator.Factory(aggregationName, config); + } + + IndexFieldData indexFieldData = context.fieldData().getForField(mapper); + config.fieldContext(new FieldContext(field, indexFieldData)); + return new ValueCountAggregator.Factory(aggregationName, config); + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/support/AggregationContext.java b/src/main/java/org/elasticsearch/search/aggregations/support/AggregationContext.java new file mode 100644 index 00000000000..54947645c56 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/support/AggregationContext.java @@ -0,0 +1,239 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.support; + +import com.carrotsearch.hppc.ObjectObjectOpenHashMap; +import org.apache.lucene.index.AtomicReaderContext; +import org.apache.lucene.search.Scorer; +import org.apache.lucene.util.ArrayUtil; +import org.apache.lucene.util.RamUsageEstimator; +import org.elasticsearch.cache.recycler.CacheRecycler; +import org.elasticsearch.common.lucene.ReaderContextAware; +import org.elasticsearch.common.lucene.ScorerAware; +import org.elasticsearch.index.fielddata.IndexGeoPointFieldData; +import org.elasticsearch.index.fielddata.IndexNumericFieldData; +import org.elasticsearch.search.aggregations.AggregationExecutionException; +import org.elasticsearch.search.aggregations.support.bytes.BytesValuesSource; +import org.elasticsearch.search.aggregations.support.geopoints.GeoPointValuesSource; +import org.elasticsearch.search.aggregations.support.numeric.NumericValuesSource; +import org.elasticsearch.search.internal.SearchContext; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; + +/** + * + */ +@SuppressWarnings({"unchecked", "ForLoopReplaceableByForEach"}) +public class AggregationContext implements ReaderContextAware, ScorerAware { + + private final SearchContext searchContext; + + private ObjectObjectOpenHashMap[] perDepthFieldDataSources = new ObjectObjectOpenHashMap[4]; + private List readerAwares = new ArrayList(); + private List scorerAwares = new ArrayList(); + + private AtomicReaderContext reader; + private Scorer scorer; + + public AggregationContext(SearchContext searchContext) { + this.searchContext = searchContext; + } + + public SearchContext searchContext() { + return searchContext; + } + + public CacheRecycler cacheRecycler() { + return searchContext.cacheRecycler(); + } + + public AtomicReaderContext currentReader() { + return reader; + } + + public Scorer currentScorer() { + return scorer; + } + + public void setNextReader(AtomicReaderContext reader) { + this.reader = reader; + for (ReaderContextAware aware : readerAwares) { + aware.setNextReader(reader); + } + } + + public void setScorer(Scorer scorer) { + this.scorer = scorer; + for (ScorerAware scorerAware : scorerAwares) { + scorerAware.setScorer(scorer); + } + } + + /** Get a value source given its configuration and the depth of the aggregator in the aggregation tree. */ + public VS valuesSource(ValuesSourceConfig config, int depth) { + assert config.valid() : "value source config is invalid - must have either a field context or a script or marked as unmapped"; + assert !config.unmapped : "value source should not be created for unmapped fields"; + + if (perDepthFieldDataSources.length <= depth) { + perDepthFieldDataSources = Arrays.copyOf(perDepthFieldDataSources, ArrayUtil.oversize(1 + depth, RamUsageEstimator.NUM_BYTES_OBJECT_REF)); + } + if (perDepthFieldDataSources[depth] == null) { + perDepthFieldDataSources[depth] = new ObjectObjectOpenHashMap(); + } + final ObjectObjectOpenHashMap fieldDataSources = perDepthFieldDataSources[depth]; + + if (config.fieldContext == null) { + if (NumericValuesSource.class.isAssignableFrom(config.valueSourceType)) { + return (VS) numericScript(config); + } + if (BytesValuesSource.class.isAssignableFrom(config.valueSourceType)) { + return (VS) bytesScript(config); + } + throw new AggregationExecutionException("value source of type [" + config.valueSourceType.getSimpleName() + "] is not supported by scripts"); + } + + if (NumericValuesSource.class.isAssignableFrom(config.valueSourceType)) { + return (VS) numericField(fieldDataSources, config); + } + if (GeoPointValuesSource.class.isAssignableFrom(config.valueSourceType)) { + return (VS) geoPointField(fieldDataSources, config); + } + // falling back to bytes values + return (VS) bytesField(fieldDataSources, config); + } + + private NumericValuesSource numericScript(ValuesSourceConfig config) { + setScorerIfNeeded(config.script); + setReaderIfNeeded(config.script); + scorerAwares.add(config.script); + readerAwares.add(config.script); + FieldDataSource.Numeric source = new FieldDataSource.Numeric.Script(config.script, config.scriptValueType); + if (config.ensureUnique || config.ensureSorted) { + source = new FieldDataSource.Numeric.SortedAndUnique(source); + readerAwares.add((ReaderContextAware) source); + } + return new NumericValuesSource(source, config.formatter(), config.parser()); + } + + private NumericValuesSource numericField(ObjectObjectOpenHashMap fieldDataSources, ValuesSourceConfig config) { + FieldDataSource.Numeric dataSource = (FieldDataSource.Numeric) fieldDataSources.get(config.fieldContext.field()); + if (dataSource == null) { + dataSource = new FieldDataSource.Numeric.FieldData((IndexNumericFieldData) config.fieldContext.indexFieldData()); + setReaderIfNeeded((ReaderContextAware) dataSource); + readerAwares.add((ReaderContextAware) dataSource); + fieldDataSources.put(config.fieldContext.field(), dataSource); + } + if (config.script != null) { + setScorerIfNeeded(config.script); + setReaderIfNeeded(config.script); + scorerAwares.add(config.script); + readerAwares.add(config.script); + dataSource = new FieldDataSource.Numeric.WithScript(dataSource, config.script); + + if (config.ensureUnique || config.ensureSorted) { + dataSource = new FieldDataSource.Numeric.SortedAndUnique(dataSource); + readerAwares.add((ReaderContextAware) dataSource); + } + } + if (config.needsHashes) { + dataSource.setNeedsHashes(true); + } + return new NumericValuesSource(dataSource, config.formatter(), config.parser()); + } + + private BytesValuesSource bytesField(ObjectObjectOpenHashMap fieldDataSources, ValuesSourceConfig config) { + FieldDataSource dataSource = fieldDataSources.get(config.fieldContext.field()); + if (dataSource == null) { + dataSource = new FieldDataSource.Bytes.FieldData(config.fieldContext.indexFieldData()); + setReaderIfNeeded((ReaderContextAware) dataSource); + readerAwares.add((ReaderContextAware) dataSource); + fieldDataSources.put(config.fieldContext.field(), dataSource); + } + if (config.script != null) { + setScorerIfNeeded(config.script); + setReaderIfNeeded(config.script); + scorerAwares.add(config.script); + readerAwares.add(config.script); + dataSource = new FieldDataSource.WithScript(dataSource, config.script); + } + // Even in case we wrap field data, we might still need to wrap for sorting, because the wrapped field data might be + // eg. a numeric field data that doesn't sort according to the byte order. However field data values are unique so no + // need to wrap for uniqueness + if ((config.ensureUnique && !(dataSource instanceof FieldDataSource.Bytes.FieldData)) || config.ensureSorted) { + dataSource = new FieldDataSource.Bytes.SortedAndUnique(dataSource); + readerAwares.add((ReaderContextAware) dataSource); + } + if (config.needsHashes) { // the data source needs hash if at least one consumer needs hashes + dataSource.setNeedsHashes(true); + } + return new BytesValuesSource(dataSource); + } + + private BytesValuesSource bytesScript(ValuesSourceConfig config) { + setScorerIfNeeded(config.script); + setReaderIfNeeded(config.script); + scorerAwares.add(config.script); + readerAwares.add(config.script); + FieldDataSource.Bytes source = new FieldDataSource.Bytes.Script(config.script); + if (config.ensureUnique || config.ensureSorted) { + source = new FieldDataSource.Bytes.SortedAndUnique(source); + readerAwares.add((ReaderContextAware) source); + } + return new BytesValuesSource(source); + } + + private GeoPointValuesSource geoPointField(ObjectObjectOpenHashMap fieldDataSources, ValuesSourceConfig config) { + FieldDataSource.GeoPoint dataSource = (FieldDataSource.GeoPoint) fieldDataSources.get(config.fieldContext.field()); + if (dataSource == null) { + dataSource = new FieldDataSource.GeoPoint((IndexGeoPointFieldData) config.fieldContext.indexFieldData()); + setReaderIfNeeded(dataSource); + readerAwares.add(dataSource); + fieldDataSources.put(config.fieldContext.field(), dataSource); + } + if (config.needsHashes) { + dataSource.setNeedsHashes(true); + } + return new GeoPointValuesSource(dataSource); + } + + public void registerReaderContextAware(ReaderContextAware readerContextAware) { + setReaderIfNeeded(readerContextAware); + readerAwares.add(readerContextAware); + } + + public void registerScorerAware(ScorerAware scorerAware) { + setScorerIfNeeded(scorerAware); + scorerAwares.add(scorerAware); + } + + private void setReaderIfNeeded(ReaderContextAware readerAware) { + if (reader != null) { + readerAware.setNextReader(reader); + } + } + + private void setScorerIfNeeded(ScorerAware scorerAware) { + if (scorer != null) { + scorerAware.setScorer(scorer); + } + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/support/FieldContext.java b/src/main/java/org/elasticsearch/search/aggregations/support/FieldContext.java new file mode 100644 index 00000000000..fd1b80ad7cb --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/support/FieldContext.java @@ -0,0 +1,55 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.support; + +import org.elasticsearch.index.fielddata.IndexFieldData; + +/** + * Used by all field data based aggregators. This determine the context of the field data the aggregators are operating + * in. I holds both the field names and the index field datas that are associated with them. + */ +public class FieldContext { + + private final String field; + private final IndexFieldData indexFieldData; + + /** + * Constructs a field data context for the given field and its index field data + * + * @param field The name of the field + * @param indexFieldData The index field data of the field + */ + public FieldContext(String field, IndexFieldData indexFieldData) { + this.field = field; + this.indexFieldData = indexFieldData; + } + + public String field() { + return field; + } + + /** + * @return The index field datas in this context + */ + public IndexFieldData indexFieldData() { + return indexFieldData; + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/support/FieldDataSource.java b/src/main/java/org/elasticsearch/search/aggregations/support/FieldDataSource.java new file mode 100644 index 00000000000..1d93458f369 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/support/FieldDataSource.java @@ -0,0 +1,668 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.support; + +import org.apache.lucene.index.AtomicReaderContext; +import org.apache.lucene.util.ArrayUtil; +import org.apache.lucene.util.BytesRef; +import org.apache.lucene.util.BytesRefHash; +import org.apache.lucene.util.InPlaceMergeSorter; +import org.elasticsearch.common.lucene.ReaderContextAware; +import org.elasticsearch.index.fielddata.*; +import org.elasticsearch.index.fielddata.AtomicFieldData.Order; +import org.elasticsearch.script.SearchScript; +import org.elasticsearch.search.aggregations.support.FieldDataSource.Bytes.SortedAndUnique.SortedUniqueBytesValues; +import org.elasticsearch.search.aggregations.support.bytes.ScriptBytesValues; +import org.elasticsearch.search.aggregations.support.numeric.ScriptDoubleValues; +import org.elasticsearch.search.aggregations.support.numeric.ScriptLongValues; + +/** + * + */ +public abstract class FieldDataSource { + + /** Whether values are unique or not per document. */ + public enum Uniqueness { + UNIQUE, + NOT_UNIQUE, + UNKNOWN; + } + + /** Return whether values are unique. */ + public Uniqueness getUniqueness() { + return Uniqueness.UNKNOWN; + } + + /** Get the current {@link BytesValues}. */ + public abstract BytesValues bytesValues(); + + /** Ask the underlying data source to provide pre-computed hashes, optional operation. */ + public void setNeedsHashes(boolean needsHashes) {} + + public static abstract class Bytes extends FieldDataSource { + + public static class FieldData extends Bytes implements ReaderContextAware { + + protected boolean needsHashes; + protected final IndexFieldData indexFieldData; + protected AtomicFieldData atomicFieldData; + private BytesValues bytesValues; + + public FieldData(IndexFieldData indexFieldData) { + this.indexFieldData = indexFieldData; + needsHashes = false; + } + + @Override + public Uniqueness getUniqueness() { + return Uniqueness.UNIQUE; + } + + public final void setNeedsHashes(boolean needsHashes) { + this.needsHashes = needsHashes; + } + + @Override + public void setNextReader(AtomicReaderContext reader) { + atomicFieldData = indexFieldData.load(reader); + if (bytesValues != null) { + bytesValues = atomicFieldData.getBytesValues(needsHashes); + } + } + + @Override + public org.elasticsearch.index.fielddata.BytesValues bytesValues() { + if (bytesValues == null) { + bytesValues = atomicFieldData.getBytesValues(needsHashes); + } + return bytesValues; + } + } + + public static class Script extends Bytes { + + private final ScriptBytesValues values; + + public Script(SearchScript script) { + values = new ScriptBytesValues(script); + } + + @Override + public org.elasticsearch.index.fielddata.BytesValues bytesValues() { + return values; + } + + } + + public static class SortedAndUnique extends Bytes implements ReaderContextAware { + + private final FieldDataSource delegate; + private BytesValues bytesValues; + + public SortedAndUnique(FieldDataSource delegate) { + this.delegate = delegate; + } + + @Override + public Uniqueness getUniqueness() { + return Uniqueness.UNIQUE; + } + + @Override + public void setNextReader(AtomicReaderContext reader) { + bytesValues = null; // order may change per-segment -> reset + } + + @Override + public org.elasticsearch.index.fielddata.BytesValues bytesValues() { + if (bytesValues == null) { + bytesValues = delegate.bytesValues(); + if (bytesValues.isMultiValued() && + (delegate.getUniqueness() != Uniqueness.UNIQUE || bytesValues.getOrder() != Order.BYTES)) { + bytesValues = new SortedUniqueBytesValues(bytesValues); + } + } + return bytesValues; + } + + static class SortedUniqueBytesValues extends FilterBytesValues { + + final BytesRef spare; + int[] sortedIds; + final BytesRefHash bytes; + int numUniqueValues; + int pos = Integer.MAX_VALUE; + + public SortedUniqueBytesValues(BytesValues delegate) { + super(delegate); + bytes = new BytesRefHash(); + spare = new BytesRef(); + } + + @Override + public int setDocument(int docId) { + final int numValues = super.setDocument(docId); + if (numValues == 0) { + sortedIds = null; + return 0; + } + bytes.clear(); + bytes.reinit(); + for (int i = 0; i < numValues; ++i) { + bytes.add(super.nextValue(), super.hashCode()); + } + numUniqueValues = bytes.size(); + sortedIds = bytes.sort(BytesRef.getUTF8SortedAsUnicodeComparator()); + pos = 0; + return numUniqueValues; + } + + @Override + public BytesRef nextValue() { + bytes.get(sortedIds[pos++], spare); + return spare; + } + + @Override + public Order getOrder() { + return Order.BYTES; + } + + } + + } + + } + + public static abstract class Numeric extends FieldDataSource { + + /** Whether the underlying data is floating-point or not. */ + public abstract boolean isFloatingPoint(); + + /** Get the current {@link LongValues}. */ + public abstract LongValues longValues(); + + /** Get the current {@link DoubleValues}. */ + public abstract DoubleValues doubleValues(); + + public static class WithScript extends Numeric { + + private final Numeric delegate; + private final LongValues longValues; + private final DoubleValues doubleValues; + private final FieldDataSource.WithScript.BytesValues bytesValues; + + public WithScript(Numeric delegate, SearchScript script) { + this.delegate = delegate; + this.longValues = new LongValues(delegate, script); + this.doubleValues = new DoubleValues(delegate, script); + this.bytesValues = new FieldDataSource.WithScript.BytesValues(delegate, script); + } + + @Override + public boolean isFloatingPoint() { + return true; // even if the underlying source produces longs, scripts can change them to doubles + } + + @Override + public BytesValues bytesValues() { + return bytesValues; + } + + @Override + public LongValues longValues() { + return longValues; + } + + @Override + public DoubleValues doubleValues() { + return doubleValues; + } + + static class LongValues extends org.elasticsearch.index.fielddata.LongValues { + + private final Numeric source; + private final SearchScript script; + + public LongValues(Numeric source, SearchScript script) { + super(true); + this.source = source; + this.script = script; + } + + @Override + public int setDocument(int docId) { + return source.longValues().setDocument(docId); + } + + @Override + public long nextValue() { + script.setNextVar("_value", source.longValues().nextValue()); + return script.runAsLong(); + } + } + + static class DoubleValues extends org.elasticsearch.index.fielddata.DoubleValues { + + private final Numeric source; + private final SearchScript script; + + public DoubleValues(Numeric source, SearchScript script) { + super(true); + this.source = source; + this.script = script; + } + + @Override + public int setDocument(int docId) { + return source.doubleValues().setDocument(docId); + } + + @Override + public double nextValue() { + script.setNextVar("_value", source.doubleValues().nextValue()); + return script.runAsDouble(); + } + } + } + + public static class FieldData extends Numeric implements ReaderContextAware { + + protected boolean needsHashes; + protected final IndexNumericFieldData indexFieldData; + protected AtomicNumericFieldData atomicFieldData; + private BytesValues bytesValues; + private LongValues longValues; + private DoubleValues doubleValues; + + public FieldData(IndexNumericFieldData indexFieldData) { + this.indexFieldData = indexFieldData; + needsHashes = false; + } + + @Override + public Uniqueness getUniqueness() { + return Uniqueness.UNIQUE; + } + + @Override + public boolean isFloatingPoint() { + return indexFieldData.getNumericType().isFloatingPoint(); + } + + @Override + public final void setNeedsHashes(boolean needsHashes) { + this.needsHashes = needsHashes; + } + + @Override + public void setNextReader(AtomicReaderContext reader) { + atomicFieldData = indexFieldData.load(reader); + if (bytesValues != null) { + bytesValues = atomicFieldData.getBytesValues(needsHashes); + } + if (longValues != null) { + longValues = atomicFieldData.getLongValues(); + } + if (doubleValues != null) { + doubleValues = atomicFieldData.getDoubleValues(); + } + } + + @Override + public org.elasticsearch.index.fielddata.BytesValues bytesValues() { + if (bytesValues == null) { + bytesValues = atomicFieldData.getBytesValues(needsHashes); + } + return bytesValues; + } + + @Override + public org.elasticsearch.index.fielddata.LongValues longValues() { + if (longValues == null) { + longValues = atomicFieldData.getLongValues(); + } + assert longValues.getOrder() == Order.NUMERIC; + return longValues; + } + + @Override + public org.elasticsearch.index.fielddata.DoubleValues doubleValues() { + if (doubleValues == null) { + doubleValues = atomicFieldData.getDoubleValues(); + } + assert doubleValues.getOrder() == Order.NUMERIC; + return doubleValues; + } + } + + public static class Script extends Numeric { + private final ScriptValueType scriptValueType; + + private final ScriptDoubleValues doubleValues; + private final ScriptLongValues longValues; + private final ScriptBytesValues bytesValues; + + public Script(SearchScript script, ScriptValueType scriptValueType) { + this.scriptValueType = scriptValueType; + longValues = new ScriptLongValues(script); + doubleValues = new ScriptDoubleValues(script); + bytesValues = new ScriptBytesValues(script); + } + + @Override + public boolean isFloatingPoint() { + return scriptValueType != null ? scriptValueType.isFloatingPoint() : true; + } + + @Override + public LongValues longValues() { + return longValues; + } + + @Override + public DoubleValues doubleValues() { + return doubleValues; + } + + @Override + public BytesValues bytesValues() { + return bytesValues; + } + + } + + public static class SortedAndUnique extends Numeric implements ReaderContextAware { + + private final Numeric delegate; + private LongValues longValues; + private DoubleValues doubleValues; + private BytesValues bytesValues; + + public SortedAndUnique(Numeric delegate) { + this.delegate = delegate; + } + + @Override + public Uniqueness getUniqueness() { + return Uniqueness.UNIQUE; + } + + @Override + public boolean isFloatingPoint() { + return delegate.isFloatingPoint(); + } + + @Override + public void setNextReader(AtomicReaderContext reader) { + longValues = null; // order may change per-segment -> reset + doubleValues = null; + bytesValues = null; + } + + @Override + public org.elasticsearch.index.fielddata.LongValues longValues() { + if (longValues == null) { + longValues = delegate.longValues(); + if (longValues.isMultiValued() && + (delegate.getUniqueness() != Uniqueness.UNIQUE || longValues.getOrder() != Order.NUMERIC)) { + longValues = new SortedUniqueLongValues(longValues); + } + } + return longValues; + } + + @Override + public org.elasticsearch.index.fielddata.DoubleValues doubleValues() { + if (doubleValues == null) { + doubleValues = delegate.doubleValues(); + if (doubleValues.isMultiValued() && + (delegate.getUniqueness() != Uniqueness.UNIQUE || doubleValues.getOrder() != Order.NUMERIC)) { + doubleValues = new SortedUniqueDoubleValues(doubleValues); + } + } + return doubleValues; + } + + @Override + public org.elasticsearch.index.fielddata.BytesValues bytesValues() { + if (bytesValues == null) { + bytesValues = delegate.bytesValues(); + if (bytesValues.isMultiValued() && + (delegate.getUniqueness() != Uniqueness.UNIQUE || bytesValues.getOrder() != Order.BYTES)) { + bytesValues = new SortedUniqueBytesValues(bytesValues); + } + } + return bytesValues; + } + + private static class SortedUniqueLongValues extends FilterLongValues { + + int numUniqueValues; + long[] array = new long[2]; + int pos = Integer.MAX_VALUE; + + final InPlaceMergeSorter sorter = new InPlaceMergeSorter() { + @Override + protected void swap(int i, int j) { + final long tmp = array[i]; + array[i] = array[j]; + array[j] = tmp; + } + @Override + protected int compare(int i, int j) { + final long l1 = array[i]; + final long l2 = array[j]; + return l1 < l2 ? -1 : l1 == l2 ? 0 : 1; + } + }; + + protected SortedUniqueLongValues(LongValues delegate) { + super(delegate); + } + + @Override + public int setDocument(int docId) { + final int numValues = super.setDocument(docId); + if (numValues == 0) { + return numUniqueValues = 0; + } + array = ArrayUtil.grow(array, numValues); + for (int i = 0; i < numValues; ++i) { + array[i] = super.nextValue(); + } + sorter.sort(0, numValues); + numUniqueValues = 1; + for (int i = 1; i < numValues; ++i) { + if (array[i] != array[i-1]) { + array[numUniqueValues++] = array[i]; + } + } + pos = 0; + return numUniqueValues; + } + + @Override + public long nextValue() { + assert pos < numUniqueValues; + return array[pos++]; + } + + @Override + public Order getOrder() { + return Order.NUMERIC; + } + + } + + private static class SortedUniqueDoubleValues extends FilterDoubleValues { + + int numUniqueValues; + double[] array = new double[2]; + int pos = Integer.MAX_VALUE; + + final InPlaceMergeSorter sorter = new InPlaceMergeSorter() { + @Override + protected void swap(int i, int j) { + final double tmp = array[i]; + array[i] = array[j]; + array[j] = tmp; + } + @Override + protected int compare(int i, int j) { + return Double.compare(array[i], array[j]); + } + }; + + SortedUniqueDoubleValues(DoubleValues delegate) { + super(delegate); + } + + @Override + public int setDocument(int docId) { + final int numValues = super.setDocument(docId); + if (numValues == 0) { + return numUniqueValues = 0; + } + array = ArrayUtil.grow(array, numValues); + for (int i = 0; i < numValues; ++i) { + array[i] = super.nextValue(); + } + sorter.sort(0, numValues); + numUniqueValues = 1; + for (int i = 1; i < numValues; ++i) { + if (array[i] != array[i-1]) { + array[numUniqueValues++] = array[i]; + } + } + pos = 0; + return numUniqueValues; + } + + @Override + public double nextValue() { + assert pos < numUniqueValues; + return array[pos++]; + } + + @Override + public Order getOrder() { + return Order.NUMERIC; + } + + } + + } + + } + + // No need to implement ReaderContextAware here, the delegate already takes care of updating data structures + public static class WithScript extends Bytes { + + private final BytesValues bytesValues; + + public WithScript(FieldDataSource delegate, SearchScript script) { + this.bytesValues = new BytesValues(delegate, script); + } + + @Override + public BytesValues bytesValues() { + return bytesValues; + } + + static class BytesValues extends org.elasticsearch.index.fielddata.BytesValues { + + private final FieldDataSource source; + private final SearchScript script; + private final BytesRef scratch; + + public BytesValues(FieldDataSource source, SearchScript script) { + super(true); + this.source = source; + this.script = script; + scratch = new BytesRef(); + } + + @Override + public int setDocument(int docId) { + return source.bytesValues().setDocument(docId); + } + + @Override + public BytesRef nextValue() { + BytesRef value = source.bytesValues().nextValue(); + script.setNextVar("_value", value.utf8ToString()); + scratch.copyChars(script.run().toString()); + return scratch; + } + } + } + + public static class GeoPoint extends FieldDataSource implements ReaderContextAware { + + protected boolean needsHashes; + protected final IndexGeoPointFieldData indexFieldData; + protected AtomicGeoPointFieldData atomicFieldData; + private BytesValues bytesValues; + private GeoPointValues geoPointValues; + + public GeoPoint(IndexGeoPointFieldData indexFieldData) { + this.indexFieldData = indexFieldData; + needsHashes = false; + } + + @Override + public Uniqueness getUniqueness() { + return Uniqueness.UNIQUE; + } + + @Override + public final void setNeedsHashes(boolean needsHashes) { + this.needsHashes = needsHashes; + } + + @Override + public void setNextReader(AtomicReaderContext reader) { + atomicFieldData = indexFieldData.load(reader); + if (bytesValues != null) { + bytesValues = atomicFieldData.getBytesValues(needsHashes); + } + if (geoPointValues != null) { + geoPointValues = atomicFieldData.getGeoPointValues(); + } + } + + @Override + public org.elasticsearch.index.fielddata.BytesValues bytesValues() { + if (bytesValues == null) { + bytesValues = atomicFieldData.getBytesValues(needsHashes); + } + return bytesValues; + } + + public org.elasticsearch.index.fielddata.GeoPointValues geoPointValues() { + if (geoPointValues == null) { + geoPointValues = atomicFieldData.getGeoPointValues(); + } + return geoPointValues; + } + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/support/ScriptValueType.java b/src/main/java/org/elasticsearch/search/aggregations/support/ScriptValueType.java new file mode 100644 index 00000000000..3cfd7a29461 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/support/ScriptValueType.java @@ -0,0 +1,32 @@ +package org.elasticsearch.search.aggregations.support; + +import org.elasticsearch.search.aggregations.support.bytes.BytesValuesSource; +import org.elasticsearch.search.aggregations.support.numeric.NumericValuesSource; + +/** + * + */ +public enum ScriptValueType { + + STRING(BytesValuesSource.class), + LONG(NumericValuesSource.class), + DOUBLE(NumericValuesSource.class); + + final Class valuesSourceType; + + private ScriptValueType(Class valuesSourceType) { + this.valuesSourceType = valuesSourceType; + } + + public Class getValuesSourceType() { + return valuesSourceType; + } + + public boolean isNumeric() { + return this != STRING; + } + + public boolean isFloatingPoint() { + return this == DOUBLE; + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/support/ScriptValues.java b/src/main/java/org/elasticsearch/search/aggregations/support/ScriptValues.java new file mode 100644 index 00000000000..624a00c5f4e --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/support/ScriptValues.java @@ -0,0 +1,31 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.support; + +import org.elasticsearch.script.SearchScript; + +/** + * + */ +public interface ScriptValues { + + SearchScript script(); + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/support/ValueSourceAggregatorFactory.java b/src/main/java/org/elasticsearch/search/aggregations/support/ValueSourceAggregatorFactory.java new file mode 100644 index 00000000000..59acff07288 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/support/ValueSourceAggregatorFactory.java @@ -0,0 +1,88 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.support; + +import org.elasticsearch.search.aggregations.*; + +/** + * + */ +public abstract class ValueSourceAggregatorFactory extends AggregatorFactory implements ValuesSourceBased { + + public static abstract class LeafOnly extends ValueSourceAggregatorFactory { + + protected LeafOnly(String name, String type, ValuesSourceConfig valuesSourceConfig) { + super(name, type, valuesSourceConfig); + } + + @Override + public AggregatorFactory subFactories(AggregatorFactories subFactories) { + throw new AggregationInitializationException("Aggregator [" + name + "] of type [" + type + "] cannot accept sub-aggregations"); + } + } + + protected ValuesSourceConfig valuesSourceConfig; + + protected ValueSourceAggregatorFactory(String name, String type, ValuesSourceConfig valuesSourceConfig) { + super(name, type); + this.valuesSourceConfig = valuesSourceConfig; + } + + @Override + public ValuesSourceConfig valuesSourceConfig() { + return valuesSourceConfig; + } + + @Override + public Aggregator create(AggregationContext context, Aggregator parent, long expectedBucketsCount) { + if (valuesSourceConfig.unmapped()) { + return createUnmapped(context, parent); + } + VS vs = context.valuesSource(valuesSourceConfig, parent == null ? 0 : 1 + parent.depth()); + return create(vs, expectedBucketsCount, context, parent); + } + + @Override + public void doValidate() { + if (valuesSourceConfig == null || !valuesSourceConfig.valid()) { + valuesSourceConfig = resolveValuesSourceConfigFromAncestors(name, parent, valuesSourceConfig.valueSourceType()); + } + } + + protected abstract Aggregator createUnmapped(AggregationContext aggregationContext, Aggregator parent); + + protected abstract Aggregator create(VS valuesSource, long expectedBucketsCount, AggregationContext aggregationContext, Aggregator parent); + + private static ValuesSourceConfig resolveValuesSourceConfigFromAncestors(String aggName, AggregatorFactory parent, Class requiredValuesSourceType) { + ValuesSourceConfig config; + while (parent != null) { + if (parent instanceof ValuesSourceBased) { + config = ((ValuesSourceBased) parent).valuesSourceConfig(); + if (config != null && config.valid()) { + if (requiredValuesSourceType == null || requiredValuesSourceType.isAssignableFrom(config.valueSourceType())) { + return (ValuesSourceConfig) config; + } + } + } + parent = parent.parent(); + } + throw new AggregationExecutionException("could not find the appropriate value context to perform aggregation [" + aggName + "]"); + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/support/ValuesSource.java b/src/main/java/org/elasticsearch/search/aggregations/support/ValuesSource.java new file mode 100644 index 00000000000..db188dbe364 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/support/ValuesSource.java @@ -0,0 +1,34 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.support; + +import org.elasticsearch.index.fielddata.BytesValues; + +/** + * An abstraction of a source from which values are resolved per document. + */ +public interface ValuesSource { + + /** + * @return A {@link org.apache.lucene.util.BytesRef bytesref} view over the values that are resolved from this value source. + */ + BytesValues bytesValues(); + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/support/ValuesSourceBased.java b/src/main/java/org/elasticsearch/search/aggregations/support/ValuesSourceBased.java new file mode 100644 index 00000000000..75505488a98 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/support/ValuesSourceBased.java @@ -0,0 +1,29 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.support; + +/** + * + */ +public interface ValuesSourceBased { + + ValuesSourceConfig valuesSourceConfig(); + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/support/ValuesSourceConfig.java b/src/main/java/org/elasticsearch/search/aggregations/support/ValuesSourceConfig.java new file mode 100644 index 00000000000..a29537d29e6 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/support/ValuesSourceConfig.java @@ -0,0 +1,118 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.support; + +import org.elasticsearch.script.SearchScript; +import org.elasticsearch.search.aggregations.support.numeric.ValueFormatter; +import org.elasticsearch.search.aggregations.support.numeric.ValueParser; + +/** + * + */ +public class ValuesSourceConfig { + + final Class valueSourceType; + FieldContext fieldContext; + SearchScript script; + ValueFormatter formatter; + ValueParser parser; + ScriptValueType scriptValueType; + boolean unmapped = false; + boolean needsHashes = false; + boolean ensureUnique = false; + boolean ensureSorted = false; + + public ValuesSourceConfig(Class valueSourceType) { + this.valueSourceType = valueSourceType; + } + + public Class valueSourceType() { + return valueSourceType; + } + + public FieldContext fieldContext() { + return fieldContext; + } + + public boolean unmapped() { + return unmapped; + } + + public boolean valid() { + return fieldContext != null || script != null || unmapped; + } + + public ValuesSourceConfig fieldContext(FieldContext fieldContext) { + this.fieldContext = fieldContext; + return this; + } + + public ValuesSourceConfig script(SearchScript script) { + this.script = script; + return this; + } + + public ValuesSourceConfig formatter(ValueFormatter formatter) { + this.formatter = formatter; + return this; + } + + public ValueFormatter formatter() { + return formatter; + } + + public ValuesSourceConfig parser(ValueParser parser) { + this.parser = parser; + return this; + } + + public ValueParser parser() { + return parser; + } + + public ValuesSourceConfig scriptValueType(ScriptValueType scriptValueType) { + this.scriptValueType = scriptValueType; + return this; + } + + public ScriptValueType scriptValueType() { + return scriptValueType; + } + + public ValuesSourceConfig unmapped(boolean unmapped) { + this.unmapped = unmapped; + return this; + } + + public ValuesSourceConfig needsHashes(boolean needsHashes) { + this.needsHashes = needsHashes; + return this; + } + + public ValuesSourceConfig ensureUnique(boolean unique) { + this.ensureUnique = unique; + return this; + } + + public ValuesSourceConfig ensureSorted(boolean sorted) { + this.ensureSorted = sorted; + return this; + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/support/bytes/BytesValuesSource.java b/src/main/java/org/elasticsearch/search/aggregations/support/bytes/BytesValuesSource.java new file mode 100644 index 00000000000..ca159298019 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/support/bytes/BytesValuesSource.java @@ -0,0 +1,42 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.support.bytes; + +import org.elasticsearch.index.fielddata.BytesValues; +import org.elasticsearch.search.aggregations.support.FieldDataSource; +import org.elasticsearch.search.aggregations.support.ValuesSource; + +/** + * + */ +public final class BytesValuesSource implements ValuesSource { + + private final FieldDataSource source; + + public BytesValuesSource(FieldDataSource source) { + this.source = source; + } + + @Override + public BytesValues bytesValues() { + return source.bytesValues(); + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/support/bytes/ScriptBytesValues.java b/src/main/java/org/elasticsearch/search/aggregations/support/bytes/ScriptBytesValues.java new file mode 100644 index 00000000000..8ad640a9e6e --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/support/bytes/ScriptBytesValues.java @@ -0,0 +1,107 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.support.bytes; + +import com.google.common.collect.Iterators; +import org.apache.lucene.util.BytesRef; +import org.elasticsearch.index.fielddata.BytesValues; +import org.elasticsearch.script.SearchScript; +import org.elasticsearch.search.aggregations.support.ScriptValues; + +import java.lang.reflect.Array; +import java.util.Collection; +import java.util.Iterator; + +/** + * + */ +public class ScriptBytesValues extends BytesValues implements ScriptValues { + + final SearchScript script; + + private Iterator iter; + private Object value; + private BytesRef scratch = new BytesRef(); + + public ScriptBytesValues(SearchScript script) { + super(true); // assume multi-valued + this.script = script; + } + + @Override + public SearchScript script() { + return script; + } + + @Override + public int setDocument(int docId) { + this.docId = docId; + script.setNextDocId(docId); + value = script.run(); + + if (value == null) { + iter = Iterators.emptyIterator(); + return 0; + } + + if (value.getClass().isArray()) { + final int length = Array.getLength(value); + // don't use Arrays.asList because the array may be an array of primitives? + iter = new Iterator() { + + int i = 0; + + @Override + public boolean hasNext() { + return i < length; + } + + @Override + public Object next() { + return Array.get(value, i++); + } + + @Override + public void remove() { + throw new UnsupportedOperationException(); + } + + }; + return length; + } + + if (value instanceof Collection) { + final Collection coll = (Collection) value; + iter = coll.iterator(); + return coll.size(); + } + + iter = Iterators.singletonIterator(value); + return 1; + } + + @Override + public BytesRef nextValue() { + final String next = iter.next().toString(); + scratch.copyChars(next); + return scratch; + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/support/geopoints/GeoPointValuesSource.java b/src/main/java/org/elasticsearch/search/aggregations/support/geopoints/GeoPointValuesSource.java new file mode 100644 index 00000000000..2134897df46 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/support/geopoints/GeoPointValuesSource.java @@ -0,0 +1,47 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.support.geopoints; + +import org.elasticsearch.index.fielddata.BytesValues; +import org.elasticsearch.index.fielddata.GeoPointValues; +import org.elasticsearch.search.aggregations.support.FieldDataSource; +import org.elasticsearch.search.aggregations.support.ValuesSource; + +/** + * A source of geo points. + */ +public final class GeoPointValuesSource implements ValuesSource { + + private final FieldDataSource.GeoPoint source; + + public GeoPointValuesSource(FieldDataSource.GeoPoint source) { + this.source = source; + } + + @Override + public BytesValues bytesValues() { + return source.bytesValues(); + } + + public final GeoPointValues values() { + return source.geoPointValues(); + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/support/numeric/NumericValuesSource.java b/src/main/java/org/elasticsearch/search/aggregations/support/numeric/NumericValuesSource.java new file mode 100644 index 00000000000..e01bf478c00 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/support/numeric/NumericValuesSource.java @@ -0,0 +1,69 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.support.numeric; + +import org.elasticsearch.common.Nullable; +import org.elasticsearch.index.fielddata.BytesValues; +import org.elasticsearch.index.fielddata.DoubleValues; +import org.elasticsearch.index.fielddata.LongValues; +import org.elasticsearch.search.aggregations.support.FieldDataSource; +import org.elasticsearch.search.aggregations.support.ValuesSource; + +/** + * A source of numeric data. + */ +public final class NumericValuesSource implements ValuesSource { + + private final FieldDataSource.Numeric source; + private final ValueFormatter formatter; + private final ValueParser parser; + + public NumericValuesSource(FieldDataSource.Numeric source, @Nullable ValueFormatter formatter, @Nullable ValueParser parser) { + this.source = source; + this.formatter = formatter; + this.parser = parser; + } + + @Override + public BytesValues bytesValues() { + return source.bytesValues(); + } + + public boolean isFloatingPoint() { + return source.isFloatingPoint(); + } + + public LongValues longValues() { + return source.longValues(); + } + + public DoubleValues doubleValues() { + return source.doubleValues(); + } + + public ValueFormatter formatter() { + return formatter; + } + + public ValueParser parser() { + return parser; + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/support/numeric/ScriptDoubleValues.java b/src/main/java/org/elasticsearch/search/aggregations/support/numeric/ScriptDoubleValues.java new file mode 100644 index 00000000000..4c1b26eba4f --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/support/numeric/ScriptDoubleValues.java @@ -0,0 +1,100 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.support.numeric; + +import org.apache.lucene.util.ArrayUtil; +import org.elasticsearch.index.fielddata.DoubleValues; +import org.elasticsearch.script.SearchScript; +import org.elasticsearch.search.aggregations.AggregationExecutionException; +import org.elasticsearch.search.aggregations.support.ScriptValues; + +import java.lang.reflect.Array; +import java.util.Collection; +import java.util.Iterator; + +/** + * {@link DoubleValues} implementation which is based on a script + */ +public class ScriptDoubleValues extends DoubleValues implements ScriptValues { + + final SearchScript script; + + private Object value; + private double[] values = new double[4]; + private int valueCount; + private int valueOffset; + + + public ScriptDoubleValues(SearchScript script) { + super(true); // assume multi-valued + this.script = script; + } + + @Override + public SearchScript script() { + return script; + } + + @Override + public int setDocument(int docId) { + this.docId = docId; + script.setNextDocId(docId); + value = script.run(); + + if (value == null) { + valueCount = 0; + } + + else if (value instanceof Number) { + valueCount = 1; + values[0] = ((Number) value).doubleValue(); + } + + else if (value.getClass().isArray()) { + valueCount = Array.getLength(value); + values = ArrayUtil.grow(values, valueCount); + for (int i = 0; i < valueCount; ++i) { + values[i] = ((Number) Array.get(value, i)).doubleValue(); + } + } + + else if (value instanceof Collection) { + valueCount = ((Collection) value).size(); + int i = 0; + for (Iterator it = ((Collection) value).iterator(); it.hasNext(); ++i) { + values[i] = ((Number) it.next()).doubleValue(); + } + assert i == valueCount; + } + + else { + throw new AggregationExecutionException("Unsupported script value [" + value + "]"); + } + + valueOffset = 0; + return valueCount; + } + + @Override + public double nextValue() { + assert valueOffset < valueCount; + return values[valueOffset++]; + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/support/numeric/ScriptLongValues.java b/src/main/java/org/elasticsearch/search/aggregations/support/numeric/ScriptLongValues.java new file mode 100644 index 00000000000..39bdbb0ef4a --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/support/numeric/ScriptLongValues.java @@ -0,0 +1,100 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.support.numeric; + +import org.apache.lucene.util.ArrayUtil; +import org.elasticsearch.index.fielddata.LongValues; +import org.elasticsearch.script.SearchScript; +import org.elasticsearch.search.aggregations.AggregationExecutionException; +import org.elasticsearch.search.aggregations.support.ScriptValues; + +import java.lang.reflect.Array; +import java.util.Collection; +import java.util.Iterator; + +/** + * {@link LongValues} implementation which is based on a script + */ +public class ScriptLongValues extends LongValues implements ScriptValues { + + final SearchScript script; + + private Object value; + private long[] values = new long[4]; + private int valueCount; + private int valueOffset; + + public ScriptLongValues(SearchScript script) { + super(true); // assume multi-valued + this.script = script; + } + + @Override + public SearchScript script() { + return script; + } + + @Override + public int setDocument(int docId) { + this.docId = docId; + script.setNextDocId(docId); + value = script.run(); + + if (value == null) { + valueCount = 0; + } + + else if (value instanceof Number) { + valueCount = 1; + values[0] = ((Number) value).longValue(); + } + + else if (value.getClass().isArray()) { + valueCount = Array.getLength(value); + values = ArrayUtil.grow(values, valueCount); + for (int i = 0; i < valueCount; ++i) { + values[i] = ((Number) Array.get(value, i++)).longValue(); + } + } + + else if (value instanceof Collection) { + valueCount = ((Collection) value).size(); + int i = 0; + for (Iterator it = ((Collection) value).iterator(); it.hasNext(); ++i) { + values[i] = ((Number) it.next()).longValue(); + } + assert i == valueCount; + } + + else { + throw new AggregationExecutionException("Unsupported script value [" + value + "]"); + } + + valueOffset = 0; + return valueCount; + } + + @Override + public long nextValue() { + assert valueOffset < valueCount; + return values[valueOffset++]; + } + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/support/numeric/ValueFormatter.java b/src/main/java/org/elasticsearch/search/aggregations/support/numeric/ValueFormatter.java new file mode 100644 index 00000000000..b44f8aa9a62 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/support/numeric/ValueFormatter.java @@ -0,0 +1,225 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.support.numeric; + +import org.elasticsearch.common.io.stream.StreamInput; +import org.elasticsearch.common.io.stream.StreamOutput; +import org.elasticsearch.common.io.stream.Streamable; +import org.elasticsearch.common.joda.FormatDateTimeFormatter; +import org.elasticsearch.common.joda.Joda; +import org.elasticsearch.index.mapper.core.DateFieldMapper; +import org.elasticsearch.index.mapper.ip.IpFieldMapper; + +import java.io.IOException; +import java.text.DecimalFormat; +import java.text.DecimalFormatSymbols; +import java.text.NumberFormat; +import java.util.Locale; + +/** + * A strategy for formatting time represented as millis long value to string + */ +public interface ValueFormatter extends Streamable { + + public final static ValueFormatter RAW = new Raw(); + public final static ValueFormatter IPv4 = new IPv4Formatter(); + + /** + * Uniquely identifies this formatter (used for efficient serialization) + * + * @return The id of this formatter + */ + byte id(); + + /** + * Formats the given millis time value (since the epoch) to string. + * + * @param value The long value to format. + * @return The formatted value as string. + */ + String format(long value); + + /** + * The + * @param value double The double value to format. + * @return The formatted value as string + */ + String format(double value); + + + static class Raw implements ValueFormatter { + + static final byte ID = 1; + + @Override + public String format(long value) { + return String.valueOf(value); + } + + @Override + public String format(double value) { + return String.valueOf(value); + } + + @Override + public byte id() { + return ID; + } + + @Override + public void readFrom(StreamInput in) throws IOException { + } + + @Override + public void writeTo(StreamOutput out) throws IOException { + } + } + + /** + * A time formatter which is based on date/time format. + */ + public static class DateTime implements ValueFormatter { + + public static final ValueFormatter DEFAULT = new ValueFormatter.DateTime(DateFieldMapper.Defaults.DATE_TIME_FORMATTER); + + static final byte ID = 2; + + FormatDateTimeFormatter formatter; + + DateTime() {} // for serialization + + public DateTime(String format) { + this.formatter = Joda.forPattern(format); + } + + public DateTime(FormatDateTimeFormatter formatter) { + this.formatter = formatter; + } + + @Override + public String format(long time) { + return formatter.printer().print(time); + } + + public String format(double value) { + return format((long) value); + } + + @Override + public byte id() { + return ID; + } + + @Override + public void readFrom(StreamInput in) throws IOException { + formatter = Joda.forPattern(in.readString()); + } + + @Override + public void writeTo(StreamOutput out) throws IOException { + out.writeString(formatter.format()); + } + } + + public static abstract class Number implements ValueFormatter { + + NumberFormat format; + + Number() {} // for serialization + + Number(NumberFormat format) { + this.format = format; + } + + @Override + public String format(long value) { + return format.format(value); + } + + @Override + public String format(double value) { + return format.format(value); + } + + public static class Pattern extends Number { + + private static final DecimalFormatSymbols SYMBOLS = new DecimalFormatSymbols(Locale.ROOT); + + static final byte ID = 4; + + String pattern; + + Pattern() {} // for serialization + + public Pattern(String pattern) { + super(new DecimalFormat(pattern, SYMBOLS)); + this.pattern = pattern; + } + + @Override + public byte id() { + return ID; + } + + @Override + public void readFrom(StreamInput in) throws IOException { + pattern = in.readString(); + format = new DecimalFormat(pattern, SYMBOLS); + } + + @Override + public void writeTo(StreamOutput out) throws IOException { + out.writeString(pattern); + } + } + } + + static class IPv4Formatter implements ValueFormatter { + + static final byte ID = 6; + + @Override + public byte id() { + return ID; + } + + @Override + public String format(long value) { + return IpFieldMapper.longToIp(value); + } + + @Override + public String format(double value) { + return format((long) value); + } + + @Override + public void readFrom(StreamInput in) throws IOException { + } + + @Override + public void writeTo(StreamOutput out) throws IOException { + } + } + + + + +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/support/numeric/ValueFormatterStreams.java b/src/main/java/org/elasticsearch/search/aggregations/support/numeric/ValueFormatterStreams.java new file mode 100644 index 00000000000..c31b573ae19 --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/support/numeric/ValueFormatterStreams.java @@ -0,0 +1,65 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.support.numeric; + +import org.elasticsearch.ElasticSearchIllegalArgumentException; +import org.elasticsearch.common.io.stream.StreamInput; +import org.elasticsearch.common.io.stream.StreamOutput; + +import java.io.IOException; + +/** + * + */ +public class ValueFormatterStreams { + + public static ValueFormatter read(StreamInput in) throws IOException { + byte id = in.readByte(); + ValueFormatter formatter = null; + switch (id) { + case ValueFormatter.Raw.ID: return ValueFormatter.RAW; + case ValueFormatter.IPv4Formatter.ID: return ValueFormatter.IPv4; + case ValueFormatter.DateTime.ID: formatter = new ValueFormatter.DateTime(); break; + case ValueFormatter.Number.Pattern.ID: formatter = new ValueFormatter.Number.Pattern(); break; + default: throw new ElasticSearchIllegalArgumentException("Unknown value formatter with id [" + id + "]"); + } + formatter.readFrom(in); + return formatter; + } + + public static ValueFormatter readOptional(StreamInput in) throws IOException { + if (!in.readBoolean()) { + return null; + } + return read(in); + } + + public static void write(ValueFormatter formatter, StreamOutput out) throws IOException { + out.writeByte(formatter.id()); + formatter.writeTo(out); + } + + public static void writeOptional(ValueFormatter formatter, StreamOutput out) throws IOException { + out.writeBoolean(formatter != null); + if (formatter != null) { + write(formatter, out); + } + } +} diff --git a/src/main/java/org/elasticsearch/search/aggregations/support/numeric/ValueParser.java b/src/main/java/org/elasticsearch/search/aggregations/support/numeric/ValueParser.java new file mode 100644 index 00000000000..04bce46c37b --- /dev/null +++ b/src/main/java/org/elasticsearch/search/aggregations/support/numeric/ValueParser.java @@ -0,0 +1,115 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.support.numeric; + +import org.elasticsearch.common.joda.DateMathParser; +import org.elasticsearch.common.joda.FormatDateTimeFormatter; +import org.elasticsearch.common.joda.Joda; +import org.elasticsearch.index.mapper.core.DateFieldMapper; +import org.elasticsearch.index.mapper.ip.IpFieldMapper; +import org.elasticsearch.search.internal.SearchContext; + +import java.util.concurrent.TimeUnit; + +/** + * + */ +public interface ValueParser { + + static final ValueParser IPv4 = new ValueParser.IPv4(); + + long parseLong(String value, SearchContext searchContext); + + double parseDouble(String value, SearchContext searchContext); + + + /** + * Knows how to parse datatime values based on date/time format + */ + static class DateTime implements ValueParser { + + private FormatDateTimeFormatter formatter; + + public DateTime(String format) { + this(Joda.forPattern(format)); + } + + public DateTime(FormatDateTimeFormatter formatter) { + this.formatter = formatter; + } + + @Override + public long parseLong(String value, SearchContext searchContext) { + return formatter.parser().parseMillis(value); + } + + @Override + public double parseDouble(String value, SearchContext searchContext) { + return parseLong(value, searchContext); + } + } + + /** + * Knows how to parse datatime values based on elasticsearch's date math expression + */ + static class DateMath implements ValueParser { + + public static final DateMath DEFAULT = new ValueParser.DateMath(new DateMathParser(DateFieldMapper.Defaults.DATE_TIME_FORMATTER, DateFieldMapper.Defaults.TIME_UNIT)); + + private DateMathParser parser; + + public DateMath(String format, TimeUnit timeUnit) { + this(new DateMathParser(Joda.forPattern(format), timeUnit)); + } + + public DateMath(DateMathParser parser) { + this.parser = parser; + } + + @Override + public long parseLong(String value, SearchContext searchContext) { + return parser.parse(value, searchContext.nowInMillis()); + } + + @Override + public double parseDouble(String value, SearchContext searchContext) { + return parseLong(value, searchContext); + } + } + + /** + * Knows how to parse IPv4 formats + */ + static class IPv4 implements ValueParser { + + private IPv4() { + } + + @Override + public long parseLong(String value, SearchContext searchContext) { + return IpFieldMapper.ipToLong(value); + } + + @Override + public double parseDouble(String value, SearchContext searchContext) { + return parseLong(value, searchContext); + } + } +} diff --git a/src/main/java/org/elasticsearch/search/builder/SearchSourceBuilder.java b/src/main/java/org/elasticsearch/search/builder/SearchSourceBuilder.java index e95060db058..5967e59f523 100644 --- a/src/main/java/org/elasticsearch/search/builder/SearchSourceBuilder.java +++ b/src/main/java/org/elasticsearch/search/builder/SearchSourceBuilder.java @@ -36,6 +36,7 @@ import org.elasticsearch.common.xcontent.XContentFactory; import org.elasticsearch.common.xcontent.XContentType; import org.elasticsearch.index.query.FilterBuilder; import org.elasticsearch.index.query.QueryBuilder; +import org.elasticsearch.search.aggregations.AbstractAggregationBuilder; import org.elasticsearch.search.facet.FacetBuilder; import org.elasticsearch.search.fetch.source.FetchSourceContext; import org.elasticsearch.search.highlight.HighlightBuilder; @@ -102,9 +103,12 @@ public class SearchSourceBuilder implements ToXContent { private FetchSourceContext fetchSourceContext; private List facets; - private BytesReference facetsBinary; + private List aggregations; + private BytesReference aggregationsBinary; + + private HighlightBuilder highlightBuilder; private SuggestBuilder suggestBuilder; @@ -393,6 +397,59 @@ public class SearchSourceBuilder implements ToXContent { } } + /** + * Add an get to perform as part of the search. + */ + public SearchSourceBuilder aggregation(AbstractAggregationBuilder aggregation) { + if (aggregations == null) { + aggregations = Lists.newArrayList(); + } + aggregations.add(aggregation); + return this; + } + + /** + * Sets a raw (xcontent / json) addAggregation. + */ + public SearchSourceBuilder aggregations(byte[] aggregationsBinary) { + return aggregations(aggregationsBinary, 0, aggregationsBinary.length); + } + + /** + * Sets a raw (xcontent / json) addAggregation. + */ + public SearchSourceBuilder aggregations(byte[] aggregationsBinary, int aggregationsBinaryOffset, int aggregationsBinaryLength) { + return aggregations(new BytesArray(aggregationsBinary, aggregationsBinaryOffset, aggregationsBinaryLength)); + } + + /** + * Sets a raw (xcontent / json) addAggregation. + */ + public SearchSourceBuilder aggregations(BytesReference aggregationsBinary) { + this.aggregationsBinary = aggregationsBinary; + return this; + } + + /** + * Sets a raw (xcontent / json) addAggregation. + */ + public SearchSourceBuilder aggregations(XContentBuilder facets) { + return aggregations(facets.bytes()); + } + + /** + * Sets a raw (xcontent / json) addAggregation. + */ + public SearchSourceBuilder aggregations(Map aggregations) { + try { + XContentBuilder builder = XContentFactory.contentBuilder(Requests.CONTENT_TYPE); + builder.map(aggregations); + return aggregations(builder); + } catch (IOException e) { + throw new ElasticSearchGenerationException("Failed to generate [" + aggregations + "]", e); + } + } + public HighlightBuilder highlighter() { if (highlightBuilder == null) { highlightBuilder = new HighlightBuilder(); @@ -789,6 +846,23 @@ public class SearchSourceBuilder implements ToXContent { } } + if (aggregations != null) { + builder.field("aggregations"); + builder.startObject(); + for (AbstractAggregationBuilder aggregation : aggregations) { + aggregation.toXContent(builder, params); + } + builder.endObject(); + } + + if (aggregationsBinary != null) { + if (XContentFactory.xContentType(aggregationsBinary) == builder.contentType()) { + builder.rawField("aggregations", aggregationsBinary); + } else { + builder.field("aggregations_binary", aggregationsBinary); + } + } + if (highlightBuilder != null) { highlightBuilder.toXContent(builder, params); } diff --git a/src/main/java/org/elasticsearch/search/controller/SearchPhaseController.java b/src/main/java/org/elasticsearch/search/controller/SearchPhaseController.java index 249fb97e071..c3544f4932c 100644 --- a/src/main/java/org/elasticsearch/search/controller/SearchPhaseController.java +++ b/src/main/java/org/elasticsearch/search/controller/SearchPhaseController.java @@ -31,6 +31,7 @@ import org.elasticsearch.common.hppc.HppcMaps; import org.elasticsearch.common.inject.Inject; import org.elasticsearch.common.settings.Settings; import org.elasticsearch.common.util.concurrent.AtomicArray; +import org.elasticsearch.search.aggregations.InternalAggregations; import org.elasticsearch.search.dfs.AggregatedDfs; import org.elasticsearch.search.dfs.DfsSearchResult; import org.elasticsearch.search.facet.Facet; @@ -422,8 +423,21 @@ public class SearchPhaseController extends AbstractComponent { suggest = hasSuggestions ? new Suggest(Suggest.Fields.SUGGEST, Suggest.reduce(groupedSuggestions)) : null; } + // merge addAggregation + InternalAggregations aggregations = null; + if (!queryResults.isEmpty()) { + if (firstResult.aggregations() != null && firstResult.aggregations().asList() != null) { + List aggregationsList = new ArrayList(queryResults.size()); + for (AtomicArray.Entry entry : queryResults) { + aggregationsList.add((InternalAggregations) entry.value.queryResult().aggregations()); + } + aggregations = InternalAggregations.reduce(aggregationsList, cacheRecycler); + } + } + InternalSearchHits searchHits = new InternalSearchHits(hits.toArray(new InternalSearchHit[hits.size()]), totalHits, maxScore); - return new InternalSearchResponse(searchHits, facets, suggest, timedOut); + + return new InternalSearchResponse(searchHits, facets, aggregations, suggest, timedOut); } } diff --git a/src/main/java/org/elasticsearch/search/facet/datehistogram/CountDateHistogramFacetExecutor.java b/src/main/java/org/elasticsearch/search/facet/datehistogram/CountDateHistogramFacetExecutor.java index 9d60bb4970f..2ae0b2c2d64 100644 --- a/src/main/java/org/elasticsearch/search/facet/datehistogram/CountDateHistogramFacetExecutor.java +++ b/src/main/java/org/elasticsearch/search/facet/datehistogram/CountDateHistogramFacetExecutor.java @@ -22,8 +22,8 @@ package org.elasticsearch.search.facet.datehistogram; import com.carrotsearch.hppc.LongLongOpenHashMap; import org.apache.lucene.index.AtomicReaderContext; import org.elasticsearch.cache.recycler.CacheRecycler; -import org.elasticsearch.common.joda.TimeZoneRounding; import org.elasticsearch.common.recycler.Recycler; +import org.elasticsearch.common.rounding.TimeZoneRounding; import org.elasticsearch.index.fielddata.IndexNumericFieldData; import org.elasticsearch.index.fielddata.LongValues; import org.elasticsearch.search.facet.FacetExecutor; @@ -110,7 +110,7 @@ public class CountDateHistogramFacetExecutor extends FacetExecutor { @Override public void onValue(int docId, long value) { - counts.addTo(tzRounding.calc(value), 1); + counts.addTo(tzRounding.round(value), 1); } public LongLongOpenHashMap counts() { diff --git a/src/main/java/org/elasticsearch/search/facet/datehistogram/DateHistogramFacetParser.java b/src/main/java/org/elasticsearch/search/facet/datehistogram/DateHistogramFacetParser.java index 3b874191608..c8c38a7f4c4 100644 --- a/src/main/java/org/elasticsearch/search/facet/datehistogram/DateHistogramFacetParser.java +++ b/src/main/java/org/elasticsearch/search/facet/datehistogram/DateHistogramFacetParser.java @@ -23,8 +23,8 @@ import com.google.common.collect.ImmutableMap; import org.elasticsearch.common.collect.MapBuilder; import org.elasticsearch.common.component.AbstractComponent; import org.elasticsearch.common.inject.Inject; -import org.elasticsearch.common.joda.Joda; -import org.elasticsearch.common.joda.TimeZoneRounding; +import org.elasticsearch.common.rounding.DateTimeUnit; +import org.elasticsearch.common.rounding.TimeZoneRounding; import org.elasticsearch.common.settings.Settings; import org.elasticsearch.common.unit.TimeValue; import org.elasticsearch.common.xcontent.XContentParser; @@ -36,7 +36,6 @@ import org.elasticsearch.search.facet.FacetParser; import org.elasticsearch.search.facet.FacetPhaseExecutionException; import org.elasticsearch.search.internal.SearchContext; import org.joda.time.Chronology; -import org.joda.time.DateTimeField; import org.joda.time.DateTimeZone; import org.joda.time.chrono.ISOChronology; @@ -48,29 +47,30 @@ import java.util.Map; */ public class DateHistogramFacetParser extends AbstractComponent implements FacetParser { - private final ImmutableMap dateFieldParsers; + private final ImmutableMap dateTimeUnits; @Inject public DateHistogramFacetParser(Settings settings) { super(settings); InternalDateHistogramFacet.registerStreams(); - dateFieldParsers = MapBuilder.newMapBuilder() - .put("year", new DateFieldParser.YearOfCentury()) - .put("1y", new DateFieldParser.YearOfCentury()) - .put("quarter", new DateFieldParser.Quarter()) - .put("month", new DateFieldParser.MonthOfYear()) - .put("1m", new DateFieldParser.MonthOfYear()) - .put("week", new DateFieldParser.WeekOfWeekyear()) - .put("1w", new DateFieldParser.WeekOfWeekyear()) - .put("day", new DateFieldParser.DayOfMonth()) - .put("1d", new DateFieldParser.DayOfMonth()) - .put("hour", new DateFieldParser.HourOfDay()) - .put("1h", new DateFieldParser.HourOfDay()) - .put("minute", new DateFieldParser.MinuteOfHour()) - .put("1m", new DateFieldParser.MinuteOfHour()) - .put("second", new DateFieldParser.SecondOfMinute()) - .put("1s", new DateFieldParser.SecondOfMinute()) + dateTimeUnits = MapBuilder.newMapBuilder() + .put("year", DateTimeUnit.YEAR_OF_CENTURY) + .put("1y", DateTimeUnit.YEAR_OF_CENTURY) + .put("quarter", DateTimeUnit.QUARTER) + .put("1q", DateTimeUnit.QUARTER) + .put("month", DateTimeUnit.MONTH_OF_YEAR) + .put("1M", DateTimeUnit.MONTH_OF_YEAR) + .put("week", DateTimeUnit.WEEK_OF_WEEKYEAR) + .put("1w", DateTimeUnit.WEEK_OF_WEEKYEAR) + .put("day", DateTimeUnit.DAY_OF_MONTH) + .put("1d", DateTimeUnit.DAY_OF_MONTH) + .put("hour", DateTimeUnit.HOUR_OF_DAY) + .put("1h", DateTimeUnit.HOUR_OF_DAY) + .put("minute", DateTimeUnit.MINUTES_OF_HOUR) + .put("1m", DateTimeUnit.MINUTES_OF_HOUR) + .put("second", DateTimeUnit.SECOND_OF_MINUTE) + .put("1s", DateTimeUnit.SECOND_OF_MINUTE) .immutableMap(); } @@ -162,9 +162,9 @@ public class DateHistogramFacetParser extends AbstractComponent implements Facet IndexNumericFieldData keyIndexFieldData = context.fieldData().getForField(keyMapper); TimeZoneRounding.Builder tzRoundingBuilder; - DateFieldParser fieldParser = dateFieldParsers.get(interval); - if (fieldParser != null) { - tzRoundingBuilder = TimeZoneRounding.builder(fieldParser.parse(chronology)); + DateTimeUnit dateTimeUnit = dateTimeUnits.get(interval); + if (dateTimeUnit != null) { + tzRoundingBuilder = TimeZoneRounding.builder(dateTimeUnit); } else { // the interval is a time value? tzRoundingBuilder = TimeZoneRounding.builder(TimeValue.parseTimeValue(interval, null)); @@ -220,64 +220,4 @@ public class DateHistogramFacetParser extends AbstractComponent implements Facet } } - static interface DateFieldParser { - - DateTimeField parse(Chronology chronology); - - static class WeekOfWeekyear implements DateFieldParser { - @Override - public DateTimeField parse(Chronology chronology) { - return chronology.weekOfWeekyear(); - } - } - - static class YearOfCentury implements DateFieldParser { - @Override - public DateTimeField parse(Chronology chronology) { - return chronology.yearOfCentury(); - } - } - - static class Quarter implements DateFieldParser { - @Override - public DateTimeField parse(Chronology chronology) { - return Joda.QuarterOfYear.getField(chronology); - } - } - - static class MonthOfYear implements DateFieldParser { - @Override - public DateTimeField parse(Chronology chronology) { - return chronology.monthOfYear(); - } - } - - static class DayOfMonth implements DateFieldParser { - @Override - public DateTimeField parse(Chronology chronology) { - return chronology.dayOfMonth(); - } - } - - static class HourOfDay implements DateFieldParser { - @Override - public DateTimeField parse(Chronology chronology) { - return chronology.hourOfDay(); - } - } - - static class MinuteOfHour implements DateFieldParser { - @Override - public DateTimeField parse(Chronology chronology) { - return chronology.minuteOfHour(); - } - } - - static class SecondOfMinute implements DateFieldParser { - @Override - public DateTimeField parse(Chronology chronology) { - return chronology.secondOfMinute(); - } - } - } } diff --git a/src/main/java/org/elasticsearch/search/facet/datehistogram/ValueDateHistogramFacetExecutor.java b/src/main/java/org/elasticsearch/search/facet/datehistogram/ValueDateHistogramFacetExecutor.java index b580f0ea57e..fd89f61a959 100644 --- a/src/main/java/org/elasticsearch/search/facet/datehistogram/ValueDateHistogramFacetExecutor.java +++ b/src/main/java/org/elasticsearch/search/facet/datehistogram/ValueDateHistogramFacetExecutor.java @@ -22,8 +22,8 @@ package org.elasticsearch.search.facet.datehistogram; import com.carrotsearch.hppc.LongObjectOpenHashMap; import org.apache.lucene.index.AtomicReaderContext; import org.elasticsearch.cache.recycler.CacheRecycler; -import org.elasticsearch.common.joda.TimeZoneRounding; import org.elasticsearch.common.recycler.Recycler; +import org.elasticsearch.common.rounding.TimeZoneRounding; import org.elasticsearch.index.fielddata.DoubleValues; import org.elasticsearch.index.fielddata.IndexNumericFieldData; import org.elasticsearch.index.fielddata.LongValues; @@ -118,7 +118,7 @@ public class ValueDateHistogramFacetExecutor extends FacetExecutor { @Override public void onValue(int docId, long value) { - long time = tzRounding.calc(value); + long time = tzRounding.round(value); InternalFullDateHistogramFacet.FullEntry entry = entries.get(time); if (entry == null) { diff --git a/src/main/java/org/elasticsearch/search/facet/datehistogram/ValueScriptDateHistogramFacetExecutor.java b/src/main/java/org/elasticsearch/search/facet/datehistogram/ValueScriptDateHistogramFacetExecutor.java index 0a1c847d361..0ac45ba0498 100644 --- a/src/main/java/org/elasticsearch/search/facet/datehistogram/ValueScriptDateHistogramFacetExecutor.java +++ b/src/main/java/org/elasticsearch/search/facet/datehistogram/ValueScriptDateHistogramFacetExecutor.java @@ -23,8 +23,8 @@ import com.carrotsearch.hppc.LongObjectOpenHashMap; import org.apache.lucene.index.AtomicReaderContext; import org.apache.lucene.search.Scorer; import org.elasticsearch.cache.recycler.CacheRecycler; -import org.elasticsearch.common.joda.TimeZoneRounding; import org.elasticsearch.common.recycler.Recycler; +import org.elasticsearch.common.rounding.TimeZoneRounding; import org.elasticsearch.index.fielddata.IndexNumericFieldData; import org.elasticsearch.index.fielddata.LongValues; import org.elasticsearch.script.SearchScript; @@ -124,7 +124,7 @@ public class ValueScriptDateHistogramFacetExecutor extends FacetExecutor { @Override public void onValue(int docId, long value) { valueScript.setNextDocId(docId); - long time = tzRounding.calc(value); + long time = tzRounding.round(value); double scriptValue = valueScript.runAsDouble(); InternalFullDateHistogramFacet.FullEntry entry = entries.get(time); diff --git a/src/main/java/org/elasticsearch/search/facet/terms/strings/TermsStringOrdinalsFacetExecutor.java b/src/main/java/org/elasticsearch/search/facet/terms/strings/TermsStringOrdinalsFacetExecutor.java index 685ab672cb8..2d541ce8869 100644 --- a/src/main/java/org/elasticsearch/search/facet/terms/strings/TermsStringOrdinalsFacetExecutor.java +++ b/src/main/java/org/elasticsearch/search/facet/terms/strings/TermsStringOrdinalsFacetExecutor.java @@ -27,8 +27,8 @@ import org.apache.lucene.util.PriorityQueue; import org.apache.lucene.util.UnicodeUtil; import org.elasticsearch.cache.recycler.CacheRecycler; import org.elasticsearch.common.collect.BoundedTreeSet; +import org.elasticsearch.common.util.BigArrays; import org.elasticsearch.common.util.IntArray; -import org.elasticsearch.common.util.IntArrays; import org.elasticsearch.index.fielddata.BytesValues; import org.elasticsearch.index.fielddata.IndexFieldData; import org.elasticsearch.index.fielddata.ordinals.Ordinals; @@ -249,7 +249,7 @@ public class TermsStringOrdinalsFacetExecutor extends FacetExecutor { public ReaderAggregator(BytesValues.WithOrdinals values, int ordinalsCacheLimit, CacheRecycler cacheRecycler) { this.values = values; this.maxOrd = values.ordinals().getMaxOrd(); - this.counts = IntArrays.allocate(maxOrd); + this.counts = BigArrays.newIntArray(maxOrd); } final void onOrdinal(int docId, long ordinal) { diff --git a/src/main/java/org/elasticsearch/search/internal/DefaultSearchContext.java b/src/main/java/org/elasticsearch/search/internal/DefaultSearchContext.java index cea1d3f8706..5640d42bd05 100644 --- a/src/main/java/org/elasticsearch/search/internal/DefaultSearchContext.java +++ b/src/main/java/org/elasticsearch/search/internal/DefaultSearchContext.java @@ -53,6 +53,7 @@ import org.elasticsearch.index.similarity.SimilarityService; import org.elasticsearch.script.ScriptService; import org.elasticsearch.search.Scroll; import org.elasticsearch.search.SearchShardTarget; +import org.elasticsearch.search.aggregations.SearchContextAggregations; import org.elasticsearch.search.dfs.DfsSearchResult; import org.elasticsearch.search.facet.SearchContextFacets; import org.elasticsearch.search.fetch.FetchSearchResult; @@ -146,6 +147,8 @@ public class DefaultSearchContext extends SearchContext { private int docsIdsToLoadSize; + private SearchContextAggregations aggregations; + private SearchContextFacets facets; private SearchContextHighlight highlight; @@ -297,6 +300,17 @@ public class DefaultSearchContext extends SearchContext { return this; } + @Override + public SearchContextAggregations aggregations() { + return aggregations; + } + + @Override + public SearchContext aggregations(SearchContextAggregations aggregations) { + this.aggregations = aggregations; + return this; + } + public SearchContextFacets facets() { return facets; } diff --git a/src/main/java/org/elasticsearch/search/internal/InternalSearchResponse.java b/src/main/java/org/elasticsearch/search/internal/InternalSearchResponse.java index fa5698b3181..b855010b5b0 100644 --- a/src/main/java/org/elasticsearch/search/internal/InternalSearchResponse.java +++ b/src/main/java/org/elasticsearch/search/internal/InternalSearchResponse.java @@ -25,6 +25,8 @@ import org.elasticsearch.common.io.stream.Streamable; import org.elasticsearch.common.xcontent.ToXContent; import org.elasticsearch.common.xcontent.XContentBuilder; import org.elasticsearch.search.SearchHits; +import org.elasticsearch.search.aggregations.Aggregations; +import org.elasticsearch.search.aggregations.InternalAggregations; import org.elasticsearch.search.facet.Facets; import org.elasticsearch.search.facet.InternalFacets; import org.elasticsearch.search.suggest.Suggest; @@ -42,18 +44,21 @@ public class InternalSearchResponse implements Streamable, ToXContent { private InternalFacets facets; + private InternalAggregations aggregations; + private Suggest suggest; private boolean timedOut; - public static final InternalSearchResponse EMPTY = new InternalSearchResponse(new InternalSearchHits(new InternalSearchHit[0], 0, 0), null, null, false); + public static final InternalSearchResponse EMPTY = new InternalSearchResponse(new InternalSearchHits(new InternalSearchHit[0], 0, 0), null, null, null, false); private InternalSearchResponse() { } - public InternalSearchResponse(InternalSearchHits hits, InternalFacets facets, Suggest suggest, boolean timedOut) { + public InternalSearchResponse(InternalSearchHits hits, InternalFacets facets, InternalAggregations aggregations, Suggest suggest, boolean timedOut) { this.hits = hits; this.facets = facets; + this.aggregations = aggregations; this.suggest = suggest; this.timedOut = timedOut; } @@ -70,6 +75,10 @@ public class InternalSearchResponse implements Streamable, ToXContent { return facets; } + public Aggregations aggregations() { + return aggregations; + } + public Suggest suggest() { return suggest; } @@ -80,6 +89,9 @@ public class InternalSearchResponse implements Streamable, ToXContent { if (facets != null) { facets.toXContent(builder, params); } + if (aggregations != null) { + aggregations.toXContent(builder, params); + } if (suggest != null) { suggest.toXContent(builder, params); } @@ -98,6 +110,9 @@ public class InternalSearchResponse implements Streamable, ToXContent { if (in.readBoolean()) { facets = InternalFacets.readFacets(in); } + if (in.readBoolean()) { + aggregations = InternalAggregations.readAggregations(in); + } if (in.readBoolean()) { suggest = Suggest.readSuggest(Suggest.Fields.SUGGEST, in); } @@ -113,6 +128,12 @@ public class InternalSearchResponse implements Streamable, ToXContent { out.writeBoolean(true); facets.writeTo(out); } + if (aggregations == null) { + out.writeBoolean(false); + } else { + out.writeBoolean(true); + aggregations.writeTo(out); + } if (suggest == null) { out.writeBoolean(false); } else { diff --git a/src/main/java/org/elasticsearch/search/internal/SearchContext.java b/src/main/java/org/elasticsearch/search/internal/SearchContext.java index 068ea8aff63..ce48be5e1c1 100644 --- a/src/main/java/org/elasticsearch/search/internal/SearchContext.java +++ b/src/main/java/org/elasticsearch/search/internal/SearchContext.java @@ -42,6 +42,7 @@ import org.elasticsearch.index.similarity.SimilarityService; import org.elasticsearch.script.ScriptService; import org.elasticsearch.search.Scroll; import org.elasticsearch.search.SearchShardTarget; +import org.elasticsearch.search.aggregations.SearchContextAggregations; import org.elasticsearch.search.dfs.DfsSearchResult; import org.elasticsearch.search.facet.SearchContextFacets; import org.elasticsearch.search.fetch.FetchSearchResult; @@ -114,6 +115,10 @@ public abstract class SearchContext implements Releasable { public abstract SearchContext scroll(Scroll scroll); + public abstract SearchContextAggregations aggregations(); + + public abstract SearchContext aggregations(SearchContextAggregations aggregations); + public abstract SearchContextFacets facets(); public abstract SearchContext facets(SearchContextFacets facets); diff --git a/src/main/java/org/elasticsearch/search/query/QueryPhase.java b/src/main/java/org/elasticsearch/search/query/QueryPhase.java index a11fd375527..276f241ca35 100644 --- a/src/main/java/org/elasticsearch/search/query/QueryPhase.java +++ b/src/main/java/org/elasticsearch/search/query/QueryPhase.java @@ -28,6 +28,7 @@ import org.elasticsearch.common.inject.Inject; import org.elasticsearch.common.lucene.Lucene; import org.elasticsearch.search.SearchParseElement; import org.elasticsearch.search.SearchPhase; +import org.elasticsearch.search.aggregations.AggregationPhase; import org.elasticsearch.search.facet.FacetPhase; import org.elasticsearch.search.internal.ContextIndexSearcher; import org.elasticsearch.search.internal.SearchContext; @@ -44,12 +45,14 @@ import java.util.Map; public class QueryPhase implements SearchPhase { private final FacetPhase facetPhase; + private final AggregationPhase aggregationPhase; private final SuggestPhase suggestPhase; private RescorePhase rescorePhase; @Inject - public QueryPhase(FacetPhase facetPhase, SuggestPhase suggestPhase, RescorePhase rescorePhase) { + public QueryPhase(FacetPhase facetPhase, AggregationPhase aggregationPhase, SuggestPhase suggestPhase, RescorePhase rescorePhase) { this.facetPhase = facetPhase; + this.aggregationPhase = aggregationPhase; this.suggestPhase = suggestPhase; this.rescorePhase = rescorePhase; } @@ -73,6 +76,7 @@ public class QueryPhase implements SearchPhase { .put("minScore", new MinScoreParseElement()) .put("timeout", new TimeoutParseElement()) .putAll(facetPhase.parseElements()) + .putAll(aggregationPhase.parseElements()) .putAll(suggestPhase.parseElements()) .putAll(rescorePhase.parseElements()); return parseElements.build(); @@ -82,6 +86,7 @@ public class QueryPhase implements SearchPhase { public void preProcess(SearchContext context) { context.preProcess(); facetPhase.preProcess(context); + aggregationPhase.preProcess(context); } public void execute(SearchContext searchContext) throws QueryPhaseExecutionException { @@ -129,5 +134,6 @@ public class QueryPhase implements SearchPhase { } suggestPhase.execute(searchContext); facetPhase.execute(searchContext); + aggregationPhase.execute(searchContext); } } diff --git a/src/main/java/org/elasticsearch/search/query/QuerySearchResult.java b/src/main/java/org/elasticsearch/search/query/QuerySearchResult.java index 7e0b3c27ee1..8f651d8142a 100644 --- a/src/main/java/org/elasticsearch/search/query/QuerySearchResult.java +++ b/src/main/java/org/elasticsearch/search/query/QuerySearchResult.java @@ -23,6 +23,8 @@ import org.apache.lucene.search.TopDocs; import org.elasticsearch.common.io.stream.StreamInput; import org.elasticsearch.common.io.stream.StreamOutput; import org.elasticsearch.search.SearchShardTarget; +import org.elasticsearch.search.aggregations.Aggregations; +import org.elasticsearch.search.aggregations.InternalAggregations; import org.elasticsearch.search.facet.Facets; import org.elasticsearch.search.facet.InternalFacets; import org.elasticsearch.search.suggest.Suggest; @@ -44,6 +46,7 @@ public class QuerySearchResult extends TransportResponse implements QuerySearchR private int size; private TopDocs topDocs; private InternalFacets facets; + private InternalAggregations aggregations; private Suggest suggest; private boolean searchTimedOut; @@ -103,6 +106,14 @@ public class QuerySearchResult extends TransportResponse implements QuerySearchR this.facets = facets; } + public Aggregations aggregations() { + return aggregations; + } + + public void aggregations(InternalAggregations aggregations) { + this.aggregations = aggregations; + } + public Suggest suggest() { return suggest; } @@ -146,6 +157,9 @@ public class QuerySearchResult extends TransportResponse implements QuerySearchR if (in.readBoolean()) { facets = InternalFacets.readFacets(in); } + if (in.readBoolean()) { + aggregations = InternalAggregations.readAggregations(in); + } if (in.readBoolean()) { suggest = Suggest.readSuggest(Suggest.Fields.SUGGEST, in); } @@ -166,6 +180,12 @@ public class QuerySearchResult extends TransportResponse implements QuerySearchR out.writeBoolean(true); facets.writeTo(out); } + if (aggregations == null) { + out.writeBoolean(false); + } else { + out.writeBoolean(true); + aggregations.writeTo(out); + } if (suggest == null) { out.writeBoolean(false); } else { diff --git a/src/test/java/org/elasticsearch/benchmark/search/facet/HistogramFacetSearchBenchmark.java b/src/test/java/org/elasticsearch/benchmark/search/aggregations/HistogramAggregationSearchBenchmark.java similarity index 74% rename from src/test/java/org/elasticsearch/benchmark/search/facet/HistogramFacetSearchBenchmark.java rename to src/test/java/org/elasticsearch/benchmark/search/aggregations/HistogramAggregationSearchBenchmark.java index 7f1bbcaee2a..9a05e9f2cf4 100644 --- a/src/test/java/org/elasticsearch/benchmark/search/facet/HistogramFacetSearchBenchmark.java +++ b/src/test/java/org/elasticsearch/benchmark/search/aggregations/HistogramAggregationSearchBenchmark.java @@ -17,7 +17,7 @@ * under the License. */ -package org.elasticsearch.benchmark.search.facet; +package org.elasticsearch.benchmark.search.aggregations; import org.elasticsearch.action.admin.cluster.health.ClusterHealthResponse; import org.elasticsearch.action.bulk.BulkRequestBuilder; @@ -40,13 +40,20 @@ import static org.elasticsearch.common.settings.ImmutableSettings.settingsBuilde import static org.elasticsearch.common.xcontent.XContentFactory.jsonBuilder; import static org.elasticsearch.index.query.QueryBuilders.matchAllQuery; import static org.elasticsearch.node.NodeBuilder.nodeBuilder; +import static org.elasticsearch.search.aggregations.AggregationBuilders.*; import static org.elasticsearch.search.facet.FacetBuilders.dateHistogramFacet; import static org.elasticsearch.search.facet.FacetBuilders.histogramFacet; /** * */ -public class HistogramFacetSearchBenchmark { +public class HistogramAggregationSearchBenchmark { + + static final long COUNT = SizeValue.parseSizeValue("20m").singles(); + static final int BATCH = 1000; + static final int QUERY_WARMUP = 5; + static final int QUERY_COUNT = 20; + static final int NUMBER_OF_TERMS = 1000; public static void main(String[] args) throws Exception { Settings settings = settingsBuilder() @@ -56,7 +63,7 @@ public class HistogramFacetSearchBenchmark { .put(SETTING_NUMBER_OF_REPLICAS, 1) .build(); - String clusterName = HistogramFacetSearchBenchmark.class.getSimpleName(); + String clusterName = HistogramAggregationSearchBenchmark.class.getSimpleName(); Node node1 = nodeBuilder() .clusterName(clusterName) .settings(settingsBuilder().put(settings).put("name", "node1")).node(); @@ -65,12 +72,6 @@ public class HistogramFacetSearchBenchmark { Client client = node1.client(); - long COUNT = SizeValue.parseSizeValue("20m").singles(); - int BATCH = 500; - int QUERY_WARMUP = 20; - int QUERY_COUNT = 200; - int NUMBER_OF_TERMS = 1000; - long[] lValues = new long[NUMBER_OF_TERMS]; for (int i = 0; i < NUMBER_OF_TERMS; i++) { lValues[i] = i; @@ -108,10 +109,10 @@ public class HistogramFacetSearchBenchmark { StopWatch stopWatch = new StopWatch().start(); System.out.println("--> Indexing [" + COUNT + "] ..."); - long ITERS = COUNT / BATCH; + long iters = COUNT / BATCH; long i = 1; int counter = 0; - for (; i <= ITERS; i++) { + for (; i <= iters; i++) { BulkRequestBuilder request = client.prepareBulk(); for (int j = 0; j < BATCH; j++) { counter++; @@ -145,7 +146,9 @@ public class HistogramFacetSearchBenchmark { } } client.admin().indices().prepareRefresh().execute().actionGet(); - COUNT = client.prepareCount().setQuery(matchAllQuery()).execute().actionGet().getCount(); + if (client.prepareCount().setQuery(matchAllQuery()).execute().actionGet().getCount() != COUNT) { + throw new Error(); + } System.out.println("--> Number of docs in index: " + COUNT); System.out.println("--> Warmup..."); @@ -158,6 +161,11 @@ public class HistogramFacetSearchBenchmark { .addFacet(histogramFacet("s_value").field("s_value").interval(4)) .addFacet(histogramFacet("b_value").field("b_value").interval(4)) .addFacet(histogramFacet("date").field("date").interval(1000)) + .addAggregation(histogram("l_value").field("l_value").interval(4)) + .addAggregation(histogram("i_value").field("i_value").interval(4)) + .addAggregation(histogram("s_value").field("s_value").interval(4)) + .addAggregation(histogram("b_value").field("b_value").interval(4)) + .addAggregation(histogram("date").field("date").interval(1000)) .execute().actionGet(); if (j == 0) { System.out.println("--> Warmup took: " + searchResponse.getTook()); @@ -183,11 +191,23 @@ public class HistogramFacetSearchBenchmark { } System.out.println("--> Histogram Facet (" + field + ") " + (totalQueryTime / QUERY_COUNT) + "ms"); + for (int j = 0; j < QUERY_COUNT; j++) { + SearchResponse searchResponse = client.prepareSearch() + .setQuery(matchAllQuery()) + .addAggregation(histogram(field).field(field).interval(4)) + .execute().actionGet(); + if (searchResponse.getHits().totalHits() != COUNT) { + System.err.println("--> mismatch on hits"); + } + totalQueryTime += searchResponse.getTookInMillis(); + } + System.out.println("--> Histogram Aggregation (" + field + ") " + (totalQueryTime / QUERY_COUNT) + "ms"); + totalQueryTime = 0; for (int j = 0; j < QUERY_COUNT; j++) { SearchResponse searchResponse = client.prepareSearch() .setQuery(matchAllQuery()) - .addFacet(histogramFacet("l_value").field("l_value").valueField("l_value").interval(4)) + .addFacet(histogramFacet(field).field(field).valueField(field).interval(4)) .execute().actionGet(); if (searchResponse.getHits().totalHits() != COUNT) { System.err.println("--> mismatch on hits"); @@ -195,6 +215,19 @@ public class HistogramFacetSearchBenchmark { totalQueryTime += searchResponse.getTookInMillis(); } System.out.println("--> Histogram Facet (" + field + "/" + field + ") " + (totalQueryTime / QUERY_COUNT) + "ms"); + + totalQueryTime = 0; + for (int j = 0; j < QUERY_COUNT; j++) { + SearchResponse searchResponse = client.prepareSearch() + .setQuery(matchAllQuery()) + .addAggregation(histogram(field).field(field).subAggregation(stats(field).field(field)).interval(4)) + .execute().actionGet(); + if (searchResponse.getHits().totalHits() != COUNT) { + System.err.println("--> mismatch on hits"); + } + totalQueryTime += searchResponse.getTookInMillis(); + } + System.out.println("--> Histogram Aggregation (" + field + "/" + field + ") " + (totalQueryTime / QUERY_COUNT) + "ms"); } totalQueryTime = 0; @@ -210,6 +243,19 @@ public class HistogramFacetSearchBenchmark { } System.out.println("--> Histogram Facet (date) " + (totalQueryTime / QUERY_COUNT) + "ms"); + totalQueryTime = 0; + for (int j = 0; j < QUERY_COUNT; j++) { + SearchResponse searchResponse = client.prepareSearch() + .setQuery(matchAllQuery()) + .addAggregation(dateHistogram("date").field("date").interval(1000)) + .execute().actionGet(); + if (searchResponse.getHits().totalHits() != COUNT) { + System.err.println("--> mismatch on hits"); + } + totalQueryTime += searchResponse.getTookInMillis(); + } + System.out.println("--> Histogram Aggregation (date) " + (totalQueryTime / QUERY_COUNT) + "ms"); + totalQueryTime = 0; for (int j = 0; j < QUERY_COUNT; j++) { SearchResponse searchResponse = client.prepareSearch() @@ -223,6 +269,18 @@ public class HistogramFacetSearchBenchmark { } System.out.println("--> Histogram Facet (date/l_value) " + (totalQueryTime / QUERY_COUNT) + "ms"); + totalQueryTime = 0; + for (int j = 0; j < QUERY_COUNT; j++) { + SearchResponse searchResponse = client.prepareSearch() + .setQuery(matchAllQuery()) + .addAggregation(dateHistogram("date").field("date").interval(1000).subAggregation(stats("stats").field("l_value"))) + .execute().actionGet(); + if (searchResponse.getHits().totalHits() != COUNT) { + System.err.println("--> mismatch on hits"); + } + totalQueryTime += searchResponse.getTookInMillis(); + } + System.out.println("--> Histogram Aggregation (date/l_value) " + (totalQueryTime / QUERY_COUNT) + "ms"); totalQueryTime = 0; for (int j = 0; j < QUERY_COUNT; j++) { diff --git a/src/test/java/org/elasticsearch/benchmark/search/facet/QueryFilterFacetSearchBenchmark.java b/src/test/java/org/elasticsearch/benchmark/search/aggregations/QueryFilterAggregationSearchBenchmark.java similarity index 74% rename from src/test/java/org/elasticsearch/benchmark/search/facet/QueryFilterFacetSearchBenchmark.java rename to src/test/java/org/elasticsearch/benchmark/search/aggregations/QueryFilterAggregationSearchBenchmark.java index b7d960028d1..002bb563fba 100644 --- a/src/test/java/org/elasticsearch/benchmark/search/facet/QueryFilterFacetSearchBenchmark.java +++ b/src/test/java/org/elasticsearch/benchmark/search/aggregations/QueryFilterAggregationSearchBenchmark.java @@ -17,7 +17,7 @@ * under the License. */ -package org.elasticsearch.benchmark.search.facet; +package org.elasticsearch.benchmark.search.aggregations; import jsr166y.ThreadLocalRandom; import org.elasticsearch.action.admin.cluster.health.ClusterHealthResponse; @@ -31,7 +31,9 @@ import org.elasticsearch.common.StopWatch; import org.elasticsearch.common.settings.Settings; import org.elasticsearch.common.unit.SizeValue; import org.elasticsearch.common.xcontent.XContentBuilder; +import org.elasticsearch.index.query.FilterBuilders; import org.elasticsearch.node.Node; +import org.elasticsearch.search.aggregations.AggregationBuilders; import org.elasticsearch.search.facet.FacetBuilder; import org.elasticsearch.search.facet.FacetBuilders; @@ -44,12 +46,12 @@ import static org.elasticsearch.index.query.QueryBuilders.matchAllQuery; import static org.elasticsearch.index.query.QueryBuilders.termQuery; import static org.elasticsearch.node.NodeBuilder.nodeBuilder; -public class QueryFilterFacetSearchBenchmark { +public class QueryFilterAggregationSearchBenchmark { - static long COUNT = SizeValue.parseSizeValue("1m").singles(); - static int BATCH = 100; - static int QUERY_COUNT = 200; - static int NUMBER_OF_TERMS = 200; + static final long COUNT = SizeValue.parseSizeValue("5m").singles(); + static final int BATCH = 1000; + static final int QUERY_COUNT = 200; + static final int NUMBER_OF_TERMS = 200; static Client client; @@ -61,7 +63,7 @@ public class QueryFilterFacetSearchBenchmark { .put(SETTING_NUMBER_OF_REPLICAS, 0) .build(); - String clusterName = QueryFilterFacetSearchBenchmark.class.getSimpleName(); + String clusterName = QueryFilterAggregationSearchBenchmark.class.getSimpleName(); Node node1 = nodeBuilder() .clusterName(clusterName) .settings(settingsBuilder().put(settings).put("name", "node1")).node(); @@ -89,7 +91,7 @@ public class QueryFilterFacetSearchBenchmark { XContentBuilder builder = jsonBuilder().startObject(); builder.field("id", Integer.toString(counter)); - builder.field("l_value", lValues[counter % lValues.length]); + builder.field("l_value", lValues[ThreadLocalRandom.current().nextInt(NUMBER_OF_TERMS)]); builder.endObject(); @@ -100,7 +102,7 @@ public class QueryFilterFacetSearchBenchmark { if (response.hasFailures()) { System.err.println("--> failures..."); } - if (((i * BATCH) % 10000) == 0) { + if (((i * BATCH) % 100000) == 0) { System.out.println("--> Indexed " + (i * BATCH) + " took " + stopWatch.stop().lastTaskTime()); stopWatch.start(); } @@ -114,53 +116,67 @@ public class QueryFilterFacetSearchBenchmark { } } client.admin().indices().prepareRefresh().execute().actionGet(); - COUNT = client.prepareCount().setQuery(matchAllQuery()).execute().actionGet().getCount(); + if (client.prepareCount().setQuery(matchAllQuery()).execute().actionGet().getCount() != COUNT) { + throw new Error(); + } System.out.println("--> Number of docs in index: " + COUNT); - + final long anyValue = ((Number) client.prepareSearch().execute().actionGet().getHits().hits()[0].sourceAsMap().get("l_value")).longValue(); + long totalQueryTime = 0; totalQueryTime = 0; for (int j = 0; j < QUERY_COUNT; j++) { SearchResponse searchResponse = client.prepareSearch() .setSearchType(SearchType.COUNT) - .setQuery(termQuery("l_value", lValues[0])) + .setQuery(termQuery("l_value", anyValue)) .execute().actionGet(); totalQueryTime += searchResponse.getTookInMillis(); } - System.out.println("--> Simple Query on first l_value " + (totalQueryTime / QUERY_COUNT) + "ms"); + System.out.println("--> Simple Query on first l_value " + totalQueryTime + "ms"); totalQueryTime = 0; for (int j = 0; j < QUERY_COUNT; j++) { SearchResponse searchResponse = client.prepareSearch() .setSearchType(SearchType.COUNT) - .setQuery(termQuery("l_value", lValues[0])) - .addFacet(FacetBuilders.queryFacet("query").query(termQuery("l_value", lValues[0]))) + .setQuery(termQuery("l_value", anyValue)) + .addFacet(FacetBuilders.queryFacet("query").query(termQuery("l_value", anyValue))) .execute().actionGet(); totalQueryTime += searchResponse.getTookInMillis(); } - System.out.println("--> Query facet first l_value " + (totalQueryTime / QUERY_COUNT) + "ms"); + System.out.println("--> Query facet first l_value " + totalQueryTime + "ms"); totalQueryTime = 0; for (int j = 0; j < QUERY_COUNT; j++) { SearchResponse searchResponse = client.prepareSearch() .setSearchType(SearchType.COUNT) - .setQuery(termQuery("l_value", lValues[0])) - .addFacet(FacetBuilders.queryFacet("query").query(termQuery("l_value", lValues[0])).global(true).mode(FacetBuilder.Mode.COLLECTOR)) + .setQuery(termQuery("l_value", anyValue)) + .addAggregation(AggregationBuilders.filter("filter").filter(FilterBuilders.termFilter("l_value", anyValue))) .execute().actionGet(); totalQueryTime += searchResponse.getTookInMillis(); } - System.out.println("--> Query facet first l_value (global) (mode/collector) " + (totalQueryTime / QUERY_COUNT) + "ms"); + System.out.println("--> Filter agg first l_value " + totalQueryTime + "ms"); totalQueryTime = 0; for (int j = 0; j < QUERY_COUNT; j++) { SearchResponse searchResponse = client.prepareSearch() .setSearchType(SearchType.COUNT) - .setQuery(termQuery("l_value", lValues[0])) - .addFacet(FacetBuilders.queryFacet("query").query(termQuery("l_value", lValues[0])).global(true).mode(FacetBuilder.Mode.POST)) + .setQuery(termQuery("l_value", anyValue)) + .addFacet(FacetBuilders.queryFacet("query").query(termQuery("l_value", anyValue)).global(true).mode(FacetBuilder.Mode.COLLECTOR)) .execute().actionGet(); totalQueryTime += searchResponse.getTookInMillis(); } - System.out.println("--> Query facet first l_value (global) (mode/post) " + (totalQueryTime / QUERY_COUNT) + "ms"); + System.out.println("--> Query facet first l_value (global) (mode/collector) " + totalQueryTime + "ms"); + + totalQueryTime = 0; + for (int j = 0; j < QUERY_COUNT; j++) { + SearchResponse searchResponse = client.prepareSearch() + .setSearchType(SearchType.COUNT) + .setQuery(termQuery("l_value", anyValue)) + .addFacet(FacetBuilders.queryFacet("query").query(termQuery("l_value", anyValue)).global(true).mode(FacetBuilder.Mode.POST)) + .execute().actionGet(); + totalQueryTime += searchResponse.getTookInMillis(); + } + System.out.println("--> Query facet first l_value (global) (mode/post) " + totalQueryTime + "ms"); } } diff --git a/src/test/java/org/elasticsearch/benchmark/search/facet/TermsFacetSearchBenchmark.java b/src/test/java/org/elasticsearch/benchmark/search/aggregations/TermsAggregationSearchBenchmark.java similarity index 67% rename from src/test/java/org/elasticsearch/benchmark/search/facet/TermsFacetSearchBenchmark.java rename to src/test/java/org/elasticsearch/benchmark/search/aggregations/TermsAggregationSearchBenchmark.java index cad42e68865..3b501711ea8 100644 --- a/src/test/java/org/elasticsearch/benchmark/search/facet/TermsFacetSearchBenchmark.java +++ b/src/test/java/org/elasticsearch/benchmark/search/aggregations/TermsAggregationSearchBenchmark.java @@ -17,7 +17,7 @@ * under the License. */ -package org.elasticsearch.benchmark.search.facet; +package org.elasticsearch.benchmark.search.aggregations; import com.carrotsearch.randomizedtesting.generators.RandomStrings; import com.google.common.collect.Lists; @@ -25,6 +25,7 @@ import jsr166y.ThreadLocalRandom; import org.elasticsearch.action.admin.cluster.health.ClusterHealthResponse; import org.elasticsearch.action.bulk.BulkRequestBuilder; import org.elasticsearch.action.bulk.BulkResponse; +import org.elasticsearch.action.search.SearchRequestBuilder; import org.elasticsearch.action.search.SearchResponse; import org.elasticsearch.action.search.SearchType; import org.elasticsearch.client.Client; @@ -35,6 +36,7 @@ import org.elasticsearch.common.unit.SizeValue; import org.elasticsearch.common.unit.TimeValue; import org.elasticsearch.common.xcontent.XContentBuilder; import org.elasticsearch.node.Node; +import org.elasticsearch.search.aggregations.AggregationBuilders; import java.util.List; import java.util.Random; @@ -52,18 +54,45 @@ import static org.elasticsearch.search.facet.FacetBuilders.termsStatsFacet; /** * */ -public class TermsFacetSearchBenchmark { +public class TermsAggregationSearchBenchmark { static long COUNT = SizeValue.parseSizeValue("2m").singles(); - static int BATCH = 100; - static int QUERY_WARMUP = 20; - static int QUERY_COUNT = 200; + static int BATCH = 1000; + static int QUERY_WARMUP = 10; + static int QUERY_COUNT = 100; static int NUMBER_OF_TERMS = 200; static int NUMBER_OF_MULTI_VALUE_TERMS = 10; static int STRING_TERM_SIZE = 5; static Client client; + private enum Method { + FACET { + @Override + SearchRequestBuilder addTermsAgg(SearchRequestBuilder builder, String name, String field, String executionHint) { + return builder.addFacet(termsFacet(name).field(field).executionHint(executionHint)); + } + + @Override + SearchRequestBuilder addTermsStatsAgg(SearchRequestBuilder builder, String name, String keyField, String valueField) { + return builder.addFacet(termsStatsFacet(name).keyField(keyField).valueField(valueField)); + } + }, + AGGREGATION { + @Override + SearchRequestBuilder addTermsAgg(SearchRequestBuilder builder, String name, String field, String executionHint) { + return builder.addAggregation(AggregationBuilders.terms(name).field(field)); + } + + @Override + SearchRequestBuilder addTermsStatsAgg(SearchRequestBuilder builder, String name, String keyField, String valueField) { + return builder.addAggregation(AggregationBuilders.terms(name).field(keyField).subAggregation(AggregationBuilders.stats("stats").field(valueField))); + } + }; + abstract SearchRequestBuilder addTermsAgg(SearchRequestBuilder builder, String name, String field, String executionHint); + abstract SearchRequestBuilder addTermsStatsAgg(SearchRequestBuilder builder, String name, String keyField, String valueField); + } + public static void main(String[] args) throws Exception { Random random = new Random(); @@ -74,7 +103,7 @@ public class TermsFacetSearchBenchmark { .put(SETTING_NUMBER_OF_REPLICAS, 0) .build(); - String clusterName = TermsFacetSearchBenchmark.class.getSimpleName(); + String clusterName = TermsAggregationSearchBenchmark.class.getSimpleName(); Node[] nodes = new Node[1]; for (int i = 0; i < nodes.length; i++) { nodes[i] = nodeBuilder().clusterName(clusterName) @@ -199,29 +228,39 @@ public class TermsFacetSearchBenchmark { List stats = Lists.newArrayList(); - stats.add(terms("terms_s", "s_value", null)); - stats.add(terms("terms_s_dv", "s_value_dv", null)); - stats.add(terms("terms_map_s", "s_value", "map")); - stats.add(terms("terms_map_s_dv", "s_value_dv", "map")); - stats.add(terms("terms_l", "l_value", null)); - stats.add(terms("terms_l_dv", "l_value_dv", null)); - stats.add(terms("terms_map_l", "l_value", "map")); - stats.add(terms("terms_map_l_dv", "l_value_dv", "map")); - stats.add(terms("terms_sm", "sm_value", null)); - stats.add(terms("terms_sm_dv", "sm_value_dv", null)); - stats.add(terms("terms_map_sm", "sm_value", "map")); - stats.add(terms("terms_map_sm_dv", "sm_value_dv", "map")); - stats.add(terms("terms_lm", "lm_value", null)); - stats.add(terms("terms_lm_dv", "lm_value_dv", null)); - stats.add(terms("terms_map_lm", "lm_value", "map")); - stats.add(terms("terms_map_lm_dv", "lm_value_dv", "map")); + stats.add(terms("terms_facet_s", Method.FACET, "s_value", null)); + stats.add(terms("terms_facet_s_dv", Method.FACET, "s_value_dv", null)); + stats.add(terms("terms_facet_map_s", Method.FACET, "s_value", "map")); + stats.add(terms("terms_facet_map_s_dv", Method.FACET, "s_value_dv", "map")); + stats.add(terms("terms_agg_s", Method.AGGREGATION, "s_value", null)); + stats.add(terms("terms_agg_s_dv", Method.AGGREGATION, "s_value_dv", null)); + stats.add(terms("terms_facet_l", Method.FACET, "l_value", null)); + stats.add(terms("terms_facet_l_dv", Method.FACET, "l_value_dv", null)); + stats.add(terms("terms_agg_l", Method.AGGREGATION, "l_value", null)); + stats.add(terms("terms_agg_l_dv", Method.AGGREGATION, "l_value_dv", null)); + stats.add(terms("terms_facet_sm", Method.FACET, "sm_value", null)); + stats.add(terms("terms_facet_sm_dv", Method.FACET, "sm_value_dv", null)); + stats.add(terms("terms_facet_map_sm", Method.FACET, "sm_value", "map")); + stats.add(terms("terms_facet_map_sm_dv", Method.FACET, "sm_value_dv", "map")); + stats.add(terms("terms_agg_sm", Method.AGGREGATION, "sm_value", null)); + stats.add(terms("terms_agg_sm_dv", Method.AGGREGATION, "sm_value_dv", null)); + stats.add(terms("terms_facet_lm", Method.FACET, "lm_value", null)); + stats.add(terms("terms_facet_lm_dv", Method.FACET, "lm_value_dv", null)); + stats.add(terms("terms_agg_lm", Method.AGGREGATION, "lm_value", null)); + stats.add(terms("terms_agg_lm_dv", Method.AGGREGATION, "lm_value_dv", null)); - stats.add(termsStats("terms_stats_s_l", "s_value", "l_value", null)); - stats.add(termsStats("terms_stats_s_l_dv", "s_value_dv", "l_value_dv", null)); - stats.add(termsStats("terms_stats_s_lm", "s_value", "lm_value", null)); - stats.add(termsStats("terms_stats_s_lm_dv", "s_value_dv", "lm_value_dv", null)); - stats.add(termsStats("terms_stats_sm_l", "sm_value", "l_value", null)); - stats.add(termsStats("terms_stats_sm_l_dv", "sm_value_dv", "l_value_dv", null)); + stats.add(termsStats("terms_stats_facet_s_l", Method.FACET, "s_value", "l_value", null)); + stats.add(termsStats("terms_stats_facet_s_l_dv", Method.FACET, "s_value_dv", "l_value_dv", null)); + stats.add(termsStats("terms_stats_agg_s_l", Method.AGGREGATION, "s_value", "l_value", null)); + stats.add(termsStats("terms_stats_agg_s_l_dv", Method.AGGREGATION, "s_value_dv", "l_value_dv", null)); + stats.add(termsStats("terms_stats_facet_s_lm", Method.FACET, "s_value", "lm_value", null)); + stats.add(termsStats("terms_stats_facet_s_lm_dv", Method.FACET, "s_value_dv", "lm_value_dv", null)); + stats.add(termsStats("terms_stats_agg_s_lm", Method.AGGREGATION, "s_value", "lm_value", null)); + stats.add(termsStats("terms_stats_agg_s_lm_dv", Method.AGGREGATION, "s_value_dv", "lm_value_dv", null)); + stats.add(termsStats("terms_stats_facet_sm_l", Method.FACET, "sm_value", "l_value", null)); + stats.add(termsStats("terms_stats_facet_sm_l_dv", Method.FACET, "sm_value_dv", "l_value_dv", null)); + stats.add(termsStats("terms_stats_agg_sm_l", Method.AGGREGATION, "sm_value", "l_value", null)); + stats.add(termsStats("terms_stats_agg_sm_l_dv", Method.AGGREGATION, "sm_value_dv", "l_value_dv", null)); System.out.println("------------------ SUMMARY -------------------------------"); System.out.format("%25s%10s%10s\n", "name", "took", "millis"); @@ -247,7 +286,7 @@ public class TermsFacetSearchBenchmark { } } - private static StatsResult terms(String name, String field, String executionHint) { + private static StatsResult terms(String name, Method method, String field, String executionHint) { long totalQueryTime;// LM VALUE client.admin().indices().prepareClearCache().setFieldDataCache(true).execute().actionGet(); @@ -255,10 +294,9 @@ public class TermsFacetSearchBenchmark { System.out.println("--> Warmup (" + name + ")..."); // run just the child query, warm up first for (int j = 0; j < QUERY_WARMUP; j++) { - SearchResponse searchResponse = client.prepareSearch() + SearchResponse searchResponse = method.addTermsAgg(client.prepareSearch() .setSearchType(SearchType.COUNT) - .setQuery(matchAllQuery()) - .addFacet(termsFacet(field).field(field).executionHint(executionHint)) + .setQuery(matchAllQuery()), name, field, executionHint) .execute().actionGet(); if (j == 0) { System.out.println("--> Loading (" + field + "): took: " + searchResponse.getTook()); @@ -273,21 +311,20 @@ public class TermsFacetSearchBenchmark { System.out.println("--> Running (" + name + ")..."); totalQueryTime = 0; for (int j = 0; j < QUERY_COUNT; j++) { - SearchResponse searchResponse = client.prepareSearch() + SearchResponse searchResponse = method.addTermsAgg(client.prepareSearch() .setSearchType(SearchType.COUNT) - .setQuery(matchAllQuery()) - .addFacet(termsFacet(field).field(field).executionHint(executionHint)) + .setQuery(matchAllQuery()), name, field, executionHint) .execute().actionGet(); if (searchResponse.getHits().totalHits() != COUNT) { System.err.println("--> mismatch on hits"); } totalQueryTime += searchResponse.getTookInMillis(); } - System.out.println("--> Terms Facet (" + field + "), hint(" + executionHint + "): " + (totalQueryTime / QUERY_COUNT) + "ms"); + System.out.println("--> Terms Agg (" + name + "): " + (totalQueryTime / QUERY_COUNT) + "ms"); return new StatsResult(name, totalQueryTime); } - private static StatsResult termsStats(String name, String keyField, String valueField, String executionHint) { + private static StatsResult termsStats(String name, Method method, String keyField, String valueField, String executionHint) { long totalQueryTime; client.admin().indices().prepareClearCache().setFieldDataCache(true).execute().actionGet(); @@ -295,10 +332,9 @@ public class TermsFacetSearchBenchmark { System.out.println("--> Warmup (" + name + ")..."); // run just the child query, warm up first for (int j = 0; j < QUERY_WARMUP; j++) { - SearchResponse searchResponse = client.prepareSearch() + SearchResponse searchResponse = method.addTermsStatsAgg(client.prepareSearch() .setSearchType(SearchType.COUNT) - .setQuery(matchAllQuery()) - .addFacet(termsStatsFacet(name).keyField(keyField).valueField(valueField)) + .setQuery(matchAllQuery()), name, keyField, valueField) .execute().actionGet(); if (j == 0) { System.out.println("--> Loading (" + name + "): took: " + searchResponse.getTook()); @@ -313,17 +349,16 @@ public class TermsFacetSearchBenchmark { System.out.println("--> Running (" + name + ")..."); totalQueryTime = 0; for (int j = 0; j < QUERY_COUNT; j++) { - SearchResponse searchResponse = client.prepareSearch() + SearchResponse searchResponse = method.addTermsStatsAgg(client.prepareSearch() .setSearchType(SearchType.COUNT) - .setQuery(matchAllQuery()) - .addFacet(termsStatsFacet(name).keyField(keyField).valueField(valueField)) + .setQuery(matchAllQuery()), name, keyField, valueField) .execute().actionGet(); if (searchResponse.getHits().totalHits() != COUNT) { System.err.println("--> mismatch on hits"); } totalQueryTime += searchResponse.getTookInMillis(); } - System.out.println("--> Terms Facet (" + name + "), hint(" + executionHint + "): " + (totalQueryTime / QUERY_COUNT) + "ms"); + System.out.println("--> Terms stats agg (" + name + "): " + (totalQueryTime / QUERY_COUNT) + "ms"); return new StatsResult(name, totalQueryTime); } } diff --git a/src/test/java/org/elasticsearch/common/rounding/RoundingTests.java b/src/test/java/org/elasticsearch/common/rounding/RoundingTests.java new file mode 100644 index 00000000000..03e086f35ff --- /dev/null +++ b/src/test/java/org/elasticsearch/common/rounding/RoundingTests.java @@ -0,0 +1,44 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.common.rounding; + +import org.elasticsearch.test.ElasticsearchTestCase; + +import static org.hamcrest.Matchers.greaterThan; +import static org.hamcrest.Matchers.lessThanOrEqualTo; + + + +public class RoundingTests extends ElasticsearchTestCase { + + public void testInterval() { + final long interval = randomIntBetween(1, 100); + Rounding.Interval rounding = new Rounding.Interval(interval); + for (int i = 0; i < 1000; ++i) { + long l = Math.max(randomLong(), Long.MIN_VALUE + interval); + final long r = rounding.round(l); + String message = "round(" + l + ", interval=" + interval + ") = " + r; + assertEquals(message, 0, r % interval); + assertThat(message, r, lessThanOrEqualTo(l)); + assertThat(message, r + interval, greaterThan(l)); + } + } + +} diff --git a/src/test/java/org/elasticsearch/common/rounding/TimeZoneRoundingTests.java b/src/test/java/org/elasticsearch/common/rounding/TimeZoneRoundingTests.java new file mode 100644 index 00000000000..74988060624 --- /dev/null +++ b/src/test/java/org/elasticsearch/common/rounding/TimeZoneRoundingTests.java @@ -0,0 +1,94 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.common.rounding; + +import org.elasticsearch.common.unit.TimeValue; +import org.elasticsearch.test.ElasticsearchTestCase; +import org.joda.time.DateTimeZone; +import org.joda.time.format.ISODateTimeFormat; +import org.junit.Test; + +import static org.hamcrest.Matchers.equalTo; + +/** + */ +public class TimeZoneRoundingTests extends ElasticsearchTestCase { + + @Test + public void testUTCMonthRounding() { + TimeZoneRounding tzRounding = TimeZoneRounding.builder(DateTimeUnit.MONTH_OF_YEAR).build(); + assertThat(tzRounding.round(utc("2009-02-03T01:01:01")), equalTo(utc("2009-02-01T00:00:00.000Z"))); + assertThat(tzRounding.nextRoundingValue(utc("2009-02-01T00:00:00.000Z")), equalTo(utc("2009-03-01T00:00:00.000Z"))); + + tzRounding = TimeZoneRounding.builder(DateTimeUnit.WEEK_OF_WEEKYEAR).build(); + assertThat(tzRounding.round(utc("2012-01-10T01:01:01")), equalTo(utc("2012-01-09T00:00:00.000Z"))); + assertThat(tzRounding.nextRoundingValue(utc("2012-01-09T00:00:00.000Z")), equalTo(utc("2012-01-16T00:00:00.000Z"))); + + tzRounding = TimeZoneRounding.builder(DateTimeUnit.WEEK_OF_WEEKYEAR).postOffset(-TimeValue.timeValueHours(24).millis()).build(); + assertThat(tzRounding.round(utc("2012-01-10T01:01:01")), equalTo(utc("2012-01-08T00:00:00.000Z"))); + assertThat(tzRounding.nextRoundingValue(utc("2012-01-08T00:00:00.000Z")), equalTo(utc("2012-01-15T00:00:00.000Z"))); + } + + @Test + public void testDayTimeZoneRounding() { + TimeZoneRounding tzRounding = TimeZoneRounding.builder(DateTimeUnit.DAY_OF_MONTH).preZone(DateTimeZone.forOffsetHours(-2)).build(); + assertThat(tzRounding.round(0), equalTo(0l - TimeValue.timeValueHours(24).millis())); + assertThat(tzRounding.nextRoundingValue(0l - TimeValue.timeValueHours(24).millis()), equalTo(0l)); + + tzRounding = TimeZoneRounding.builder(DateTimeUnit.DAY_OF_MONTH).preZone(DateTimeZone.forOffsetHours(-2)).postZone(DateTimeZone.forOffsetHours(-2)).build(); + assertThat(tzRounding.round(0), equalTo(0l - TimeValue.timeValueHours(26).millis())); + assertThat(tzRounding.nextRoundingValue(0l - TimeValue.timeValueHours(26).millis()), equalTo(-TimeValue.timeValueHours(2).millis())); + + tzRounding = TimeZoneRounding.builder(DateTimeUnit.DAY_OF_MONTH).preZone(DateTimeZone.forOffsetHours(-2)).build(); + assertThat(tzRounding.round(utc("2009-02-03T01:01:01")), equalTo(utc("2009-02-02T00:00:00"))); + assertThat(tzRounding.nextRoundingValue(utc("2009-02-02T00:00:00")), equalTo(utc("2009-02-03T00:00:00"))); + + tzRounding = TimeZoneRounding.builder(DateTimeUnit.DAY_OF_MONTH).preZone(DateTimeZone.forOffsetHours(-2)).postZone(DateTimeZone.forOffsetHours(-2)).build(); + assertThat(tzRounding.round(utc("2009-02-03T01:01:01")), equalTo(time("2009-02-02T00:00:00", DateTimeZone.forOffsetHours(+2)))); + assertThat(tzRounding.nextRoundingValue(time("2009-02-02T00:00:00", DateTimeZone.forOffsetHours(+2))), equalTo(time("2009-02-03T00:00:00", DateTimeZone.forOffsetHours(+2)))); + } + + @Test + public void testTimeTimeZoneRounding() { + TimeZoneRounding tzRounding = TimeZoneRounding.builder(DateTimeUnit.HOUR_OF_DAY).preZone(DateTimeZone.forOffsetHours(-2)).build(); + assertThat(tzRounding.round(0), equalTo(0l)); + assertThat(tzRounding.nextRoundingValue(0l), equalTo(TimeValue.timeValueHours(1l).getMillis())); + + tzRounding = TimeZoneRounding.builder(DateTimeUnit.HOUR_OF_DAY).preZone(DateTimeZone.forOffsetHours(-2)).postZone(DateTimeZone.forOffsetHours(-2)).build(); + assertThat(tzRounding.round(0), equalTo(0l - TimeValue.timeValueHours(2).millis())); + assertThat(tzRounding.nextRoundingValue(0l - TimeValue.timeValueHours(2).millis()), equalTo(0l - TimeValue.timeValueHours(1).millis())); + + tzRounding = TimeZoneRounding.builder(DateTimeUnit.HOUR_OF_DAY).preZone(DateTimeZone.forOffsetHours(-2)).build(); + assertThat(tzRounding.round(utc("2009-02-03T01:01:01")), equalTo(utc("2009-02-03T01:00:00"))); + assertThat(tzRounding.nextRoundingValue(utc("2009-02-03T01:00:00")), equalTo(utc("2009-02-03T02:00:00"))); + + tzRounding = TimeZoneRounding.builder(DateTimeUnit.HOUR_OF_DAY).preZone(DateTimeZone.forOffsetHours(-2)).postZone(DateTimeZone.forOffsetHours(-2)).build(); + assertThat(tzRounding.round(utc("2009-02-03T01:01:01")), equalTo(time("2009-02-03T01:00:00", DateTimeZone.forOffsetHours(+2)))); + assertThat(tzRounding.nextRoundingValue(time("2009-02-03T01:00:00", DateTimeZone.forOffsetHours(+2))), equalTo(time("2009-02-03T02:00:00", DateTimeZone.forOffsetHours(+2)))); + } + + private long utc(String time) { + return time(time, DateTimeZone.UTC); + } + + private long time(String time, DateTimeZone zone) { + return ISODateTimeFormat.dateOptionalTimeParser().withZone(zone).parseMillis(time); + } +} diff --git a/src/test/java/org/elasticsearch/common/util/BigArraysTests.java b/src/test/java/org/elasticsearch/common/util/BigArraysTests.java new file mode 100644 index 00000000000..ba1825897a0 --- /dev/null +++ b/src/test/java/org/elasticsearch/common/util/BigArraysTests.java @@ -0,0 +1,112 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.common.util; + +import org.elasticsearch.test.ElasticsearchTestCase; + +import java.util.Arrays; + +public class BigArraysTests extends ElasticsearchTestCase { + + public void testIntArrayGrowth() { + final int totalLen = randomIntBetween(1, 1000000); + final int startLen = randomIntBetween(1, randomBoolean() ? 1000 : totalLen); + IntArray array = BigArrays.newIntArray(startLen); + int[] ref = new int[totalLen]; + for (int i = 0; i < totalLen; ++i) { + ref[i] = randomInt(); + array = BigArrays.grow(array, i + 1); + array.set(i, ref[i]); + } + for (int i = 0; i < totalLen; ++i) { + assertEquals(ref[i], array.get(i)); + } + } + + public void testLongArrayGrowth() { + final int totalLen = randomIntBetween(1, 1000000); + final int startLen = randomIntBetween(1, randomBoolean() ? 1000 : totalLen); + LongArray array = BigArrays.newLongArray(startLen); + long[] ref = new long[totalLen]; + for (int i = 0; i < totalLen; ++i) { + ref[i] = randomLong(); + array = BigArrays.grow(array, i + 1); + array.set(i, ref[i]); + } + for (int i = 0; i < totalLen; ++i) { + assertEquals(ref[i], array.get(i)); + } + } + + public void testDoubleArrayGrowth() { + final int totalLen = randomIntBetween(1, 1000000); + final int startLen = randomIntBetween(1, randomBoolean() ? 1000 : totalLen); + DoubleArray array = BigArrays.newDoubleArray(startLen); + double[] ref = new double[totalLen]; + for (int i = 0; i < totalLen; ++i) { + ref[i] = randomDouble(); + array = BigArrays.grow(array, i + 1); + array.set(i, ref[i]); + } + for (int i = 0; i < totalLen; ++i) { + assertEquals(ref[i], array.get(i), 0.001d); + } + } + + public void testObjectArrayGrowth() { + final int totalLen = randomIntBetween(1, 1000000); + final int startLen = randomIntBetween(1, randomBoolean() ? 1000 : totalLen); + ObjectArray array = BigArrays.newObjectArray(startLen); + final Object[] pool = new Object[100]; + for (int i = 0; i < pool.length; ++i) { + pool[i] = new Object(); + } + Object[] ref = new Object[totalLen]; + for (int i = 0; i < totalLen; ++i) { + ref[i] = randomFrom(pool); + array = BigArrays.grow(array, i + 1); + array.set(i, ref[i]); + } + for (int i = 0; i < totalLen; ++i) { + assertSame(ref[i], array.get(i)); + } + } + + public void testDoubleArrayFill() { + final int len = randomIntBetween(1, 100000); + final int fromIndex = randomIntBetween(0, len - 1); + final int toIndex = randomBoolean() + ? Math.min(fromIndex + randomInt(100), len) // single page + : randomIntBetween(fromIndex, len); // likely multiple pages + final DoubleArray array2 = BigArrays.newDoubleArray(len); + final double[] array1 = new double[len]; + for (int i = 0; i < len; ++i) { + array1[i] = randomDouble(); + array2.set(i, array1[i]); + } + final double rand = randomDouble(); + Arrays.fill(array1, fromIndex, toIndex, rand); + array2.fill(fromIndex, toIndex, rand); + for (int i = 0; i < len; ++i) { + assertEquals(array1[i], array2.get(i), 0.001d); + } + } + +} diff --git a/src/test/java/org/elasticsearch/deps/joda/TimeZoneRoundingTests.java b/src/test/java/org/elasticsearch/deps/joda/TimeZoneRoundingTests.java deleted file mode 100644 index c1ae14e0fed..00000000000 --- a/src/test/java/org/elasticsearch/deps/joda/TimeZoneRoundingTests.java +++ /dev/null @@ -1,91 +0,0 @@ -/* - * Licensed to ElasticSearch and Shay Banon under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. ElasticSearch licenses this - * file to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ - -package org.elasticsearch.deps.joda; - -import org.elasticsearch.common.joda.TimeZoneRounding; -import org.elasticsearch.common.unit.TimeValue; -import org.elasticsearch.test.ElasticsearchTestCase; -import org.joda.time.Chronology; -import org.joda.time.DateTimeZone; -import org.joda.time.chrono.ISOChronology; -import org.joda.time.format.ISODateTimeFormat; -import org.junit.Test; - -import static org.hamcrest.MatcherAssert.assertThat; -import static org.hamcrest.Matchers.equalTo; - -/** - */ -public class TimeZoneRoundingTests extends ElasticsearchTestCase { - - @Test - public void testUTCMonthRounding() { - TimeZoneRounding tzRounding = TimeZoneRounding.builder(chronology().monthOfYear()).build(); - assertThat(tzRounding.calc(utc("2009-02-03T01:01:01")), equalTo(utc("2009-02-01T00:00:00.000Z"))); - - tzRounding = TimeZoneRounding.builder(chronology().weekOfWeekyear()).build(); - assertThat(tzRounding.calc(utc("2012-01-10T01:01:01")), equalTo(utc("2012-01-09T00:00:00.000Z"))); - - tzRounding = TimeZoneRounding.builder(chronology().weekOfWeekyear()).postOffset(-TimeValue.timeValueHours(24).millis()).build(); - assertThat(tzRounding.calc(utc("2012-01-10T01:01:01")), equalTo(utc("2012-01-08T00:00:00.000Z"))); - } - - @Test - public void testDayTimeZoneRounding() { - TimeZoneRounding tzRounding = TimeZoneRounding.builder(chronology().dayOfMonth()).preZone(DateTimeZone.forOffsetHours(-2)).build(); - assertThat(tzRounding.calc(0), equalTo(0l - TimeValue.timeValueHours(24).millis())); - - tzRounding = TimeZoneRounding.builder(chronology().dayOfMonth()).preZone(DateTimeZone.forOffsetHours(-2)).postZone(DateTimeZone.forOffsetHours(-2)).build(); - assertThat(tzRounding.calc(0), equalTo(0l - TimeValue.timeValueHours(26).millis())); - - tzRounding = TimeZoneRounding.builder(chronology().dayOfMonth()).preZone(DateTimeZone.forOffsetHours(-2)).build(); - assertThat(tzRounding.calc(utc("2009-02-03T01:01:01")), equalTo(utc("2009-02-02T00:00:00"))); - - tzRounding = TimeZoneRounding.builder(chronology().dayOfMonth()).preZone(DateTimeZone.forOffsetHours(-2)).postZone(DateTimeZone.forOffsetHours(-2)).build(); - assertThat(tzRounding.calc(utc("2009-02-03T01:01:01")), equalTo(time("2009-02-02T00:00:00", DateTimeZone.forOffsetHours(+2)))); - } - - @Test - public void testTimeTimeZoneRounding() { - TimeZoneRounding tzRounding = TimeZoneRounding.builder(chronology().hourOfDay()).preZone(DateTimeZone.forOffsetHours(-2)).build(); - assertThat(tzRounding.calc(0), equalTo(0l)); - - tzRounding = TimeZoneRounding.builder(chronology().hourOfDay()).preZone(DateTimeZone.forOffsetHours(-2)).postZone(DateTimeZone.forOffsetHours(-2)).build(); - assertThat(tzRounding.calc(0), equalTo(0l - TimeValue.timeValueHours(2).millis())); - - tzRounding = TimeZoneRounding.builder(chronology().hourOfDay()).preZone(DateTimeZone.forOffsetHours(-2)).build(); - assertThat(tzRounding.calc(utc("2009-02-03T01:01:01")), equalTo(utc("2009-02-03T01:00:00"))); - - tzRounding = TimeZoneRounding.builder(chronology().hourOfDay()).preZone(DateTimeZone.forOffsetHours(-2)).postZone(DateTimeZone.forOffsetHours(-2)).build(); - assertThat(tzRounding.calc(utc("2009-02-03T01:01:01")), equalTo(time("2009-02-03T01:00:00", DateTimeZone.forOffsetHours(+2)))); - } - - private static Chronology chronology() { - return ISOChronology.getInstanceUTC(); - } - - private long utc(String time) { - return time(time, DateTimeZone.UTC); - } - - private long time(String time, DateTimeZone zone) { - return ISODateTimeFormat.dateOptionalTimeParser().withZone(zone).parseMillis(time); - } -} diff --git a/src/test/java/org/elasticsearch/index/search/child/TestSearchContext.java b/src/test/java/org/elasticsearch/index/search/child/TestSearchContext.java index db5c3e4c8b3..f361361bbdc 100644 --- a/src/test/java/org/elasticsearch/index/search/child/TestSearchContext.java +++ b/src/test/java/org/elasticsearch/index/search/child/TestSearchContext.java @@ -42,6 +42,7 @@ import org.elasticsearch.index.similarity.SimilarityService; import org.elasticsearch.script.ScriptService; import org.elasticsearch.search.Scroll; import org.elasticsearch.search.SearchShardTarget; +import org.elasticsearch.search.aggregations.SearchContextAggregations; import org.elasticsearch.search.dfs.DfsSearchResult; import org.elasticsearch.search.facet.SearchContextFacets; import org.elasticsearch.search.fetch.FetchSearchResult; @@ -169,6 +170,16 @@ class TestSearchContext extends SearchContext { return null; } + @Override + public SearchContextAggregations aggregations() { + return null; + } + + @Override + public SearchContext aggregations(SearchContextAggregations aggregations) { + return null; + } + @Override public SearchContextHighlight highlight() { return null; diff --git a/src/test/java/org/elasticsearch/search/aggregations/RandomTests.java b/src/test/java/org/elasticsearch/search/aggregations/RandomTests.java new file mode 100644 index 00000000000..8381f670b34 --- /dev/null +++ b/src/test/java/org/elasticsearch/search/aggregations/RandomTests.java @@ -0,0 +1,263 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations; + +import com.carrotsearch.hppc.IntOpenHashSet; +import org.elasticsearch.action.search.SearchRequestBuilder; +import org.elasticsearch.action.search.SearchResponse; +import org.elasticsearch.action.support.IgnoreIndices; +import org.elasticsearch.common.settings.ImmutableSettings; +import org.elasticsearch.common.settings.Settings; +import org.elasticsearch.common.xcontent.XContentBuilder; +import org.elasticsearch.index.query.FilterBuilders; +import org.elasticsearch.index.query.RangeFilterBuilder; +import org.elasticsearch.search.aggregations.bucket.histogram.Histogram; +import org.elasticsearch.search.aggregations.bucket.range.Range; +import org.elasticsearch.search.aggregations.bucket.range.RangeBuilder; +import org.elasticsearch.search.aggregations.bucket.terms.Terms; +import org.elasticsearch.search.aggregations.bucket.filter.Filter; +import org.elasticsearch.test.ElasticsearchIntegrationTest; + +import static org.elasticsearch.common.xcontent.XContentFactory.jsonBuilder; +import static org.elasticsearch.search.aggregations.AggregationBuilders.*; +import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertNoFailures; +import static org.hamcrest.Matchers.equalTo; +import static org.hamcrest.core.IsNull.notNullValue; + +/** Additional tests that aim at testing more complex aggregation trees on larger random datasets, so that things like the growth of dynamic arrays is tested. */ +public class RandomTests extends ElasticsearchIntegrationTest { + + @Override + public Settings indexSettings() { + return ImmutableSettings.builder() + .put("index.number_of_shards", between(1, 5)) + .put("index.number_of_replicas", between(0, 1)) + .build(); + } + + // Make sure that unordered, reversed, disjoint and/or overlapping ranges are supported + // Duel with filters + public void testRandomRanges() throws Exception { + final int numDocs = atLeast(1000); + final double[][] docs = new double[numDocs][]; + for (int i = 0; i < numDocs; ++i) { + final int numValues = randomInt(5); + docs[i] = new double[numValues]; + for (int j = 0; j < numValues; ++j) { + docs[i][j] = randomDouble() * 100; + } + } + + createIndex("idx"); + for (int i = 0; i < docs.length; ++i) { + XContentBuilder source = jsonBuilder() + .startObject() + .startArray("values"); + for (int j = 0; j < docs[i].length; ++j) { + source = source.value(docs[i][j]); + } + source = source.endArray().endObject(); + client().prepareIndex("idx", "type").setSource(source).execute().actionGet(); + } + assertNoFailures(client().admin().indices().prepareRefresh("idx").setIgnoreIndices(IgnoreIndices.MISSING).execute().get()); + + final int numRanges = randomIntBetween(1, 20); + final double[][] ranges = new double[numRanges][]; + for (int i = 0; i < ranges.length; ++i) { + switch (randomInt(2)) { + case 0: + ranges[i] = new double[] { Double.NEGATIVE_INFINITY, randomInt(100) }; + break; + case 1: + ranges[i] = new double[] { randomInt(100), Double.POSITIVE_INFINITY }; + break; + case 2: + ranges[i] = new double[] { randomInt(100), randomInt(100) }; + break; + default: + throw new AssertionError(); + } + } + + RangeBuilder query = range("range").field("values"); + for (int i = 0; i < ranges.length; ++i) { + String key = Integer.toString(i); + if (ranges[i][0] == Double.NEGATIVE_INFINITY) { + query.addUnboundedTo(key, ranges[i][1]); + } else if (ranges[i][1] == Double.POSITIVE_INFINITY) { + query.addUnboundedFrom(key, ranges[i][0]); + } else { + query.addRange(key, ranges[i][0], ranges[i][1]); + } + } + + SearchRequestBuilder reqBuilder = client().prepareSearch("idx").addAggregation(query); + for (int i = 0; i < ranges.length; ++i) { + RangeFilterBuilder filter = FilterBuilders.rangeFilter("values"); + if (ranges[i][0] != Double.NEGATIVE_INFINITY) { + filter = filter.from(ranges[i][0]); + } + if (ranges[i][1] != Double.POSITIVE_INFINITY){ + filter = filter.to(ranges[i][1]); + } + reqBuilder = reqBuilder.addAggregation(filter("filter" + i).filter(filter)); + } + + SearchResponse resp = reqBuilder.execute().actionGet(); + Range range = resp.getAggregations().get("range"); + + for (int i = 0; i < ranges.length; ++i) { + + long count = 0; + for (double[] values : docs) { + for (double value : values) { + if (value >= ranges[i][0] && value < ranges[i][1]) { + ++count; + break; + } + } + } + + final Range.Bucket bucket = range.getByKey(Integer.toString(i)); + assertEquals(bucket.getKey(), count, bucket.getDocCount()); + + final Filter filter = resp.getAggregations().get("filter" + i); + assertThat(filter.getDocCount(), equalTo(count)); + } + } + + // test long/double/string terms aggs with high number of buckets that require array growth + public void testDuelTerms() throws Exception { + final int numDocs = atLeast(1000); + final int maxNumTerms = randomIntBetween(10, 10000); + + final IntOpenHashSet valuesSet = new IntOpenHashSet(); + wipeIndices("idx"); + prepareCreate("idx").addMapping("type", jsonBuilder().startObject() + .startObject("type") + .startObject("properties") + .startObject("string_values") + .field("type", "string") + .field("index", "not_analyzed") + .endObject() + .startObject("long_values") + .field("type", "long") + .endObject() + .startObject("double_values") + .field("type", "double") + .endObject() + .endObject() + .endObject()).execute().actionGet(); + for (int i = 0; i < numDocs; ++i) { + final int[] values = new int[randomInt(4)]; + for (int j = 0; j < values.length; ++j) { + values[j] = randomInt(maxNumTerms - 1) - 1000; + valuesSet.add(values[j]); + } + XContentBuilder source = jsonBuilder() + .startObject() + .field("num", randomDouble()) + .startArray("long_values"); + for (int j = 0; j < values.length; ++j) { + source = source.value(values[j]); + } + source = source.endArray().startArray("double_values"); + for (int j = 0; j < values.length; ++j) { + source = source.value((double) values[j]); + } + source = source.endArray().startArray("string_values"); + for (int j = 0; j < values.length; ++j) { + source = source.value(Integer.toString(values[j])); + } + source = source.endArray().endObject(); + client().prepareIndex("idx", "type").setSource(source).execute().actionGet(); + } + assertNoFailures(client().admin().indices().prepareRefresh("idx").setIgnoreIndices(IgnoreIndices.MISSING).execute().get()); + + SearchResponse resp = client().prepareSearch("idx") + .addAggregation(terms("long").field("long_values").size(maxNumTerms).subAggregation(min("min").field("num"))) + .addAggregation(terms("double").field("double_values").size(maxNumTerms).subAggregation(max("max").field("num"))) + .addAggregation(terms("string").field("string_values").size(maxNumTerms).subAggregation(stats("stats").field("num"))).execute().actionGet(); + assertEquals(0, resp.getFailedShards()); + + final Terms longTerms = resp.getAggregations().get("long"); + final Terms doubleTerms = resp.getAggregations().get("double"); + final Terms stringTerms = resp.getAggregations().get("string"); + + assertEquals(valuesSet.size(), longTerms.buckets().size()); + assertEquals(valuesSet.size(), doubleTerms.buckets().size()); + assertEquals(valuesSet.size(), stringTerms.buckets().size()); + for (Terms.Bucket bucket : longTerms.buckets()) { + final Terms.Bucket doubleBucket = doubleTerms.getByTerm(Double.toString(Long.parseLong(bucket.getKey().string()))); + final Terms.Bucket stringBucket = stringTerms.getByTerm(bucket.getKey().string()); + assertNotNull(doubleBucket); + assertNotNull(stringBucket); + assertEquals(bucket.getDocCount(), doubleBucket.getDocCount()); + assertEquals(bucket.getDocCount(), stringBucket.getDocCount()); + } + } + + // Duel between histograms and scripted terms + public void testDuelTermsHistogram() throws Exception { + createIndex("idx"); + + final int numDocs = atLeast(1000); + final int maxNumTerms = randomIntBetween(10, 2000); + final int interval = randomIntBetween(1, 100); + + final Integer[] values = new Integer[maxNumTerms]; + for (int i = 0; i < values.length; ++i) { + values[i] = randomInt(maxNumTerms * 3) - maxNumTerms; + } + + for (int i = 0; i < numDocs; ++i) { + XContentBuilder source = jsonBuilder() + .startObject() + .field("num", randomDouble()) + .startArray("values"); + final int numValues = randomInt(4); + for (int j = 0; j < numValues; ++j) { + source = source.value(randomFrom(values)); + } + source = source.endArray().endObject(); + client().prepareIndex("idx", "type").setSource(source).execute().actionGet(); + } + assertNoFailures(client().admin().indices().prepareRefresh("idx").setIgnoreIndices(IgnoreIndices.MISSING).execute().get()); + + SearchResponse resp = client().prepareSearch("idx") + .addAggregation(terms("terms").field("values").script("floor(_value / interval)").param("interval", interval).size(maxNumTerms)) + .addAggregation(histogram("histo").field("values").interval(interval)) + .execute().actionGet(); + + assertThat(resp.getFailedShards(), equalTo(0)); + + Terms terms = resp.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + Histogram histo = resp.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + assertThat(terms.buckets().size(), equalTo(histo.buckets().size())); + for (Terms.Bucket bucket : terms) { + final long key = bucket.getKeyAsNumber().longValue() * interval; + final Histogram.Bucket histoBucket = histo.getByKey(key); + assertEquals(bucket.getDocCount(), histoBucket.getDocCount()); + } + } + +} diff --git a/src/test/java/org/elasticsearch/search/aggregations/bucket/DateHistogramTests.java b/src/test/java/org/elasticsearch/search/aggregations/bucket/DateHistogramTests.java new file mode 100644 index 00000000000..a6e7fe697d5 --- /dev/null +++ b/src/test/java/org/elasticsearch/search/aggregations/bucket/DateHistogramTests.java @@ -0,0 +1,887 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket; + +import org.elasticsearch.action.index.IndexRequestBuilder; +import org.elasticsearch.action.search.SearchResponse; +import org.elasticsearch.common.settings.ImmutableSettings; +import org.elasticsearch.common.settings.Settings; +import org.elasticsearch.search.aggregations.bucket.histogram.DateHistogram; +import org.elasticsearch.search.aggregations.bucket.histogram.Histogram; +import org.elasticsearch.search.aggregations.metrics.max.Max; +import org.elasticsearch.search.aggregations.metrics.sum.Sum; +import org.elasticsearch.test.ElasticsearchIntegrationTest; +import org.hamcrest.Matchers; +import org.joda.time.DateTime; +import org.joda.time.DateTimeZone; +import org.junit.Before; +import org.junit.Test; + +import java.util.ArrayList; +import java.util.List; + +import static org.elasticsearch.common.xcontent.XContentFactory.jsonBuilder; +import static org.elasticsearch.index.query.QueryBuilders.matchAllQuery; +import static org.elasticsearch.search.aggregations.AggregationBuilders.*; +import static org.hamcrest.Matchers.equalTo; +import static org.hamcrest.Matchers.is; +import static org.hamcrest.core.IsNull.notNullValue; + +/** + * + */ +public class DateHistogramTests extends ElasticsearchIntegrationTest { + + @Override + public Settings indexSettings() { + return ImmutableSettings.builder() + .put("index.number_of_shards", between(1, 5)) + .put("index.number_of_replicas", between(0, 1)) + .build(); + } + + private DateTime date(int month, int day) { + return new DateTime(2012, month, day, 0, 0, DateTimeZone.UTC); + } + + private IndexRequestBuilder indexDoc(int month, int day, int value) throws Exception { + return client().prepareIndex("idx", "type").setSource(jsonBuilder() + .startObject() + .field("value", value) + .field("date", date(month, day)) + .startArray("dates").value(date(month, day)).value(date(month + 1, day + 1)).endArray() + .endObject()); + } + + @Before + public void init() throws Exception { + createIndex("idx"); + createIndex("idx_unmapped"); + // TODO: would be nice to have more random data here + indexRandom(true, + indexDoc(1, 2, 1), // date: Jan 2, dates: Jan 2, Feb 3 + indexDoc(2, 2, 2), // date: Feb 2, dates: Feb 2, Mar 3 + indexDoc(2, 15, 3), // date: Feb 15, dates: Feb 15, Mar 16 + indexDoc(3, 2, 4), // date: Mar 2, dates: Mar 2, Apr 3 + indexDoc(3, 15, 5), // date: Mar 15, dates: Mar 15, Apr 16 + indexDoc(3, 23, 6)); // date: Mar 23, dates: Mar 23, Apr 24 + } + + @Test + public void singleValuedField() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(dateHistogram("histo").field("date").interval(DateHistogram.Interval.MONTH)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + DateHistogram histo = response.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + assertThat(histo.getName(), equalTo("histo")); + assertThat(histo.buckets().size(), equalTo(3)); + + long key = new DateTime(2012, 1, 1, 0, 0, DateTimeZone.UTC).getMillis(); + DateHistogram.Bucket bucket = histo.getByKey(key); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo(key)); + assertThat(bucket.getDocCount(), equalTo(1l)); + + key = new DateTime(2012, 2, 1, 0, 0, DateTimeZone.UTC).getMillis(); + bucket = histo.getByKey(key); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo(key)); + assertThat(bucket.getDocCount(), equalTo(2l)); + + key = new DateTime(2012, 3, 1, 0, 0, DateTimeZone.UTC).getMillis(); + bucket = histo.getByKey(key); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo(key)); + assertThat(bucket.getDocCount(), equalTo(3l)); + } + + @Test + public void singleValuedField_OrderedByKeyAsc() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(dateHistogram("histo") + .field("date") + .interval(DateHistogram.Interval.MONTH) + .order(DateHistogram.Order.KEY_ASC)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + DateHistogram histo = response.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + assertThat(histo.getName(), equalTo("histo")); + assertThat(histo.buckets().size(), equalTo(3)); + + int i = 0; + for (DateHistogram.Bucket bucket : histo.buckets()) { + assertThat(bucket.getKey(), equalTo(new DateTime(2012, i+1, 1, 0, 0, DateTimeZone.UTC).getMillis())); + i++; + } + } + + @Test + public void singleValuedField_OrderedByKeyDesc() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(dateHistogram("histo") + .field("date") + .interval(DateHistogram.Interval.MONTH) + .order(DateHistogram.Order.KEY_DESC)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + DateHistogram histo = response.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + assertThat(histo.getName(), equalTo("histo")); + assertThat(histo.buckets().size(), equalTo(3)); + + int i = 2; + for (DateHistogram.Bucket bucket : histo.buckets()) { + assertThat(bucket.getKey(), equalTo(new DateTime(2012, i+1, 1, 0, 0, DateTimeZone.UTC).getMillis())); + i--; + } + } + + @Test + public void singleValuedField_OrderedByCountAsc() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(dateHistogram("histo") + .field("date") + .interval(DateHistogram.Interval.MONTH) + .order(DateHistogram.Order.COUNT_ASC)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + DateHistogram histo = response.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + assertThat(histo.getName(), equalTo("histo")); + assertThat(histo.buckets().size(), equalTo(3)); + + int i = 0; + for (DateHistogram.Bucket bucket : histo.buckets()) { + assertThat(bucket.getKey(), equalTo(new DateTime(2012, i+1, 1, 0, 0, DateTimeZone.UTC).getMillis())); + i++; + } + } + + @Test + public void singleValuedField_OrderedByCountDesc() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(dateHistogram("histo") + .field("date") + .interval(DateHistogram.Interval.MONTH) + .order(DateHistogram.Order.COUNT_DESC)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + DateHistogram histo = response.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + assertThat(histo.getName(), equalTo("histo")); + assertThat(histo.buckets().size(), equalTo(3)); + + int i = 2; + for (DateHistogram.Bucket bucket : histo.buckets()) { + assertThat(bucket.getKey(), equalTo(new DateTime(2012, i+1, 1, 0, 0, DateTimeZone.UTC).getMillis())); + i--; + } + } + + @Test + public void singleValuedField_WithSubAggregation() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(dateHistogram("histo").field("date").interval(DateHistogram.Interval.MONTH) + .subAggregation(sum("sum").field("value"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + DateHistogram histo = response.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + assertThat(histo.getName(), equalTo("histo")); + assertThat(histo.buckets().size(), equalTo(3)); + + long key = new DateTime(2012, 1, 1, 0, 0, DateTimeZone.UTC).getMillis(); + DateHistogram.Bucket bucket = histo.getByKey(key); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo(key)); + assertThat(bucket.getDocCount(), equalTo(1l)); + Sum sum = bucket.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + assertThat(sum.getValue(), equalTo(1.0)); + + key = new DateTime(2012, 2, 1, 0, 0, DateTimeZone.UTC).getMillis(); + bucket = histo.getByKey(key); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo(key)); + assertThat(bucket.getDocCount(), equalTo(2l)); + sum = bucket.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + assertThat(sum.getValue(), equalTo(5.0)); + + key = new DateTime(2012, 3, 1, 0, 0, DateTimeZone.UTC).getMillis(); + bucket = histo.getByKey(key); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo(key)); + assertThat(bucket.getDocCount(), equalTo(3l)); + sum = bucket.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + assertThat(sum.getValue(), equalTo(15.0)); + } + + @Test + public void singleValuedField_WithSubAggregation_Inherited() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(dateHistogram("histo").field("date").interval(DateHistogram.Interval.MONTH) + .subAggregation(max("max"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + DateHistogram histo = response.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + assertThat(histo.getName(), equalTo("histo")); + assertThat(histo.buckets().size(), equalTo(3)); + + long key = new DateTime(2012, 1, 1, 0, 0, DateTimeZone.UTC).getMillis(); + DateHistogram.Bucket bucket = histo.getByKey(key); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo(key)); + assertThat(bucket.getDocCount(), equalTo(1l)); + Max max = bucket.getAggregations().get("max"); + assertThat(max, notNullValue()); + assertThat(max.getValue(), equalTo((double) new DateTime(2012, 1, 2, 0, 0, DateTimeZone.UTC).getMillis())); + + key = new DateTime(2012, 2, 1, 0, 0, DateTimeZone.UTC).getMillis(); + bucket = histo.getByKey(key); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo(key)); + assertThat(bucket.getDocCount(), equalTo(2l)); + max = bucket.getAggregations().get("max"); + assertThat(max, notNullValue()); + assertThat(max.getValue(), equalTo((double) new DateTime(2012, 2, 15, 0, 0, DateTimeZone.UTC).getMillis())); + + key = new DateTime(2012, 3, 1, 0, 0, DateTimeZone.UTC).getMillis(); + bucket = histo.getByKey(key); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo(key)); + assertThat(bucket.getDocCount(), equalTo(3l)); + max = bucket.getAggregations().get("max"); + assertThat(max, notNullValue()); + assertThat(max.getValue(), equalTo((double) new DateTime(2012, 3, 23, 0, 0, DateTimeZone.UTC).getMillis())); + } + + @Test + public void singleValuedField_OrderedBySubAggregationAsc() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(dateHistogram("histo") + .field("date") + .interval(DateHistogram.Interval.MONTH) + .order(DateHistogram.Order.aggregation("sum", true)) + .subAggregation(max("sum").field("value"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + DateHistogram histo = response.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + assertThat(histo.getName(), equalTo("histo")); + assertThat(histo.buckets().size(), equalTo(3)); + + int i = 0; + for (DateHistogram.Bucket bucket : histo.buckets()) { + assertThat(bucket.getKey(), equalTo(new DateTime(2012, i+1, 1, 0, 0, DateTimeZone.UTC).getMillis())); + i++; + } + } + + @Test + public void singleValuedField_OrderedBySubAggregationDesc() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(dateHistogram("histo") + .field("date") + .interval(DateHistogram.Interval.MONTH) + .order(DateHistogram.Order.aggregation("sum", false)) + .subAggregation(max("sum").field("value"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + DateHistogram histo = response.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + assertThat(histo.getName(), equalTo("histo")); + assertThat(histo.buckets().size(), equalTo(3)); + + int i = 2; + for (DateHistogram.Bucket bucket : histo.buckets()) { + assertThat(bucket.getKey(), equalTo(new DateTime(2012, i+1, 1, 0, 0, DateTimeZone.UTC).getMillis())); + i--; + } + } + + @Test + public void singleValuedField_OrderedByMultiValuedSubAggregationAsc_Inherited() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(dateHistogram("histo") + .field("date") + .interval(DateHistogram.Interval.MONTH) + .order(DateHistogram.Order.aggregation("stats", "sum", true)) + .subAggregation(stats("stats").field("value"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + DateHistogram histo = response.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + assertThat(histo.getName(), equalTo("histo")); + assertThat(histo.buckets().size(), equalTo(3)); + + int i = 0; + for (DateHistogram.Bucket bucket : histo.buckets()) { + assertThat(bucket.getKey(), equalTo(new DateTime(2012, i+1, 1, 0, 0, DateTimeZone.UTC).getMillis())); + i++; + } + } + + @Test + public void singleValuedField_OrderedByMultiValuedSubAggregationDesc() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(dateHistogram("histo") + .field("date") + .interval(DateHistogram.Interval.MONTH) + .order(DateHistogram.Order.aggregation("stats", "sum", false)) + .subAggregation(stats("stats").field("value"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + DateHistogram histo = response.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + assertThat(histo.getName(), equalTo("histo")); + assertThat(histo.buckets().size(), equalTo(3)); + + int i = 2; + for (DateHistogram.Bucket bucket : histo.buckets()) { + assertThat(bucket.getKey(), equalTo(new DateTime(2012, i+1, 1, 0, 0, DateTimeZone.UTC).getMillis())); + i--; + } + } + + @Test + public void singleValuedField_WithValueScript() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(dateHistogram("histo") + .field("date") + .script("new DateTime(_value).plusMonths(1).getMillis()") + .interval(DateHistogram.Interval.MONTH)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + DateHistogram histo = response.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + assertThat(histo.getName(), equalTo("histo")); + + assertThat(histo.buckets().size(), equalTo(3)); + + long key = new DateTime(2012, 2, 1, 0, 0, DateTimeZone.UTC).getMillis(); + DateHistogram.Bucket bucket = histo.getByKey(key); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo(key)); + assertThat(bucket.getDocCount(), equalTo(1l)); + + key = new DateTime(2012, 3, 1, 0, 0, DateTimeZone.UTC).getMillis(); + bucket = histo.getByKey(key); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo(key)); + assertThat(bucket.getDocCount(), equalTo(2l)); + + key = new DateTime(2012, 4, 1, 0, 0, DateTimeZone.UTC).getMillis(); + bucket = histo.getByKey(key); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo(key)); + assertThat(bucket.getDocCount(), equalTo(3l)); + } + + /* + [ Jan 2, Feb 3] + [ Feb 2, Mar 3] + [ Feb 15, Mar 16] + [ Mar 2, Apr 3] + [ Mar 15, Apr 16] + [ Mar 23, Apr 24] + */ + + @Test + public void multiValuedField() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(dateHistogram("histo").field("dates").interval(DateHistogram.Interval.MONTH)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + DateHistogram histo = response.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + assertThat(histo.getName(), equalTo("histo")); + assertThat(histo.buckets().size(), equalTo(4)); + + long key = new DateTime(2012, 1, 1, 0, 0, DateTimeZone.UTC).getMillis(); + DateHistogram.Bucket bucket = histo.getByKey(key); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo(key)); + assertThat(bucket.getDocCount(), equalTo(1l)); + + key = new DateTime(2012, 2, 1, 0, 0, DateTimeZone.UTC).getMillis(); + bucket = histo.getByKey(key); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo(key)); + assertThat(bucket.getDocCount(), equalTo(3l)); + + key = new DateTime(2012, 3, 1, 0, 0, DateTimeZone.UTC).getMillis(); + bucket = histo.getByKey(key); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo(key)); + assertThat(bucket.getDocCount(), equalTo(5l)); + + key = new DateTime(2012, 4, 1, 0, 0, DateTimeZone.UTC).getMillis(); + bucket = histo.getByKey(key); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo(key)); + assertThat(bucket.getDocCount(), equalTo(3l)); + } + + @Test + public void multiValuedField_OrderedByKeyDesc() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(dateHistogram("histo") + .field("dates") + .interval(DateHistogram.Interval.MONTH) + .order(DateHistogram.Order.COUNT_DESC)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + DateHistogram histo = response.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + assertThat(histo.getName(), equalTo("histo")); + assertThat(histo.buckets().size(), equalTo(4)); + + DateHistogram.Bucket bucket = histo.buckets().get(0); + assertThat(bucket, notNullValue()); + assertThat(bucket.getDocCount(), equalTo(5l)); + + bucket = histo.buckets().get(1); + assertThat(bucket, notNullValue()); + assertThat(bucket.getDocCount(), equalTo(3l)); + + bucket = histo.buckets().get(2); + assertThat(bucket, notNullValue()); + assertThat(bucket.getDocCount(), equalTo(3l)); + + bucket = histo.buckets().get(3); + assertThat(bucket, notNullValue()); + assertThat(bucket.getDocCount(), equalTo(1l)); + } + + + /** + * The script will change to document date values to the following: + * + * doc 1: [ Feb 2, Mar 3] + * doc 2: [ Mar 2, Apr 3] + * doc 3: [ Mar 15, Apr 16] + * doc 4: [ Apr 2, May 3] + * doc 5: [ Apr 15, May 16] + * doc 6: [ Apr 23, May 24] + */ + @Test + public void multiValuedField_WithValueScript() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(dateHistogram("histo") + .field("dates") + .script("new DateTime(_value, DateTimeZone.UTC).plusMonths(1).getMillis()") + .interval(DateHistogram.Interval.MONTH)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + DateHistogram histo = response.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + assertThat(histo.getName(), equalTo("histo")); + assertThat(histo.buckets().size(), equalTo(4)); + + long key = new DateTime(2012, 2, 1, 0, 0, DateTimeZone.UTC).getMillis(); + DateHistogram.Bucket bucket = histo.getByKey(key); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo(key)); + assertThat(bucket.getDocCount(), equalTo(1l)); + + key = new DateTime(2012, 3, 1, 0, 0, DateTimeZone.UTC).getMillis(); + bucket = histo.getByKey(key); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo(key)); + assertThat(bucket.getDocCount(), equalTo(3l)); + + key = new DateTime(2012, 4, 1, 0, 0, DateTimeZone.UTC).getMillis(); + bucket = histo.getByKey(key); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo(key)); + assertThat(bucket.getDocCount(), equalTo(5l)); + + key = new DateTime(2012, 5, 1, 0, 0, DateTimeZone.UTC).getMillis(); + bucket = histo.getByKey(key); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo(key)); + assertThat(bucket.getDocCount(), equalTo(3l)); + } + + /** + * The script will change to document date values to the following: + * + * doc 1: [ Feb 2, Mar 3] + * doc 2: [ Mar 2, Apr 3] + * doc 3: [ Mar 15, Apr 16] + * doc 4: [ Apr 2, May 3] + * doc 5: [ Apr 15, May 16] + * doc 6: [ Apr 23, May 24] + * + */ + @Test + public void multiValuedField_WithValueScript_WithInheritedSubAggregator() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(dateHistogram("histo") + .field("dates") + .script("new DateTime(_value, DateTimeZone.UTC).plusMonths(1).getMillis()") + .interval(DateHistogram.Interval.MONTH) + .subAggregation(max("max"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + DateHistogram histo = response.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + assertThat(histo.getName(), equalTo("histo")); + assertThat(histo.buckets().size(), equalTo(4)); + + long key = new DateTime(2012, 2, 1, 0, 0, DateTimeZone.UTC).getMillis(); + DateHistogram.Bucket bucket = histo.getByKey(key); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo(key)); + assertThat(bucket.getDocCount(), equalTo(1l)); + Max max = bucket.getAggregations().get("max"); + assertThat(max, notNullValue()); + assertThat((long) max.getValue(), equalTo(new DateTime(2012, 3, 3, 0, 0, DateTimeZone.UTC).getMillis())); + + key = new DateTime(2012, 3, 1, 0, 0, DateTimeZone.UTC).getMillis(); + bucket = histo.getByKey(key); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo(key)); + assertThat(bucket.getDocCount(), equalTo(3l)); + max = bucket.getAggregations().get("max"); + assertThat(max, notNullValue()); + assertThat((long) max.getValue(), equalTo(new DateTime(2012, 4, 16, 0, 0, DateTimeZone.UTC).getMillis())); + + key = new DateTime(2012, 4, 1, 0, 0, DateTimeZone.UTC).getMillis(); + bucket = histo.getByKey(key); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo(key)); + assertThat(bucket.getDocCount(), equalTo(5l)); + max = bucket.getAggregations().get("max"); + assertThat(max, notNullValue()); + assertThat((long) max.getValue(), equalTo(new DateTime(2012, 5, 24, 0, 0, DateTimeZone.UTC).getMillis())); + + key = new DateTime(2012, 5, 1, 0, 0, DateTimeZone.UTC).getMillis(); + bucket = histo.getByKey(key); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo(key)); + assertThat(bucket.getDocCount(), equalTo(3l)); + max = bucket.getAggregations().get("max"); + assertThat(max, notNullValue()); + assertThat((long) max.getValue(), equalTo(new DateTime(2012, 5, 24, 0, 0, DateTimeZone.UTC).getMillis())); + } + + /** + * Jan 2 + * Feb 2 + * Feb 15 + * Mar 2 + * Mar 15 + * Mar 23 + */ + @Test + public void script_SingleValue() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(dateHistogram("histo").script("doc['date'].value").interval(DateHistogram.Interval.MONTH)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + DateHistogram histo = response.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + assertThat(histo.getName(), equalTo("histo")); + assertThat(histo.buckets().size(), equalTo(3)); + + long key = new DateTime(2012, 1, 1, 0, 0, DateTimeZone.UTC).getMillis(); + DateHistogram.Bucket bucket = histo.getByKey(key); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo(key)); + assertThat(bucket.getDocCount(), equalTo(1l)); + + key = new DateTime(2012, 2, 1, 0, 0, DateTimeZone.UTC).getMillis(); + bucket = histo.getByKey(key); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo(key)); + assertThat(bucket.getDocCount(), equalTo(2l)); + + key = new DateTime(2012, 3, 1, 0, 0, DateTimeZone.UTC).getMillis(); + bucket = histo.getByKey(key); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo(key)); + assertThat(bucket.getDocCount(), equalTo(3l)); + } + + @Test + public void script_SingleValue_WithSubAggregator_Inherited() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(dateHistogram("histo") + .script("doc['date'].value") + .interval(DateHistogram.Interval.MONTH) + .subAggregation(max("max"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + DateHistogram histo = response.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + assertThat(histo.getName(), equalTo("histo")); + assertThat(histo.buckets().size(), equalTo(3)); + + long key = new DateTime(2012, 1, 1, 0, 0, DateTimeZone.UTC).getMillis(); + DateHistogram.Bucket bucket = histo.getByKey(key); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo(key)); + assertThat(bucket.getDocCount(), equalTo(1l)); + Max max = bucket.getAggregations().get("max"); + assertThat(max, notNullValue()); + assertThat(max.getValue(), equalTo((double) new DateTime(2012, 1, 2, 0, 0, DateTimeZone.UTC).getMillis())); + + key = new DateTime(2012, 2, 1, 0, 0, DateTimeZone.UTC).getMillis(); + bucket = histo.getByKey(key); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo(key)); + assertThat(bucket.getDocCount(), equalTo(2l)); + max = bucket.getAggregations().get("max"); + assertThat(max, notNullValue()); + assertThat(max.getValue(), equalTo((double) new DateTime(2012, 2, 15, 0, 0, DateTimeZone.UTC).getMillis())); + + key = new DateTime(2012, 3, 1, 0, 0, DateTimeZone.UTC).getMillis(); + bucket = histo.getByKey(key); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo(key)); + assertThat(bucket.getDocCount(), equalTo(3l)); + max = bucket.getAggregations().get("max"); + assertThat(max, notNullValue()); + assertThat(max.getValue(), equalTo((double) new DateTime(2012, 3, 23, 0, 0, DateTimeZone.UTC).getMillis())); + } + + @Test + public void script_MultiValued() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(dateHistogram("histo").script("doc['dates'].values").interval(DateHistogram.Interval.MONTH)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + DateHistogram histo = response.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + assertThat(histo.getName(), equalTo("histo")); + assertThat(histo.buckets().size(), equalTo(4)); + + long key = new DateTime(2012, 1, 1, 0, 0, DateTimeZone.UTC).getMillis(); + DateHistogram.Bucket bucket = histo.getByKey(key); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo(key)); + assertThat(bucket.getDocCount(), equalTo(1l)); + + key = new DateTime(2012, 2, 1, 0, 0, DateTimeZone.UTC).getMillis(); + bucket = histo.getByKey(key); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo(key)); + assertThat(bucket.getDocCount(), equalTo(3l)); + + key = new DateTime(2012, 3, 1, 0, 0, DateTimeZone.UTC).getMillis(); + bucket = histo.getByKey(key); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo(key)); + assertThat(bucket.getDocCount(), equalTo(5l)); + + key = new DateTime(2012, 4, 1, 0, 0, DateTimeZone.UTC).getMillis(); + bucket = histo.getByKey(key); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo(key)); + assertThat(bucket.getDocCount(), equalTo(3l)); + } + + /* + [ Jan 2, Feb 3] + [ Feb 2, Mar 3] + [ Feb 15, Mar 16] + [ Mar 2, Apr 3] + [ Mar 15, Apr 16] + [ Mar 23, Apr 24] + */ + + @Test + public void script_MultiValued_WithAggregatorInherited() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(dateHistogram("histo") + .script("doc['dates'].values") + .interval(DateHistogram.Interval.MONTH) + .subAggregation(max("max"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + DateHistogram histo = response.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + assertThat(histo.getName(), equalTo("histo")); + assertThat(histo.buckets().size(), equalTo(4)); + + long key = new DateTime(2012, 1, 1, 0, 0, DateTimeZone.UTC).getMillis(); + DateHistogram.Bucket bucket = histo.getByKey(key); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo(key)); + assertThat(bucket.getDocCount(), equalTo(1l)); + Max max = bucket.getAggregations().get("max"); + assertThat(max, notNullValue()); + assertThat((long) max.getValue(), equalTo(new DateTime(2012, 2, 3, 0, 0, DateTimeZone.UTC).getMillis())); + + key = new DateTime(2012, 2, 1, 0, 0, DateTimeZone.UTC).getMillis(); + bucket = histo.getByKey(key); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo(key)); + assertThat(bucket.getDocCount(), equalTo(3l)); + max = bucket.getAggregations().get("max"); + assertThat(max, notNullValue()); + assertThat((long) max.getValue(), equalTo(new DateTime(2012, 3, 16, 0, 0, DateTimeZone.UTC).getMillis())); + + key = new DateTime(2012, 3, 1, 0, 0, DateTimeZone.UTC).getMillis(); + bucket = histo.getByKey(key); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo(key)); + assertThat(bucket.getDocCount(), equalTo(5l)); + max = bucket.getAggregations().get("max"); + assertThat(max, notNullValue()); + assertThat((long) max.getValue(), equalTo(new DateTime(2012, 4, 24, 0, 0, DateTimeZone.UTC).getMillis())); + + key = new DateTime(2012, 4, 1, 0, 0, DateTimeZone.UTC).getMillis(); + bucket = histo.getByKey(key); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo(key)); + assertThat(bucket.getDocCount(), equalTo(3l)); + max = bucket.getAggregations().get("max"); + assertThat(max, notNullValue()); + assertThat((long) max.getValue(), equalTo(new DateTime(2012, 4, 24, 0, 0, DateTimeZone.UTC).getMillis())); + } + + @Test + public void unmapped() throws Exception { + client().admin().cluster().prepareHealth("idx_unmapped").setWaitForYellowStatus().execute().actionGet(); + + SearchResponse response = client().prepareSearch("idx_unmapped") + .addAggregation(dateHistogram("histo").field("date").interval(DateHistogram.Interval.MONTH)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + DateHistogram histo = response.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + assertThat(histo.getName(), equalTo("histo")); + assertThat(histo.buckets().size(), equalTo(0)); + } + + @Test + public void partiallyUnmapped() throws Exception { + client().admin().cluster().prepareHealth("idx_unmapped").setWaitForGreenStatus().execute().actionGet(); + + SearchResponse response = client().prepareSearch("idx", "idx_unmapped") + .addAggregation(dateHistogram("histo").field("date").interval(DateHistogram.Interval.MONTH)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + DateHistogram histo = response.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + assertThat(histo.getName(), equalTo("histo")); + assertThat(histo.buckets().size(), equalTo(3)); + + long key = new DateTime(2012, 1, 1, 0, 0, DateTimeZone.UTC).getMillis(); + DateHistogram.Bucket bucket = histo.getByKey(key); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo(key)); + assertThat(bucket.getDocCount(), equalTo(1l)); + + key = new DateTime(2012, 2, 1, 0, 0, DateTimeZone.UTC).getMillis(); + bucket = histo.getByKey(key); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo(key)); + assertThat(bucket.getDocCount(), equalTo(2l)); + + key = new DateTime(2012, 3, 1, 0, 0, DateTimeZone.UTC).getMillis(); + bucket = histo.getByKey(key); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo(key)); + assertThat(bucket.getDocCount(), equalTo(3l)); + } + + @Test + public void emptyAggregation() throws Exception { + prepareCreate("empty_bucket_idx").addMapping("type", "value", "type=integer").execute().actionGet(); + List builders = new ArrayList(); + for (int i = 0; i < 2; i++) { + builders.add(client().prepareIndex("empty_bucket_idx", "type", ""+i).setSource(jsonBuilder() + .startObject() + .field("value", i*2) + .endObject())); + } + indexRandom(true, builders.toArray(new IndexRequestBuilder[builders.size()])); + + SearchResponse searchResponse = client().prepareSearch("empty_bucket_idx") + .setQuery(matchAllQuery()) + .addAggregation(histogram("histo").field("value").interval(1l).emptyBuckets(true).subAggregation(dateHistogram("date_histo").interval(1))) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(2l)); + Histogram histo = searchResponse.getAggregations().get("histo"); + assertThat(histo, Matchers.notNullValue()); + Histogram.Bucket bucket = histo.getByKey(1l); + assertThat(bucket, Matchers.notNullValue()); + + DateHistogram dateHisto = bucket.getAggregations().get("date_histo"); + assertThat(dateHisto, Matchers.notNullValue()); + assertThat(dateHisto.getName(), equalTo("date_histo")); + assertThat(dateHisto.buckets().isEmpty(), is(true)); + + } +} diff --git a/src/test/java/org/elasticsearch/search/aggregations/bucket/DateRangeTests.java b/src/test/java/org/elasticsearch/search/aggregations/bucket/DateRangeTests.java new file mode 100644 index 00000000000..27bcb4e6a49 --- /dev/null +++ b/src/test/java/org/elasticsearch/search/aggregations/bucket/DateRangeTests.java @@ -0,0 +1,1005 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket; + +import org.elasticsearch.action.index.IndexRequestBuilder; +import org.elasticsearch.action.search.SearchResponse; +import org.elasticsearch.common.settings.ImmutableSettings; +import org.elasticsearch.common.settings.Settings; +import org.elasticsearch.search.aggregations.bucket.histogram.Histogram; +import org.elasticsearch.search.aggregations.bucket.range.date.DateRange; +import org.elasticsearch.search.aggregations.metrics.max.Max; +import org.elasticsearch.search.aggregations.metrics.min.Min; +import org.elasticsearch.search.aggregations.metrics.sum.Sum; +import org.elasticsearch.test.ElasticsearchIntegrationTest; +import org.hamcrest.Matchers; +import org.joda.time.DateTime; +import org.joda.time.DateTimeZone; +import org.junit.Before; +import org.junit.Test; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; + +import static org.elasticsearch.common.xcontent.XContentFactory.jsonBuilder; +import static org.elasticsearch.index.query.QueryBuilders.matchAllQuery; +import static org.elasticsearch.search.aggregations.AggregationBuilders.*; +import static org.hamcrest.Matchers.equalTo; +import static org.hamcrest.Matchers.is; +import static org.hamcrest.core.IsNull.notNullValue; +import static org.hamcrest.core.IsNull.nullValue; + +/** + * + */ +public class DateRangeTests extends ElasticsearchIntegrationTest { + + @Override + public Settings indexSettings() { + return ImmutableSettings.builder() + .put("index.number_of_shards", between(1, 5)) + .put("index.number_of_replicas", between(0, 1)) + .build(); + } + + private static DateTime date(int month, int day) { + return new DateTime(2012, month, day, 0, 0, DateTimeZone.UTC); + } + + private static IndexRequestBuilder indexDoc(int month, int day, int value) throws Exception { + return client().prepareIndex("idx", "type").setSource(jsonBuilder() + .startObject() + .field("value", value) + .field("date", date(month, day)) + .startArray("dates").value(date(month, day)).value(date(month + 1, day + 1)).endArray() + .endObject()); + } + + int numDocs; + + @Before + public void init() throws Exception { + createIndex("idx"); + createIndex("idx_unmapped"); + + numDocs = randomIntBetween(7, 20); + + List docs = new ArrayList(); + docs.addAll(Arrays.asList( + indexDoc(1, 2, 1), // Jan 2 + indexDoc(2, 2, 2), // Feb 2 + indexDoc(2, 15, 3), // Feb 15 + indexDoc(3, 2, 4), // Mar 2 + indexDoc(3, 15, 5), // Mar 15 + indexDoc(3, 23, 6))); // Mar 23 + + // dummy docs + for (int i = docs.size(); i < numDocs; ++i) { + docs.add(indexDoc(randomIntBetween(6, 10), randomIntBetween(1, 20), randomInt(100))); + } + + indexRandom(true, docs); + } + + @Test + public void singleValueField() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(dateRange("range") + .field("date") + .addUnboundedTo(date(2, 15)) + .addRange(date(2, 15), date(3, 15)) + .addUnboundedFrom(date(3, 15))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + DateRange range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(3)); + + DateRange.Bucket bucket = range.getByKey("*-2012-02-15T00:00:00.000Z"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("*-2012-02-15T00:00:00.000Z")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getFromAsDate(), nullValue()); + assertThat(bucket.getTo(), equalTo((double) date(2, 15).getMillis())); + assertThat(bucket.getToAsDate(), equalTo(date(2, 15))); + assertThat(bucket.getDocCount(), equalTo(2l)); + + bucket = range.getByKey("2012-02-15T00:00:00.000Z-2012-03-15T00:00:00.000Z"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("2012-02-15T00:00:00.000Z-2012-03-15T00:00:00.000Z")); + assertThat(bucket.getFrom(), equalTo((double) date(2, 15).getMillis())); + assertThat(bucket.getFromAsDate(), equalTo(date(2, 15))); + assertThat(bucket.getTo(), equalTo((double) date(3, 15).getMillis())); + assertThat(bucket.getToAsDate(), equalTo(date(3, 15))); + assertThat(bucket.getDocCount(), equalTo(2l)); + + bucket = range.getByKey("2012-03-15T00:00:00.000Z-*"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("2012-03-15T00:00:00.000Z-*")); + assertThat(bucket.getFrom(), equalTo((double) date(3, 15).getMillis())); + assertThat(bucket.getFromAsDate(), equalTo(date(3, 15))); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getToAsDate(), nullValue()); + assertThat(bucket.getDocCount(), equalTo(numDocs - 4l)); + } + + @Test + public void singleValueField_WithStringDates() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(dateRange("range") + .field("date") + .addUnboundedTo("2012-02-15") + .addRange("2012-02-15", "2012-03-15") + .addUnboundedFrom("2012-03-15")) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + DateRange range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(3)); + + DateRange.Bucket bucket = range.getByKey("*-2012-02-15T00:00:00.000Z"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("*-2012-02-15T00:00:00.000Z")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getFromAsDate(), nullValue()); + assertThat(bucket.getTo(), equalTo((double) date(2, 15).getMillis())); + assertThat(bucket.getToAsDate(), equalTo(date(2, 15))); + assertThat(bucket.getDocCount(), equalTo(2l)); + + bucket = range.getByKey("2012-02-15T00:00:00.000Z-2012-03-15T00:00:00.000Z"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("2012-02-15T00:00:00.000Z-2012-03-15T00:00:00.000Z")); + assertThat(bucket.getFrom(), equalTo((double) date(2, 15).getMillis())); + assertThat(bucket.getFromAsDate(), equalTo(date(2, 15))); + assertThat(bucket.getTo(), equalTo((double) date(3, 15).getMillis())); + assertThat(bucket.getToAsDate(), equalTo(date(3, 15))); + assertThat(bucket.getDocCount(), equalTo(2l)); + + bucket = range.getByKey("2012-03-15T00:00:00.000Z-*"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("2012-03-15T00:00:00.000Z-*")); + assertThat(bucket.getFrom(), equalTo((double) date(3, 15).getMillis())); + assertThat(bucket.getFromAsDate(), equalTo(date(3, 15))); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getToAsDate(), nullValue()); + assertThat(bucket.getDocCount(), equalTo(numDocs - 4l)); + } + + @Test + public void singleValueField_WithStringDates_WithCustomFormat() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(dateRange("range") + .field("date") + .format("yyyy-MM-dd") + .addUnboundedTo("2012-02-15") + .addRange("2012-02-15", "2012-03-15") + .addUnboundedFrom("2012-03-15")) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + DateRange range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(3)); + + DateRange.Bucket bucket = range.getByKey("*-2012-02-15"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("*-2012-02-15")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getFromAsDate(), nullValue()); + assertThat(bucket.getTo(), equalTo((double) date(2, 15).getMillis())); + assertThat(bucket.getToAsDate(), equalTo(date(2, 15))); + assertThat(bucket.getDocCount(), equalTo(2l)); + + bucket = range.getByKey("2012-02-15-2012-03-15"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("2012-02-15-2012-03-15")); + assertThat(bucket.getFrom(), equalTo((double) date(2, 15).getMillis())); + assertThat(bucket.getFromAsDate(), equalTo(date(2, 15))); + assertThat(bucket.getTo(), equalTo((double) date(3, 15).getMillis())); + assertThat(bucket.getToAsDate(), equalTo(date(3, 15))); + assertThat(bucket.getDocCount(), equalTo(2l)); + + bucket = range.getByKey("2012-03-15-*"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("2012-03-15-*")); + assertThat(bucket.getFrom(), equalTo((double) date(3, 15).getMillis())); + assertThat(bucket.getFromAsDate(), equalTo(date(3, 15))); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getToAsDate(), nullValue()); + assertThat(bucket.getDocCount(), equalTo(numDocs - 4l)); + } + + @Test + public void singleValueField_WithDateMath() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(dateRange("range") + .field("date") + .addUnboundedTo("2012-02-15") + .addRange("2012-02-15", "2012-02-15||+1M") + .addUnboundedFrom("2012-02-15||+1M")) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + DateRange range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(3)); + + DateRange.Bucket bucket = range.getByKey("*-2012-02-15T00:00:00.000Z"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("*-2012-02-15T00:00:00.000Z")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getFromAsDate(), nullValue()); + assertThat(bucket.getTo(), equalTo((double) date(2, 15).getMillis())); + assertThat(bucket.getToAsDate(), equalTo(date(2, 15))); + assertThat(bucket.getDocCount(), equalTo(2l)); + + bucket = range.getByKey("2012-02-15T00:00:00.000Z-2012-03-15T00:00:00.000Z"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("2012-02-15T00:00:00.000Z-2012-03-15T00:00:00.000Z")); + assertThat(bucket.getFrom(), equalTo((double) date(2, 15).getMillis())); + assertThat(bucket.getFromAsDate(), equalTo(date(2, 15))); + assertThat(bucket.getTo(), equalTo((double) date(3, 15).getMillis())); + assertThat(bucket.getToAsDate(), equalTo(date(3, 15))); + assertThat(bucket.getDocCount(), equalTo(2l)); + + bucket = range.getByKey("2012-03-15T00:00:00.000Z-*"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("2012-03-15T00:00:00.000Z-*")); + assertThat(bucket.getFrom(), equalTo((double) date(3, 15).getMillis())); + assertThat(bucket.getFromAsDate(), equalTo(date(3, 15))); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getToAsDate(), nullValue()); + assertThat(bucket.getDocCount(), equalTo(numDocs - 4l)); + } + + @Test + public void singleValueField_WithCustomKey() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(dateRange("range") + .field("date") + .addUnboundedTo("r1", date(2, 15)) + .addRange("r2", date(2, 15), date(3, 15)) + .addUnboundedFrom("r3", date(3, 15))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + DateRange range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(3)); + + DateRange.Bucket bucket = range.getByKey("r1"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("r1")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getFromAsDate(), nullValue()); + assertThat(bucket.getTo(), equalTo((double) date(2, 15).getMillis())); + assertThat(bucket.getToAsDate(), equalTo(date(2, 15))); + assertThat(bucket.getDocCount(), equalTo(2l)); + + bucket = range.getByKey("r2"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("r2")); + assertThat(bucket.getFrom(), equalTo((double) date(2, 15).getMillis())); + assertThat(bucket.getFromAsDate(), equalTo(date(2, 15))); + assertThat(bucket.getTo(), equalTo((double) date(3, 15).getMillis())); + assertThat(bucket.getToAsDate(), equalTo(date(3, 15))); + assertThat(bucket.getDocCount(), equalTo(2l)); + + bucket = range.getByKey("r3"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("r3")); + assertThat(bucket.getFrom(), equalTo((double) date(3, 15).getMillis())); + assertThat(bucket.getFromAsDate(), equalTo(date(3, 15))); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getToAsDate(), nullValue()); + assertThat(bucket.getDocCount(), equalTo(numDocs - 4l)); + } + + /* + Jan 2, 1 + Feb 2, 2 + Feb 15, 3 + Mar 2, 4 + Mar 15, 5 + Mar 23, 6 + */ + + @Test + public void singleValuedField_WithSubAggregation() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(dateRange("range") + .field("date") + .addUnboundedTo("r1", date(2, 15)) + .addRange("r2", date(2, 15), date(3, 15)) + .addUnboundedFrom("r3", date(3, 15)) + .subAggregation(sum("sum").field("value"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + DateRange range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(3)); + + DateRange.Bucket bucket = range.getByKey("r1"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("r1")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getFromAsDate(), nullValue()); + assertThat(bucket.getTo(), equalTo((double) date(2, 15).getMillis())); + assertThat(bucket.getToAsDate(), equalTo(date(2, 15))); + assertThat(bucket.getDocCount(), equalTo(2l)); + Sum sum = bucket.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + assertThat(sum.getValue(), equalTo((double) 1 + 2)); + + bucket = range.getByKey("r2"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("r2")); + assertThat(bucket.getFrom(), equalTo((double) date(2, 15).getMillis())); + assertThat(bucket.getFromAsDate(), equalTo(date(2, 15))); + assertThat(bucket.getTo(), equalTo((double) date(3, 15).getMillis())); + assertThat(bucket.getToAsDate(), equalTo(date(3, 15))); + assertThat(bucket.getDocCount(), equalTo(2l)); + sum = bucket.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + assertThat(sum.getValue(), equalTo((double) 3 + 4)); + + bucket = range.getByKey("r3"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("r3")); + assertThat(bucket.getFrom(), equalTo((double) date(3, 15).getMillis())); + assertThat(bucket.getFromAsDate(), equalTo(date(3, 15))); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getToAsDate(), nullValue()); + assertThat(bucket.getDocCount(), equalTo(numDocs - 4l)); + sum = bucket.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + } + + @Test + public void singleValuedField_WithSubAggregation_Inherited() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(dateRange("range") + .field("date") + .addUnboundedTo("r1", date(2, 15)) + .addRange("r2", date(2, 15), date(3, 15)) + .addUnboundedFrom("r3", date(3, 15)) + .subAggregation(min("min"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + DateRange range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(3)); + + DateRange.Bucket bucket = range.getByKey("r1"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("r1")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getFromAsDate(), nullValue()); + assertThat(bucket.getTo(), equalTo((double) date(2, 15).getMillis())); + assertThat(bucket.getToAsDate(), equalTo(date(2, 15))); + assertThat(bucket.getDocCount(), equalTo(2l)); + Min min = bucket.getAggregations().get("min"); + assertThat(min, notNullValue()); + assertThat(min.getValue(), equalTo((double) date(1, 2).getMillis())); + + bucket = range.getByKey("r2"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("r2")); + assertThat(bucket.getFrom(), equalTo((double) date(2, 15).getMillis())); + assertThat(bucket.getFromAsDate(), equalTo(date(2, 15))); + assertThat(bucket.getTo(), equalTo((double) date(3, 15).getMillis())); + assertThat(bucket.getToAsDate(), equalTo(date(3, 15))); + assertThat(bucket.getDocCount(), equalTo(2l)); + min = bucket.getAggregations().get("min"); + assertThat(min, notNullValue()); + assertThat(min.getValue(), equalTo((double) date(2, 15).getMillis())); + + bucket = range.getByKey("r3"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("r3")); + assertThat(bucket.getFrom(), equalTo((double) date(3, 15).getMillis())); + assertThat(bucket.getFromAsDate(), equalTo(date(3, 15))); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getToAsDate(), nullValue()); + assertThat(bucket.getDocCount(), equalTo(numDocs - 4l)); + min = bucket.getAggregations().get("min"); + assertThat(min, notNullValue()); + assertThat(min.getValue(), equalTo((double) date(3, 15).getMillis())); + } + + /* + Jan 2, Feb 3, 1 + Feb 2, Mar 3, 2 + Feb 15, Mar 16, 3 + Mar 2, Apr 3, 4 + Mar 15, Apr 16 5 + Mar 23, Apr 24 6 + */ + + @Test + public void multiValuedField() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(dateRange("range") + .field("dates") + .addUnboundedTo(date(2, 15)) + .addRange(date(2, 15), date(3, 15)) + .addUnboundedFrom(date(3, 15))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + DateRange range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(3)); + + DateRange.Bucket bucket = range.getByKey("*-2012-02-15T00:00:00.000Z"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("*-2012-02-15T00:00:00.000Z")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getFromAsDate(), nullValue()); + assertThat(bucket.getTo(), equalTo((double) date(2, 15).getMillis())); + assertThat(bucket.getToAsDate(), equalTo(date(2, 15))); + assertThat(bucket.getDocCount(), equalTo(2l)); + + bucket = range.getByKey("2012-02-15T00:00:00.000Z-2012-03-15T00:00:00.000Z"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("2012-02-15T00:00:00.000Z-2012-03-15T00:00:00.000Z")); + assertThat(bucket.getFrom(), equalTo((double) date(2, 15).getMillis())); + assertThat(bucket.getFromAsDate(), equalTo(date(2, 15))); + assertThat(bucket.getTo(), equalTo((double) date(3, 15).getMillis())); + assertThat(bucket.getToAsDate(), equalTo(date(3, 15))); + assertThat(bucket.getDocCount(), equalTo(3l)); + + bucket = range.getByKey("2012-03-15T00:00:00.000Z-*"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("2012-03-15T00:00:00.000Z-*")); + assertThat(bucket.getFrom(), equalTo((double) date(3, 15).getMillis())); + assertThat(bucket.getFromAsDate(), equalTo(date(3, 15))); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getToAsDate(), nullValue()); + assertThat(bucket.getDocCount(), equalTo(numDocs - 2l)); + } + + /* + Feb 2, Mar 3, 1 + Mar 2, Apr 3, 2 + Mar 15, Apr 16, 3 + Apr 2, May 3, 4 + Apr 15, May 16 5 + Apr 23, May 24 6 + */ + + + @Test + public void multiValuedField_WithValueScript() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(dateRange("range") + .field("dates") + .script("new DateTime(_value.longValue(), DateTimeZone.UTC).plusMonths(1).getMillis()") + .addUnboundedTo(date(2, 15)) + .addRange(date(2, 15), date(3, 15)) + .addUnboundedFrom(date(3, 15))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + DateRange range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(3)); + + DateRange.Bucket bucket = range.getByKey("*-2012-02-15T00:00:00.000Z"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("*-2012-02-15T00:00:00.000Z")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getFromAsDate(), nullValue()); + assertThat(bucket.getTo(), equalTo((double) date(2, 15).getMillis())); + assertThat(bucket.getToAsDate(), equalTo(date(2, 15))); + assertThat(bucket.getDocCount(), equalTo(1l)); + + bucket = range.getByKey("2012-02-15T00:00:00.000Z-2012-03-15T00:00:00.000Z"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("2012-02-15T00:00:00.000Z-2012-03-15T00:00:00.000Z")); + assertThat(bucket.getFrom(), equalTo((double) date(2, 15).getMillis())); + assertThat(bucket.getFromAsDate(), equalTo(date(2, 15))); + assertThat(bucket.getTo(), equalTo((double) date(3, 15).getMillis())); + assertThat(bucket.getToAsDate(), equalTo(date(3, 15))); + assertThat(bucket.getDocCount(), equalTo(2l)); + + bucket = range.getByKey("2012-03-15T00:00:00.000Z-*"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("2012-03-15T00:00:00.000Z-*")); + assertThat(bucket.getFrom(), equalTo((double) date(3, 15).getMillis())); + assertThat(bucket.getFromAsDate(), equalTo(date(3, 15))); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getToAsDate(), nullValue()); + assertThat(bucket.getDocCount(), equalTo(numDocs - 1l)); + } + + /* + Feb 2, Mar 3, 1 + Mar 2, Apr 3, 2 + Mar 15, Apr 16, 3 + Apr 2, May 3, 4 + Apr 15, May 16 5 + Apr 23, May 24 6 + */ + + @Test + public void multiValuedField_WithValueScript_WithInheritedSubAggregator() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(dateRange("range") + .field("dates") + .script("new DateTime(_value.longValue(), DateTimeZone.UTC).plusMonths(1).getMillis()") + .addUnboundedTo(date(2, 15)) + .addRange(date(2, 15), date(3, 15)) + .addUnboundedFrom(date(3, 15)) + .subAggregation(max("max"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + DateRange range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(3)); + + DateRange.Bucket bucket = range.getByKey("*-2012-02-15T00:00:00.000Z"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("*-2012-02-15T00:00:00.000Z")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getFromAsDate(), nullValue()); + assertThat(bucket.getTo(), equalTo((double) date(2, 15).getMillis())); + assertThat(bucket.getToAsDate(), equalTo(date(2, 15))); + assertThat(bucket.getDocCount(), equalTo(1l)); + Max max = bucket.getAggregations().get("max"); + assertThat(max, notNullValue()); + assertThat(max.getValue(), equalTo((double) date(3, 3).getMillis())); + + bucket = range.getByKey("2012-02-15T00:00:00.000Z-2012-03-15T00:00:00.000Z"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("2012-02-15T00:00:00.000Z-2012-03-15T00:00:00.000Z")); + assertThat(bucket.getFrom(), equalTo((double) date(2, 15).getMillis())); + assertThat(bucket.getFromAsDate(), equalTo(date(2, 15))); + assertThat(bucket.getTo(), equalTo((double) date(3, 15).getMillis())); + assertThat(bucket.getToAsDate(), equalTo(date(3, 15))); + assertThat(bucket.getDocCount(), equalTo(2l)); + max = bucket.getAggregations().get("max"); + assertThat(max, notNullValue()); + assertThat(max.getValue(), equalTo((double) date(4, 3).getMillis())); + + bucket = range.getByKey("2012-03-15T00:00:00.000Z-*"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("2012-03-15T00:00:00.000Z-*")); + assertThat(bucket.getFrom(), equalTo((double) date(3, 15).getMillis())); + assertThat(bucket.getFromAsDate(), equalTo(date(3, 15))); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getToAsDate(), nullValue()); + assertThat(bucket.getDocCount(), equalTo(numDocs - 1l)); + max = bucket.getAggregations().get("max"); + assertThat(max, notNullValue()); + } + + @Test + public void script_SingleValue() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(dateRange("range") + .script("doc['date'].value") + .addUnboundedTo(date(2, 15)) + .addRange(date(2, 15), date(3, 15)) + .addUnboundedFrom(date(3, 15))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + DateRange range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(3)); + + DateRange.Bucket bucket = range.getByKey("*-2012-02-15T00:00:00.000Z"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("*-2012-02-15T00:00:00.000Z")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getFromAsDate(), nullValue()); + assertThat(bucket.getTo(), equalTo((double) date(2, 15).getMillis())); + assertThat(bucket.getToAsDate(), equalTo(date(2, 15))); + assertThat(bucket.getDocCount(), equalTo(2l)); + + bucket = range.getByKey("2012-02-15T00:00:00.000Z-2012-03-15T00:00:00.000Z"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("2012-02-15T00:00:00.000Z-2012-03-15T00:00:00.000Z")); + assertThat(bucket.getFrom(), equalTo((double) date(2, 15).getMillis())); + assertThat(bucket.getFromAsDate(), equalTo(date(2, 15))); + assertThat(bucket.getTo(), equalTo((double) date(3, 15).getMillis())); + assertThat(bucket.getToAsDate(), equalTo(date(3, 15))); + assertThat(bucket.getDocCount(), equalTo(2l)); + + bucket = range.getByKey("2012-03-15T00:00:00.000Z-*"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("2012-03-15T00:00:00.000Z-*")); + assertThat(bucket.getFrom(), equalTo((double) date(3, 15).getMillis())); + assertThat(bucket.getFromAsDate(), equalTo(date(3, 15))); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getToAsDate(), nullValue()); + assertThat(bucket.getDocCount(), equalTo(numDocs - 4l)); + } + + @Test + public void script_SingleValue_WithSubAggregator_Inherited() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(dateRange("range") + .script("doc['date'].value") + .addUnboundedTo(date(2, 15)) + .addRange(date(2, 15), date(3, 15)) + .addUnboundedFrom(date(3, 15)) + .subAggregation(max("max"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + DateRange range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(3)); + + DateRange.Bucket bucket = range.getByKey("*-2012-02-15T00:00:00.000Z"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("*-2012-02-15T00:00:00.000Z")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getFromAsDate(), nullValue()); + assertThat(bucket.getTo(), equalTo((double) date(2, 15).getMillis())); + assertThat(bucket.getToAsDate(), equalTo(date(2, 15))); + assertThat(bucket.getDocCount(), equalTo(2l)); + Max max = bucket.getAggregations().get("max"); + assertThat(max, notNullValue()); + assertThat(max.getValue(), equalTo((double) date(2, 2).getMillis())); + + bucket = range.getByKey("2012-02-15T00:00:00.000Z-2012-03-15T00:00:00.000Z"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("2012-02-15T00:00:00.000Z-2012-03-15T00:00:00.000Z")); + assertThat(bucket.getFrom(), equalTo((double) date(2, 15).getMillis())); + assertThat(bucket.getFromAsDate(), equalTo(date(2, 15))); + assertThat(bucket.getTo(), equalTo((double) date(3, 15).getMillis())); + assertThat(bucket.getToAsDate(), equalTo(date(3, 15))); + assertThat(bucket.getDocCount(), equalTo(2l)); + max = bucket.getAggregations().get("max"); + assertThat(max, notNullValue()); + assertThat(max.getValue(), equalTo((double) date(3, 2).getMillis())); + + bucket = range.getByKey("2012-03-15T00:00:00.000Z-*"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("2012-03-15T00:00:00.000Z-*")); + assertThat(bucket.getFrom(), equalTo((double) date(3, 15).getMillis())); + assertThat(bucket.getFromAsDate(), equalTo(date(3, 15))); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getToAsDate(), nullValue()); + assertThat(bucket.getDocCount(), equalTo(numDocs - 4l)); + max = bucket.getAggregations().get("max"); + assertThat(max, notNullValue()); + } + + /* + Jan 2, Feb 3, 1 + Feb 2, Mar 3, 2 + Feb 15, Mar 16, 3 + Mar 2, Apr 3, 4 + Mar 15, Apr 16 5 + Mar 23, Apr 24 6 + */ + + @Test + public void script_MultiValued() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(dateRange("range") + .script("doc['dates'].values") + .addUnboundedTo(date(2, 15)) + .addRange(date(2, 15), date(3, 15)) + .addUnboundedFrom(date(3, 15))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + DateRange range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(3)); + + DateRange.Bucket bucket = range.getByKey("*-2012-02-15T00:00:00.000Z"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("*-2012-02-15T00:00:00.000Z")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getFromAsDate(), nullValue()); + assertThat(bucket.getTo(), equalTo((double) date(2, 15).getMillis())); + assertThat(bucket.getToAsDate(), equalTo(date(2, 15))); + assertThat(bucket.getDocCount(), equalTo(2l)); + + bucket = range.getByKey("2012-02-15T00:00:00.000Z-2012-03-15T00:00:00.000Z"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("2012-02-15T00:00:00.000Z-2012-03-15T00:00:00.000Z")); + assertThat(bucket.getFrom(), equalTo((double) date(2, 15).getMillis())); + assertThat(bucket.getFromAsDate(), equalTo(date(2, 15))); + assertThat(bucket.getTo(), equalTo((double) date(3, 15).getMillis())); + assertThat(bucket.getToAsDate(), equalTo(date(3, 15))); + assertThat(bucket.getDocCount(), equalTo(3l)); + + bucket = range.getByKey("2012-03-15T00:00:00.000Z-*"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("2012-03-15T00:00:00.000Z-*")); + assertThat(bucket.getFrom(), equalTo((double) date(3, 15).getMillis())); + assertThat(bucket.getFromAsDate(), equalTo(date(3, 15))); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getToAsDate(), nullValue()); + assertThat(bucket.getDocCount(), equalTo(numDocs - 2l)); + } + + @Test + public void script_MultiValued_WithAggregatorInherited() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(dateRange("range") + .script("doc['dates'].values") + .addUnboundedTo(date(2, 15)) + .addRange(date(2, 15), date(3, 15)) + .addUnboundedFrom(date(3, 15)) + .subAggregation(min("min"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + DateRange range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(3)); + + DateRange.Bucket bucket = range.getByKey("*-2012-02-15T00:00:00.000Z"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("*-2012-02-15T00:00:00.000Z")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getFromAsDate(), nullValue()); + assertThat(bucket.getTo(), equalTo((double) date(2, 15).getMillis())); + assertThat(bucket.getToAsDate(), equalTo(date(2, 15))); + assertThat(bucket.getDocCount(), equalTo(2l)); + Min min = bucket.getAggregations().get("min"); + assertThat(min, notNullValue()); + assertThat(min.getValue(), equalTo((double) date(1, 2).getMillis())); + + bucket = range.getByKey("2012-02-15T00:00:00.000Z-2012-03-15T00:00:00.000Z"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("2012-02-15T00:00:00.000Z-2012-03-15T00:00:00.000Z")); + assertThat(bucket.getFrom(), equalTo((double) date(2, 15).getMillis())); + assertThat(bucket.getFromAsDate(), equalTo(date(2, 15))); + assertThat(bucket.getTo(), equalTo((double) date(3, 15).getMillis())); + assertThat(bucket.getToAsDate(), equalTo(date(3, 15))); + assertThat(bucket.getDocCount(), equalTo(3l)); + min = bucket.getAggregations().get("min"); + assertThat(min, notNullValue()); + assertThat(min.getValue(), equalTo((double) date(2, 2).getMillis())); + + bucket = range.getByKey("2012-03-15T00:00:00.000Z-*"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("2012-03-15T00:00:00.000Z-*")); + assertThat(bucket.getFrom(), equalTo((double) date(3, 15).getMillis())); + assertThat(bucket.getFromAsDate(), equalTo(date(3, 15))); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getToAsDate(), nullValue()); + assertThat(bucket.getDocCount(), equalTo(numDocs - 2l)); + min = bucket.getAggregations().get("min"); + assertThat(min, notNullValue()); + assertThat(min.getValue(), equalTo((double) date(2, 15).getMillis())); + } + + @Test + public void unmapped() throws Exception { + client().admin().cluster().prepareHealth("idx_unmapped").setWaitForYellowStatus().execute().actionGet(); + + SearchResponse response = client().prepareSearch("idx_unmapped") + .addAggregation(dateRange("range") + .field("date") + .addUnboundedTo(date(2, 15)) + .addRange(date(2, 15), date(3, 15)) + .addUnboundedFrom(date(3, 15))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + DateRange range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(3)); + + DateRange.Bucket bucket = range.getByKey("*-2012-02-15T00:00:00.000Z"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("*-2012-02-15T00:00:00.000Z")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getFromAsDate(), nullValue()); + assertThat(bucket.getTo(), equalTo((double) date(2, 15).getMillis())); + assertThat(bucket.getToAsDate(), equalTo(date(2, 15))); + assertThat(bucket.getDocCount(), equalTo(0l)); + + bucket = range.getByKey("2012-02-15T00:00:00.000Z-2012-03-15T00:00:00.000Z"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("2012-02-15T00:00:00.000Z-2012-03-15T00:00:00.000Z")); + assertThat(bucket.getFrom(), equalTo((double) date(2, 15).getMillis())); + assertThat(bucket.getFromAsDate(), equalTo(date(2, 15))); + assertThat(bucket.getTo(), equalTo((double) date(3, 15).getMillis())); + assertThat(bucket.getToAsDate(), equalTo(date(3, 15))); + assertThat(bucket.getDocCount(), equalTo(0l)); + + bucket = range.getByKey("2012-03-15T00:00:00.000Z-*"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("2012-03-15T00:00:00.000Z-*")); + assertThat(bucket.getFrom(), equalTo((double) date(3, 15).getMillis())); + assertThat(bucket.getFromAsDate(), equalTo(date(3, 15))); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getToAsDate(), nullValue()); + assertThat(bucket.getDocCount(), equalTo(0l)); + } + + @Test + public void unmapped_WithStringDates() throws Exception { + client().admin().cluster().prepareHealth("idx_unmapped").setWaitForGreenStatus().execute().actionGet(); + + SearchResponse response = client().prepareSearch("idx_unmapped") + .addAggregation(dateRange("range") + .field("date") + .addUnboundedTo("2012-02-15") + .addRange("2012-02-15", "2012-03-15") + .addUnboundedFrom("2012-03-15")) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + DateRange range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(3)); + + DateRange.Bucket bucket = range.getByKey("*-2012-02-15T00:00:00.000Z"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("*-2012-02-15T00:00:00.000Z")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getFromAsDate(), nullValue()); + assertThat(bucket.getTo(), equalTo((double) date(2, 15).getMillis())); + assertThat(bucket.getToAsDate(), equalTo(date(2, 15))); + assertThat(bucket.getDocCount(), equalTo(0l)); + + bucket = range.getByKey("2012-02-15T00:00:00.000Z-2012-03-15T00:00:00.000Z"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("2012-02-15T00:00:00.000Z-2012-03-15T00:00:00.000Z")); + assertThat(bucket.getFrom(), equalTo((double) date(2, 15).getMillis())); + assertThat(bucket.getFromAsDate(), equalTo(date(2, 15))); + assertThat(bucket.getTo(), equalTo((double) date(3, 15).getMillis())); + assertThat(bucket.getToAsDate(), equalTo(date(3, 15))); + assertThat(bucket.getDocCount(), equalTo(0l)); + + bucket = range.getByKey("2012-03-15T00:00:00.000Z-*"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("2012-03-15T00:00:00.000Z-*")); + assertThat(bucket.getFrom(), equalTo((double) date(3, 15).getMillis())); + assertThat(bucket.getFromAsDate(), equalTo(date(3, 15))); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getToAsDate(), nullValue()); + assertThat(bucket.getDocCount(), equalTo(0l)); + } + + @Test + public void partiallyUnmapped() throws Exception { + client().admin().cluster().prepareHealth("idx_unmapped").setWaitForGreenStatus().execute().actionGet(); + + SearchResponse response = client().prepareSearch("idx", "idx_unmapped") + .addAggregation(dateRange("range") + .field("date") + .addUnboundedTo(date(2, 15)) + .addRange(date(2, 15), date(3, 15)) + .addUnboundedFrom(date(3, 15))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + DateRange range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(3)); + + DateRange.Bucket bucket = range.getByKey("*-2012-02-15T00:00:00.000Z"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("*-2012-02-15T00:00:00.000Z")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getFromAsDate(), nullValue()); + assertThat(bucket.getTo(), equalTo((double) date(2, 15).getMillis())); + assertThat(bucket.getToAsDate(), equalTo(date(2, 15))); + assertThat(bucket.getDocCount(), equalTo(2l)); + + bucket = range.getByKey("2012-02-15T00:00:00.000Z-2012-03-15T00:00:00.000Z"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("2012-02-15T00:00:00.000Z-2012-03-15T00:00:00.000Z")); + assertThat(bucket.getFrom(), equalTo((double) date(2, 15).getMillis())); + assertThat(bucket.getFromAsDate(), equalTo(date(2, 15))); + assertThat(bucket.getTo(), equalTo((double) date(3, 15).getMillis())); + assertThat(bucket.getToAsDate(), equalTo(date(3, 15))); + assertThat(bucket.getDocCount(), equalTo(2l)); + + bucket = range.getByKey("2012-03-15T00:00:00.000Z-*"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("2012-03-15T00:00:00.000Z-*")); + assertThat(bucket.getFrom(), equalTo((double) date(3, 15).getMillis())); + assertThat(bucket.getFromAsDate(), equalTo(date(3, 15))); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getToAsDate(), nullValue()); + assertThat(bucket.getDocCount(), equalTo(numDocs - 4l)); + } + + @Test + public void emptyAggregation() throws Exception { + prepareCreate("empty_bucket_idx").addMapping("type", "value", "type=integer").execute().actionGet(); + List builders = new ArrayList(); + for (int i = 0; i < 2; i++) { + builders.add(client().prepareIndex("empty_bucket_idx", "type", ""+i).setSource(jsonBuilder() + .startObject() + .field("value", i*2) + .endObject())); + } + indexRandom(true, builders.toArray(new IndexRequestBuilder[builders.size()])); + + SearchResponse searchResponse = client().prepareSearch("empty_bucket_idx") + .setQuery(matchAllQuery()) + .addAggregation(histogram("histo").field("value").interval(1l).emptyBuckets(true).subAggregation(dateRange("date_range").addRange("0-1", 0, 1))) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(2l)); + Histogram histo = searchResponse.getAggregations().get("histo"); + assertThat(histo, Matchers.notNullValue()); + Histogram.Bucket bucket = histo.getByKey(1l); + assertThat(bucket, Matchers.notNullValue()); + + DateRange dateRange = bucket.getAggregations().get("date_range"); + assertThat(dateRange, Matchers.notNullValue()); + assertThat(dateRange.getName(), equalTo("date_range")); + assertThat(dateRange.buckets().size(), is(1)); + assertThat(dateRange.buckets().get(0).getKey(), equalTo("0-1")); + assertThat(dateRange.buckets().get(0).getFrom(), equalTo(0.0)); + assertThat(dateRange.buckets().get(0).getTo(), equalTo(1.0)); + assertThat(dateRange.buckets().get(0).getDocCount(), equalTo(0l)); + assertThat(dateRange.buckets().get(0).getAggregations().asList().isEmpty(), is(true)); + + } + + +} diff --git a/src/test/java/org/elasticsearch/search/aggregations/bucket/DoubleTermsTests.java b/src/test/java/org/elasticsearch/search/aggregations/bucket/DoubleTermsTests.java new file mode 100644 index 00000000000..34171351259 --- /dev/null +++ b/src/test/java/org/elasticsearch/search/aggregations/bucket/DoubleTermsTests.java @@ -0,0 +1,602 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket; + +import org.elasticsearch.action.index.IndexRequestBuilder; +import org.elasticsearch.action.search.SearchResponse; +import org.elasticsearch.common.settings.ImmutableSettings; +import org.elasticsearch.common.settings.Settings; +import org.elasticsearch.search.aggregations.bucket.histogram.Histogram; +import org.elasticsearch.search.aggregations.bucket.terms.Terms; +import org.elasticsearch.search.aggregations.metrics.sum.Sum; +import org.elasticsearch.test.ElasticsearchIntegrationTest; +import org.hamcrest.Matchers; +import org.junit.Before; +import org.junit.Test; + +import java.util.ArrayList; +import java.util.List; + +import static org.elasticsearch.common.xcontent.XContentFactory.jsonBuilder; +import static org.elasticsearch.index.query.QueryBuilders.matchAllQuery; +import static org.elasticsearch.search.aggregations.AggregationBuilders.*; +import static org.hamcrest.Matchers.equalTo; +import static org.hamcrest.Matchers.is; +import static org.hamcrest.core.IsNull.notNullValue; + +/** + * + */ +public class DoubleTermsTests extends ElasticsearchIntegrationTest { + + private static final int NUM_DOCS = 5; // TODO: randomize the size? + + @Override + public Settings indexSettings() { + return ImmutableSettings.builder() + .put("index.number_of_shards", between(1, 5)) + .put("index.number_of_replicas", between(0, 1)) + .build(); + } + + @Before + public void init() throws Exception { + createIndex("idx"); + + IndexRequestBuilder[] lowcardBuilders = new IndexRequestBuilder[NUM_DOCS]; + for (int i = 0; i < lowcardBuilders.length; i++) { + lowcardBuilders[i] = client().prepareIndex("idx", "type").setSource(jsonBuilder() + .startObject() + .field("value", (double) i) + .startArray("values").value((double)i).value(i + 1d).endArray() + .endObject()); + + } + indexRandom(randomBoolean(), lowcardBuilders); + IndexRequestBuilder[] highCardBuilders = new IndexRequestBuilder[100]; // TODO: randomize the size? + for (int i = 0; i < highCardBuilders.length; i++) { + highCardBuilders[i] = client().prepareIndex("idx", "high_card_type").setSource(jsonBuilder() + .startObject() + .field("value", (double) i) + .startArray("values").value((double)i).value(i + 1d).endArray() + .endObject()); + } + indexRandom(true, highCardBuilders); + + createIndex("idx_unmapped"); + + } + + @Test + public void singleValueField() throws Exception { + SearchResponse response = client().prepareSearch("idx").setTypes("type") + .addAggregation(terms("terms") + .field("value")) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(5)); + + for (int i = 0; i < 5; i++) { + Terms.Bucket bucket = terms.getByTerm("" + (double)i); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("" + (double)i)); + assertThat(bucket.getKeyAsNumber().intValue(), equalTo(i)); + assertThat(bucket.getDocCount(), equalTo(1l)); + } + } + + @Test + public void singleValueField_WithMaxSize() throws Exception { + SearchResponse response = client().prepareSearch("idx").setTypes("high_card_type") + .addAggregation(terms("terms") + .field("value") + .size(20) + .order(Terms.Order.TERM_ASC)) // we need to sort by terms cause we're checking the first 20 values + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(20)); + + for (int i = 0; i < 20; i++) { + Terms.Bucket bucket = terms.getByTerm("" + (double) i); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("" + (double) i)); + assertThat(bucket.getKeyAsNumber().intValue(), equalTo(i)); + assertThat(bucket.getDocCount(), equalTo(1l)); + } + } + + @Test + public void singleValueField_OrderedByTermAsc() throws Exception { + SearchResponse response = client().prepareSearch("idx").setTypes("type") + .addAggregation(terms("terms") + .field("value") + .order(Terms.Order.TERM_ASC)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(5)); + + int i = 0; + for (Terms.Bucket bucket : terms.buckets()) { + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("" + (double)i)); + assertThat(bucket.getKeyAsNumber().intValue(), equalTo(i)); + assertThat(bucket.getDocCount(), equalTo(1l)); + i++; + } + } + + @Test + public void singleValueField_OrderedByTermDesc() throws Exception { + SearchResponse response = client().prepareSearch("idx").setTypes("type") + .addAggregation(terms("terms") + .field("value") + .order(Terms.Order.TERM_DESC)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(5)); + + int i = 4; + for (Terms.Bucket bucket : terms.buckets()) { + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("" + (double) i)); + assertThat(bucket.getKeyAsNumber().intValue(), equalTo(i)); + assertThat(bucket.getDocCount(), equalTo(1l)); + i--; + } + } + + @Test + public void singleValuedField_WithSubAggregation() throws Exception { + SearchResponse response = client().prepareSearch("idx").setTypes("type") + .addAggregation(terms("terms") + .field("value") + .subAggregation(sum("sum").field("values"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(5)); + + for (int i = 0; i < 5; i++) { + Terms.Bucket bucket = terms.getByTerm("" + (double) i); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("" + (double) i)); + assertThat(bucket.getKeyAsNumber().intValue(), equalTo(i)); + assertThat(bucket.getDocCount(), equalTo(1l)); + Sum sum = bucket.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + assertThat((long) sum.getValue(), equalTo(i+i+1l)); + } + } + + @Test + public void singleValuedField_WithSubAggregation_Inherited() throws Exception { + SearchResponse response = client().prepareSearch("idx").setTypes("type") + .addAggregation(terms("terms") + .field("value") + .subAggregation(sum("sum"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(5)); + + for (int i = 0; i < 5; i++) { + Terms.Bucket bucket = terms.getByTerm("" + (double) i); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("" + (double) i)); + assertThat(bucket.getKeyAsNumber().intValue(), equalTo(i)); + assertThat(bucket.getDocCount(), equalTo(1l)); + Sum sum = bucket.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + assertThat(sum.getValue(), equalTo((double) i)); + } + } + + @Test + public void singleValuedField_WithValueScript() throws Exception { + SearchResponse response = client().prepareSearch("idx").setTypes("type") + .addAggregation(terms("terms") + .field("value") + .script("_value + 1")) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(5)); + + for (int i = 0; i < 5; i++) { + Terms.Bucket bucket = terms.getByTerm("" + (i+1d)); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("" + (i+1d))); + assertThat(bucket.getKeyAsNumber().intValue(), equalTo(i+1)); + assertThat(bucket.getDocCount(), equalTo(1l)); + } + } + + @Test + public void multiValuedField() throws Exception { + SearchResponse response = client().prepareSearch("idx").setTypes("type") + .addAggregation(terms("terms") + .field("values")) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(6)); + + for (int i = 0; i < 6; i++) { + Terms.Bucket bucket = terms.getByTerm("" + (double) i); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("" + (double) i)); + assertThat(bucket.getKeyAsNumber().intValue(), equalTo(i)); + if (i == 0 || i == 5) { + assertThat(bucket.getDocCount(), equalTo(1l)); + } else { + assertThat(bucket.getDocCount(), equalTo(2l)); + } + } + } + + @Test + public void multiValuedField_WithValueScript() throws Exception { + SearchResponse response = client().prepareSearch("idx").setTypes("type") + .addAggregation(terms("terms") + .field("values") + .script("_value + 1")) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(6)); + + for (int i = 0; i < 6; i++) { + Terms.Bucket bucket = terms.getByTerm("" + (i+1d)); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("" + (i+1d))); + assertThat(bucket.getKeyAsNumber().intValue(), equalTo(i+1)); + if (i == 0 || i == 5) { + assertThat(bucket.getDocCount(), equalTo(1l)); + } else { + assertThat(bucket.getDocCount(), equalTo(2l)); + } + } + } + + @Test + public void multiValuedField_WithValueScript_NotUnique() throws Exception { + SearchResponse response = client().prepareSearch("idx").setTypes("type") + .addAggregation(terms("terms") + .field("values") + .script("(long) _value / 1000 + 1")) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(1)); + + Terms.Bucket bucket = terms.getByTerm("1.0"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("1.0")); + assertThat(bucket.getKeyAsNumber().intValue(), equalTo(1)); + assertThat(bucket.getDocCount(), equalTo(5l)); + } + + /* + + [1, 2] + [2, 3] + [3, 4] + [4, 5] + [5, 6] + + 1 - count: 1 - sum: 1 + 2 - count: 2 - sum: 4 + 3 - count: 2 - sum: 6 + 4 - count: 2 - sum: 8 + 5 - count: 2 - sum: 10 + 6 - count: 1 - sum: 6 + + */ + + @Test + public void multiValuedField_WithValueScript_WithInheritedSubAggregator() throws Exception { + SearchResponse response = client().prepareSearch("idx").setTypes("type") + .addAggregation(terms("terms") + .field("values") + .script("_value + 1") + .subAggregation(sum("sum"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(6)); + + for (int i = 0; i < 6; i++) { + Terms.Bucket bucket = terms.getByTerm("" + (i+1d)); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("" + (i+1d))); + assertThat(bucket.getKeyAsNumber().doubleValue(), equalTo(i+1d)); + final long count = i == 0 || i == 5 ? 1 : 2; + double s = 0; + for (int j = 0; j < NUM_DOCS; ++j) { + if (i == j || i == j+1) { + s += j + 1; + s += j+1 + 1; + } + } + assertThat(bucket.getDocCount(), equalTo(count)); + Sum sum = bucket.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + assertThat(sum.getValue(), equalTo(s)); + } + } + + @Test + public void script_SingleValue() throws Exception { + SearchResponse response = client().prepareSearch("idx").setTypes("type") + .addAggregation(terms("terms") + .script("doc['value'].value")) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(5)); + + for (int i = 0; i < 5; i++) { + Terms.Bucket bucket = terms.getByTerm("" + (double) i); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("" + (double) i)); + assertThat(bucket.getKeyAsNumber().intValue(), equalTo(i)); + assertThat(bucket.getDocCount(), equalTo(1l)); + } + } + + @Test + public void script_SingleValue_WithSubAggregator_Inherited() throws Exception { + SearchResponse response = client().prepareSearch("idx").setTypes("type") + .addAggregation(terms("terms") + .field("value") + .subAggregation(sum("sum"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(5)); + + for (int i = 0; i < 5; i++) { + Terms.Bucket bucket = terms.getByTerm("" + (double) i); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("" + (double) i)); + assertThat(bucket.getKeyAsNumber().intValue(), equalTo(i)); + assertThat(bucket.getDocCount(), equalTo(1l)); + Sum sum = bucket.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + assertThat(sum.getValue(), equalTo((double) i)); + } + } + + @Test + public void script_MultiValued() throws Exception { + SearchResponse response = client().prepareSearch("idx").setTypes("type") + .addAggregation(terms("terms") + .script("doc['values'].values")) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(6)); + + for (int i = 0; i < 6; i++) { + Terms.Bucket bucket = terms.getByTerm("" + (double) i); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("" + (double) i)); + assertThat(bucket.getKeyAsNumber().intValue(), equalTo(i)); + if (i == 0 || i == 5) { + assertThat(bucket.getDocCount(), equalTo(1l)); + } else { + assertThat(bucket.getDocCount(), equalTo(2l)); + } + } + } + + @Test + public void script_MultiValued_WithAggregatorInherited_NoExplicitType() throws Exception { + + // since no type is explicitly defined, es will assume all values returned by the script to be strings (bytes), + // so the aggregation should fail, since the "sum" aggregation can only operation on numeric values. + + try { + + SearchResponse response = client().prepareSearch("idx").setTypes("type") + .addAggregation(terms("terms") + .script("doc['values'].values") + .subAggregation(sum("sum"))) + .execute().actionGet(); + + + fail("expected to fail as sub-aggregation sum requires a numeric value source context, but there is none"); + + } catch (Exception e) { + // expected + } + + } + + @Test + public void script_MultiValued_WithAggregatorInherited_WithExplicitType() throws Exception { + SearchResponse response = client().prepareSearch("idx").setTypes("type") + .addAggregation(terms("terms") + .script("doc['values'].values") + .valueType(Terms.ValueType.DOUBLE) + .subAggregation(sum("sum"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(6)); + + for (int i = 0; i < 6; i++) { + Terms.Bucket bucket = terms.getByTerm("" + i + ".0"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("" + i + ".0")); + assertThat(bucket.getKeyAsNumber().intValue(), equalTo(i)); + final long count = i == 0 || i == 5 ? 1 : 2; + double s = 0; + for (int j = 0; j < NUM_DOCS; ++j) { + if (i == j || i == j+1) { + s += j; + s += j+1; + } + } + assertThat(bucket.getDocCount(), equalTo(count)); + Sum sum = bucket.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + assertThat(sum.getValue(), equalTo(s)); + } + } + + @Test + public void unmapped() throws Exception { + client().admin().cluster().prepareHealth("idx_unmapped").setWaitForYellowStatus().execute().actionGet(); + + SearchResponse response = client().prepareSearch("idx_unmapped").setTypes("type") + .addAggregation(terms("terms") + .field("value")) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(0)); + } + + @Test + public void partiallyUnmapped() throws Exception { + client().admin().cluster().prepareHealth("idx_unmapped").setWaitForGreenStatus().execute().actionGet(); + + SearchResponse response = client().prepareSearch("idx_unmapped", "idx").setTypes("type") + .addAggregation(terms("terms") + .field("value")) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(5)); + + for (int i = 0; i < 5; i++) { + Terms.Bucket bucket = terms.getByTerm("" + (double) i); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("" + (double) i)); + assertThat(bucket.getKeyAsNumber().intValue(), equalTo(i)); + assertThat(bucket.getDocCount(), equalTo(1l)); + } + } + + @Test + public void emptyAggregation() throws Exception { + prepareCreate("empty_bucket_idx").addMapping("type", "value", "type=integer").execute().actionGet(); + List builders = new ArrayList(); + for (int i = 0; i < 2; i++) { + builders.add(client().prepareIndex("empty_bucket_idx", "type", ""+i).setSource(jsonBuilder() + .startObject() + .field("value", i*2) + .endObject())); + } + indexRandom(true, builders.toArray(new IndexRequestBuilder[builders.size()])); + + SearchResponse searchResponse = client().prepareSearch("empty_bucket_idx") + .setQuery(matchAllQuery()) + .addAggregation(histogram("histo").field("value").interval(1l).emptyBuckets(true) + .subAggregation(terms("terms"))) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(2l)); + Histogram histo = searchResponse.getAggregations().get("histo"); + assertThat(histo, Matchers.notNullValue()); + Histogram.Bucket bucket = histo.getByKey(1l); + assertThat(bucket, Matchers.notNullValue()); + + Terms terms = bucket.getAggregations().get("terms"); + assertThat(terms, Matchers.notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().isEmpty(), is(true)); + + } + +} diff --git a/src/test/java/org/elasticsearch/search/aggregations/bucket/FilterTests.java b/src/test/java/org/elasticsearch/search/aggregations/bucket/FilterTests.java new file mode 100644 index 00000000000..1661abb557a --- /dev/null +++ b/src/test/java/org/elasticsearch/search/aggregations/bucket/FilterTests.java @@ -0,0 +1,173 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket; + +import org.elasticsearch.ElasticSearchException; +import org.elasticsearch.action.index.IndexRequestBuilder; +import org.elasticsearch.action.search.SearchResponse; +import org.elasticsearch.common.settings.ImmutableSettings; +import org.elasticsearch.common.settings.Settings; +import org.elasticsearch.search.aggregations.bucket.histogram.Histogram; +import org.elasticsearch.search.aggregations.bucket.filter.Filter; +import org.elasticsearch.search.aggregations.metrics.avg.Avg; +import org.elasticsearch.test.ElasticsearchIntegrationTest; +import org.hamcrest.Matchers; +import org.junit.Before; +import org.junit.Test; + +import java.util.ArrayList; +import java.util.List; + +import static org.elasticsearch.common.xcontent.XContentFactory.jsonBuilder; +import static org.elasticsearch.index.query.FilterBuilders.matchAllFilter; +import static org.elasticsearch.index.query.FilterBuilders.termFilter; +import static org.elasticsearch.index.query.QueryBuilders.matchAllQuery; +import static org.elasticsearch.search.aggregations.AggregationBuilders.*; +import static org.hamcrest.Matchers.equalTo; +import static org.hamcrest.Matchers.is; +import static org.hamcrest.core.IsNull.notNullValue; + +/** + * + */ +public class FilterTests extends ElasticsearchIntegrationTest { + + @Override + public Settings indexSettings() { + return ImmutableSettings.builder() + .put("index.number_of_shards", between(1, 5)) + .put("index.number_of_replicas", between(0, 1)) + .build(); + } + + int numDocs, numTag1Docs; + + @Before + public void init() throws Exception { + createIndex("idx"); + createIndex("idx2"); + numDocs = randomIntBetween(5, 20); + numTag1Docs = randomIntBetween(1, numDocs - 1); + List builders = new ArrayList(); + for (int i = 0; i < numTag1Docs; i++) { + builders.add(client().prepareIndex("idx", "type", ""+i).setSource(jsonBuilder() + .startObject() + .field("value", i + 1) + .field("tag", "tag1") + .endObject())); + } + for (int i = numTag1Docs; i < numDocs; i++) { + builders.add(client().prepareIndex("idx", "type", ""+i).setSource(jsonBuilder() + .startObject() + .field("value", i) + .field("tag", "tag2") + .field("name", "name" + i) + .endObject())); + } + indexRandom(true, builders.toArray(new IndexRequestBuilder[builders.size()])); + } + + @Test + public void simple() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(filter("tag1").filter(termFilter("tag", "tag1"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Filter filter = response.getAggregations().get("tag1"); + assertThat(filter, notNullValue()); + assertThat(filter.getName(), equalTo("tag1")); + assertThat(filter.getDocCount(), equalTo((long) numTag1Docs)); + } + + @Test + public void withSubAggregation() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(filter("tag1") + .filter(termFilter("tag", "tag1")) + .subAggregation(avg("avg_value").field("value"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Filter filter = response.getAggregations().get("tag1"); + assertThat(filter, notNullValue()); + assertThat(filter.getName(), equalTo("tag1")); + assertThat(filter.getDocCount(), equalTo((long) numTag1Docs)); + + long sum = 0; + for (int i = 0; i < numTag1Docs; ++i) { + sum += i + 1; + } + assertThat(filter.getAggregations().asList().isEmpty(), is(false)); + Avg avgValue = filter.getAggregations().get("avg_value"); + assertThat(avgValue, notNullValue()); + assertThat(avgValue.getName(), equalTo("avg_value")); + assertThat(avgValue.getValue(), equalTo((double) sum / numTag1Docs)); + } + + @Test + public void withContextBasedSubAggregation() throws Exception { + + try { + client().prepareSearch("idx") + .addAggregation(filter("tag1") + .filter(termFilter("tag", "tag1")) + .subAggregation(avg("avg_value"))) + .execute().actionGet(); + + fail("expected execution to fail - an attempt to have a context based numeric sub-aggregation, but there is not value source" + + "context which the sub-aggregation can inherit"); + + } catch (ElasticSearchException ese) { + } + } + + @Test + public void emptyAggregation() throws Exception { + prepareCreate("empty_bucket_idx").addMapping("type", "value", "type=integer").execute().actionGet(); + List builders = new ArrayList(); + for (int i = 0; i < 2; i++) { + builders.add(client().prepareIndex("empty_bucket_idx", "type", ""+i).setSource(jsonBuilder() + .startObject() + .field("value", i*2) + .endObject())); + } + indexRandom(true, builders.toArray(new IndexRequestBuilder[builders.size()])); + + SearchResponse searchResponse = client().prepareSearch("empty_bucket_idx") + .setQuery(matchAllQuery()) + .addAggregation(histogram("histo").field("value").interval(1l).emptyBuckets(true) + .subAggregation(filter("filter").filter(matchAllFilter()))) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(2l)); + Histogram histo = searchResponse.getAggregations().get("histo"); + assertThat(histo, Matchers.notNullValue()); + Histogram.Bucket bucket = histo.getByKey(1l); + assertThat(bucket, Matchers.notNullValue()); + + Filter filter = bucket.getAggregations().get("filter"); + assertThat(filter, Matchers.notNullValue()); + assertThat(filter.getName(), equalTo("filter")); + assertThat(filter.getDocCount(), is(0l)); + } +} diff --git a/src/test/java/org/elasticsearch/search/aggregations/bucket/GeoDistanceTests.java b/src/test/java/org/elasticsearch/search/aggregations/bucket/GeoDistanceTests.java new file mode 100644 index 00000000000..ea9bc5a6cfe --- /dev/null +++ b/src/test/java/org/elasticsearch/search/aggregations/bucket/GeoDistanceTests.java @@ -0,0 +1,370 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket; + +import com.google.common.collect.Sets; +import org.elasticsearch.action.index.IndexRequestBuilder; +import org.elasticsearch.action.search.SearchResponse; +import org.elasticsearch.common.settings.ImmutableSettings; +import org.elasticsearch.common.settings.Settings; +import org.elasticsearch.common.unit.DistanceUnit; +import org.elasticsearch.common.xcontent.XContentBuilder; +import org.elasticsearch.search.aggregations.bucket.histogram.Histogram; +import org.elasticsearch.search.aggregations.bucket.range.geodistance.GeoDistance; +import org.elasticsearch.search.aggregations.bucket.terms.Terms; +import org.elasticsearch.test.ElasticsearchIntegrationTest; +import org.hamcrest.Matchers; +import org.junit.Before; +import org.junit.Test; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; +import java.util.Set; + +import static org.elasticsearch.common.xcontent.XContentFactory.jsonBuilder; +import static org.elasticsearch.index.query.QueryBuilders.matchAllQuery; +import static org.elasticsearch.search.aggregations.AggregationBuilders.*; +import static org.hamcrest.Matchers.equalTo; +import static org.hamcrest.Matchers.is; +import static org.hamcrest.core.IsNull.notNullValue; + +/** + * + */ +public class GeoDistanceTests extends ElasticsearchIntegrationTest { + + @Override + public Settings indexSettings() { + return ImmutableSettings.builder() + .put("index.number_of_shards", between(1, 5)) + .put("index.number_of_replicas", between(0, 1)) + .build(); + } + + private IndexRequestBuilder indexCity(String name, String latLon) throws Exception { + XContentBuilder source = jsonBuilder().startObject().field("city", name); + if (latLon != null) { + source = source.field("location", latLon); + } + source = source.endObject(); + return client().prepareIndex("idx", "type").setSource(source); + } + + @Before + public void init() throws Exception { + prepareCreate("idx") + .addMapping("type", "location", "type=geo_point", "city", "type=string,index=not_analyzed") + .execute().actionGet(); + + createIndex("idx_unmapped"); + + List cities = new ArrayList(); + cities.addAll(Arrays.asList( + // below 500km + indexCity("utrecht", "52.0945, 5.116"), + indexCity("haarlem", "52.3890, 4.637"), + // above 500km, below 1000km + indexCity("berlin", "52.540, 13.409"), + indexCity("prague", "50.086, 14.439"), + // above 1000km + indexCity("tel-aviv", "32.0741, 34.777"))); + + // random cities with no location + for (String cityName : Arrays.asList("london", "singapour", "tokyo", "milan")) { + if (randomBoolean() || true) { + cities.add(indexCity(cityName, null)); + } + } + + indexRandom(true, cities); + } + + @Test + public void simple() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(geoDistance("amsterdam_rings") + .field("location") + .unit(DistanceUnit.KILOMETERS) + .point("52.3760, 4.894") // coords of amsterdam + .addUnboundedTo(500) + .addRange(500, 1000) + .addUnboundedFrom(1000)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + GeoDistance geoDist = response.getAggregations().get("amsterdam_rings"); + assertThat(geoDist, notNullValue()); + assertThat(geoDist.getName(), equalTo("amsterdam_rings")); + assertThat(geoDist.buckets().size(), equalTo(3)); + + GeoDistance.Bucket bucket = geoDist.getByKey("*-500.0"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("*-500.0")); + assertThat(bucket.getFrom(), equalTo(0.0)); + assertThat(bucket.getTo(), equalTo(500.0)); + assertThat(bucket.getDocCount(), equalTo(2l)); + + bucket = geoDist.getByKey("500.0-1000.0"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("500.0-1000.0")); + assertThat(bucket.getFrom(), equalTo(500.0)); + assertThat(bucket.getTo(), equalTo(1000.0)); + assertThat(bucket.getDocCount(), equalTo(2l)); + + bucket = geoDist.getByKey("1000.0-*"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("1000.0-*")); + assertThat(bucket.getFrom(), equalTo(1000.0)); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getDocCount(), equalTo(1l)); + } + + @Test + public void simple_WithCustomKeys() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(geoDistance("amsterdam_rings") + .field("location") + .unit(DistanceUnit.KILOMETERS) + .point("52.3760, 4.894") // coords of amsterdam + .addUnboundedTo("ring1", 500) + .addRange("ring2", 500, 1000) + .addUnboundedFrom("ring3", 1000)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + GeoDistance geoDist = response.getAggregations().get("amsterdam_rings"); + assertThat(geoDist, notNullValue()); + assertThat(geoDist.getName(), equalTo("amsterdam_rings")); + assertThat(geoDist.buckets().size(), equalTo(3)); + + GeoDistance.Bucket bucket = geoDist.getByKey("ring1"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("ring1")); + assertThat(bucket.getFrom(), equalTo(0.0)); + assertThat(bucket.getTo(), equalTo(500.0)); + assertThat(bucket.getDocCount(), equalTo(2l)); + + bucket = geoDist.getByKey("ring2"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("ring2")); + assertThat(bucket.getFrom(), equalTo(500.0)); + assertThat(bucket.getTo(), equalTo(1000.0)); + assertThat(bucket.getDocCount(), equalTo(2l)); + + bucket = geoDist.getByKey("ring3"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("ring3")); + assertThat(bucket.getFrom(), equalTo(1000.0)); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getDocCount(), equalTo(1l)); + } + + @Test + public void unmapped() throws Exception { + client().admin().cluster().prepareHealth("idx_unmapped").setWaitForYellowStatus().execute().actionGet(); + + SearchResponse response = client().prepareSearch("idx_unmapped") + .addAggregation(geoDistance("amsterdam_rings") + .field("location") + .unit(DistanceUnit.KILOMETERS) + .point("52.3760, 4.894") // coords of amsterdam + .addUnboundedTo(500) + .addRange(500, 1000) + .addUnboundedFrom(1000)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + GeoDistance geoDist = response.getAggregations().get("amsterdam_rings"); + assertThat(geoDist, notNullValue()); + assertThat(geoDist.getName(), equalTo("amsterdam_rings")); + assertThat(geoDist.buckets().size(), equalTo(3)); + + GeoDistance.Bucket bucket = geoDist.getByKey("*-500.0"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("*-500.0")); + assertThat(bucket.getFrom(), equalTo(0.0)); + assertThat(bucket.getTo(), equalTo(500.0)); + assertThat(bucket.getDocCount(), equalTo(0l)); + + bucket = geoDist.getByKey("500.0-1000.0"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("500.0-1000.0")); + assertThat(bucket.getFrom(), equalTo(500.0)); + assertThat(bucket.getTo(), equalTo(1000.0)); + assertThat(bucket.getDocCount(), equalTo(0l)); + + bucket = geoDist.getByKey("1000.0-*"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("1000.0-*")); + assertThat(bucket.getFrom(), equalTo(1000.0)); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getDocCount(), equalTo(0l)); + } + + @Test + public void partiallyUnmapped() throws Exception { + client().admin().cluster().prepareHealth("idx_unmapped").setWaitForYellowStatus().execute().actionGet(); + + SearchResponse response = client().prepareSearch("idx", "idx_unmapped") + .addAggregation(geoDistance("amsterdam_rings") + .field("location") + .unit(DistanceUnit.KILOMETERS) + .point("52.3760, 4.894") // coords of amsterdam + .addUnboundedTo(500) + .addRange(500, 1000) + .addUnboundedFrom(1000)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + GeoDistance geoDist = response.getAggregations().get("amsterdam_rings"); + assertThat(geoDist, notNullValue()); + assertThat(geoDist.getName(), equalTo("amsterdam_rings")); + assertThat(geoDist.buckets().size(), equalTo(3)); + + GeoDistance.Bucket bucket = geoDist.getByKey("*-500.0"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("*-500.0")); + assertThat(bucket.getFrom(), equalTo(0.0)); + assertThat(bucket.getTo(), equalTo(500.0)); + assertThat(bucket.getDocCount(), equalTo(2l)); + + bucket = geoDist.getByKey("500.0-1000.0"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("500.0-1000.0")); + assertThat(bucket.getFrom(), equalTo(500.0)); + assertThat(bucket.getTo(), equalTo(1000.0)); + assertThat(bucket.getDocCount(), equalTo(2l)); + + bucket = geoDist.getByKey("1000.0-*"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("1000.0-*")); + assertThat(bucket.getFrom(), equalTo(1000.0)); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getDocCount(), equalTo(1l)); + } + + + @Test + public void withSubAggregation() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(geoDistance("amsterdam_rings") + .field("location") + .unit(DistanceUnit.KILOMETERS) + .point("52.3760, 4.894") // coords of amsterdam + .addUnboundedTo(500) + .addRange(500, 1000) + .addUnboundedFrom(1000) + .subAggregation(terms("cities").field("city"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + GeoDistance geoDist = response.getAggregations().get("amsterdam_rings"); + assertThat(geoDist, notNullValue()); + assertThat(geoDist.getName(), equalTo("amsterdam_rings")); + assertThat(geoDist.buckets().size(), equalTo(3)); + + GeoDistance.Bucket bucket = geoDist.getByKey("*-500.0"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("*-500.0")); + assertThat(bucket.getFrom(), equalTo(0.0)); + assertThat(bucket.getTo(), equalTo(500.0)); + assertThat(bucket.getDocCount(), equalTo(2l)); + assertThat(bucket.getAggregations().asList().isEmpty(), is(false)); + Terms cities = bucket.getAggregations().get("cities"); + assertThat(cities, Matchers.notNullValue()); + Set names = Sets.newHashSet(); + for (Terms.Bucket city : cities) { + names.add(city.getKey().string()); + } + assertThat(names.contains("utrecht") && names.contains("haarlem"), is(true)); + + bucket = geoDist.getByKey("500.0-1000.0"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("500.0-1000.0")); + assertThat(bucket.getFrom(), equalTo(500.0)); + assertThat(bucket.getTo(), equalTo(1000.0)); + assertThat(bucket.getDocCount(), equalTo(2l)); + assertThat(bucket.getAggregations().asList().isEmpty(), is(false)); + cities = bucket.getAggregations().get("cities"); + assertThat(cities, Matchers.notNullValue()); + names = Sets.newHashSet(); + for (Terms.Bucket city : cities) { + names.add(city.getKey().string()); + } + assertThat(names.contains("berlin") && names.contains("prague"), is(true)); + + bucket = geoDist.getByKey("1000.0-*"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("1000.0-*")); + assertThat(bucket.getFrom(), equalTo(1000.0)); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getDocCount(), equalTo(1l)); + assertThat(bucket.getAggregations().asList().isEmpty(), is(false)); + cities = bucket.getAggregations().get("cities"); + assertThat(cities, Matchers.notNullValue()); + names = Sets.newHashSet(); + for (Terms.Bucket city : cities) { + names.add(city.getKey().string()); + } + assertThat(names.contains("tel-aviv"), is(true)); + } + + @Test + public void emptyAggregation() throws Exception { + prepareCreate("empty_bucket_idx").addMapping("type", "value", "type=integer", "location", "type=geo_point").execute().actionGet(); + List builders = new ArrayList(); + for (int i = 0; i < 2; i++) { + builders.add(client().prepareIndex("empty_bucket_idx", "type", "" + i).setSource(jsonBuilder() + .startObject() + .field("value", i * 2) + .field("location", "52.0945, 5.116") + .endObject())); + } + indexRandom(true, builders.toArray(new IndexRequestBuilder[builders.size()])); + + SearchResponse searchResponse = client().prepareSearch("empty_bucket_idx") + .setQuery(matchAllQuery()) + .addAggregation(histogram("histo").field("value").interval(1l).emptyBuckets(true) + .subAggregation(geoDistance("geo_dist").field("location").point("52.3760, 4.894").addRange("0-100", 0.0, 100.0))) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(2l)); + Histogram histo = searchResponse.getAggregations().get("histo"); + assertThat(histo, Matchers.notNullValue()); + Histogram.Bucket bucket = histo.getByKey(1l); + assertThat(bucket, Matchers.notNullValue()); + + GeoDistance geoDistance = bucket.getAggregations().get("geo_dist"); + assertThat(geoDistance, Matchers.notNullValue()); + assertThat(geoDistance.getName(), equalTo("geo_dist")); + assertThat(geoDistance.buckets().size(), is(1)); + assertThat(geoDistance.buckets().get(0).getKey(), equalTo("0-100")); + assertThat(geoDistance.buckets().get(0).getFrom(), equalTo(0.0)); + assertThat(geoDistance.buckets().get(0).getTo(), equalTo(100.0)); + assertThat(geoDistance.buckets().get(0).getDocCount(), equalTo(0l)); + + } +} diff --git a/src/test/java/org/elasticsearch/search/aggregations/bucket/GlobalTests.java b/src/test/java/org/elasticsearch/search/aggregations/bucket/GlobalTests.java new file mode 100644 index 00000000000..af84352e441 --- /dev/null +++ b/src/test/java/org/elasticsearch/search/aggregations/bucket/GlobalTests.java @@ -0,0 +1,130 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket; + +import org.elasticsearch.ElasticSearchException; +import org.elasticsearch.action.index.IndexRequestBuilder; +import org.elasticsearch.action.search.SearchResponse; +import org.elasticsearch.common.settings.ImmutableSettings; +import org.elasticsearch.common.settings.Settings; +import org.elasticsearch.index.query.QueryBuilders; +import org.elasticsearch.search.aggregations.bucket.global.Global; +import org.elasticsearch.search.aggregations.metrics.stats.Stats; +import org.elasticsearch.test.ElasticsearchIntegrationTest; +import org.junit.Before; +import org.junit.Test; + +import java.util.ArrayList; +import java.util.List; + +import static org.elasticsearch.common.xcontent.XContentFactory.jsonBuilder; +import static org.elasticsearch.search.aggregations.AggregationBuilders.global; +import static org.elasticsearch.search.aggregations.AggregationBuilders.stats; +import static org.hamcrest.Matchers.equalTo; +import static org.hamcrest.Matchers.is; +import static org.hamcrest.core.IsNull.notNullValue; + +/** + * + */ +public class GlobalTests extends ElasticsearchIntegrationTest { + + int numDocs; + + @Override + public Settings indexSettings() { + return ImmutableSettings.builder() + .put("index.number_of_shards", between(1, 5)) + .put("index.number_of_replicas", between(0, 1)) + .build(); + } + + @Before + public void init() throws Exception { + createIndex("idx"); + createIndex("idx2"); + List builders = new ArrayList(); + numDocs = randomIntBetween(3, 20); + for (int i = 0; i < numDocs / 2; i++) { + builders.add(client().prepareIndex("idx", "type", ""+i+1).setSource(jsonBuilder() + .startObject() + .field("value", i + 1) + .field("tag", "tag1") + .endObject())); + } + for (int i = numDocs / 2; i < numDocs; i++) { + builders.add(client().prepareIndex("idx", "type", ""+i+1).setSource(jsonBuilder() + .startObject() + .field("value", i + 1) + .field("tag", "tag2") + .field("name", "name" + i+1) + .endObject())); + } + indexRandom(true, builders); + } + + @Test + public void withStatsSubAggregator() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .setQuery(QueryBuilders.termQuery("tag", "tag1")) + .addAggregation(global("global") + .subAggregation(stats("value_stats").field("value"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Global global = response.getAggregations().get("global"); + assertThat(global, notNullValue()); + assertThat(global.getName(), equalTo("global")); + assertThat(global.getDocCount(), equalTo((long) numDocs)); + assertThat(global.getAggregations().asList().isEmpty(), is(false)); + + Stats stats = global.getAggregations().get("value_stats"); + assertThat(stats, notNullValue()); + assertThat(stats.getName(), equalTo("value_stats")); + long sum = 0; + for (int i = 0; i < numDocs; ++i) { + sum += i + 1; + } + assertThat(stats.getAvg(), equalTo((double) sum / numDocs)); + assertThat(stats.getMin(), equalTo(1.0)); + assertThat(stats.getMax(), equalTo((double) numDocs)); + assertThat(stats.getCount(), equalTo((long) numDocs)); + assertThat(stats.getSum(), equalTo((double) sum)); + } + + @Test + public void nonTopLevel() throws Exception { + + try { + + client().prepareSearch("idx") + .setQuery(QueryBuilders.termQuery("tag", "tag1")) + .addAggregation(global("global") + .subAggregation(global("inner_global"))) + .execute().actionGet(); + + fail("expected to fail executing non-top-level global aggregator. global aggregations are only allowed as top level" + + "aggregations"); + + } catch (ElasticSearchException ese) { + } + } +} diff --git a/src/test/java/org/elasticsearch/search/aggregations/bucket/HistogramTests.java b/src/test/java/org/elasticsearch/search/aggregations/bucket/HistogramTests.java new file mode 100644 index 00000000000..375433e564f --- /dev/null +++ b/src/test/java/org/elasticsearch/search/aggregations/bucket/HistogramTests.java @@ -0,0 +1,757 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket; + +import com.carrotsearch.hppc.LongOpenHashSet; +import org.elasticsearch.action.index.IndexRequestBuilder; +import org.elasticsearch.action.search.SearchResponse; +import org.elasticsearch.common.settings.ImmutableSettings; +import org.elasticsearch.common.settings.Settings; +import org.elasticsearch.search.aggregations.bucket.histogram.Histogram; +import org.elasticsearch.search.aggregations.bucket.terms.Terms; +import org.elasticsearch.search.aggregations.metrics.stats.Stats; +import org.elasticsearch.search.aggregations.metrics.sum.Sum; +import org.elasticsearch.test.ElasticsearchIntegrationTest; +import org.hamcrest.Matchers; +import org.junit.Before; +import org.junit.Test; + +import java.util.ArrayList; +import java.util.Iterator; +import java.util.List; + +import static org.elasticsearch.common.xcontent.XContentFactory.jsonBuilder; +import static org.elasticsearch.index.query.QueryBuilders.matchAllQuery; +import static org.elasticsearch.search.aggregations.AggregationBuilders.*; +import static org.hamcrest.Matchers.*; +import static org.hamcrest.core.IsNull.notNullValue; + +/** + * + */ +public class HistogramTests extends ElasticsearchIntegrationTest { + + @Override + public Settings indexSettings() { + return ImmutableSettings.builder() + .put("index.number_of_shards", between(1, 5)) + .put("index.number_of_replicas", between(0, 1)) + .build(); + } + + int numDocs; + int interval; + int numValueBuckets, numValuesBuckets; + long[] valueCounts, valuesCounts; + + @Before + public void init() throws Exception { + createIndex("idx"); + createIndex("idx_unmapped"); + + numDocs = randomIntBetween(6, 20); + interval = randomIntBetween(2, 5); + + numValueBuckets = numDocs / interval + 1; + valueCounts = new long[numValueBuckets]; + for (int i = 0; i < numDocs; ++i) { + final int bucket = (i + 1) / interval; + ++valueCounts[bucket]; + } + + numValuesBuckets = (numDocs + 1) / interval + 1; + valuesCounts = new long[numValuesBuckets]; + for (int i = 0; i < numDocs; ++i) { + final int bucket1 = (i + 1) / interval; + final int bucket2 = (i + 2) / interval; + ++valuesCounts[bucket1]; + if (bucket1 != bucket2) { + ++valuesCounts[bucket2]; + } + } + + IndexRequestBuilder[] builders = new IndexRequestBuilder[numDocs]; + + for (int i = 0; i < builders.length; i++) { + builders[i] = client().prepareIndex("idx", "type").setSource(jsonBuilder() + .startObject() + .field("value", i + 1) + .startArray("values").value(i + 1).value(i + 2).endArray() + .field("tag", "tag" + i) + .endObject()); + } + indexRandom(true, builders); + + } + + @Test + public void singleValuedField() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(histogram("histo").field("value").interval(interval)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Histogram histo = response.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + assertThat(histo.getName(), equalTo("histo")); + assertThat(histo.buckets().size(), equalTo(numValueBuckets)); + + for (int i = 0; i < numValueBuckets; ++i) { + Histogram.Bucket bucket = histo.getByKey(i * interval); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo((long) i * interval)); + assertThat(bucket.getDocCount(), equalTo(valueCounts[i])); + } + } + + @Test + public void singleValuedField_OrderedByKeyAsc() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(histogram("histo").field("value").interval(interval).order(Histogram.Order.KEY_ASC)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Histogram histo = response.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + assertThat(histo.getName(), equalTo("histo")); + assertThat(histo.buckets().size(), equalTo(numValueBuckets)); + + for (int i = 0; i < numValueBuckets; ++i) { + Histogram.Bucket bucket = histo.buckets().get(i); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo((long) i * interval)); + assertThat(bucket.getDocCount(), equalTo(valueCounts[i])); + } + } + + @Test + public void singleValuedField_OrderedByKeyDesc() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(histogram("histo").field("value").interval(interval).order(Histogram.Order.KEY_DESC)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Histogram histo = response.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + assertThat(histo.getName(), equalTo("histo")); + assertThat(histo.buckets().size(), equalTo(numValueBuckets)); + + for (int i = 0; i < numValueBuckets; ++i) { + Histogram.Bucket bucket = histo.buckets().get(numValueBuckets -i - 1); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo((long) i * interval)); + assertThat(bucket.getDocCount(), equalTo(valueCounts[i])); + } + } + + @Test + public void singleValuedField_OrderedByCountAsc() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(histogram("histo").field("value").interval(interval).order(Histogram.Order.COUNT_ASC)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Histogram histo = response.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + assertThat(histo.getName(), equalTo("histo")); + assertThat(histo.buckets().size(), equalTo(numValueBuckets)); + + LongOpenHashSet buckets = new LongOpenHashSet(); + long previousCount = Long.MIN_VALUE; + for (int i = 0; i < numValueBuckets; ++i) { + Histogram.Bucket bucket = histo.buckets().get(i); + assertThat(bucket, notNullValue()); + long key = bucket.getKey(); + assertEquals(0, key % interval); + assertTrue(buckets.add(key)); + assertThat(bucket.getDocCount(), equalTo(valueCounts[(int) (key / interval)])); + assertThat(bucket.getDocCount(), greaterThanOrEqualTo(previousCount)); + previousCount = bucket.getDocCount(); + } + } + + @Test + public void singleValuedField_OrderedByCountDesc() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(histogram("histo").field("value").interval(interval).order(Histogram.Order.COUNT_DESC)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Histogram histo = response.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + assertThat(histo.getName(), equalTo("histo")); + assertThat(histo.buckets().size(), equalTo(numValueBuckets)); + + LongOpenHashSet buckets = new LongOpenHashSet(); + long previousCount = Long.MAX_VALUE; + for (int i = 0; i < numValueBuckets; ++i) { + Histogram.Bucket bucket = histo.buckets().get(i); + assertThat(bucket, notNullValue()); + long key = bucket.getKey(); + assertEquals(0, key % interval); + assertTrue(buckets.add(key)); + assertThat(bucket.getDocCount(), equalTo(valueCounts[(int) (key / interval)])); + assertThat(bucket.getDocCount(), lessThanOrEqualTo(previousCount)); + previousCount = bucket.getDocCount(); + } + } + + @Test + public void singleValuedField_WithSubAggregation() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(histogram("histo").field("value").interval(interval) + .subAggregation(sum("sum").field("value"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Histogram histo = response.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + assertThat(histo.getName(), equalTo("histo")); + assertThat(histo.buckets().size(), equalTo(numValueBuckets)); + + for (int i = 0; i < numValueBuckets; ++i) { + Histogram.Bucket bucket = histo.buckets().get(i); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo((long) i * interval)); + assertThat(bucket.getDocCount(), equalTo(valueCounts[i])); + assertThat(bucket.getAggregations().asList().isEmpty(), is(false)); + Sum sum = bucket.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + long s = 0; + for (int j = 0; j < numDocs; ++j) { + if ((j + 1) / interval == i) { + s += j + 1; + } + } + assertThat(sum.getValue(), equalTo((double) s)); + } + } + + @Test + public void singleValuedField_WithSubAggregation_Inherited() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(histogram("histo").field("value").interval(interval) + .subAggregation(sum("sum"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Histogram histo = response.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + assertThat(histo.getName(), equalTo("histo")); + assertThat(histo.buckets().size(), equalTo(numValueBuckets)); + + for (int i = 0; i < numValueBuckets; ++i) { + Histogram.Bucket bucket = histo.buckets().get(i); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo((long) i * interval)); + assertThat(bucket.getDocCount(), equalTo(valueCounts[i])); + assertThat(bucket.getAggregations().asList().isEmpty(), is(false)); + Sum sum = bucket.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + long s = 0; + for (int j = 0; j < numDocs; ++j) { + if ((j + 1) / interval == i) { + s += j + 1; + } + } + assertThat(sum.getValue(), equalTo((double) s)); + } + } + + @Test + public void singleValuedField_OrderedBySubAggregationAsc() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(histogram("histo").field("value").interval(interval).order(Histogram.Order.aggregation("sum", true)) + .subAggregation(sum("sum").field("value"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Histogram histo = response.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + assertThat(histo.getName(), equalTo("histo")); + assertThat(histo.buckets().size(), equalTo(numValueBuckets)); + + LongOpenHashSet visited = new LongOpenHashSet(); + double previousSum = Double.NEGATIVE_INFINITY; + for (int i = 0; i < numValueBuckets; ++i) { + Histogram.Bucket bucket = histo.buckets().get(i); + assertThat(bucket, notNullValue()); + long key = bucket.getKey(); + assertTrue(visited.add(key)); + int b = (int) (key / interval); + assertThat(bucket.getDocCount(), equalTo(valueCounts[b])); + assertThat(bucket.getAggregations().asList().isEmpty(), is(false)); + Sum sum = bucket.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + long s = 0; + for (int j = 0; j < numDocs; ++j) { + if ((j + 1) / interval == b) { + s += j + 1; + } + } + assertThat(sum.getValue(), equalTo((double) s)); + assertThat(sum.getValue(), greaterThanOrEqualTo(previousSum)); + previousSum = s; + } + } + + @Test + public void singleValuedField_OrderedBySubAggregationDesc() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(histogram("histo").field("value").interval(interval).order(Histogram.Order.aggregation("sum", false)) + .subAggregation(sum("sum").field("value"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Histogram histo = response.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + assertThat(histo.getName(), equalTo("histo")); + assertThat(histo.buckets().size(), equalTo(numValueBuckets)); + + LongOpenHashSet visited = new LongOpenHashSet(); + double previousSum = Double.POSITIVE_INFINITY; + for (int i = 0; i < numValueBuckets; ++i) { + Histogram.Bucket bucket = histo.buckets().get(i); + assertThat(bucket, notNullValue()); + long key = bucket.getKey(); + assertTrue(visited.add(key)); + int b = (int) (key / interval); + assertThat(bucket.getDocCount(), equalTo(valueCounts[b])); + assertThat(bucket.getAggregations().asList().isEmpty(), is(false)); + Sum sum = bucket.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + long s = 0; + for (int j = 0; j < numDocs; ++j) { + if ((j + 1) / interval == b) { + s += j + 1; + } + } + assertThat(sum.getValue(), equalTo((double) s)); + assertThat(sum.getValue(), lessThanOrEqualTo(previousSum)); + previousSum = s; + } + } + + @Test + public void singleValuedField_OrderedByMultiValuedSubAggregationAsc_Inherited() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(histogram("histo").field("value").interval(interval).order(Histogram.Order.aggregation("stats.sum", true)) + .subAggregation(stats("stats"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Histogram histo = response.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + assertThat(histo.getName(), equalTo("histo")); + assertThat(histo.buckets().size(), equalTo(numValueBuckets)); + + LongOpenHashSet visited = new LongOpenHashSet(); + double previousSum = Double.NEGATIVE_INFINITY; + for (int i = 0; i < numValueBuckets; ++i) { + Histogram.Bucket bucket = histo.buckets().get(i); + assertThat(bucket, notNullValue()); + long key = bucket.getKey(); + assertTrue(visited.add(key)); + int b = (int) (key / interval); + assertThat(bucket.getDocCount(), equalTo(valueCounts[b])); + assertThat(bucket.getAggregations().asList().isEmpty(), is(false)); + Stats stats = bucket.getAggregations().get("stats"); + assertThat(stats, notNullValue()); + long s = 0; + for (int j = 0; j < numDocs; ++j) { + if ((j + 1) / interval == b) { + s += j + 1; + } + } + assertThat(stats.getSum(), equalTo((double) s)); + assertThat(stats.getSum(), greaterThanOrEqualTo(previousSum)); + previousSum = s; + } + } + + @Test + public void singleValuedField_OrderedByMultiValuedSubAggregationDesc() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(histogram("histo").field("value").interval(interval).order(Histogram.Order.aggregation("stats.sum", false)) + .subAggregation(stats("stats").field("value"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Histogram histo = response.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + assertThat(histo.getName(), equalTo("histo")); + assertThat(histo.buckets().size(), equalTo(numValueBuckets)); + + LongOpenHashSet visited = new LongOpenHashSet(); + double previousSum = Double.POSITIVE_INFINITY; + for (int i = 0; i < numValueBuckets; ++i) { + Histogram.Bucket bucket = histo.buckets().get(i); + assertThat(bucket, notNullValue()); + long key = bucket.getKey(); + assertTrue(visited.add(key)); + int b = (int) (key / interval); + assertThat(bucket.getDocCount(), equalTo(valueCounts[b])); + assertThat(bucket.getAggregations().asList().isEmpty(), is(false)); + Stats stats = bucket.getAggregations().get("stats"); + assertThat(stats, notNullValue()); + long s = 0; + for (int j = 0; j < numDocs; ++j) { + if ((j + 1) / interval == b) { + s += j + 1; + } + } + assertThat(stats.getSum(), equalTo((double) s)); + assertThat(stats.getSum(), lessThanOrEqualTo(previousSum)); + previousSum = s; + } + } + + @Test + public void singleValuedField_WithValueScript() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(histogram("histo").field("value").script("_value + 1").interval(interval)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + final int numBuckets = (numDocs + 1) / interval - 2 / interval + 1; + final long[] counts = new long[(numDocs + 1) / interval + 1]; + for (int i = 0; i < numDocs ; ++i) { + ++counts[(i + 2) / interval]; + } + + Histogram histo = response.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + assertThat(histo.getName(), equalTo("histo")); + assertThat(histo.buckets().size(), equalTo(numBuckets)); + + for (int i = 2 / interval; i <= (numDocs + 1) / interval; ++i) { + Histogram.Bucket bucket = histo.getByKey(i * interval); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo((long) i * interval)); + assertThat(bucket.getDocCount(), equalTo(counts[i])); + } + } + + @Test + public void multiValuedField() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(histogram("histo").field("values").interval(interval)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Histogram histo = response.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + assertThat(histo.getName(), equalTo("histo")); + assertThat(histo.buckets().size(), equalTo(numValuesBuckets)); + + for (int i = 0; i < numValuesBuckets; ++i) { + Histogram.Bucket bucket = histo.getByKey(i * interval); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo((long) i * interval)); + assertThat(bucket.getDocCount(), equalTo(valuesCounts[i])); + } + } + + @Test + public void multiValuedField_OrderedByKeyDesc() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(histogram("histo").field("values").interval(interval).order(Histogram.Order.KEY_DESC)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Histogram histo = response.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + assertThat(histo.getName(), equalTo("histo")); + assertThat(histo.buckets().size(), equalTo(numValuesBuckets)); + + for (int i = 0; i < numValuesBuckets; ++i) { + Histogram.Bucket bucket = histo.buckets().get(numValuesBuckets -i - 1); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo((long) i * interval)); + assertThat(bucket.getDocCount(), equalTo(valuesCounts[i])); + } + } + + @Test + public void multiValuedField_WithValueScript() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(histogram("histo").field("values").script("_value + 1").interval(interval)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + final int numBuckets = (numDocs + 2) / interval - 2 / interval + 1; + final long[] counts = new long[(numDocs + 2) / interval + 1]; + for (int i = 0; i < numDocs ; ++i) { + final int bucket1 = (i + 2) / interval; + final int bucket2 = (i + 3) / interval; + ++counts[bucket1]; + if (bucket1 != bucket2) { + ++counts[bucket2]; + } + } + + Histogram histo = response.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + assertThat(histo.getName(), equalTo("histo")); + assertThat(histo.buckets().size(), equalTo(numBuckets)); + + for (int i = 2 / interval; i <= (numDocs + 2) / interval; ++i) { + Histogram.Bucket bucket = histo.getByKey(i * interval); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo((long) i * interval)); + assertThat(bucket.getDocCount(), equalTo(counts[i])); + } + } + + @Test + public void multiValuedField_WithValueScript_WithInheritedSubAggregator() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(histogram("histo").field("values").script("_value + 1").interval(interval) + .subAggregation(terms("values").order(Terms.Order.TERM_ASC))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + final int numBuckets = (numDocs + 2) / interval - 2 / interval + 1; + final long[] counts = new long[(numDocs + 2) / interval + 1]; + for (int i = 0; i < numDocs ; ++i) { + final int bucket1 = (i + 2) / interval; + final int bucket2 = (i + 3) / interval; + ++counts[bucket1]; + if (bucket1 != bucket2) { + ++counts[bucket2]; + } + } + + Histogram histo = response.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + assertThat(histo.getName(), equalTo("histo")); + assertThat(histo.buckets().size(), equalTo(numBuckets)); + + for (int i = 2 / interval; i < (numDocs + 2) / interval; ++i) { + Histogram.Bucket bucket = histo.getByKey(i * interval); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo((long) i * interval)); + assertThat(bucket.getDocCount(), equalTo(counts[i])); + Terms terms = bucket.getAggregations().get("values"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("values")); + int minTerm = Math.max(2, i * interval - 1); + int maxTerm = Math.min(numDocs + 2, (i + 1) * interval); + assertThat(terms.buckets().size(), equalTo(maxTerm - minTerm + 1)); + Iterator iter = terms.iterator(); + for (int j = minTerm; j <= maxTerm; ++j) { + assertThat(iter.next().getKeyAsNumber().longValue(), equalTo((long) j)); + } + } + } + + @Test + public void script_SingleValue() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(histogram("histo").script("doc['value'].value").interval(interval)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Histogram histo = response.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + assertThat(histo.getName(), equalTo("histo")); + assertThat(histo.buckets().size(), equalTo(numValueBuckets)); + + for (int i = 0; i < numValueBuckets; ++i) { + Histogram.Bucket bucket = histo.getByKey(i * interval); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo((long) i * interval)); + assertThat(bucket.getDocCount(), equalTo(valueCounts[i])); + } + } + + @Test + public void script_SingleValue_WithSubAggregator_Inherited() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(histogram("histo").script("doc['value'].value").interval(interval) + .subAggregation(sum("sum"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Histogram histo = response.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + assertThat(histo.getName(), equalTo("histo")); + assertThat(histo.buckets().size(), equalTo(numValueBuckets)); + + for (int i = 0; i < numValueBuckets; ++i) { + Histogram.Bucket bucket = histo.buckets().get(i); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo((long) i * interval)); + assertThat(bucket.getDocCount(), equalTo(valueCounts[i])); + assertThat(bucket.getAggregations().asList().isEmpty(), is(false)); + Sum sum = bucket.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + long s = 0; + for (int j = 0; j < numDocs; ++j) { + if ((j + 1) / interval == i) { + s += j + 1; + } + } + assertThat(sum.getValue(), equalTo((double) s)); + } + } + + @Test + public void script_MultiValued() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(histogram("histo").script("doc['values'].values").interval(interval)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Histogram histo = response.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + assertThat(histo.getName(), equalTo("histo")); + assertThat(histo.buckets().size(), equalTo(numValuesBuckets)); + + for (int i = 0; i < numValuesBuckets; ++i) { + Histogram.Bucket bucket = histo.getByKey(i * interval); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo((long) i * interval)); + assertThat(bucket.getDocCount(), equalTo(valuesCounts[i])); + } + } + + @Test + public void script_MultiValued_WithAggregatorInherited() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(histogram("histo").script("doc['values'].values").interval(interval) + .subAggregation(sum("sum"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Histogram histo = response.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + assertThat(histo.getName(), equalTo("histo")); + assertThat(histo.buckets().size(), equalTo(numValuesBuckets)); + + for (int i = 0; i < numValuesBuckets; ++i) { + Histogram.Bucket bucket = histo.getByKey(i * interval); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo((long) i * interval)); + assertThat(bucket.getDocCount(), equalTo(valuesCounts[i])); + assertThat(bucket.getAggregations().asList().isEmpty(), is(false)); + Sum sum = bucket.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + long s = 0; + for (int j = 0; j < numDocs; ++j) { + if ((j + 1) / interval == i || (j + 2) / interval == i) { + s += j + 1; + s += j + 2; + } + } + assertThat(sum.getValue(), equalTo((double) s)); + } + } + + @Test + public void unmapped() throws Exception { + client().admin().cluster().prepareHealth("idx_unmapped").setWaitForYellowStatus().execute().actionGet(); + + SearchResponse response = client().prepareSearch("idx_unmapped") + .addAggregation(histogram("histo").field("value").interval(interval)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Histogram histo = response.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + assertThat(histo.getName(), equalTo("histo")); + assertThat(histo.buckets().size(), equalTo(0)); + } + + @Test + public void partiallyUnmapped() throws Exception { + client().admin().cluster().prepareHealth("idx_unmapped").setWaitForYellowStatus().execute().actionGet(); + + SearchResponse response = client().prepareSearch("idx", "idx_unmapped") + .addAggregation(histogram("histo").field("value").interval(interval)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Histogram histo = response.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + assertThat(histo.getName(), equalTo("histo")); + assertThat(histo.buckets().size(), equalTo(numValueBuckets)); + + for (int i = 0; i < numValueBuckets; ++i) { + Histogram.Bucket bucket = histo.getByKey(i * interval); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo((long) i * interval)); + assertThat(bucket.getDocCount(), equalTo(valueCounts[i])); + } + } + + @Test + public void emptyAggregation() throws Exception { + prepareCreate("empty_bucket_idx").addMapping("type", "value", "type=integer").execute().actionGet(); + List builders = new ArrayList(); + for (int i = 0; i < 2; i++) { + builders.add(client().prepareIndex("empty_bucket_idx", "type", "" + i).setSource(jsonBuilder() + .startObject() + .field("value", i * 2) + .endObject())); + } + indexRandom(true, builders.toArray(new IndexRequestBuilder[builders.size()])); + + SearchResponse searchResponse = client().prepareSearch("empty_bucket_idx") + .setQuery(matchAllQuery()) + .addAggregation(histogram("histo").field("value").interval(1l).emptyBuckets(true) + .subAggregation(histogram("sub_histo").interval(1l))) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(2l)); + Histogram histo = searchResponse.getAggregations().get("histo"); + assertThat(histo, Matchers.notNullValue()); + Histogram.Bucket bucket = histo.getByKey(1l); + assertThat(bucket, Matchers.notNullValue()); + + histo = bucket.getAggregations().get("sub_histo"); + assertThat(histo, Matchers.notNullValue()); + assertThat(histo.getName(), equalTo("sub_histo")); + assertThat(histo.buckets().isEmpty(), is(true)); + } + +} diff --git a/src/test/java/org/elasticsearch/search/aggregations/bucket/IPv4RangeTests.java b/src/test/java/org/elasticsearch/search/aggregations/bucket/IPv4RangeTests.java new file mode 100644 index 00000000000..9e982f89a73 --- /dev/null +++ b/src/test/java/org/elasticsearch/search/aggregations/bucket/IPv4RangeTests.java @@ -0,0 +1,853 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket; + +import org.elasticsearch.action.index.IndexRequestBuilder; +import org.elasticsearch.action.search.SearchResponse; +import org.elasticsearch.common.settings.ImmutableSettings; +import org.elasticsearch.common.settings.Settings; +import org.elasticsearch.index.mapper.ip.IpFieldMapper; +import org.elasticsearch.search.aggregations.bucket.histogram.Histogram; +import org.elasticsearch.search.aggregations.bucket.range.ipv4.IPv4Range; +import org.elasticsearch.search.aggregations.metrics.max.Max; +import org.elasticsearch.search.aggregations.metrics.sum.Sum; +import org.elasticsearch.test.ElasticsearchIntegrationTest; +import org.hamcrest.Matchers; +import org.junit.Before; +import org.junit.Test; + +import java.util.ArrayList; +import java.util.List; + +import static org.elasticsearch.common.xcontent.XContentFactory.jsonBuilder; +import static org.elasticsearch.index.query.QueryBuilders.matchAllQuery; +import static org.elasticsearch.search.aggregations.AggregationBuilders.*; +import static org.hamcrest.Matchers.equalTo; +import static org.hamcrest.Matchers.is; +import static org.hamcrest.core.IsNull.notNullValue; +import static org.hamcrest.core.IsNull.nullValue; + +/** + * + */ +public class IPv4RangeTests extends ElasticsearchIntegrationTest { + + @Override + public Settings indexSettings() { + return ImmutableSettings.builder() + .put("index.number_of_shards", between(1, 5)) + .put("index.number_of_replicas", between(0, 1)) + .build(); + } + + @Before + public void init() throws Exception { + prepareCreate("idx") + .addMapping("type", "ip", "type=ip", "ips", "type=ip") + .execute().actionGet(); + IndexRequestBuilder[] builders = new IndexRequestBuilder[255]; // TODO randomize the size? + // TODO randomize the values in the docs? + for (int i = 0; i < builders.length; i++) { + builders[i] = client().prepareIndex("idx", "type").setSource(jsonBuilder() + .startObject() + .field("ip", "10.0.0." + (i)) + .startArray("ips").value("10.0.0." + i).value("10.0.0." + (i + 1)).endArray() + .field("value", (i < 100 ? 1 : i < 200 ? 2 : 3)) // 100 1's, 100 2's, and 55 3's + .endObject()); + } + indexRandom(true, builders); + createIndex("idx_unmapped"); + + } + + @Test + public void singleValueField() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(ipRange("range") + .field("ip") + .addUnboundedTo("10.0.0.100") + .addRange("10.0.0.100", "10.0.0.200") + .addUnboundedFrom("10.0.0.200")) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + IPv4Range range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(3)); + + IPv4Range.Bucket bucket = range.getByKey("*-10.0.0.100"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("*-10.0.0.100")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getFromAsString(), nullValue()); + assertThat(bucket.getToAsString(), equalTo("10.0.0.100")); + assertThat(bucket.getTo(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.100"))); + assertThat(bucket.getDocCount(), equalTo(100l)); + + bucket = range.getByKey("10.0.0.100-10.0.0.200"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("10.0.0.100-10.0.0.200")); + assertThat(bucket.getFromAsString(), equalTo("10.0.0.100")); + assertThat(bucket.getFrom(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.100"))); + assertThat(bucket.getToAsString(), equalTo("10.0.0.200")); + assertThat(bucket.getTo(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.200"))); + assertThat(bucket.getDocCount(), equalTo(100l)); + + bucket = range.getByKey("10.0.0.200-*"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("10.0.0.200-*")); + assertThat(bucket.getFromAsString(), equalTo("10.0.0.200")); + assertThat(bucket.getFrom(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.200"))); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getToAsString(), nullValue()); + assertThat(bucket.getDocCount(), equalTo(55l)); + } + + @Test + public void singleValueField_WithMaskRange() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(ipRange("range") + .field("ip") + .addMaskRange("10.0.0.0/25") + .addMaskRange("10.0.0.128/25")) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + IPv4Range range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(2)); + + IPv4Range.Bucket bucket = range.getByKey("10.0.0.0/25"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("10.0.0.0/25")); + assertThat(bucket.getFrom(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.0"))); + assertThat(bucket.getFromAsString(), equalTo("10.0.0.0")); + assertThat(bucket.getTo(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.128"))); + assertThat(bucket.getToAsString(), equalTo("10.0.0.128")); + assertThat(bucket.getDocCount(), equalTo(128l)); + + bucket = range.getByKey("10.0.0.128/25"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("10.0.0.128/25")); + assertThat((long) bucket.getFrom(), equalTo(IpFieldMapper.ipToLong("10.0.0.128"))); + assertThat(bucket.getFromAsString(), equalTo("10.0.0.128")); + assertThat((long) bucket.getTo(), equalTo(IpFieldMapper.ipToLong("10.0.1.0"))); // range is exclusive on the to side + assertThat(bucket.getToAsString(), equalTo("10.0.1.0")); + assertThat(bucket.getDocCount(), equalTo(127l)); // include 10.0.0.128 + } + + @Test + public void singleValueField_WithCustomKey() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(ipRange("range") + .field("ip") + .addUnboundedTo("r1", "10.0.0.100") + .addRange("r2", "10.0.0.100", "10.0.0.200") + .addUnboundedFrom("r3", "10.0.0.200")) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + IPv4Range range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(3)); + + IPv4Range.Bucket bucket = range.getByKey("r1"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("r1")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getFromAsString(), nullValue()); + assertThat(bucket.getToAsString(), equalTo("10.0.0.100")); + assertThat(bucket.getTo(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.100"))); + assertThat(bucket.getDocCount(), equalTo(100l)); + + bucket = range.getByKey("r2"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("r2")); + assertThat(bucket.getFromAsString(), equalTo("10.0.0.100")); + assertThat(bucket.getFrom(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.100"))); + assertThat(bucket.getToAsString(), equalTo("10.0.0.200")); + assertThat(bucket.getTo(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.200"))); + assertThat(bucket.getDocCount(), equalTo(100l)); + + bucket = range.getByKey("r3"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("r3")); + assertThat(bucket.getFromAsString(), equalTo("10.0.0.200")); + assertThat(bucket.getFrom(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.200"))); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getToAsString(), nullValue()); + assertThat(bucket.getDocCount(), equalTo(55l)); + } + + @Test + public void singleValuedField_WithSubAggregation() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(ipRange("range") + .field("ip") + .addUnboundedTo("10.0.0.100") + .addRange("10.0.0.100", "10.0.0.200") + .addUnboundedFrom("10.0.0.200") + .subAggregation(sum("sum").field("value"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + IPv4Range range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(3)); + + IPv4Range.Bucket bucket = range.getByKey("*-10.0.0.100"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("*-10.0.0.100")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getFromAsString(), nullValue()); + assertThat(bucket.getToAsString(), equalTo("10.0.0.100")); + assertThat(bucket.getTo(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.100"))); + assertThat(bucket.getDocCount(), equalTo(100l)); + Sum sum = bucket.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + assertThat(sum.getValue(), equalTo((double) 100)); + + bucket = range.getByKey("10.0.0.100-10.0.0.200"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("10.0.0.100-10.0.0.200")); + assertThat(bucket.getFromAsString(), equalTo("10.0.0.100")); + assertThat(bucket.getFrom(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.100"))); + assertThat(bucket.getToAsString(), equalTo("10.0.0.200")); + assertThat(bucket.getTo(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.200"))); + assertThat(bucket.getDocCount(), equalTo(100l)); + sum = bucket.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + assertThat(sum.getValue(), equalTo((double) 200)); + + bucket = range.getByKey("10.0.0.200-*"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("10.0.0.200-*")); + assertThat(bucket.getFromAsString(), equalTo("10.0.0.200")); + assertThat(bucket.getFrom(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.200"))); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getToAsString(), nullValue()); + assertThat(bucket.getDocCount(), equalTo(55l)); + sum = bucket.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + assertThat(sum.getValue(), equalTo((double) 55*3)); + } + + @Test + public void singleValuedField_WithSubAggregation_Inherited() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(ipRange("range") + .field("ip") + .addUnboundedTo("10.0.0.100") + .addRange("10.0.0.100", "10.0.0.200") + .addUnboundedFrom("10.0.0.200") + .subAggregation(max("max"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + IPv4Range range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(3)); + + IPv4Range.Bucket bucket = range.getByKey("*-10.0.0.100"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("*-10.0.0.100")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getFromAsString(), nullValue()); + assertThat(bucket.getToAsString(), equalTo("10.0.0.100")); + assertThat(bucket.getTo(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.100"))); + assertThat(bucket.getDocCount(), equalTo(100l)); + Max max = bucket.getAggregations().get("max"); + assertThat(max, notNullValue()); + assertThat(max.getValue(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.99"))); + + bucket = range.getByKey("10.0.0.100-10.0.0.200"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("10.0.0.100-10.0.0.200")); + assertThat(bucket.getFromAsString(), equalTo("10.0.0.100")); + assertThat(bucket.getFrom(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.100"))); + assertThat(bucket.getToAsString(), equalTo("10.0.0.200")); + assertThat(bucket.getTo(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.200"))); + assertThat(bucket.getDocCount(), equalTo(100l)); + max = bucket.getAggregations().get("max"); + assertThat(max, notNullValue()); + assertThat(max.getValue(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.199"))); + + bucket = range.getByKey("10.0.0.200-*"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("10.0.0.200-*")); + assertThat(bucket.getFromAsString(), equalTo("10.0.0.200")); + assertThat(bucket.getFrom(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.200"))); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getToAsString(), nullValue()); + assertThat(bucket.getDocCount(), equalTo(55l)); + max = bucket.getAggregations().get("max"); + assertThat(max, notNullValue()); + assertThat(max.getValue(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.254"))); + } + + @Test + public void singleValuedField_WithValueScript() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(ipRange("range") + .field("ip") + .script("_value") + .addUnboundedTo("10.0.0.100") + .addRange("10.0.0.100", "10.0.0.200") + .addUnboundedFrom("10.0.0.200")) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + IPv4Range range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(3)); + + IPv4Range.Bucket bucket = range.getByKey("*-10.0.0.100"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("*-10.0.0.100")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getFromAsString(), nullValue()); + assertThat(bucket.getToAsString(), equalTo("10.0.0.100")); + assertThat(bucket.getTo(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.100"))); + assertThat(bucket.getDocCount(), equalTo(100l)); + + bucket = range.getByKey("10.0.0.100-10.0.0.200"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("10.0.0.100-10.0.0.200")); + assertThat(bucket.getFromAsString(), equalTo("10.0.0.100")); + assertThat(bucket.getFrom(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.100"))); + assertThat(bucket.getToAsString(), equalTo("10.0.0.200")); + assertThat(bucket.getTo(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.200"))); + assertThat(bucket.getDocCount(), equalTo(100l)); + + bucket = range.getByKey("10.0.0.200-*"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("10.0.0.200-*")); + assertThat(bucket.getFromAsString(), equalTo("10.0.0.200")); + assertThat(bucket.getFrom(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.200"))); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getToAsString(), nullValue()); + assertThat(bucket.getDocCount(), equalTo(55l)); + } + + /* + [0, 1] + [1, 2] + [2, 3] + ... + [99, 100] + [100, 101] + [101, 102] + ... + [199, 200] + [200, 201] + [201, 202] + ... + [254, 255] + [255, 256] + */ + + @Test + public void multiValuedField() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(ipRange("range") + .field("ips") + .addUnboundedTo("10.0.0.100") + .addRange("10.0.0.100", "10.0.0.200") + .addUnboundedFrom("10.0.0.200")) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + IPv4Range range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(3)); + + IPv4Range.Bucket bucket = range.getByKey("*-10.0.0.100"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("*-10.0.0.100")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getFromAsString(), nullValue()); + assertThat(bucket.getToAsString(), equalTo("10.0.0.100")); + assertThat(bucket.getTo(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.100"))); + assertThat(bucket.getDocCount(), equalTo(100l)); + + bucket = range.getByKey("10.0.0.100-10.0.0.200"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("10.0.0.100-10.0.0.200")); + assertThat(bucket.getFromAsString(), equalTo("10.0.0.100")); + assertThat(bucket.getFrom(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.100"))); + assertThat(bucket.getToAsString(), equalTo("10.0.0.200")); + assertThat(bucket.getTo(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.200"))); + assertThat(bucket.getDocCount(), equalTo(101l)); + + bucket = range.getByKey("10.0.0.200-*"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("10.0.0.200-*")); + assertThat(bucket.getFromAsString(), equalTo("10.0.0.200")); + assertThat(bucket.getFrom(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.200"))); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getToAsString(), nullValue()); + assertThat(bucket.getDocCount(), equalTo(56l)); + } + + @Test + public void multiValuedField_WithValueScript() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(ipRange("range") + .field("ips") + .script("_value") + .addUnboundedTo("10.0.0.100") + .addRange("10.0.0.100", "10.0.0.200") + .addUnboundedFrom("10.0.0.200")) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + IPv4Range range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(3)); + + IPv4Range.Bucket bucket = range.getByKey("*-10.0.0.100"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("*-10.0.0.100")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getFromAsString(), nullValue()); + assertThat(bucket.getToAsString(), equalTo("10.0.0.100")); + assertThat(bucket.getTo(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.100"))); + assertThat(bucket.getDocCount(), equalTo(100l)); + + bucket = range.getByKey("10.0.0.100-10.0.0.200"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("10.0.0.100-10.0.0.200")); + assertThat(bucket.getFromAsString(), equalTo("10.0.0.100")); + assertThat(bucket.getFrom(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.100"))); + assertThat(bucket.getToAsString(), equalTo("10.0.0.200")); + assertThat(bucket.getTo(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.200"))); + assertThat(bucket.getDocCount(), equalTo(101l)); + + bucket = range.getByKey("10.0.0.200-*"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("10.0.0.200-*")); + assertThat(bucket.getFromAsString(), equalTo("10.0.0.200")); + assertThat(bucket.getFrom(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.200"))); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getToAsString(), nullValue()); + assertThat(bucket.getDocCount(), equalTo(56l)); + } + + @Test + public void multiValuedField_WithValueScript_WithInheritedSubAggregator() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(ipRange("range") + .field("ips") + .script("_value") + .addUnboundedTo("10.0.0.100") + .addRange("10.0.0.100", "10.0.0.200") + .addUnboundedFrom("10.0.0.200") + .subAggregation(max("max"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + IPv4Range range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(3)); + + IPv4Range.Bucket bucket = range.getByKey("*-10.0.0.100"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("*-10.0.0.100")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getFromAsString(), nullValue()); + assertThat(bucket.getToAsString(), equalTo("10.0.0.100")); + assertThat(bucket.getTo(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.100"))); + assertThat(bucket.getDocCount(), equalTo(100l)); + Max max = bucket.getAggregations().get("max"); + assertThat(max, Matchers.notNullValue()); + assertThat((long) max.getValue(), equalTo(IpFieldMapper.ipToLong("10.0.0.100"))); + + bucket = range.getByKey("10.0.0.100-10.0.0.200"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("10.0.0.100-10.0.0.200")); + assertThat(bucket.getFromAsString(), equalTo("10.0.0.100")); + assertThat(bucket.getFrom(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.100"))); + assertThat(bucket.getToAsString(), equalTo("10.0.0.200")); + assertThat(bucket.getTo(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.200"))); + assertThat(bucket.getDocCount(), equalTo(101l)); + max = bucket.getAggregations().get("max"); + assertThat(max, Matchers.notNullValue()); + assertThat((long) max.getValue(), equalTo(IpFieldMapper.ipToLong("10.0.0.200"))); + + bucket = range.getByKey("10.0.0.200-*"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("10.0.0.200-*")); + assertThat(bucket.getFromAsString(), equalTo("10.0.0.200")); + assertThat(bucket.getFrom(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.200"))); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getToAsString(), nullValue()); + assertThat(bucket.getDocCount(), equalTo(56l)); + max = bucket.getAggregations().get("max"); + assertThat(max, Matchers.notNullValue()); + assertThat((long) max.getValue(), equalTo(IpFieldMapper.ipToLong("10.0.0.255"))); + } + + @Test + public void script_SingleValue() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(ipRange("range") + .script("doc['ip'].value") + .addUnboundedTo("10.0.0.100") + .addRange("10.0.0.100", "10.0.0.200") + .addUnboundedFrom("10.0.0.200")) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + IPv4Range range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(3)); + + IPv4Range.Bucket bucket = range.getByKey("*-10.0.0.100"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("*-10.0.0.100")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getFromAsString(), nullValue()); + assertThat(bucket.getToAsString(), equalTo("10.0.0.100")); + assertThat(bucket.getTo(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.100"))); + assertThat(bucket.getDocCount(), equalTo(100l)); + + bucket = range.getByKey("10.0.0.100-10.0.0.200"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("10.0.0.100-10.0.0.200")); + assertThat(bucket.getFromAsString(), equalTo("10.0.0.100")); + assertThat(bucket.getFrom(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.100"))); + assertThat(bucket.getToAsString(), equalTo("10.0.0.200")); + assertThat(bucket.getTo(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.200"))); + assertThat(bucket.getDocCount(), equalTo(100l)); + + bucket = range.getByKey("10.0.0.200-*"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("10.0.0.200-*")); + assertThat(bucket.getFromAsString(), equalTo("10.0.0.200")); + assertThat(bucket.getFrom(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.200"))); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getToAsString(), nullValue()); + assertThat(bucket.getDocCount(), equalTo(55l)); + } + + @Test + public void script_SingleValue_WithSubAggregator_Inherited() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(ipRange("range") + .script("doc['ip'].value") + .addUnboundedTo("10.0.0.100") + .addRange("10.0.0.100", "10.0.0.200") + .addUnboundedFrom("10.0.0.200") + .subAggregation(max("max"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + IPv4Range range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(3)); + + IPv4Range.Bucket bucket = range.getByKey("*-10.0.0.100"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("*-10.0.0.100")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getFromAsString(), nullValue()); + assertThat(bucket.getToAsString(), equalTo("10.0.0.100")); + assertThat(bucket.getTo(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.100"))); + assertThat(bucket.getDocCount(), equalTo(100l)); + Max max = bucket.getAggregations().get("max"); + assertThat(max, notNullValue()); + assertThat(max.getValue(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.99"))); + + bucket = range.getByKey("10.0.0.100-10.0.0.200"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("10.0.0.100-10.0.0.200")); + assertThat(bucket.getFromAsString(), equalTo("10.0.0.100")); + assertThat(bucket.getFrom(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.100"))); + assertThat(bucket.getToAsString(), equalTo("10.0.0.200")); + assertThat(bucket.getTo(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.200"))); + assertThat(bucket.getDocCount(), equalTo(100l)); + max = bucket.getAggregations().get("max"); + assertThat(max, notNullValue()); + assertThat(max.getValue(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.199"))); + + bucket = range.getByKey("10.0.0.200-*"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("10.0.0.200-*")); + assertThat(bucket.getFromAsString(), equalTo("10.0.0.200")); + assertThat(bucket.getFrom(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.200"))); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getToAsString(), nullValue()); + assertThat(bucket.getDocCount(), equalTo(55l)); + max = bucket.getAggregations().get("max"); + assertThat(max, notNullValue()); + assertThat(max.getValue(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.254"))); + } + + @Test + public void script_MultiValued() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(ipRange("range") + .script("doc['ips'].values") + .addUnboundedTo("10.0.0.100") + .addRange("10.0.0.100", "10.0.0.200") + .addUnboundedFrom("10.0.0.200")) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + IPv4Range range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(3)); + + IPv4Range.Bucket bucket = range.getByKey("*-10.0.0.100"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("*-10.0.0.100")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getFromAsString(), nullValue()); + assertThat(bucket.getToAsString(), equalTo("10.0.0.100")); + assertThat(bucket.getTo(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.100"))); + assertThat(bucket.getDocCount(), equalTo(100l)); + + bucket = range.getByKey("10.0.0.100-10.0.0.200"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("10.0.0.100-10.0.0.200")); + assertThat(bucket.getFromAsString(), equalTo("10.0.0.100")); + assertThat(bucket.getFrom(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.100"))); + assertThat(bucket.getToAsString(), equalTo("10.0.0.200")); + assertThat(bucket.getTo(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.200"))); + assertThat(bucket.getDocCount(), equalTo(101l)); + + bucket = range.getByKey("10.0.0.200-*"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("10.0.0.200-*")); + assertThat(bucket.getFromAsString(), equalTo("10.0.0.200")); + assertThat(bucket.getFrom(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.200"))); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getToAsString(), nullValue()); + assertThat(bucket.getDocCount(), equalTo(56l)); + } + + @Test + public void script_MultiValued_WithAggregatorInherited() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(ipRange("range") + .script("doc['ips'].values") + .addUnboundedTo("10.0.0.100") + .addRange("10.0.0.100", "10.0.0.200") + .addUnboundedFrom("10.0.0.200") + .subAggregation(max("max"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + IPv4Range range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(3)); + + IPv4Range.Bucket bucket = range.getByKey("*-10.0.0.100"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("*-10.0.0.100")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getFromAsString(), nullValue()); + assertThat(bucket.getToAsString(), equalTo("10.0.0.100")); + assertThat(bucket.getTo(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.100"))); + assertThat(bucket.getDocCount(), equalTo(100l)); + Max max = bucket.getAggregations().get("max"); + assertThat(max, Matchers.notNullValue()); + assertThat((long) max.getValue(), equalTo(IpFieldMapper.ipToLong("10.0.0.100"))); + + bucket = range.getByKey("10.0.0.100-10.0.0.200"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("10.0.0.100-10.0.0.200")); + assertThat(bucket.getFromAsString(), equalTo("10.0.0.100")); + assertThat(bucket.getFrom(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.100"))); + assertThat(bucket.getToAsString(), equalTo("10.0.0.200")); + assertThat(bucket.getTo(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.200"))); + assertThat(bucket.getDocCount(), equalTo(101l)); + max = bucket.getAggregations().get("max"); + assertThat(max, Matchers.notNullValue()); + assertThat((long) max.getValue(), equalTo(IpFieldMapper.ipToLong("10.0.0.200"))); + + bucket = range.getByKey("10.0.0.200-*"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("10.0.0.200-*")); + assertThat(bucket.getFromAsString(), equalTo("10.0.0.200")); + assertThat(bucket.getFrom(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.200"))); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getToAsString(), nullValue()); + assertThat(bucket.getDocCount(), equalTo(56l)); + max = bucket.getAggregations().get("max"); + assertThat(max, Matchers.notNullValue()); + assertThat((long) max.getValue(), equalTo(IpFieldMapper.ipToLong("10.0.0.255"))); + } + + @Test + public void unmapped() throws Exception { + client().admin().cluster().prepareHealth("idx_unmapped").setWaitForYellowStatus().execute().actionGet(); + + SearchResponse response = client().prepareSearch("idx_unmapped") + .addAggregation(ipRange("range") + .field("ip") + .addUnboundedTo("10.0.0.100") + .addRange("10.0.0.100", "10.0.0.200") + .addUnboundedFrom("10.0.0.200")) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + IPv4Range range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(3)); + + IPv4Range.Bucket bucket = range.getByKey("*-10.0.0.100"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("*-10.0.0.100")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getFromAsString(), nullValue()); + assertThat(bucket.getToAsString(), equalTo("10.0.0.100")); + assertThat(bucket.getTo(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.100"))); + assertThat(bucket.getDocCount(), equalTo(0l)); + + bucket = range.getByKey("10.0.0.100-10.0.0.200"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("10.0.0.100-10.0.0.200")); + assertThat(bucket.getFromAsString(), equalTo("10.0.0.100")); + assertThat(bucket.getFrom(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.100"))); + assertThat(bucket.getToAsString(), equalTo("10.0.0.200")); + assertThat(bucket.getTo(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.200"))); + assertThat(bucket.getDocCount(), equalTo(0l)); + + bucket = range.getByKey("10.0.0.200-*"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("10.0.0.200-*")); + assertThat(bucket.getFromAsString(), equalTo("10.0.0.200")); + assertThat(bucket.getFrom(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.200"))); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getToAsString(), nullValue()); + assertThat(bucket.getDocCount(), equalTo(0l)); + } + + @Test + public void partiallyUnmapped() throws Exception { + client().admin().cluster().prepareHealth("idx_unmapped").setWaitForYellowStatus().execute().actionGet(); + + SearchResponse response = client().prepareSearch("idx", "idx_unmapped") + .addAggregation(ipRange("range") + .field("ip") + .addUnboundedTo("10.0.0.100") + .addRange("10.0.0.100", "10.0.0.200") + .addUnboundedFrom("10.0.0.200")) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + IPv4Range range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(3)); + + IPv4Range.Bucket bucket = range.getByKey("*-10.0.0.100"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("*-10.0.0.100")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getFromAsString(), nullValue()); + assertThat(bucket.getToAsString(), equalTo("10.0.0.100")); + assertThat(bucket.getTo(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.100"))); + assertThat(bucket.getDocCount(), equalTo(100l)); + + bucket = range.getByKey("10.0.0.100-10.0.0.200"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("10.0.0.100-10.0.0.200")); + assertThat(bucket.getFromAsString(), equalTo("10.0.0.100")); + assertThat(bucket.getFrom(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.100"))); + assertThat(bucket.getToAsString(), equalTo("10.0.0.200")); + assertThat(bucket.getTo(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.200"))); + assertThat(bucket.getDocCount(), equalTo(100l)); + + bucket = range.getByKey("10.0.0.200-*"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("10.0.0.200-*")); + assertThat(bucket.getFromAsString(), equalTo("10.0.0.200")); + assertThat(bucket.getFrom(), equalTo((double) IpFieldMapper.ipToLong("10.0.0.200"))); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getToAsString(), nullValue()); + assertThat(bucket.getDocCount(), equalTo(55l)); + } + + @Test + public void emptyAggregation() throws Exception { + prepareCreate("empty_bucket_idx").addMapping("type", "value", "type=integer", "ip", "type=ip").execute().actionGet(); + List builders = new ArrayList(); + for (int i = 0; i < 2; i++) { + builders.add(client().prepareIndex("empty_bucket_idx", "type", "" + i).setSource(jsonBuilder() + .startObject() + .field("value", i * 2) + .field("ip", "10.0.0.5") + .endObject())); + } + indexRandom(true, builders.toArray(new IndexRequestBuilder[builders.size()])); + + SearchResponse searchResponse = client().prepareSearch("empty_bucket_idx") + .setQuery(matchAllQuery()) + .addAggregation(histogram("histo").field("value").interval(1l).emptyBuckets(true) + .subAggregation(ipRange("ip_range").field("ip").addRange("r1", "10.0.0.1", "10.0.0.10"))) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(2l)); + Histogram histo = searchResponse.getAggregations().get("histo"); + assertThat(histo, Matchers.notNullValue()); + Histogram.Bucket bucket = histo.getByKey(1l); + assertThat(bucket, Matchers.notNullValue()); + + IPv4Range range = bucket.getAggregations().get("ip_range"); + assertThat(range, Matchers.notNullValue()); + assertThat(range.getName(), equalTo("ip_range")); + assertThat(range.buckets().size(), is(1)); + assertThat(range.buckets().get(0).getKey(), equalTo("r1")); + assertThat(range.buckets().get(0).getFromAsString(), equalTo("10.0.0.1")); + assertThat(range.buckets().get(0).getToAsString(), equalTo("10.0.0.10")); + assertThat(range.buckets().get(0).getDocCount(), equalTo(0l)); + } +} diff --git a/src/test/java/org/elasticsearch/search/aggregations/bucket/LongHashTests.java b/src/test/java/org/elasticsearch/search/aggregations/bucket/LongHashTests.java new file mode 100644 index 00000000000..7562f66f2ad --- /dev/null +++ b/src/test/java/org/elasticsearch/search/aggregations/bucket/LongHashTests.java @@ -0,0 +1,68 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket; + +import com.carrotsearch.hppc.LongLongMap; +import com.carrotsearch.hppc.LongLongOpenHashMap; +import com.carrotsearch.hppc.cursors.LongLongCursor; +import org.elasticsearch.search.aggregations.bucket.LongHash; +import org.elasticsearch.test.ElasticsearchTestCase; + +import java.util.Iterator; + +public class LongHashTests extends ElasticsearchTestCase { + + public void testDuell() { + final Long[] values = new Long[randomIntBetween(1, 100000)]; + for (int i = 0; i < values.length; ++i) { + values[i] = randomLong(); + } + final LongLongMap valueToId = new LongLongOpenHashMap(); + final long[] idToValue = new long[values.length]; + // Test high load factors to make sure that collision resolution works fine + final float maxLoadFactor = 0.6f + randomFloat() * 0.39f; + final LongHash longHash = new LongHash(randomIntBetween(0, 100), maxLoadFactor); + final int iters = randomInt(1000000); + for (int i = 0; i < iters; ++i) { + final Long value = randomFrom(values); + if (valueToId.containsKey(value)) { + assertEquals(- 1 - valueToId.get(value), longHash.add(value)); + } else { + assertEquals(valueToId.size(), longHash.add(value)); + idToValue[valueToId.size()] = value; + valueToId.put(value, valueToId.size()); + } + } + + assertEquals(valueToId.size(), longHash.size()); + for (Iterator iterator = valueToId.iterator(); iterator.hasNext(); ) { + final LongLongCursor next = iterator.next(); + assertEquals(next.value, longHash.get(next.key)); + } + + for (long i = 0; i < longHash.capacity(); ++i) { + final long id = longHash.id(i); + if (id >= 0) { + assertEquals(idToValue[(int) id], longHash.key(i)); + } + } + } + +} diff --git a/src/test/java/org/elasticsearch/search/aggregations/bucket/LongTermsTests.java b/src/test/java/org/elasticsearch/search/aggregations/bucket/LongTermsTests.java new file mode 100644 index 00000000000..58d399eb1a9 --- /dev/null +++ b/src/test/java/org/elasticsearch/search/aggregations/bucket/LongTermsTests.java @@ -0,0 +1,597 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket; + +import org.elasticsearch.action.index.IndexRequestBuilder; +import org.elasticsearch.action.search.SearchResponse; +import org.elasticsearch.common.settings.ImmutableSettings; +import org.elasticsearch.common.settings.Settings; +import org.elasticsearch.search.aggregations.bucket.histogram.Histogram; +import org.elasticsearch.search.aggregations.bucket.terms.Terms; +import org.elasticsearch.search.aggregations.metrics.sum.Sum; +import org.elasticsearch.test.ElasticsearchIntegrationTest; +import org.hamcrest.Matchers; +import org.junit.Before; +import org.junit.Test; + +import java.util.ArrayList; +import java.util.List; + +import static org.elasticsearch.common.xcontent.XContentFactory.jsonBuilder; +import static org.elasticsearch.index.query.QueryBuilders.matchAllQuery; +import static org.elasticsearch.search.aggregations.AggregationBuilders.*; +import static org.hamcrest.Matchers.equalTo; +import static org.hamcrest.Matchers.is; +import static org.hamcrest.core.IsNull.notNullValue; + +/** + * + */ +public class LongTermsTests extends ElasticsearchIntegrationTest { + + private static final int NUM_DOCS = 5; // TODO randomize the size? + + @Override + public Settings indexSettings() { + return ImmutableSettings.builder() + .put("index.number_of_shards", between(1, 5)) + .put("index.number_of_replicas", between(0, 1)) + .build(); + } + + @Before + public void init() throws Exception { + createIndex("idx"); + IndexRequestBuilder[] lowCardBuilders = new IndexRequestBuilder[NUM_DOCS]; + for (int i = 0; i < lowCardBuilders.length; i++) { + lowCardBuilders[i] = client().prepareIndex("idx", "type").setSource(jsonBuilder() + .startObject() + .field("value", i) + .startArray("values").value(i).value(i + 1).endArray() + .endObject()); + } + indexRandom(randomBoolean(), lowCardBuilders); + IndexRequestBuilder[] highCardBuilders = new IndexRequestBuilder[100]; // TODO randomize the size? + for (int i = 0; i < highCardBuilders.length; i++) { + highCardBuilders[i] = client().prepareIndex("idx", "high_card_type").setSource(jsonBuilder() + .startObject() + .field("value", i) + .startArray("values").value(i).value(i + 1).endArray() + .endObject()); + + } + indexRandom(true, highCardBuilders); + createIndex("idx_unmapped"); + } + + @Test + public void singleValueField() throws Exception { + SearchResponse response = client().prepareSearch("idx").setTypes("type") + .addAggregation(terms("terms") + .field("value")) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(5)); + + for (int i = 0; i < 5; i++) { + Terms.Bucket bucket = terms.getByTerm("" + i); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("" + i)); + assertThat(bucket.getKeyAsNumber().intValue(), equalTo(i)); + assertThat(bucket.getDocCount(), equalTo(1l)); + } + } + + @Test + public void singleValueField_WithMaxSize() throws Exception { + SearchResponse response = client().prepareSearch("idx").setTypes("high_card_type") + .addAggregation(terms("terms") + .field("value") + .size(20) + .order(Terms.Order.TERM_ASC)) // we need to sort by terms cause we're checking the first 20 values + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(20)); + + for (int i = 0; i < 20; i++) { + Terms.Bucket bucket = terms.getByTerm("" + i); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("" + i)); + assertThat(bucket.getKeyAsNumber().intValue(), equalTo(i)); + assertThat(bucket.getDocCount(), equalTo(1l)); + } + } + + @Test + public void singleValueField_OrderedByTermAsc() throws Exception { + SearchResponse response = client().prepareSearch("idx").setTypes("type") + .addAggregation(terms("terms") + .field("value") + .order(Terms.Order.TERM_ASC)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(5)); + + int i = 0; + for (Terms.Bucket bucket : terms.buckets()) { + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("" + i)); + assertThat(bucket.getKeyAsNumber().intValue(), equalTo(i)); + assertThat(bucket.getDocCount(), equalTo(1l)); + i++; + } + } + + @Test + public void singleValueField_OrderedByTermDesc() throws Exception { + SearchResponse response = client().prepareSearch("idx").setTypes("type") + .addAggregation(terms("terms") + .field("value") + .order(Terms.Order.TERM_DESC)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(5)); + + int i = 4; + for (Terms.Bucket bucket : terms.buckets()) { + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("" + i)); + assertThat(bucket.getKeyAsNumber().intValue(), equalTo(i)); + assertThat(bucket.getDocCount(), equalTo(1l)); + i--; + } + } + + @Test + public void singleValuedField_WithSubAggregation() throws Exception { + SearchResponse response = client().prepareSearch("idx").setTypes("type") + .addAggregation(terms("terms") + .field("value") + .subAggregation(sum("sum").field("values"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(5)); + + for (int i = 0; i < 5; i++) { + Terms.Bucket bucket = terms.getByTerm("" + i); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("" + i)); + assertThat(bucket.getKeyAsNumber().intValue(), equalTo(i)); + assertThat(bucket.getDocCount(), equalTo(1l)); + Sum sum = bucket.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + assertThat((long) sum.getValue(), equalTo(i+i+1l)); + } + } + + @Test + public void singleValuedField_WithSubAggregation_Inherited() throws Exception { + SearchResponse response = client().prepareSearch("idx").setTypes("type") + .addAggregation(terms("terms") + .field("value") + .subAggregation(sum("sum"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(5)); + + for (int i = 0; i < 5; i++) { + Terms.Bucket bucket = terms.getByTerm("" + i); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("" + i)); + assertThat(bucket.getKeyAsNumber().intValue(), equalTo(i)); + assertThat(bucket.getDocCount(), equalTo(1l)); + Sum sum = bucket.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + assertThat(sum.getValue(), equalTo((double) i)); + } + } + + @Test + public void singleValuedField_WithValueScript() throws Exception { + SearchResponse response = client().prepareSearch("idx").setTypes("type") + .addAggregation(terms("terms") + .field("value") + .script("_value + 1")) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(5)); + + for (int i = 0; i < 5; i++) { + Terms.Bucket bucket = terms.getByTerm("" + (i+1d)); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("" + (i+1d))); + assertThat(bucket.getKeyAsNumber().intValue(), equalTo(i+1)); + assertThat(bucket.getDocCount(), equalTo(1l)); + } + } + + @Test + public void multiValuedField() throws Exception { + SearchResponse response = client().prepareSearch("idx").setTypes("type") + .addAggregation(terms("terms") + .field("values")) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(6)); + + for (int i = 0; i < 6; i++) { + Terms.Bucket bucket = terms.getByTerm("" + i); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("" + i)); + assertThat(bucket.getKeyAsNumber().intValue(), equalTo(i)); + if (i == 0 || i == 5) { + assertThat(bucket.getDocCount(), equalTo(1l)); + } else { + assertThat(bucket.getDocCount(), equalTo(2l)); + } + } + } + + @Test + public void multiValuedField_WithValueScript() throws Exception { + SearchResponse response = client().prepareSearch("idx").setTypes("type") + .addAggregation(terms("terms") + .field("values") + .script("_value - 1")) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(6)); + + for (int i = 0; i < 6; i++) { + Terms.Bucket bucket = terms.getByTerm("" + (i-1d)); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("" + (i-1d))); + assertThat(bucket.getKeyAsNumber().intValue(), equalTo(i-1)); + if (i == 0 || i == 5) { + assertThat(bucket.getDocCount(), equalTo(1l)); + } else { + assertThat(bucket.getDocCount(), equalTo(2l)); + } + } + } + + @Test + public void multiValuedField_WithValueScript_NotUnique() throws Exception { + SearchResponse response = client().prepareSearch("idx").setTypes("type") + .addAggregation(terms("terms") + .field("values") + .script("floor(_value / 1000 + 1)")) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(1)); + + Terms.Bucket bucket = terms.getByTerm("1.0"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("1.0")); + assertThat(bucket.getKeyAsNumber().intValue(), equalTo(1)); + assertThat(bucket.getDocCount(), equalTo(5l)); + } + + /* + + [1, 2] + [2, 3] + [3, 4] + [4, 5] + [5, 6] + + 1 - count: 1 - sum: 1 + 2 - count: 2 - sum: 4 + 3 - count: 2 - sum: 6 + 4 - count: 2 - sum: 8 + 5 - count: 2 - sum: 10 + 6 - count: 1 - sum: 6 + + */ + + @Test + public void multiValuedField_WithValueScript_WithInheritedSubAggregator() throws Exception { + SearchResponse response = client().prepareSearch("idx").setTypes("type") + .addAggregation(terms("terms") + .field("values") + .script("_value + 1") + .subAggregation(sum("sum"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(6)); + + for (int i = 0; i < 6; i++) { + Terms.Bucket bucket = terms.getByTerm("" + (i+1d)); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("" + (i+1d))); + assertThat(bucket.getKeyAsNumber().intValue(), equalTo(i+1)); + final long count = i == 0 || i == 5 ? 1 : 2; + double s = 0; + for (int j = 0; j < NUM_DOCS; ++j) { + if (i == j || i == j+1) { + s += j + 1; + s += j+1 + 1; + } + } + assertThat(bucket.getDocCount(), equalTo(count)); + Sum sum = bucket.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + assertThat(sum.getValue(), equalTo(s)); + } + } + + @Test + public void script_SingleValue() throws Exception { + SearchResponse response = client().prepareSearch("idx").setTypes("type") + .addAggregation(terms("terms") + .script("doc['value'].value")) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(5)); + + for (int i = 0; i < 5; i++) { + Terms.Bucket bucket = terms.getByTerm("" + i); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("" + i)); + assertThat(bucket.getKeyAsNumber().intValue(), equalTo(i)); + assertThat(bucket.getDocCount(), equalTo(1l)); + } + } + + @Test + public void script_SingleValue_WithSubAggregator_Inherited() throws Exception { + SearchResponse response = client().prepareSearch("idx").setTypes("type") + .addAggregation(terms("terms") + .field("value") + .subAggregation(sum("sum"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(5)); + + for (int i = 0; i < 5; i++) { + Terms.Bucket bucket = terms.getByTerm("" + i); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("" + i)); + assertThat(bucket.getKeyAsNumber().intValue(), equalTo(i)); + assertThat(bucket.getDocCount(), equalTo(1l)); + Sum sum = bucket.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + assertThat(sum.getValue(), equalTo((double) i)); + } + } + + @Test + public void script_MultiValued() throws Exception { + SearchResponse response = client().prepareSearch("idx").setTypes("type") + .addAggregation(terms("terms") + .script("doc['values'].values")) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(6)); + + for (int i = 0; i < 6; i++) { + Terms.Bucket bucket = terms.getByTerm("" + i); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("" + i)); + assertThat(bucket.getKeyAsNumber().intValue(), equalTo(i)); + if (i == 0 || i == 5) { + assertThat(bucket.getDocCount(), equalTo(1l)); + } else { + assertThat(bucket.getDocCount(), equalTo(2l)); + } + } + } + + @Test + public void script_MultiValued_WithAggregatorInherited_NoExplicitType() throws Exception { + + // since no type ie explicitly defined, es will assume all values returned by the script to be strings (bytes), + // so the aggregation should fail, since the "sum" aggregation can only operation on numeric values. + + try { + + SearchResponse response = client().prepareSearch("idx").setTypes("type") + .addAggregation(terms("terms") + .script("doc['values'].values") + .subAggregation(sum("sum"))) + .execute().actionGet(); + + fail("expected to fail as sub-aggregation sum requires a numeric value source context, but there is none"); + + } catch (Exception e) { + // expected + } + } + + @Test + public void script_MultiValued_WithAggregatorInherited_WithExplicitType() throws Exception { + SearchResponse response = client().prepareSearch("idx").setTypes("type") + .addAggregation(terms("terms") + .script("doc['values'].values") + .valueType(Terms.ValueType.LONG) + .subAggregation(sum("sum"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(6)); + + for (int i = 0; i < 6; i++) { + Terms.Bucket bucket = terms.getByTerm("" + i); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("" + i)); + assertThat(bucket.getKeyAsNumber().intValue(), equalTo(i)); + final long count = i == 0 || i == 5 ? 1 : 2; + double s = 0; + for (int j = 0; j < NUM_DOCS; ++j) { + if (i == j || i == j+1) { + s += j; + s += j+1; + } + } + assertThat(bucket.getDocCount(), equalTo(count)); + Sum sum = bucket.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + assertThat(sum.getValue(), equalTo(s)); + } + } + + @Test + public void unmapped() throws Exception { + client().admin().cluster().prepareHealth("idx_unmapped").setWaitForYellowStatus().execute().actionGet(); + + SearchResponse response = client().prepareSearch("idx_unmapped").setTypes("type") + .addAggregation(terms("terms") + .field("value")) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(0)); + } + + @Test + public void partiallyUnmapped() throws Exception { + client().admin().cluster().prepareHealth("idx_unmapped").setWaitForYellowStatus().execute().actionGet(); + + SearchResponse response = client().prepareSearch("idx_unmapped", "idx").setTypes("type") + .addAggregation(terms("terms") + .field("value")) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(5)); + + for (int i = 0; i < 5; i++) { + Terms.Bucket bucket = terms.getByTerm("" + i); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("" + i)); + assertThat(bucket.getKeyAsNumber().intValue(), equalTo(i)); + assertThat(bucket.getDocCount(), equalTo(1l)); + } + } + + @Test + public void emptyAggregation() throws Exception { + prepareCreate("empty_bucket_idx").addMapping("type", "value", "type=integer").execute().actionGet(); + List builders = new ArrayList(); + for (int i = 0; i < 2; i++) { + builders.add(client().prepareIndex("empty_bucket_idx", "type", ""+i).setSource(jsonBuilder() + .startObject() + .field("value", i*2) + .endObject())); + } + indexRandom(true, builders.toArray(new IndexRequestBuilder[builders.size()])); + + SearchResponse searchResponse = client().prepareSearch("empty_bucket_idx") + .setQuery(matchAllQuery()) + .addAggregation(histogram("histo").field("value").interval(1l).emptyBuckets(true) + .subAggregation(terms("terms"))) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(2l)); + Histogram histo = searchResponse.getAggregations().get("histo"); + assertThat(histo, Matchers.notNullValue()); + Histogram.Bucket bucket = histo.getByKey(1l); + assertThat(bucket, Matchers.notNullValue()); + + Terms terms = bucket.getAggregations().get("terms"); + assertThat(terms, Matchers.notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().isEmpty(), is(true)); + + } + +} diff --git a/src/test/java/org/elasticsearch/search/aggregations/bucket/MissingTests.java b/src/test/java/org/elasticsearch/search/aggregations/bucket/MissingTests.java new file mode 100644 index 00000000000..c7090a1e9f7 --- /dev/null +++ b/src/test/java/org/elasticsearch/search/aggregations/bucket/MissingTests.java @@ -0,0 +1,215 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket; + +import org.elasticsearch.action.index.IndexRequestBuilder; +import org.elasticsearch.action.search.SearchResponse; +import org.elasticsearch.common.settings.ImmutableSettings; +import org.elasticsearch.common.settings.Settings; +import org.elasticsearch.search.aggregations.bucket.histogram.Histogram; +import org.elasticsearch.search.aggregations.bucket.missing.Missing; +import org.elasticsearch.search.aggregations.metrics.avg.Avg; +import org.elasticsearch.test.ElasticsearchIntegrationTest; +import org.hamcrest.Matchers; +import org.junit.Before; +import org.junit.Test; + +import java.util.ArrayList; +import java.util.List; + +import static org.elasticsearch.common.xcontent.XContentFactory.jsonBuilder; +import static org.elasticsearch.index.query.QueryBuilders.matchAllQuery; +import static org.elasticsearch.search.aggregations.AggregationBuilders.*; +import static org.hamcrest.Matchers.equalTo; +import static org.hamcrest.Matchers.is; +import static org.hamcrest.core.IsNull.notNullValue; + +/** + * + */ +public class MissingTests extends ElasticsearchIntegrationTest { + + + @Override + public Settings indexSettings() { + return ImmutableSettings.builder() + .put("index.number_of_shards", between(1, 5)) + .put("index.number_of_replicas", between(0, 1)) + .build(); + } + + int numDocs, numDocsMissing, numDocsUnmapped; + + @Before + public void init() throws Exception { + createIndex("idx"); + List builders = new ArrayList(); + numDocs = randomIntBetween(5, 20); + numDocsMissing = randomIntBetween(1, numDocs - 1); + for (int i = 0; i < numDocsMissing; i++) { + builders.add(client().prepareIndex("idx", "type", ""+i).setSource(jsonBuilder() + .startObject() + .field("value", i) + .endObject())); + } + for (int i = numDocsMissing; i < numDocs; i++) { + builders.add(client().prepareIndex("idx", "type", ""+i).setSource(jsonBuilder() + .startObject() + .field("tag", "tag1") + .endObject())); + } + + createIndex("unmapped_idx"); + numDocsUnmapped = randomIntBetween(2, 5); + for (int i = 0; i < numDocsUnmapped; i++) { + builders.add(client().prepareIndex("unmapped_idx", "type", ""+i).setSource(jsonBuilder() + .startObject() + .field("value", i) + .endObject())); + } + + indexRandom(true, builders.toArray(new IndexRequestBuilder[builders.size()])); + + } + + @Test + public void unmapped() throws Exception { + SearchResponse response = client().prepareSearch("unmapped_idx") + .addAggregation(missing("missing_tag").field("tag")) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Missing missing = response.getAggregations().get("missing_tag"); + assertThat(missing, notNullValue()); + assertThat(missing.getName(), equalTo("missing_tag")); + assertThat(missing.getDocCount(), equalTo((long) numDocsUnmapped)); + } + + @Test + public void partiallyUnmapped() throws Exception { + SearchResponse response = client().prepareSearch("idx", "unmapped_idx") + .addAggregation(missing("missing_tag").field("tag")) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Missing missing = response.getAggregations().get("missing_tag"); + assertThat(missing, notNullValue()); + assertThat(missing.getName(), equalTo("missing_tag")); + assertThat(missing.getDocCount(), equalTo((long) numDocsMissing + numDocsUnmapped)); + } + + @Test + public void simple() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(missing("missing_tag").field("tag")) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Missing missing = response.getAggregations().get("missing_tag"); + assertThat(missing, notNullValue()); + assertThat(missing.getName(), equalTo("missing_tag")); + assertThat(missing.getDocCount(), equalTo((long) numDocsMissing)); + } + + @Test + public void withSubAggregation() throws Exception { + SearchResponse response = client().prepareSearch("idx", "unmapped_idx") + .addAggregation(missing("missing_tag").field("tag") + .subAggregation(avg("avg_value").field("value"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Missing missing = response.getAggregations().get("missing_tag"); + assertThat(missing, notNullValue()); + assertThat(missing.getName(), equalTo("missing_tag")); + assertThat(missing.getDocCount(), equalTo((long) numDocsMissing + numDocsUnmapped)); + assertThat(missing.getAggregations().asList().isEmpty(), is(false)); + + long sum = 0; + for (int i = 0; i < numDocsMissing; ++i) { + sum += i; + } + for (int i = 0; i < numDocsUnmapped; ++i) { + sum += i; + } + Avg avgValue = missing.getAggregations().get("avg_value"); + assertThat(avgValue, notNullValue()); + assertThat(avgValue.getName(), equalTo("avg_value")); + assertThat(avgValue.getValue(), equalTo((double) sum / (numDocsMissing + numDocsUnmapped))); + } + + @Test + public void withInheritedSubMissing() throws Exception { + + SearchResponse response = client().prepareSearch() + .addAggregation(missing("top_missing").field("tag") + .subAggregation(missing("sub_missing"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Missing topMissing = response.getAggregations().get("top_missing"); + assertThat(topMissing, notNullValue()); + assertThat(topMissing.getName(), equalTo("top_missing")); + assertThat(topMissing.getDocCount(), equalTo((long) numDocsMissing + numDocsUnmapped)); + assertThat(topMissing.getAggregations().asList().isEmpty(), is(false)); + + Missing subMissing = topMissing.getAggregations().get("sub_missing"); + assertThat(subMissing, notNullValue()); + assertThat(subMissing.getName(), equalTo("sub_missing")); + assertThat(subMissing.getDocCount(), equalTo((long) numDocsMissing + numDocsUnmapped)); + } + + @Test + public void emptyAggregation() throws Exception { + prepareCreate("empty_bucket_idx").addMapping("type", "value", "type=integer").execute().actionGet(); + List builders = new ArrayList(); + for (int i = 0; i < 2; i++) { + builders.add(client().prepareIndex("empty_bucket_idx", "type", ""+i).setSource(jsonBuilder() + .startObject() + .field("value", i*2) + .endObject())); + } + indexRandom(true, builders.toArray(new IndexRequestBuilder[builders.size()])); + + SearchResponse searchResponse = client().prepareSearch("empty_bucket_idx") + .setQuery(matchAllQuery()) + .addAggregation(histogram("histo").field("value").interval(1l).emptyBuckets(true) + .subAggregation(missing("missing"))) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(2l)); + Histogram histo = searchResponse.getAggregations().get("histo"); + assertThat(histo, Matchers.notNullValue()); + Histogram.Bucket bucket = histo.getByKey(1l); + assertThat(bucket, Matchers.notNullValue()); + + Missing missing = bucket.getAggregations().get("missing"); + assertThat(missing, Matchers.notNullValue()); + assertThat(missing.getName(), equalTo("missing")); + assertThat(missing.getDocCount(), is(0l)); + } + + +} diff --git a/src/test/java/org/elasticsearch/search/aggregations/bucket/NestedTests.java b/src/test/java/org/elasticsearch/search/aggregations/bucket/NestedTests.java new file mode 100644 index 00000000000..f5171ddc7dd --- /dev/null +++ b/src/test/java/org/elasticsearch/search/aggregations/bucket/NestedTests.java @@ -0,0 +1,259 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket; + +import org.elasticsearch.ElasticSearchException; +import org.elasticsearch.action.index.IndexRequestBuilder; +import org.elasticsearch.action.search.SearchResponse; +import org.elasticsearch.common.settings.ImmutableSettings; +import org.elasticsearch.common.settings.Settings; +import org.elasticsearch.common.xcontent.XContentBuilder; +import org.elasticsearch.search.aggregations.bucket.histogram.Histogram; +import org.elasticsearch.search.aggregations.bucket.terms.LongTerms; +import org.elasticsearch.search.aggregations.bucket.terms.Terms.Bucket; +import org.elasticsearch.search.aggregations.bucket.nested.Nested; +import org.elasticsearch.search.aggregations.metrics.max.Max; +import org.elasticsearch.search.aggregations.metrics.stats.Stats; +import org.elasticsearch.test.ElasticsearchIntegrationTest; +import org.hamcrest.Matchers; +import org.junit.Before; +import org.junit.Test; + +import java.util.ArrayList; +import java.util.List; + +import static org.elasticsearch.common.xcontent.XContentFactory.jsonBuilder; +import static org.elasticsearch.index.query.QueryBuilders.matchAllQuery; +import static org.elasticsearch.search.aggregations.AggregationBuilders.*; +import static org.hamcrest.Matchers.equalTo; +import static org.hamcrest.Matchers.is; +import static org.hamcrest.core.IsNull.notNullValue; + +/** + * + */ +public class NestedTests extends ElasticsearchIntegrationTest { + + @Override + public Settings indexSettings() { + return ImmutableSettings.builder() + .put("index.number_of_shards", between(1, 5)) + .put("index.number_of_replicas", between(0, 1)) + .build(); + } + + int numParents; + int[] numChildren; + + @Before + public void init() throws Exception { + + prepareCreate("idx") + .addMapping("type", "nested", "type=nested") + .setSettings(indexSettings()) + .execute().actionGet(); + List builders = new ArrayList(); + + numParents = randomIntBetween(3, 10); + numChildren = new int[numParents]; + for (int i = 0; i < numParents; ++i) { + numChildren[i] = randomInt(5); + } + + for (int i = 0; i < numParents; i++) { + XContentBuilder source = jsonBuilder() + .startObject() + .field("value", i + 1) + .startArray("nested"); + for (int j = 0; j < numChildren[i]; ++j) { + source = source.startObject().field("value", i + 1 + j).endObject(); + } + source = source.endArray().endObject(); + builders.add(client().prepareIndex("idx", "type", ""+i+1).setSource(source)); + } + indexRandom(true, builders); + } + + @Test + public void simple() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(nested("nested").path("nested") + .subAggregation(stats("nested_value_stats").field("nested.value"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + double min = Double.POSITIVE_INFINITY; + double max = Double.NEGATIVE_INFINITY; + long sum = 0; + long count = 0; + for (int i = 0; i < numParents; ++i) { + for (int j = 0; j < numChildren[i]; ++j) { + final long value = i + 1 + j; + min = Math.min(min, value); + max = Math.max(max, value); + sum += value; + ++count; + } + } + + Nested nested = response.getAggregations().get("nested"); + assertThat(nested, notNullValue()); + assertThat(nested.getName(), equalTo("nested")); + assertThat(nested.getDocCount(), equalTo(count)); + assertThat(nested.getAggregations().asList().isEmpty(), is(false)); + + Stats stats = nested.getAggregations().get("nested_value_stats"); + assertThat(stats, notNullValue()); + assertThat(stats.getMin(), equalTo(min)); + assertThat(stats.getMax(), equalTo(max)); + assertThat(stats.getCount(), equalTo(count)); + assertThat(stats.getSum(), equalTo((double) sum)); + assertThat(stats.getAvg(), equalTo((double) sum / count)); + } + + @Test + public void onNonNestedField() throws Exception { + + try { + client().prepareSearch("idx") + .addAggregation(nested("nested").path("value") + .subAggregation(stats("nested_value_stats").field("nested.value"))) + .execute().actionGet(); + + fail("expected execution to fail - an attempt to nested facet on non-nested field/path"); + + } catch (ElasticSearchException ese) { + } + } + + @Test + public void nestedWithSubTermsAgg() throws Exception { + + SearchResponse response = client().prepareSearch("idx") + .addAggregation(nested("nested").path("nested") + .subAggregation(terms("values").field("nested.value").size(100))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + long docCount = 0; + long[] counts = new long[numParents + 6]; + for (int i = 0; i < numParents; ++i) { + for (int j = 0; j < numChildren[i]; ++j) { + final int value = i + 1 + j; + ++counts[value]; + ++docCount; + } + } + int uniqueValues = 0; + for (long count : counts) { + if (count > 0) { + ++uniqueValues; + } + } + + Nested nested = response.getAggregations().get("nested"); + assertThat(nested, notNullValue()); + assertThat(nested.getName(), equalTo("nested")); + assertThat(nested.getDocCount(), equalTo(docCount)); + assertThat(nested.getAggregations().asList().isEmpty(), is(false)); + + LongTerms values = nested.getAggregations().get("values"); + assertThat(values, notNullValue()); + assertThat(values.getName(), equalTo("values")); + assertThat(values.buckets(), notNullValue()); + assertThat(values.buckets().size(), equalTo(uniqueValues)); + for (int i = 0; i < counts.length; ++i) { + final String key = Long.toString(i); + if (counts[i] == 0) { + assertNull(values.getByTerm(key)); + } else { + Bucket bucket = values.getByTerm(key); + assertNotNull(bucket); + assertEquals(counts[i], bucket.getDocCount()); + } + } + } + + @Test + public void nestedAsSubAggregation() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(terms("top_values").field("value").size(100) + .subAggregation(nested("nested").path("nested") + .subAggregation(max("max_value").field("nested.value")))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + LongTerms values = response.getAggregations().get("top_values"); + assertThat(values, notNullValue()); + assertThat(values.getName(), equalTo("top_values")); + assertThat(values.buckets(), notNullValue()); + assertThat(values.buckets().size(), equalTo(numParents)); + + for (int i = 0; i < numParents; i++) { + String topValue = "" + (i + 1); + assertThat(values.getByTerm(topValue), notNullValue()); + Nested nested = values.getByTerm(topValue).getAggregations().get("nested"); + assertThat(nested, notNullValue()); + Max max = nested.getAggregations().get("max_value"); + assertThat(max, notNullValue()); + assertThat(max.getValue(), equalTo(numChildren[i] == 0 ? Double.NEGATIVE_INFINITY : (double) i + numChildren[i])); + } + } + + @Test + public void emptyAggregation() throws Exception { + prepareCreate("empty_bucket_idx").addMapping("type", "value", "type=integer", "nested", "type=nested").execute().actionGet(); + List builders = new ArrayList(); + for (int i = 0; i < 2; i++) { + builders.add(client().prepareIndex("empty_bucket_idx", "type", ""+i).setSource(jsonBuilder() + .startObject() + .field("value", i*2) + .startArray("nested") + .startObject().field("value", i + 1).endObject() + .startObject().field("value", i + 2).endObject() + .startObject().field("value", i + 3).endObject() + .startObject().field("value", i + 4).endObject() + .startObject().field("value", i + 5).endObject() + .endArray() + .endObject())); + } + indexRandom(true, builders.toArray(new IndexRequestBuilder[builders.size()])); + + SearchResponse searchResponse = client().prepareSearch("empty_bucket_idx") + .setQuery(matchAllQuery()) + .addAggregation(histogram("histo").field("value").interval(1l).emptyBuckets(true) + .subAggregation(nested("nested").path("nested"))) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(2l)); + Histogram histo = searchResponse.getAggregations().get("histo"); + assertThat(histo, Matchers.notNullValue()); + Histogram.Bucket bucket = histo.getByKey(1l); + assertThat(bucket, Matchers.notNullValue()); + + Nested nested = bucket.getAggregations().get("nested"); + assertThat(nested, Matchers.notNullValue()); + assertThat(nested.getName(), equalTo("nested")); + assertThat(nested.getDocCount(), is(0l)); + } +} diff --git a/src/test/java/org/elasticsearch/search/aggregations/bucket/RangeTests.java b/src/test/java/org/elasticsearch/search/aggregations/bucket/RangeTests.java new file mode 100644 index 00000000000..0e16c0d4825 --- /dev/null +++ b/src/test/java/org/elasticsearch/search/aggregations/bucket/RangeTests.java @@ -0,0 +1,879 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket; + +import org.elasticsearch.action.index.IndexRequestBuilder; +import org.elasticsearch.action.search.SearchResponse; +import org.elasticsearch.common.settings.ImmutableSettings; +import org.elasticsearch.common.settings.Settings; +import org.elasticsearch.search.aggregations.bucket.histogram.Histogram; +import org.elasticsearch.search.aggregations.bucket.range.Range; +import org.elasticsearch.search.aggregations.metrics.avg.Avg; +import org.elasticsearch.search.aggregations.metrics.sum.Sum; +import org.elasticsearch.test.ElasticsearchIntegrationTest; +import org.hamcrest.Matchers; +import org.junit.Before; +import org.junit.Test; + +import java.util.ArrayList; +import java.util.List; + +import static org.elasticsearch.common.xcontent.XContentFactory.jsonBuilder; +import static org.elasticsearch.index.query.QueryBuilders.matchAllQuery; +import static org.elasticsearch.search.aggregations.AggregationBuilders.*; +import static org.hamcrest.Matchers.equalTo; +import static org.hamcrest.Matchers.is; +import static org.hamcrest.core.IsNull.notNullValue; + +/** + * + */ +public class RangeTests extends ElasticsearchIntegrationTest { + + + @Override + public Settings indexSettings() { + return ImmutableSettings.builder() + .put("index.number_of_shards", between(1, 5)) + .put("index.number_of_replicas", between(0, 1)) + .build(); + } + + int numDocs; + + @Before + public void init() throws Exception { + createIndex("idx"); + numDocs = randomIntBetween(10, 20); + IndexRequestBuilder[] builders = new IndexRequestBuilder[numDocs]; + for (int i = 0; i < builders.length; i++) { + builders[i] = client().prepareIndex("idx", "type").setSource(jsonBuilder() + .startObject() + .field("value", i+1) + .startArray("values").value(i+1).value(i+2).endArray() + .endObject()); + } + indexRandom(true, builders); + createIndex("idx_unmapped"); + + } + + @Test + public void singleValueField() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(range("range") + .field("value") + .addUnboundedTo(3) + .addRange(3, 6) + .addUnboundedFrom(6)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Range range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(3)); + + Range.Bucket bucket = range.getByKey("*-3.0"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("*-3.0")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getTo(), equalTo(3.0)); + assertThat(bucket.getDocCount(), equalTo(2l)); + + bucket = range.getByKey("3.0-6.0"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("3.0-6.0")); + assertThat(bucket.getFrom(), equalTo(3.0)); + assertThat(bucket.getTo(), equalTo(6.0)); + assertThat(bucket.getDocCount(), equalTo(3l)); + + bucket = range.getByKey("6.0-*"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("6.0-*")); + assertThat(bucket.getFrom(), equalTo(6.0)); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getDocCount(), equalTo(numDocs - 5L)); + } + + @Test + public void singleValueField_WithCustomKey() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(range("range") + .field("value") + .addUnboundedTo("r1", 3) + .addRange("r2", 3, 6) + .addUnboundedFrom("r3", 6)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Range range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(3)); + + Range.Bucket bucket = range.getByKey("r1"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("r1")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getTo(), equalTo(3.0)); + assertThat(bucket.getDocCount(), equalTo(2l)); + + bucket = range.getByKey("r2"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("r2")); + assertThat(bucket.getFrom(), equalTo(3.0)); + assertThat(bucket.getTo(), equalTo(6.0)); + assertThat(bucket.getDocCount(), equalTo(3l)); + + bucket = range.getByKey("r3"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("r3")); + assertThat(bucket.getFrom(), equalTo(6.0)); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getDocCount(), equalTo(numDocs - 5L)); + } + + @Test + public void singleValuedField_WithSubAggregation() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(range("range") + .field("value") + .addUnboundedTo(3) + .addRange(3, 6) + .addUnboundedFrom(6) + .subAggregation(sum("sum").field("value"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Range range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(3)); + + Range.Bucket bucket = range.getByKey("*-3.0"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("*-3.0")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getTo(), equalTo(3.0)); + assertThat(bucket.getDocCount(), equalTo(2l)); + Sum sum = bucket.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + assertThat(sum.getValue(), equalTo(3.0)); // 1 + 2 + + bucket = range.getByKey("3.0-6.0"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("3.0-6.0")); + assertThat(bucket.getFrom(), equalTo(3.0)); + assertThat(bucket.getTo(), equalTo(6.0)); + assertThat(bucket.getDocCount(), equalTo(3l)); + sum = bucket.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + assertThat(sum.getValue(), equalTo(12.0)); // 3 + 4 + 5 + + bucket = range.getByKey("6.0-*"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("6.0-*")); + assertThat(bucket.getFrom(), equalTo(6.0)); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getDocCount(), equalTo(numDocs - 5l)); + sum = bucket.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + long total = 0; + for (int i = 5; i < numDocs; ++i) { + total += i + 1; + } + assertThat(sum.getValue(), equalTo((double) total)); + } + + @Test + public void singleValuedField_WithSubAggregation_Inherited() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(range("range") + .field("value") + .addUnboundedTo(3) + .addRange(3, 6) + .addUnboundedFrom(6) + .subAggregation(avg("avg"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Range range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(3)); + + Range.Bucket bucket = range.getByKey("*-3.0"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("*-3.0")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getTo(), equalTo(3.0)); + assertThat(bucket.getDocCount(), equalTo(2l)); + Avg avg = bucket.getAggregations().get("avg"); + assertThat(avg, notNullValue()); + assertThat(avg.getValue(), equalTo(1.5)); // (1 + 2) / 2 + + bucket = range.getByKey("3.0-6.0"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("3.0-6.0")); + assertThat(bucket.getFrom(), equalTo(3.0)); + assertThat(bucket.getTo(), equalTo(6.0)); + assertThat(bucket.getDocCount(), equalTo(3l)); + avg = bucket.getAggregations().get("avg"); + assertThat(avg, notNullValue()); + assertThat(avg.getValue(), equalTo(4.0)); // (3 + 4 + 5) / 3 + + bucket = range.getByKey("6.0-*"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("6.0-*")); + assertThat(bucket.getFrom(), equalTo(6.0)); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getDocCount(), equalTo(numDocs - 5l)); + avg = bucket.getAggregations().get("avg"); + assertThat(avg, notNullValue()); + long total = 0; + for (int i = 5; i < numDocs; ++i) { + total += i + 1; + } + assertThat(avg.getValue(), equalTo((double) total / (numDocs - 5))); // (6 + 7 + 8 + 9 + 10) / 5 + } + + @Test + public void singleValuedField_WithValueScript() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(range("range") + .field("value") + .script("_value + 1") + .addUnboundedTo(3) + .addRange(3, 6) + .addUnboundedFrom(6)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Range range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(3)); + + Range.Bucket bucket = range.getByKey("*-3.0"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("*-3.0")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getTo(), equalTo(3.0)); + assertThat(bucket.getDocCount(), equalTo(1l)); // 2 + + bucket = range.getByKey("3.0-6.0"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("3.0-6.0")); + assertThat(bucket.getFrom(), equalTo(3.0)); + assertThat(bucket.getTo(), equalTo(6.0)); + assertThat(bucket.getDocCount(), equalTo(3l)); // 3, 4, 5 + + bucket = range.getByKey("6.0-*"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("6.0-*")); + assertThat(bucket.getFrom(), equalTo(6.0)); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getDocCount(), equalTo(numDocs - 4l)); + } + + /* + [1, 2] + [2, 3] + [3, 4] + [4, 5] + [5, 6] + [6, 7] + [7, 8j + [8, 9] + [9, 10] + [10, 11] + */ + + @Test + public void multiValuedField() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(range("range") + .field("values") + .addUnboundedTo(3) + .addRange(3, 6) + .addUnboundedFrom(6)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Range range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(3)); + + Range.Bucket bucket = range.getByKey("*-3.0"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("*-3.0")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getTo(), equalTo(3.0)); + assertThat(bucket.getDocCount(), equalTo(2l)); + + bucket = range.getByKey("3.0-6.0"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("3.0-6.0")); + assertThat(bucket.getFrom(), equalTo(3.0)); + assertThat(bucket.getTo(), equalTo(6.0)); + assertThat(bucket.getDocCount(), equalTo(4l)); + + bucket = range.getByKey("6.0-*"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("6.0-*")); + assertThat(bucket.getFrom(), equalTo(6.0)); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getDocCount(), equalTo(numDocs - 4l)); + } + + /* + [2, 3] + [3, 4] + [4, 5] + [5, 6] + [6, 7] + [7, 8j + [8, 9] + [9, 10] + [10, 11] + [11, 12] + */ + + @Test + public void multiValuedField_WithValueScript() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(range("range") + .field("values") + .script("_value + 1") + .addUnboundedTo(3) + .addRange(3, 6) + .addUnboundedFrom(6)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Range range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(3)); + + Range.Bucket bucket = range.getByKey("*-3.0"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("*-3.0")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getTo(), equalTo(3.0)); + assertThat(bucket.getDocCount(), equalTo(1l)); + + bucket = range.getByKey("3.0-6.0"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("3.0-6.0")); + assertThat(bucket.getFrom(), equalTo(3.0)); + assertThat(bucket.getTo(), equalTo(6.0)); + assertThat(bucket.getDocCount(), equalTo(4l)); + + bucket = range.getByKey("6.0-*"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("6.0-*")); + assertThat(bucket.getFrom(), equalTo(6.0)); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getDocCount(), equalTo(numDocs - 3l)); + } + + /* + [2, 3] + [3, 4] + [4, 5] + [5, 6] + [6, 7] + [7, 8j + [8, 9] + [9, 10] + [10, 11] + [11, 12] + + r1: 2 + r2: 3, 3, 4, 4, 5, 5 + r3: 6, 6, 7, 7, 8, 8, 9, 9, 10, 10, 11, 11, 12 + */ + + @Test + public void multiValuedField_WithValueScript_WithInheritedSubAggregator() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(range("range") + .field("values") + .script("_value + 1") + .addUnboundedTo(3) + .addRange(3, 6) + .addUnboundedFrom(6) + .subAggregation(sum("sum"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Range range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(3)); + + Range.Bucket bucket = range.getByKey("*-3.0"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("*-3.0")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getTo(), equalTo(3.0)); + assertThat(bucket.getDocCount(), equalTo(1l)); + Sum sum = bucket.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + assertThat(sum.getName(), equalTo("sum")); + assertThat(sum.getValue(), equalTo(2d+3d)); + + bucket = range.getByKey("3.0-6.0"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("3.0-6.0")); + assertThat(bucket.getFrom(), equalTo(3.0)); + assertThat(bucket.getTo(), equalTo(6.0)); + assertThat(bucket.getDocCount(), equalTo(4l)); + sum = bucket.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + assertThat(sum.getName(), equalTo("sum")); + assertThat(sum.getValue(), equalTo((double) 2+3+3+4+4+5+5+6)); + + bucket = range.getByKey("6.0-*"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("6.0-*")); + assertThat(bucket.getFrom(), equalTo(6.0)); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getDocCount(), equalTo(numDocs - 3L)); + sum = bucket.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + assertThat(sum.getName(), equalTo("sum")); + long total = 0; + for (int i = 3; i < numDocs; ++i) { + total += ((i + 1) + 1) + ((i + 1) + 2); + } + assertThat(sum.getValue(), equalTo((double) total)); + } + + @Test + public void script_SingleValue() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(range("range") + .script("doc['value'].value") + .addUnboundedTo(3) + .addRange(3, 6) + .addUnboundedFrom(6)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Range range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(3)); + + Range.Bucket bucket = range.getByKey("*-3.0"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("*-3.0")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getTo(), equalTo(3.0)); + assertThat(bucket.getDocCount(), equalTo(2l)); + + bucket = range.getByKey("3.0-6.0"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("3.0-6.0")); + assertThat(bucket.getFrom(), equalTo(3.0)); + assertThat(bucket.getTo(), equalTo(6.0)); + assertThat(bucket.getDocCount(), equalTo(3l)); + + bucket = range.getByKey("6.0-*"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("6.0-*")); + assertThat(bucket.getFrom(), equalTo(6.0)); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getDocCount(), equalTo(numDocs - 5l)); + } + + @Test + public void script_SingleValue_WithSubAggregator_Inherited() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(range("range") + .script("doc['value'].value") + .addUnboundedTo(3) + .addRange(3, 6) + .addUnboundedFrom(6) + .subAggregation(avg("avg"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Range range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(3)); + + Range.Bucket bucket = range.getByKey("*-3.0"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("*-3.0")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getTo(), equalTo(3.0)); + assertThat(bucket.getDocCount(), equalTo(2l)); + Avg avg = bucket.getAggregations().get("avg"); + assertThat(avg, notNullValue()); + assertThat(avg.getValue(), equalTo(1.5)); // (1 + 2) / 2 + + bucket = range.getByKey("3.0-6.0"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("3.0-6.0")); + assertThat(bucket.getFrom(), equalTo(3.0)); + assertThat(bucket.getTo(), equalTo(6.0)); + assertThat(bucket.getDocCount(), equalTo(3l)); + avg = bucket.getAggregations().get("avg"); + assertThat(avg, notNullValue()); + assertThat(avg.getValue(), equalTo(4.0)); // (3 + 4 + 5) / 3 + + bucket = range.getByKey("6.0-*"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("6.0-*")); + assertThat(bucket.getFrom(), equalTo(6.0)); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getDocCount(), equalTo(numDocs - 5l)); + avg = bucket.getAggregations().get("avg"); + assertThat(avg, notNullValue()); + long total = 0; + for (int i = 5; i < numDocs; ++i) { + total += i + 1; + } + assertThat(avg.getValue(), equalTo((double) total / (numDocs - 5))); // (6 + 7 + 8 + 9 + 10) / 5 + } + + @Test + public void emptyRange() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(range("range") + .field("values") + .addUnboundedTo(-1) + .addUnboundedFrom(1000)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Range range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(2)); + + Range.Bucket bucket = range.getByKey("*--1.0"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("*--1.0")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getTo(), equalTo(-1.0)); + assertThat(bucket.getDocCount(), equalTo(0l)); + + bucket = range.getByKey("1000.0-*"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("1000.0-*")); + assertThat(bucket.getFrom(), equalTo(1000d)); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getDocCount(), equalTo(0l)); + } + + @Test + public void script_MultiValued() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(range("range") + .script("doc['values'].values") + .addUnboundedTo(3) + .addRange(3, 6) + .addUnboundedFrom(6)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Range range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(3)); + + Range.Bucket bucket = range.getByKey("*-3.0"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("*-3.0")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getTo(), equalTo(3.0)); + assertThat(bucket.getDocCount(), equalTo(2l)); + + bucket = range.getByKey("3.0-6.0"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("3.0-6.0")); + assertThat(bucket.getFrom(), equalTo(3.0)); + assertThat(bucket.getTo(), equalTo(6.0)); + assertThat(bucket.getDocCount(), equalTo(4l)); + + bucket = range.getByKey("6.0-*"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("6.0-*")); + assertThat(bucket.getFrom(), equalTo(6.0)); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getDocCount(), equalTo(numDocs - 4l)); + } + + /* + [1, 2] + [2, 3] + [3, 4] + [4, 5] + [5, 6] + [6, 7] + [7, 8j + [8, 9] + [9, 10] + [10, 11] + + r1: 1, 2, 2 + r2: 3, 3, 4, 4, 5, 5 + r3: 6, 6, 7, 7, 8, 8, 9, 9, 10, 10, 11 + */ + + @Test + public void script_MultiValued_WithAggregatorInherited() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(range("range") + .script("doc['values'].values") + .addUnboundedTo("r1", 3) + .addRange("r2", 3, 6) + .addUnboundedFrom("r3", 6) + .subAggregation(sum("sum"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Range range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(3)); + + Range.Bucket bucket = range.getByKey("r1"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("r1")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getTo(), equalTo(3.0)); + assertThat(bucket.getDocCount(), equalTo(2l)); + Sum sum = bucket.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + assertThat(sum.getName(), equalTo("sum")); + assertThat(sum.getValue(), equalTo((double) 1+2+2+3)); + + bucket = range.getByKey("r2"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("r2")); + assertThat(bucket.getFrom(), equalTo(3.0)); + assertThat(bucket.getTo(), equalTo(6.0)); + assertThat(bucket.getDocCount(), equalTo(4l)); + sum = bucket.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + assertThat(sum.getName(), equalTo("sum")); + assertThat(sum.getValue(), equalTo((double) 2+3+3+4+4+5+5+6)); + + bucket = range.getByKey("r3"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("r3")); + assertThat(bucket.getFrom(), equalTo(6.0)); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getDocCount(), equalTo(numDocs - 4l)); + sum = bucket.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + assertThat(sum.getName(), equalTo("sum")); + long total = 0; + for (int i = 4; i < numDocs; ++i) { + total += (i + 1) + (i + 2); + } + assertThat(sum.getValue(), equalTo((double) total)); + } + + @Test + public void unmapped() throws Exception { + client().admin().cluster().prepareHealth("idx_unmapped").setWaitForYellowStatus().execute().actionGet(); + + SearchResponse response = client().prepareSearch("idx_unmapped") + .addAggregation(range("range") + .field("value") + .addUnboundedTo(3) + .addRange(3, 6) + .addUnboundedFrom(6)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Range range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(3)); + + Range.Bucket bucket = range.getByKey("*-3.0"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("*-3.0")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getTo(), equalTo(3.0)); + assertThat(bucket.getDocCount(), equalTo(0l)); + + bucket = range.getByKey("3.0-6.0"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("3.0-6.0")); + assertThat(bucket.getFrom(), equalTo(3.0)); + assertThat(bucket.getTo(), equalTo(6.0)); + assertThat(bucket.getDocCount(), equalTo(0l)); + + bucket = range.getByKey("6.0-*"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("6.0-*")); + assertThat(bucket.getFrom(), equalTo(6.0)); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getDocCount(), equalTo(0l)); + } + + @Test + public void partiallyUnmapped() throws Exception { + client().admin().cluster().prepareHealth("idx_unmapped").setWaitForYellowStatus().execute().actionGet(); + + SearchResponse response = client().prepareSearch("idx", "idx_unmapped") + .addAggregation(range("range") + .field("value") + .addUnboundedTo(3) + .addRange(3, 6) + .addUnboundedFrom(6)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Range range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(3)); + + Range.Bucket bucket = range.getByKey("*-3.0"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("*-3.0")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getTo(), equalTo(3.0)); + assertThat(bucket.getDocCount(), equalTo(2l)); + + bucket = range.getByKey("3.0-6.0"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("3.0-6.0")); + assertThat(bucket.getFrom(), equalTo(3.0)); + assertThat(bucket.getTo(), equalTo(6.0)); + assertThat(bucket.getDocCount(), equalTo(3l)); + + bucket = range.getByKey("6.0-*"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("6.0-*")); + assertThat(bucket.getFrom(), equalTo(6.0)); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getDocCount(), equalTo(numDocs - 5l)); + } + + @Test + public void overlappingRanges() throws Exception { + SearchResponse response = client().prepareSearch("idx") + .addAggregation(range("range") + .field("values") + .addUnboundedTo(5) + .addRange(3, 6) + .addRange(4, 5) + .addUnboundedFrom(4)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Range range = response.getAggregations().get("range"); + assertThat(range, notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), equalTo(4)); + + Range.Bucket bucket = range.getByKey("*-5.0"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("*-5.0")); + assertThat(bucket.getFrom(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(bucket.getTo(), equalTo(5.0)); + assertThat(bucket.getDocCount(), equalTo(4l)); + + bucket = range.getByKey("3.0-6.0"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("3.0-6.0")); + assertThat(bucket.getFrom(), equalTo(3.0)); + assertThat(bucket.getTo(), equalTo(6.0)); + assertThat(bucket.getDocCount(), equalTo(4l)); + + bucket = range.getByKey("4.0-5.0"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("4.0-5.0")); + assertThat(bucket.getFrom(), equalTo(4.0)); + assertThat(bucket.getTo(), equalTo(5.0)); + assertThat(bucket.getDocCount(), equalTo(2l)); + + bucket = range.getByKey("4.0-*"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey(), equalTo("4.0-*")); + assertThat(bucket.getFrom(), equalTo(4.0)); + assertThat(bucket.getTo(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(bucket.getDocCount(), equalTo(numDocs - 2l)); + } + + @Test + public void emptyAggregation() throws Exception { + prepareCreate("empty_bucket_idx").addMapping("type", "value", "type=integer").execute().actionGet(); + List builders = new ArrayList(); + for (int i = 0; i < 2; i++) { + builders.add(client().prepareIndex("empty_bucket_idx", "type", "" + i).setSource(jsonBuilder() + .startObject() + .field("value", i * 2) + .endObject())); + } + indexRandom(true, builders.toArray(new IndexRequestBuilder[builders.size()])); + + SearchResponse searchResponse = client().prepareSearch("empty_bucket_idx") + .setQuery(matchAllQuery()) + .addAggregation(histogram("histo").field("value").interval(1l).emptyBuckets(true) + .subAggregation(range("range").addRange("0-2", 0.0, 2.0))) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(2l)); + Histogram histo = searchResponse.getAggregations().get("histo"); + assertThat(histo, Matchers.notNullValue()); + Histogram.Bucket bucket = histo.getByKey(1l); + assertThat(bucket, Matchers.notNullValue()); + + Range range = bucket.getAggregations().get("range"); + assertThat(range, Matchers.notNullValue()); + assertThat(range.getName(), equalTo("range")); + assertThat(range.buckets().size(), is(1)); + assertThat(range.buckets().get(0).getKey(), equalTo("0-2")); + assertThat(range.buckets().get(0).getFrom(), equalTo(0.0)); + assertThat(range.buckets().get(0).getTo(), equalTo(2.0)); + assertThat(range.buckets().get(0).getDocCount(), equalTo(0l)); + + } +} diff --git a/src/test/java/org/elasticsearch/search/aggregations/bucket/StringTermsTests.java b/src/test/java/org/elasticsearch/search/aggregations/bucket/StringTermsTests.java new file mode 100644 index 00000000000..f59e831131c --- /dev/null +++ b/src/test/java/org/elasticsearch/search/aggregations/bucket/StringTermsTests.java @@ -0,0 +1,580 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.bucket; + +import com.google.common.base.Strings; +import org.elasticsearch.action.index.IndexRequestBuilder; +import org.elasticsearch.action.search.SearchResponse; +import org.elasticsearch.common.settings.ImmutableSettings; +import org.elasticsearch.common.settings.Settings; +import org.elasticsearch.search.aggregations.bucket.histogram.Histogram; +import org.elasticsearch.search.aggregations.bucket.terms.Terms; +import org.elasticsearch.search.aggregations.metrics.valuecount.ValueCount; +import org.elasticsearch.test.ElasticsearchIntegrationTest; +import org.hamcrest.Matchers; +import org.junit.Before; +import org.junit.Test; + +import java.util.ArrayList; +import java.util.List; + +import static org.elasticsearch.common.xcontent.XContentFactory.jsonBuilder; +import static org.elasticsearch.index.query.QueryBuilders.matchAllQuery; +import static org.elasticsearch.search.aggregations.AggregationBuilders.*; +import static org.hamcrest.Matchers.equalTo; +import static org.hamcrest.Matchers.is; +import static org.hamcrest.core.IsNull.notNullValue; + +/** + * + */ +public class StringTermsTests extends ElasticsearchIntegrationTest { + + + @Override + public Settings indexSettings() { + return ImmutableSettings.builder() + .put("index.number_of_shards", between(1, 5)) + .put("index.number_of_replicas", between(0, 1)) + .build(); + } + + @Before + public void init() throws Exception { + createIndex("idx"); + IndexRequestBuilder[] lowCardBuilders = new IndexRequestBuilder[5]; // TODO randomize the size? + for (int i = 0; i < lowCardBuilders.length; i++) { + lowCardBuilders[i] = client().prepareIndex("idx", "type").setSource(jsonBuilder() + .startObject() + .field("value", "val" + i) + .startArray("values").value("val" + i).value("val" + (i + 1)).endArray() + .endObject()); + } + indexRandom(true, lowCardBuilders); + IndexRequestBuilder[] highCardBuilders = new IndexRequestBuilder[100]; // TODO randomize the size? + + for (int i = 0; i < highCardBuilders.length; i++) { + highCardBuilders[i] = client().prepareIndex("idx", "high_card_type").setSource(jsonBuilder() + .startObject() + .field("value", "val" + Strings.padStart(i+"", 3, '0')) + .startArray("values").value("val" + Strings.padStart(i+"", 3, '0')).value("val" + Strings.padStart((i+1)+"", 3, '0')).endArray() + .endObject()); + } + indexRandom(true, highCardBuilders); + createIndex("idx_unmapped"); + + } + + @Test + public void singleValueField() throws Exception { + SearchResponse response = client().prepareSearch("idx").setTypes("type") + .addAggregation(terms("terms") + .field("value")) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(5)); + + for (int i = 0; i < 5; i++) { + Terms.Bucket bucket = terms.getByTerm("val" + i); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("val" + i)); + assertThat(bucket.getDocCount(), equalTo(1l)); + } + } + + @Test + public void singleValueField_WithMaxSize() throws Exception { + SearchResponse response = client().prepareSearch("idx").setTypes("high_card_type") + .addAggregation(terms("terms") + .field("value") + .size(20) + .order(Terms.Order.TERM_ASC)) // we need to sort by terms cause we're checking the first 20 values + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(20)); + + for (int i = 0; i < 20; i++) { + Terms.Bucket bucket = terms.getByTerm("val" + Strings.padStart(i+"", 3, '0')); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("val" + Strings.padStart(i+"", 3, '0'))); + assertThat(bucket.getDocCount(), equalTo(1l)); + } + } + + @Test + public void singleValueField_OrderedByTermAsc() throws Exception { + SearchResponse response = client().prepareSearch("idx").setTypes("type") + .addAggregation(terms("terms") + .field("value") + .order(Terms.Order.TERM_ASC)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(5)); + + int i = 0; + for (Terms.Bucket bucket : terms.buckets()) { + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("val" + i)); + assertThat(bucket.getDocCount(), equalTo(1l)); + i++; + } + } + + @Test + public void singleValueField_OrderedByTermDesc() throws Exception { + SearchResponse response = client().prepareSearch("idx").setTypes("type") + .addAggregation(terms("terms") + .field("value") + .order(Terms.Order.TERM_DESC)) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(5)); + + int i = 4; + for (Terms.Bucket bucket : terms.buckets()) { + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("val" + i)); + assertThat(bucket.getDocCount(), equalTo(1l)); + i--; + } + } + + @Test + public void singleValuedField_WithSubAggregation() throws Exception { + SearchResponse response = client().prepareSearch("idx").setTypes("type") + .addAggregation(terms("terms") + .field("value") + .subAggregation(count("count").field("values"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(5)); + + for (int i = 0; i < 5; i++) { + Terms.Bucket bucket = terms.getByTerm("val" + i); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("val" + i)); + assertThat(bucket.getDocCount(), equalTo(1l)); + ValueCount valueCount = bucket.getAggregations().get("count"); + assertThat(valueCount, notNullValue()); + assertThat(valueCount.getValue(), equalTo(2l)); + } + } + + @Test + public void singleValuedField_WithSubAggregation_Inherited() throws Exception { + SearchResponse response = client().prepareSearch("idx").setTypes("type") + .addAggregation(terms("terms") + .field("value") + .subAggregation(count("count"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(5)); + + for (int i = 0; i < 5; i++) { + Terms.Bucket bucket = terms.getByTerm("val" + i); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("val" + i)); + assertThat(bucket.getDocCount(), equalTo(1l)); + ValueCount valueCount = bucket.getAggregations().get("count"); + assertThat(valueCount, notNullValue()); + assertThat(valueCount.getValue(), equalTo(1l)); + } + } + + @Test + public void singleValuedField_WithValueScript() throws Exception { + SearchResponse response = client().prepareSearch("idx").setTypes("type") + .addAggregation(terms("terms") + .field("value") + .script("'foo_' + _value")) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(5)); + + for (int i = 0; i < 5; i++) { + Terms.Bucket bucket = terms.getByTerm("foo_val" + i); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("foo_val" + i)); + assertThat(bucket.getDocCount(), equalTo(1l)); + } + } + + @Test + public void multiValuedField_WithValueScript_NotUnique() throws Exception { + SearchResponse response = client().prepareSearch("idx").setTypes("type") + .addAggregation(terms("terms") + .field("values") + .script("_value.substring(0,3)")) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(1)); + + Terms.Bucket bucket = terms.getByTerm("val"); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("val")); + assertThat(bucket.getDocCount(), equalTo(5l)); + } + + @Test + public void multiValuedField() throws Exception { + SearchResponse response = client().prepareSearch("idx").setTypes("type") + .addAggregation(terms("terms") + .field("values")) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(6)); + + for (int i = 0; i < 6; i++) { + Terms.Bucket bucket = terms.getByTerm("val" + i); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("val" + i)); + if (i == 0 || i == 5) { + assertThat(bucket.getDocCount(), equalTo(1l)); + } else { + assertThat(bucket.getDocCount(), equalTo(2l)); + } + } + } + + @Test + public void multiValuedField_WithValueScript() throws Exception { + SearchResponse response = client().prepareSearch("idx").setTypes("type") + .addAggregation(terms("terms") + .field("values") + .script("'foo_' + _value")) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(6)); + + for (int i = 0; i < 6; i++) { + Terms.Bucket bucket = terms.getByTerm("foo_val" + i); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("foo_val" + i)); + if (i == 0 || i == 5) { + assertThat(bucket.getDocCount(), equalTo(1l)); + } else { + assertThat(bucket.getDocCount(), equalTo(2l)); + } + } + } + + /* + + [foo_val0, foo_val1] + [foo_val1, foo_val2] + [foo_val2, foo_val3] + [foo_val3, foo_val4] + [foo_val4, foo_val5] + + + foo_val0 - doc_count: 1 - val_count: 2 + foo_val1 - doc_count: 2 - val_count: 4 + foo_val2 - doc_count: 2 - val_count: 4 + foo_val3 - doc_count: 2 - val_count: 4 + foo_val4 - doc_count: 2 - val_count: 4 + foo_val5 - doc_count: 1 - val_count: 2 + + */ + + @Test + public void multiValuedField_WithValueScript_WithInheritedSubAggregator() throws Exception { + SearchResponse response = client().prepareSearch("idx").setTypes("type") + .addAggregation(terms("terms") + .field("values") + .script("'foo_' + _value") + .subAggregation(count("count"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(6)); + + for (int i = 0; i < 6; i++) { + Terms.Bucket bucket = terms.getByTerm("foo_val" + i); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("foo_val" + i)); + if (i == 0 | i == 5) { + assertThat(bucket.getDocCount(), equalTo(1l)); + ValueCount valueCount = bucket.getAggregations().get("count"); + assertThat(valueCount, notNullValue()); + assertThat(valueCount.getValue(), equalTo(2l)); + } else { + assertThat(bucket.getDocCount(), equalTo(2l)); + ValueCount valueCount = bucket.getAggregations().get("count"); + assertThat(valueCount, notNullValue()); + assertThat("term[" + bucket.getKey().string() + "]", valueCount.getValue(), equalTo(4l)); + } + } + } + + @Test + public void script_SingleValue() throws Exception { + SearchResponse response = client().prepareSearch("idx").setTypes("type") + .addAggregation(terms("terms") + .script("doc['value'].value")) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(5)); + + for (int i = 0; i < 5; i++) { + Terms.Bucket bucket = terms.getByTerm("val" + i); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("val" + i)); + assertThat(bucket.getDocCount(), equalTo(1l)); + } + } + + @Test + public void script_SingleValue_ExplicitSingleValue() throws Exception { + SearchResponse response = client().prepareSearch("idx").setTypes("type") + .addAggregation(terms("terms") + .script("doc['value'].value")) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(5)); + + for (int i = 0; i < 5; i++) { + Terms.Bucket bucket = terms.getByTerm("val" + i); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("val" + i)); + assertThat(bucket.getDocCount(), equalTo(1l)); + } + } + + @Test + public void script_SingleValue_WithSubAggregator_Inherited() throws Exception { + SearchResponse response = client().prepareSearch("idx").setTypes("type") + .addAggregation(terms("terms") + .script("doc['value'].value") + .subAggregation(count("count"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(5)); + + for (int i = 0; i < 5; i++) { + Terms.Bucket bucket = terms.getByTerm("val" + i); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("val" + i)); + assertThat(bucket.getDocCount(), equalTo(1l)); + ValueCount valueCount = bucket.getAggregations().get("count"); + assertThat(valueCount, notNullValue()); + assertThat(valueCount.getValue(), equalTo(1l)); + } + } + + @Test + public void script_MultiValued() throws Exception { + SearchResponse response = client().prepareSearch("idx").setTypes("type") + .addAggregation(terms("terms") + .script("doc['values'].values")) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(6)); + + for (int i = 0; i < 6; i++) { + Terms.Bucket bucket = terms.getByTerm("val" + i); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("val" + i)); + if (i == 0 || i == 5) { + assertThat(bucket.getDocCount(), equalTo(1l)); + } else { + assertThat(bucket.getDocCount(), equalTo(2l)); + } + } + } + + @Test + public void script_MultiValued_WithAggregatorInherited() throws Exception { + SearchResponse response = client().prepareSearch("idx").setTypes("type") + .addAggregation(terms("terms") + .script("doc['values'].values") + .subAggregation(count("count"))) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(6)); + + for (int i = 0; i < 6; i++) { + Terms.Bucket bucket = terms.getByTerm("val" + i); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("val" + i)); + if (i == 0 | i == 5) { + assertThat(bucket.getDocCount(), equalTo(1l)); + ValueCount valueCount = bucket.getAggregations().get("count"); + assertThat(valueCount, notNullValue()); + assertThat(valueCount.getValue(), equalTo(2l)); + } else { + assertThat(bucket.getDocCount(), equalTo(2l)); + ValueCount valueCount = bucket.getAggregations().get("count"); + assertThat(valueCount, notNullValue()); + assertThat(valueCount.getValue(), equalTo(4l)); + } + } + } + + @Test + public void unmapped() throws Exception { + client().admin().cluster().prepareHealth("idx_unmapped").setWaitForYellowStatus().execute().actionGet(); + + SearchResponse response = client().prepareSearch("idx_unmapped").setTypes("type") + .addAggregation(terms("terms") + .field("value")) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(0)); + } + + @Test + public void partiallyUnmapped() throws Exception { + client().admin().cluster().prepareHealth("idx_unmapped").setWaitForYellowStatus().execute().actionGet(); + + SearchResponse response = client().prepareSearch("idx", "idx_unmapped").setTypes("type") + .addAggregation(terms("terms") + .field("value")) + .execute().actionGet(); + + assertThat(response.getFailedShards(), equalTo(0)); + + Terms terms = response.getAggregations().get("terms"); + assertThat(terms, notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().size(), equalTo(5)); + + for (int i = 0; i < 5; i++) { + Terms.Bucket bucket = terms.getByTerm("val" + i); + assertThat(bucket, notNullValue()); + assertThat(bucket.getKey().string(), equalTo("val" + i)); + assertThat(bucket.getDocCount(), equalTo(1l)); + } + } + + @Test + public void emptyAggregation() throws Exception { + prepareCreate("empty_bucket_idx").addMapping("type", "value", "type=integer").execute().actionGet(); + List builders = new ArrayList(); + for (int i = 0; i < 2; i++) { + builders.add(client().prepareIndex("empty_bucket_idx", "type", ""+i).setSource(jsonBuilder() + .startObject() + .field("value", i*2) + .endObject())); + } + indexRandom(true, builders.toArray(new IndexRequestBuilder[builders.size()])); + + SearchResponse searchResponse = client().prepareSearch("empty_bucket_idx") + .setQuery(matchAllQuery()) + .addAggregation(histogram("histo").field("value").interval(1l).emptyBuckets(true) + .subAggregation(terms("terms"))) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(2l)); + Histogram histo = searchResponse.getAggregations().get("histo"); + assertThat(histo, Matchers.notNullValue()); + Histogram.Bucket bucket = histo.getByKey(1l); + assertThat(bucket, Matchers.notNullValue()); + + Terms terms = bucket.getAggregations().get("terms"); + assertThat(terms, Matchers.notNullValue()); + assertThat(terms.getName(), equalTo("terms")); + assertThat(terms.buckets().isEmpty(), is(true)); + } + +} diff --git a/src/test/java/org/elasticsearch/search/aggregations/metrics/AbstractNumericTests.java b/src/test/java/org/elasticsearch/search/aggregations/metrics/AbstractNumericTests.java new file mode 100644 index 00000000000..38ea10fc89f --- /dev/null +++ b/src/test/java/org/elasticsearch/search/aggregations/metrics/AbstractNumericTests.java @@ -0,0 +1,109 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.metrics; + +import org.elasticsearch.action.index.IndexRequestBuilder; +import org.elasticsearch.common.settings.ImmutableSettings; +import org.elasticsearch.common.settings.Settings; +import org.elasticsearch.test.ElasticsearchIntegrationTest; +import org.junit.Before; + +import java.util.ArrayList; +import java.util.List; + +import static org.elasticsearch.common.xcontent.XContentFactory.jsonBuilder; + +/** + * + */ +public abstract class AbstractNumericTests extends ElasticsearchIntegrationTest { + + @Override + public Settings indexSettings() { + return ImmutableSettings.builder() + .put("index.number_of_shards", between(1, 5)) + .put("index.number_of_replicas", between(0, 1)) + .build(); + } + + @Before + public void init() throws Exception { + createIndex("idx"); + createIndex("idx_unmapped"); + + List builders = new ArrayList(); + + for (int i = 0; i < 10; i++) { // TODO randomize the size and the params in here? + builders.add(client().prepareIndex("idx", "type", ""+i).setSource(jsonBuilder() + .startObject() + .field("value", i+1) + .startArray("values").value(i+2).value(i+3).endArray() + .endObject())); + } + indexRandom(true, builders); + + // creating an index to test the empty buckets functionality. The way it works is by indexing + // two docs {value: 0} and {value : 2}, then building a histogram agg with interval 1 and with empty + // buckets computed.. the empty bucket is the one associated with key "1". then each test will have + // to check that this bucket exists with the appropriate sub aggregations. + prepareCreate("empty_bucket_idx").addMapping("type", "value", "type=integer").execute().actionGet(); + builders = new ArrayList(); + for (int i = 0; i < 2; i++) { + builders.add(client().prepareIndex("empty_bucket_idx", "type", ""+i).setSource(jsonBuilder() + .startObject() + .field("value", i*2) + .endObject())); + } + indexRandom(true, builders); + + } + + public abstract void testEmptyAggregation() throws Exception; + + public abstract void testUnmapped() throws Exception; + + public abstract void testSingleValuedField() throws Exception; + + public abstract void testSingleValuedField_PartiallyUnmapped() throws Exception; + + public abstract void testSingleValuedField_WithValueScript() throws Exception; + + public abstract void testSingleValuedField_WithValueScript_WithParams() throws Exception; + + public abstract void testMultiValuedField() throws Exception; + + public abstract void testMultiValuedField_WithValueScript() throws Exception; + + public abstract void testMultiValuedField_WithValueScript_WithParams() throws Exception; + + public abstract void testScript_SingleValued() throws Exception; + + public abstract void testScript_SingleValued_WithParams() throws Exception; + + public abstract void testScript_ExplicitSingleValued_WithParams() throws Exception; + + public abstract void testScript_MultiValued() throws Exception; + + public abstract void testScript_ExplicitMultiValued() throws Exception; + + public abstract void testScript_MultiValued_WithParams() throws Exception; + + +} \ No newline at end of file diff --git a/src/test/java/org/elasticsearch/search/aggregations/metrics/AvgTests.java b/src/test/java/org/elasticsearch/search/aggregations/metrics/AvgTests.java new file mode 100644 index 00000000000..7d879ab3603 --- /dev/null +++ b/src/test/java/org/elasticsearch/search/aggregations/metrics/AvgTests.java @@ -0,0 +1,270 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.metrics; + +import org.elasticsearch.action.search.SearchResponse; +import org.elasticsearch.search.aggregations.bucket.histogram.Histogram; +import org.elasticsearch.search.aggregations.metrics.avg.Avg; +import org.junit.Test; + +import static org.elasticsearch.index.query.QueryBuilders.matchAllQuery; +import static org.elasticsearch.search.aggregations.AggregationBuilders.avg; +import static org.elasticsearch.search.aggregations.AggregationBuilders.histogram; +import static org.hamcrest.Matchers.*; + +/** + * + */ +public class AvgTests extends AbstractNumericTests { + + @Test + public void testEmptyAggregation() throws Exception { + + SearchResponse searchResponse = client().prepareSearch("empty_bucket_idx") + .setQuery(matchAllQuery()) + .addAggregation(histogram("histo").field("value").interval(1l).emptyBuckets(true).subAggregation(avg("avg"))) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(2l)); + Histogram histo = searchResponse.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + Histogram.Bucket bucket = histo.getByKey(1l); + assertThat(bucket, notNullValue()); + + Avg avg = bucket.getAggregations().get("avg"); + assertThat(avg, notNullValue()); + assertThat(avg.getName(), equalTo("avg")); + assertThat(Double.isNaN(avg.getValue()), is(true)); + } + + @Test + public void testUnmapped() throws Exception { + client().admin().cluster().prepareHealth("idx_unmapped").setWaitForGreenStatus().execute().actionGet(); + + SearchResponse searchResponse = client().prepareSearch("idx_unmapped") + .setQuery(matchAllQuery()) + .addAggregation(avg("avg").field("value")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(0l)); + + Avg avg = searchResponse.getAggregations().get("avg"); + assertThat(avg, notNullValue()); + assertThat(avg.getName(), equalTo("avg")); + assertThat(avg.getValue(), equalTo(Double.NaN)); + } + + @Test + public void testSingleValuedField() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(avg("avg").field("value")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Avg avg = searchResponse.getAggregations().get("avg"); + assertThat(avg, notNullValue()); + assertThat(avg.getName(), equalTo("avg")); + assertThat(avg.getValue(), equalTo((double) (1+2+3+4+5+6+7+8+9+10) / 10)); + } + + @Override + public void testSingleValuedField_PartiallyUnmapped() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx", "idx_unmapped") + .setQuery(matchAllQuery()) + .addAggregation(avg("avg").field("value")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Avg avg = searchResponse.getAggregations().get("avg"); + assertThat(avg, notNullValue()); + assertThat(avg.getName(), equalTo("avg")); + assertThat(avg.getValue(), equalTo((double) (1+2+3+4+5+6+7+8+9+10) / 10)); + } + + @Test + public void testSingleValuedField_WithValueScript() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(avg("avg").field("value").script("_value + 1")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Avg avg = searchResponse.getAggregations().get("avg"); + assertThat(avg, notNullValue()); + assertThat(avg.getName(), equalTo("avg")); + assertThat(avg.getValue(), equalTo((double) (2+3+4+5+6+7+8+9+10+11) / 10)); + } + + @Test + public void testSingleValuedField_WithValueScript_WithParams() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(avg("avg").field("value").script("_value + inc").param("inc", 1)) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Avg avg = searchResponse.getAggregations().get("avg"); + assertThat(avg, notNullValue()); + assertThat(avg.getName(), equalTo("avg")); + assertThat(avg.getValue(), equalTo((double) (2+3+4+5+6+7+8+9+10+11) / 10)); + } + + + @Test + public void testMultiValuedField() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(avg("avg").field("values")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Avg avg = searchResponse.getAggregations().get("avg"); + assertThat(avg, notNullValue()); + assertThat(avg.getName(), equalTo("avg")); + assertThat(avg.getValue(), equalTo((double) (2+3+3+4+4+5+5+6+6+7+7+8+8+9+9+10+10+11+11+12) / 20)); + } + + @Test + public void testMultiValuedField_WithValueScript() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(avg("avg").field("values").script("_value + 1")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Avg avg = searchResponse.getAggregations().get("avg"); + assertThat(avg, notNullValue()); + assertThat(avg.getName(), equalTo("avg")); + assertThat(avg.getValue(), equalTo((double) (3+4+4+5+5+6+6+7+7+8+8+9+9+10+10+11+11+12+12+13) / 20)); + } + + @Test + public void testMultiValuedField_WithValueScript_WithParams() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(avg("avg").field("values").script("_value + inc").param("inc", 1)) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Avg avg = searchResponse.getAggregations().get("avg"); + assertThat(avg, notNullValue()); + assertThat(avg.getName(), equalTo("avg")); + assertThat(avg.getValue(), equalTo((double) (3+4+4+5+5+6+6+7+7+8+8+9+9+10+10+11+11+12+12+13) / 20)); + } + + @Test + public void testScript_SingleValued() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(avg("avg").script("doc['value'].value")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Avg avg = searchResponse.getAggregations().get("avg"); + assertThat(avg, notNullValue()); + assertThat(avg.getName(), equalTo("avg")); + assertThat(avg.getValue(), equalTo((double) (1+2+3+4+5+6+7+8+9+10) / 10)); + } + + @Test + public void testScript_SingleValued_WithParams() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(avg("avg").script("doc['value'].value + inc").param("inc", 1)) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Avg avg = searchResponse.getAggregations().get("avg"); + assertThat(avg, notNullValue()); + assertThat(avg.getName(), equalTo("avg")); + assertThat(avg.getValue(), equalTo((double) (2+3+4+5+6+7+8+9+10+11) / 10)); + } + + @Test + public void testScript_ExplicitSingleValued_WithParams() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(avg("avg").script("doc['value'].value + inc").param("inc", 1)) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Avg avg = searchResponse.getAggregations().get("avg"); + assertThat(avg, notNullValue()); + assertThat(avg.getName(), equalTo("avg")); + assertThat(avg.getValue(), equalTo((double) (2+3+4+5+6+7+8+9+10+11) / 10)); + } + + @Test + public void testScript_MultiValued() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(avg("avg").script("new double[] { doc['value'].value, doc['value'].value + 1 }")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Avg avg = searchResponse.getAggregations().get("avg"); + assertThat(avg, notNullValue()); + assertThat(avg.getName(), equalTo("avg")); + assertThat(avg.getValue(), equalTo((double) (1+2+2+3+3+4+4+5+5+6+6+7+7+8+8+9+9+10+10+11) / 20)); + } + + @Test + public void testScript_ExplicitMultiValued() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(avg("avg").script("new double[] { doc['value'].value, doc['value'].value + 1 }")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Avg avg = searchResponse.getAggregations().get("avg"); + assertThat(avg, notNullValue()); + assertThat(avg.getName(), equalTo("avg")); + assertThat(avg.getValue(), equalTo((double) (1+2+2+3+3+4+4+5+5+6+6+7+7+8+8+9+9+10+10+11) / 20)); + } + + @Test + public void testScript_MultiValued_WithParams() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(avg("avg").script("new double[] { doc['value'].value, doc['value'].value + inc }").param("inc", 1)) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Avg avg = searchResponse.getAggregations().get("avg"); + assertThat(avg, notNullValue()); + assertThat(avg.getName(), equalTo("avg")); + assertThat(avg.getValue(), equalTo((double) (1+2+2+3+3+4+4+5+5+6+6+7+7+8+8+9+9+10+10+11) / 20)); + } + +} \ No newline at end of file diff --git a/src/test/java/org/elasticsearch/search/aggregations/metrics/ExtendedStatsTests.java b/src/test/java/org/elasticsearch/search/aggregations/metrics/ExtendedStatsTests.java new file mode 100644 index 00000000000..d45e9ec6e33 --- /dev/null +++ b/src/test/java/org/elasticsearch/search/aggregations/metrics/ExtendedStatsTests.java @@ -0,0 +1,388 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.metrics; + +import org.elasticsearch.action.search.SearchResponse; +import org.elasticsearch.search.aggregations.bucket.histogram.Histogram; +import org.elasticsearch.search.aggregations.metrics.stats.extended.ExtendedStats; +import org.junit.Test; + +import static org.elasticsearch.index.query.QueryBuilders.matchAllQuery; +import static org.elasticsearch.search.aggregations.AggregationBuilders.extendedStats; +import static org.elasticsearch.search.aggregations.AggregationBuilders.histogram; +import static org.hamcrest.Matchers.*; + +/** + * + */ +public class ExtendedStatsTests extends AbstractNumericTests { + + private static double stdDev(int... vals) { + return Math.sqrt(variance(vals)); + } + + private static double variance(int... vals) { + double sum = 0; + double sumOfSqrs = 0; + for (int val : vals) { + sum += val; + sumOfSqrs += val * val; + } + return (sumOfSqrs - ((sum * sum) / vals.length)) / vals.length; + } + + @Test + public void testEmptyAggregation() throws Exception { + + SearchResponse searchResponse = client().prepareSearch("empty_bucket_idx") + .setQuery(matchAllQuery()) + .addAggregation(histogram("histo").field("value").interval(1l).emptyBuckets(true).subAggregation(extendedStats("stats"))) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(2l)); + Histogram histo = searchResponse.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + Histogram.Bucket bucket = histo.getByKey(1l); + assertThat(bucket, notNullValue()); + + ExtendedStats stats = bucket.getAggregations().get("stats"); + assertThat(stats, notNullValue()); + assertThat(stats.getName(), equalTo("stats")); + assertThat(stats.getSumOfSquares(), equalTo(0.0)); + assertThat(stats.getCount(), equalTo(0l)); + assertThat(stats.getSum(), equalTo(0.0)); + assertThat(stats.getMin(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(stats.getMax(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(Double.isNaN(stats.getStdDeviation()), is(true)); + assertThat(Double.isNaN(stats.getAvg()), is(true)); + } + + @Test + public void testUnmapped() throws Exception { + client().admin().cluster().prepareHealth("idx_unmapped").setWaitForGreenStatus().execute().actionGet(); + + SearchResponse searchResponse = client().prepareSearch("idx_unmapped") + .setQuery(matchAllQuery()) + .addAggregation(extendedStats("stats").field("value")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(0l)); + + ExtendedStats stats = searchResponse.getAggregations().get("stats"); + assertThat(stats, notNullValue()); + assertThat(stats.getName(), equalTo("stats")); + assertThat(stats.getAvg(), equalTo(Double.NaN)); + assertThat(stats.getMin(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(stats.getMax(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(stats.getSum(), equalTo(0.0)); + assertThat(stats.getCount(), equalTo(0l)); + assertThat(stats.getSumOfSquares(), equalTo(0.0)); + assertThat(stats.getVariance(), equalTo(Double.NaN)); + assertThat(stats.getStdDeviation(), equalTo(Double.NaN)); + } + + @Test + public void testSingleValuedField() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(extendedStats("stats").field("value")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + ExtendedStats stats = searchResponse.getAggregations().get("stats"); + assertThat(stats, notNullValue()); + assertThat(stats.getName(), equalTo("stats")); + assertThat(stats.getAvg(), equalTo((double) (1+2+3+4+5+6+7+8+9+10) / 10)); + assertThat(stats.getMin(), equalTo(1.0)); + assertThat(stats.getMax(), equalTo(10.0)); + assertThat(stats.getSum(), equalTo((double) 1+2+3+4+5+6+7+8+9+10)); + assertThat(stats.getCount(), equalTo(10l)); + assertThat(stats.getSumOfSquares(), equalTo((double) 1+4+9+16+25+36+49+64+81+100)); + assertThat(stats.getVariance(), equalTo(variance(1, 2, 3, 4, 5, 6, 7, 8 ,9, 10))); + assertThat(stats.getStdDeviation(), equalTo(stdDev(1, 2, 3, 4, 5, 6, 7, 8 ,9, 10))); + } + + @Test + public void testSingleValuedField_PartiallyUnmapped() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx", "idx_unmapped") + .setQuery(matchAllQuery()) + .addAggregation(extendedStats("stats").field("value")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + ExtendedStats stats = searchResponse.getAggregations().get("stats"); + assertThat(stats, notNullValue()); + assertThat(stats.getName(), equalTo("stats")); + assertThat(stats.getAvg(), equalTo((double) (1+2+3+4+5+6+7+8+9+10) / 10)); + assertThat(stats.getMin(), equalTo(1.0)); + assertThat(stats.getMax(), equalTo(10.0)); + assertThat(stats.getSum(), equalTo((double) 1+2+3+4+5+6+7+8+9+10)); + assertThat(stats.getCount(), equalTo(10l)); + assertThat(stats.getSumOfSquares(), equalTo((double) 1+4+9+16+25+36+49+64+81+100)); + assertThat(stats.getVariance(), equalTo(variance(1, 2, 3, 4, 5, 6, 7, 8 ,9, 10))); + assertThat(stats.getStdDeviation(), equalTo(stdDev(1, 2, 3, 4, 5, 6, 7, 8 ,9, 10))); + } + + @Test + public void testSingleValuedField_WithValueScript() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(extendedStats("stats").field("value").script("_value + 1")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + ExtendedStats stats = searchResponse.getAggregations().get("stats"); + assertThat(stats, notNullValue()); + assertThat(stats.getName(), equalTo("stats")); + assertThat(stats.getAvg(), equalTo((double) (2+3+4+5+6+7+8+9+10+11) / 10)); + assertThat(stats.getMin(), equalTo(2.0)); + assertThat(stats.getMax(), equalTo(11.0)); + assertThat(stats.getSum(), equalTo((double) 2+3+4+5+6+7+8+9+10+11)); + assertThat(stats.getCount(), equalTo(10l)); + assertThat(stats.getSumOfSquares(), equalTo((double) 4+9+16+25+36+49+64+81+100+121)); + assertThat(stats.getVariance(), equalTo(variance(2, 3, 4, 5, 6, 7, 8 ,9, 10, 11))); + assertThat(stats.getStdDeviation(), equalTo(stdDev(2, 3, 4, 5, 6, 7, 8 ,9, 10, 11))); + } + + @Test + public void testSingleValuedField_WithValueScript_WithParams() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(extendedStats("stats").field("value").script("_value + inc").param("inc", 1)) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + ExtendedStats stats = searchResponse.getAggregations().get("stats"); + assertThat(stats, notNullValue()); + assertThat(stats.getName(), equalTo("stats")); + assertThat(stats.getAvg(), equalTo((double) (2+3+4+5+6+7+8+9+10+11) / 10)); + assertThat(stats.getMin(), equalTo(2.0)); + assertThat(stats.getMax(), equalTo(11.0)); + assertThat(stats.getSum(), equalTo((double) 2+3+4+5+6+7+8+9+10+11)); + assertThat(stats.getCount(), equalTo(10l)); + assertThat(stats.getSumOfSquares(), equalTo((double) 4+9+16+25+36+49+64+81+100+121)); + assertThat(stats.getVariance(), equalTo(variance(2, 3, 4, 5, 6, 7, 8 ,9, 10, 11))); + assertThat(stats.getStdDeviation(), equalTo(stdDev(2, 3, 4, 5, 6, 7, 8 ,9, 10, 11))); + } + + @Test + public void testMultiValuedField() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(extendedStats("stats").field("values")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + ExtendedStats stats = searchResponse.getAggregations().get("stats"); + assertThat(stats, notNullValue()); + assertThat(stats.getName(), equalTo("stats")); + assertThat(stats.getAvg(), equalTo((double) (2+3+4+5+6+7+8+9+10+11+3+4+5+6+7+8+9+10+11+12) / 20)); + assertThat(stats.getMin(), equalTo(2.0)); + assertThat(stats.getMax(), equalTo(12.0)); + assertThat(stats.getSum(), equalTo((double) 2+3+4+5+6+7+8+9+10+11+3+4+5+6+7+8+9+10+11+12)); + assertThat(stats.getCount(), equalTo(20l)); + assertThat(stats.getSumOfSquares(), equalTo((double) 4+9+16+25+36+49+64+81+100+121+9+16+25+36+49+64+81+100+121+144)); + assertThat(stats.getVariance(), equalTo(variance(2, 3, 4, 5, 6, 7, 8 ,9, 10, 11, 3, 4, 5, 6, 7, 8 ,9, 10, 11, 12))); + assertThat(stats.getStdDeviation(), equalTo(stdDev(2, 3, 4, 5, 6, 7, 8 ,9, 10, 11, 3, 4, 5, 6, 7, 8 ,9, 10, 11, 12))); + } + + @Test + public void testMultiValuedField_WithValueScript() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(extendedStats("stats").field("values").script("_value - 1")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + ExtendedStats stats = searchResponse.getAggregations().get("stats"); + assertThat(stats, notNullValue()); + assertThat(stats.getName(), equalTo("stats")); + assertThat(stats.getAvg(), equalTo((double) (1+2+3+4+5+6+7+8+9+10+2+3+4+5+6+7+8+9+10+11) / 20)); + assertThat(stats.getMin(), equalTo(1.0)); + assertThat(stats.getMax(), equalTo(11.0)); + assertThat(stats.getSum(), equalTo((double) 1+2+3+4+5+6+7+8+9+10+2+3+4+5+6+7+8+9+10+11)); + assertThat(stats.getCount(), equalTo(20l)); + assertThat(stats.getSumOfSquares(), equalTo((double) 1+4+9+16+25+36+49+64+81+100+4+9+16+25+36+49+64+81+100+121)); + assertThat(stats.getVariance(), equalTo(variance(1, 2, 3, 4, 5, 6, 7, 8 ,9, 10, 2, 3, 4, 5, 6, 7, 8 ,9, 10, 11))); + assertThat(stats.getStdDeviation(), equalTo(stdDev(1, 2, 3, 4, 5, 6, 7, 8 ,9, 10, 2, 3, 4, 5, 6, 7, 8 ,9, 10, 11))); + } + + @Test + public void testMultiValuedField_WithValueScript_WithParams() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(extendedStats("stats").field("values").script("_value - dec").param("dec", 1)) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + ExtendedStats stats = searchResponse.getAggregations().get("stats"); + assertThat(stats, notNullValue()); + assertThat(stats.getName(), equalTo("stats")); + assertThat(stats.getAvg(), equalTo((double) (1+2+3+4+5+6+7+8+9+10+2+3+4+5+6+7+8+9+10+11) / 20)); + assertThat(stats.getMin(), equalTo(1.0)); + assertThat(stats.getMax(), equalTo(11.0)); + assertThat(stats.getSum(), equalTo((double) 1+2+3+4+5+6+7+8+9+10+2+3+4+5+6+7+8+9+10+11)); + assertThat(stats.getCount(), equalTo(20l)); + assertThat(stats.getSumOfSquares(), equalTo((double) 1+4+9+16+25+36+49+64+81+100+4+9+16+25+36+49+64+81+100+121)); + assertThat(stats.getVariance(), equalTo(variance(1, 2, 3, 4, 5, 6, 7, 8 ,9, 10, 2, 3, 4, 5, 6, 7, 8 ,9, 10, 11))); + assertThat(stats.getStdDeviation(), equalTo(stdDev(1, 2, 3, 4, 5, 6, 7, 8 ,9, 10, 2, 3, 4, 5, 6, 7, 8 ,9, 10, 11))); + } + + @Test + public void testScript_SingleValued() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(extendedStats("stats").script("doc['value'].value")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + ExtendedStats stats = searchResponse.getAggregations().get("stats"); + assertThat(stats, notNullValue()); + assertThat(stats.getName(), equalTo("stats")); + assertThat(stats.getAvg(), equalTo((double) (1+2+3+4+5+6+7+8+9+10) / 10)); + assertThat(stats.getMin(), equalTo(1.0)); + assertThat(stats.getMax(), equalTo(10.0)); + assertThat(stats.getSum(), equalTo((double) 1+2+3+4+5+6+7+8+9+10)); + assertThat(stats.getCount(), equalTo(10l)); + assertThat(stats.getSumOfSquares(), equalTo((double) 1+4+9+16+25+36+49+64+81+100)); + assertThat(stats.getVariance(), equalTo(variance(1, 2, 3, 4, 5, 6, 7, 8 ,9, 10))); + assertThat(stats.getStdDeviation(), equalTo(stdDev(1, 2, 3, 4, 5, 6, 7, 8 ,9, 10))); + } + + @Test + public void testScript_SingleValued_WithParams() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(extendedStats("stats").script("doc['value'].value + inc").param("inc", 1)) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + ExtendedStats stats = searchResponse.getAggregations().get("stats"); + assertThat(stats, notNullValue()); + assertThat(stats.getName(), equalTo("stats")); + assertThat(stats.getAvg(), equalTo((double) (2+3+4+5+6+7+8+9+10+11) / 10)); + assertThat(stats.getMin(), equalTo(2.0)); + assertThat(stats.getMax(), equalTo(11.0)); + assertThat(stats.getSum(), equalTo((double) 2+3+4+5+6+7+8+9+10+11)); + assertThat(stats.getCount(), equalTo(10l)); + assertThat(stats.getSumOfSquares(), equalTo((double) 4+9+16+25+36+49+64+81+100+121)); + assertThat(stats.getVariance(), equalTo(variance(2, 3, 4, 5, 6, 7, 8 ,9, 10, 11))); + assertThat(stats.getStdDeviation(), equalTo(stdDev(2, 3, 4, 5, 6, 7, 8 ,9, 10, 11))); + } + + @Test + public void testScript_ExplicitSingleValued_WithParams() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(extendedStats("stats").script("doc['value'].value + inc").param("inc", 1)) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + ExtendedStats stats = searchResponse.getAggregations().get("stats"); + assertThat(stats, notNullValue()); + assertThat(stats.getName(), equalTo("stats")); + assertThat(stats.getAvg(), equalTo((double) (2+3+4+5+6+7+8+9+10+11) / 10)); + assertThat(stats.getMin(), equalTo(2.0)); + assertThat(stats.getMax(), equalTo(11.0)); + assertThat(stats.getSum(), equalTo((double) 2+3+4+5+6+7+8+9+10+11)); + assertThat(stats.getCount(), equalTo(10l)); + assertThat(stats.getSumOfSquares(), equalTo((double) 4+9+16+25+36+49+64+81+100+121)); + assertThat(stats.getVariance(), equalTo(variance(2, 3, 4, 5, 6, 7, 8 ,9, 10, 11))); + assertThat(stats.getStdDeviation(), equalTo(stdDev(2, 3, 4, 5, 6, 7, 8 ,9, 10, 11))); + } + + @Test + public void testScript_MultiValued() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(extendedStats("stats").script("doc['values'].values")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + ExtendedStats stats = searchResponse.getAggregations().get("stats"); + assertThat(stats, notNullValue()); + assertThat(stats.getName(), equalTo("stats")); + assertThat(stats.getAvg(), equalTo((double) (2+3+4+5+6+7+8+9+10+11+3+4+5+6+7+8+9+10+11+12) / 20)); + assertThat(stats.getMin(), equalTo(2.0)); + assertThat(stats.getMax(), equalTo(12.0)); + assertThat(stats.getSum(), equalTo((double) 2+3+4+5+6+7+8+9+10+11+3+4+5+6+7+8+9+10+11+12)); + assertThat(stats.getCount(), equalTo(20l)); + assertThat(stats.getSumOfSquares(), equalTo((double) 4+9+16+25+36+49+64+81+100+121+9+16+25+36+49+64+81+100+121+144)); + assertThat(stats.getVariance(), equalTo(variance(2, 3, 4, 5, 6, 7, 8 ,9, 10, 11, 3, 4, 5, 6, 7, 8 ,9, 10, 11, 12))); + assertThat(stats.getStdDeviation(), equalTo(stdDev(2, 3, 4, 5, 6, 7, 8 ,9, 10, 11, 3, 4, 5, 6, 7, 8 ,9, 10, 11, 12))); + } + + @Test + public void testScript_ExplicitMultiValued() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(extendedStats("stats").script("doc['values'].values")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + ExtendedStats stats = searchResponse.getAggregations().get("stats"); + assertThat(stats, notNullValue()); + assertThat(stats.getName(), equalTo("stats")); + assertThat(stats.getAvg(), equalTo((double) (2+3+4+5+6+7+8+9+10+11+3+4+5+6+7+8+9+10+11+12) / 20)); + assertThat(stats.getMin(), equalTo(2.0)); + assertThat(stats.getMax(), equalTo(12.0)); + assertThat(stats.getSum(), equalTo((double) 2+3+4+5+6+7+8+9+10+11+3+4+5+6+7+8+9+10+11+12)); + assertThat(stats.getCount(), equalTo(20l)); + assertThat(stats.getSumOfSquares(), equalTo((double) 4+9+16+25+36+49+64+81+100+121+9+16+25+36+49+64+81+100+121+144)); + assertThat(stats.getVariance(), equalTo(variance(2, 3, 4, 5, 6, 7, 8 ,9, 10, 11, 3, 4, 5, 6, 7, 8 ,9, 10, 11, 12))); + assertThat(stats.getStdDeviation(), equalTo(stdDev(2, 3, 4, 5, 6, 7, 8 ,9, 10, 11, 3, 4, 5, 6, 7, 8 ,9, 10, 11, 12))); + + } + + @Test + public void testScript_MultiValued_WithParams() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(extendedStats("stats").script("new double[] { doc['value'].value, doc['value'].value - dec }").param("dec", 1)) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + ExtendedStats stats = searchResponse.getAggregations().get("stats"); + assertThat(stats, notNullValue()); + assertThat(stats.getName(), equalTo("stats")); + assertThat(stats.getAvg(), equalTo((double) (1+2+3+4+5+6+7+8+9+10+0+1+2+3+4+5+6+7+8+9) / 20)); + assertThat(stats.getMin(), equalTo(0.0)); + assertThat(stats.getMax(), equalTo(10.0)); + assertThat(stats.getSum(), equalTo((double) 1+2+3+4+5+6+7+8+9+10+0+1+2+3+4+5+6+7+8+9)); + assertThat(stats.getCount(), equalTo(20l)); + assertThat(stats.getSumOfSquares(), equalTo((double) 1+4+9+16+25+36+49+64+81+100+0+1+4+9+16+25+36+49+64+81)); + assertThat(stats.getVariance(), equalTo(variance(1, 2, 3, 4, 5, 6, 7, 8 ,9, 10, 0, 1, 2, 3, 4, 5, 6, 7, 8 ,9))); + assertThat(stats.getStdDeviation(), equalTo(stdDev(1, 2, 3, 4, 5, 6, 7, 8 ,9, 10, 0, 1, 2, 3, 4, 5, 6, 7, 8 ,9))); + } + +} \ No newline at end of file diff --git a/src/test/java/org/elasticsearch/search/aggregations/metrics/MaxTests.java b/src/test/java/org/elasticsearch/search/aggregations/metrics/MaxTests.java new file mode 100644 index 00000000000..7867dc52089 --- /dev/null +++ b/src/test/java/org/elasticsearch/search/aggregations/metrics/MaxTests.java @@ -0,0 +1,271 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.metrics; + +import org.elasticsearch.action.search.SearchResponse; +import org.elasticsearch.search.aggregations.bucket.histogram.Histogram; +import org.elasticsearch.search.aggregations.metrics.max.Max; +import org.junit.Test; + +import static org.elasticsearch.index.query.QueryBuilders.matchAllQuery; +import static org.elasticsearch.search.aggregations.AggregationBuilders.histogram; +import static org.elasticsearch.search.aggregations.AggregationBuilders.max; +import static org.hamcrest.Matchers.equalTo; +import static org.hamcrest.Matchers.notNullValue; + +/** + * + */ +public class MaxTests extends AbstractNumericTests { + + @Test + public void testEmptyAggregation() throws Exception { + + SearchResponse searchResponse = client().prepareSearch("empty_bucket_idx") + .setQuery(matchAllQuery()) + .addAggregation(histogram("histo").field("value").interval(1l).emptyBuckets(true).subAggregation(max("max"))) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(2l)); + Histogram histo = searchResponse.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + Histogram.Bucket bucket = histo.getByKey(1l); + assertThat(bucket, notNullValue()); + + Max max = bucket.getAggregations().get("max"); + assertThat(max, notNullValue()); + assertThat(max.getName(), equalTo("max")); + assertThat(max.getValue(), equalTo(Double.NEGATIVE_INFINITY)); + } + @Test + public void testUnmapped() throws Exception { + client().admin().cluster().prepareHealth("idx_unmapped").setWaitForGreenStatus().execute().actionGet(); + + SearchResponse searchResponse = client().prepareSearch("idx_unmapped") + .setQuery(matchAllQuery()) + .addAggregation(max("max").field("value")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(0l)); + + Max max = searchResponse.getAggregations().get("max"); + assertThat(max, notNullValue()); + assertThat(max.getName(), equalTo("max")); + assertThat(max.getValue(), equalTo(Double.NEGATIVE_INFINITY)); + } + + @Test + public void testSingleValuedField() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(max("max").field("value")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Max max = searchResponse.getAggregations().get("max"); + assertThat(max, notNullValue()); + assertThat(max.getName(), equalTo("max")); + assertThat(max.getValue(), equalTo(10.0)); + } + + + @Test + public void testSingleValuedField_PartiallyUnmapped() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx", "idx_unmapped") + .setQuery(matchAllQuery()) + .addAggregation(max("max").field("value")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Max max = searchResponse.getAggregations().get("max"); + assertThat(max, notNullValue()); + assertThat(max.getName(), equalTo("max")); + assertThat(max.getValue(), equalTo(10.0)); + } + + @Test + public void testSingleValuedField_WithValueScript() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(max("max").field("value").script("_value + 1")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Max max = searchResponse.getAggregations().get("max"); + assertThat(max, notNullValue()); + assertThat(max.getName(), equalTo("max")); + assertThat(max.getValue(), equalTo(11.0)); + } + + @Test + public void testSingleValuedField_WithValueScript_WithParams() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(max("max").field("value").script("_value + inc").param("inc", 1)) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Max max = searchResponse.getAggregations().get("max"); + assertThat(max, notNullValue()); + assertThat(max.getName(), equalTo("max")); + assertThat(max.getValue(), equalTo(11.0)); + } + + @Test + public void testMultiValuedField() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(max("max").field("values")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Max max = searchResponse.getAggregations().get("max"); + assertThat(max, notNullValue()); + assertThat(max.getName(), equalTo("max")); + assertThat(max.getValue(), equalTo(12.0)); + } + + @Test + public void testMultiValuedField_WithValueScript() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(max("max").field("values").script("_value + 1")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Max max = searchResponse.getAggregations().get("max"); + assertThat(max, notNullValue()); + assertThat(max.getName(), equalTo("max")); + assertThat(max.getValue(), equalTo(13.0)); + } + + @Test + public void testMultiValuedField_WithValueScript_WithParams() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(max("max").field("values").script("_value + inc").param("inc", 1)) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Max max = searchResponse.getAggregations().get("max"); + assertThat(max, notNullValue()); + assertThat(max.getName(), equalTo("max")); + assertThat(max.getValue(), equalTo(13.0)); + } + + @Test + public void testScript_SingleValued() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(max("max").script("doc['value'].value")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Max max = searchResponse.getAggregations().get("max"); + assertThat(max, notNullValue()); + assertThat(max.getName(), equalTo("max")); + assertThat(max.getValue(), equalTo(10.0)); + } + + @Test + public void testScript_SingleValued_WithParams() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(max("max").script("doc['value'].value + inc").param("inc", 1)) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Max max = searchResponse.getAggregations().get("max"); + assertThat(max, notNullValue()); + assertThat(max.getName(), equalTo("max")); + assertThat(max.getValue(), equalTo(11.0)); + } + + @Test + public void testScript_ExplicitSingleValued_WithParams() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(max("max").script("doc['value'].value + inc").param("inc", 1)) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Max max = searchResponse.getAggregations().get("max"); + assertThat(max, notNullValue()); + assertThat(max.getName(), equalTo("max")); + assertThat(max.getValue(), equalTo(11.0)); + } + + @Test + public void testScript_MultiValued() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(max("max").script("doc['values'].values")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Max max = searchResponse.getAggregations().get("max"); + assertThat(max, notNullValue()); + assertThat(max.getName(), equalTo("max")); + assertThat(max.getValue(), equalTo(12.0)); + } + + @Test + public void testScript_ExplicitMultiValued() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(max("max").script("doc['values'].values")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Max max = searchResponse.getAggregations().get("max"); + assertThat(max, notNullValue()); + assertThat(max.getName(), equalTo("max")); + assertThat(max.getValue(), equalTo(12.0)); + } + + @Test + public void testScript_MultiValued_WithParams() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(max("max").script("new double[] { doc['value'].value, doc['value'].value + inc }").param("inc", 1)) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Max max = searchResponse.getAggregations().get("max"); + assertThat(max, notNullValue()); + assertThat(max.getName(), equalTo("max")); + assertThat(max.getValue(), equalTo(11.0)); + } + + +} \ No newline at end of file diff --git a/src/test/java/org/elasticsearch/search/aggregations/metrics/MinTests.java b/src/test/java/org/elasticsearch/search/aggregations/metrics/MinTests.java new file mode 100644 index 00000000000..ca87ce93853 --- /dev/null +++ b/src/test/java/org/elasticsearch/search/aggregations/metrics/MinTests.java @@ -0,0 +1,286 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.metrics; + +import org.elasticsearch.action.search.SearchResponse; +import org.elasticsearch.search.aggregations.bucket.histogram.Histogram; +import org.elasticsearch.search.aggregations.metrics.min.Min; +import org.junit.Test; + +import static org.elasticsearch.index.query.QueryBuilders.matchAllQuery; +import static org.elasticsearch.search.aggregations.AggregationBuilders.histogram; +import static org.elasticsearch.search.aggregations.AggregationBuilders.min; +import static org.hamcrest.Matchers.equalTo; +import static org.hamcrest.Matchers.notNullValue; + +/** + * + */ +public class MinTests extends AbstractNumericTests { + + @Test + public void testEmptyAggregation() throws Exception { + + SearchResponse searchResponse = client().prepareSearch("empty_bucket_idx") + .setQuery(matchAllQuery()) + .addAggregation(histogram("histo").field("value").interval(1l).emptyBuckets(true).subAggregation(min("min"))) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(2l)); + Histogram histo = searchResponse.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + Histogram.Bucket bucket = histo.getByKey(1l); + assertThat(bucket, notNullValue()); + + Min min = bucket.getAggregations().get("min"); + assertThat(min, notNullValue()); + assertThat(min.getName(), equalTo("min")); + assertThat(min.getValue(), equalTo(Double.POSITIVE_INFINITY)); + } + + @Test + public void testUnmapped() throws Exception { + client().admin().cluster().prepareHealth("idx_unmapped").setWaitForGreenStatus().execute().actionGet(); + + SearchResponse searchResponse = client().prepareSearch("idx_unmapped") + .setQuery(matchAllQuery()) + .addAggregation(min("min").field("value")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(0l)); + + Min min = searchResponse.getAggregations().get("min"); + assertThat(min, notNullValue()); + assertThat(min.getName(), equalTo("min")); + assertThat(min.getValue(), equalTo(Double.POSITIVE_INFINITY)); + } + + @Test + public void testSingleValuedField() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(min("min").field("value")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Min min = searchResponse.getAggregations().get("min"); + assertThat(min, notNullValue()); + assertThat(min.getName(), equalTo("min")); + assertThat(min.getValue(), equalTo(1.0)); + } + + @Test + public void testSingleValuedField_PartiallyUnmapped() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx", "idx_unmapped") + .setQuery(matchAllQuery()) + .addAggregation(min("min").field("value")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Min min = searchResponse.getAggregations().get("min"); + assertThat(min, notNullValue()); + assertThat(min.getName(), equalTo("min")); + assertThat(min.getValue(), equalTo(1.0)); + } + + @Test + public void testSingleValuedField_WithValueScript() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(min("min").field("value").script("_value - 1")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Min min = searchResponse.getAggregations().get("min"); + assertThat(min, notNullValue()); + assertThat(min.getName(), equalTo("min")); + assertThat(min.getValue(), equalTo(0.0)); + } + + @Test + public void testSingleValuedField_WithValueScript_WithParams() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(min("min").field("value").script("_value - dec").param("dec", 1)) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Min min = searchResponse.getAggregations().get("min"); + assertThat(min, notNullValue()); + assertThat(min.getName(), equalTo("min")); + assertThat(min.getValue(), equalTo(0.0)); + } + + @Test + public void testMultiValuedField() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(min("min").field("values")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Min min = searchResponse.getAggregations().get("min"); + assertThat(min, notNullValue()); + assertThat(min.getName(), equalTo("min")); + assertThat(min.getValue(), equalTo(2.0)); + } + + @Test + public void testMultiValuedField_WithValueScript() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(min("min").field("values").script("_value - 1")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Min min = searchResponse.getAggregations().get("min"); + assertThat(min, notNullValue()); + assertThat(min.getName(), equalTo("min")); + assertThat(min.getValue(), equalTo(1.0)); + } + + @Test + public void testMultiValuedField_WithValueScript_Reverse() throws Exception { + // test what happens when values arrive in reverse order since the min aggregator is optimized to work on sorted values + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(min("min").field("values").script("_value * -1")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Min min = searchResponse.getAggregations().get("min"); + assertThat(min, notNullValue()); + assertThat(min.getName(), equalTo("min")); + assertThat(min.getValue(), equalTo(-12d)); + } + + @Test + public void testMultiValuedField_WithValueScript_WithParams() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(min("min").field("values").script("_value - dec").param("dec", 1)) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Min min = searchResponse.getAggregations().get("min"); + assertThat(min, notNullValue()); + assertThat(min.getName(), equalTo("min")); + assertThat(min.getValue(), equalTo(1.0)); + } + + @Test + public void testScript_SingleValued() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(min("min").script("doc['value'].value")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Min min = searchResponse.getAggregations().get("min"); + assertThat(min, notNullValue()); + assertThat(min.getName(), equalTo("min")); + assertThat(min.getValue(), equalTo(1.0)); + } + + @Test + public void testScript_SingleValued_WithParams() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(min("min").script("doc['value'].value - dec").param("dec", 1)) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Min min = searchResponse.getAggregations().get("min"); + assertThat(min, notNullValue()); + assertThat(min.getName(), equalTo("min")); + assertThat(min.getValue(), equalTo(0.0)); + } + + @Test + public void testScript_ExplicitSingleValued_WithParams() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(min("min").script("doc['value'].value - dec").param("dec", 1)) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Min min = searchResponse.getAggregations().get("min"); + assertThat(min, notNullValue()); + assertThat(min.getName(), equalTo("min")); + assertThat(min.getValue(), equalTo(0.0)); + } + + @Test + public void testScript_MultiValued() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(min("min").script("doc['values'].values")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Min min = searchResponse.getAggregations().get("min"); + assertThat(min, notNullValue()); + assertThat(min.getName(), equalTo("min")); + assertThat(min.getValue(), equalTo(2.0)); + } + + @Test + public void testScript_ExplicitMultiValued() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(min("min").script("doc['values'].values")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Min min = searchResponse.getAggregations().get("min"); + assertThat(min, notNullValue()); + assertThat(min.getName(), equalTo("min")); + assertThat(min.getValue(), equalTo(2.0)); + } + + @Test + public void testScript_MultiValued_WithParams() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(min("min").script("List values = doc['values'].values; double[] res = new double[values.length]; for (int i = 0; i < res.length; i++) { res[i] = values.get(i) - dec; }; return res;").param("dec", 1)) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Min min = searchResponse.getAggregations().get("min"); + assertThat(min, notNullValue()); + assertThat(min.getName(), equalTo("min")); + assertThat(min.getValue(), equalTo(1.0)); + } + +} \ No newline at end of file diff --git a/src/test/java/org/elasticsearch/search/aggregations/metrics/StatsTests.java b/src/test/java/org/elasticsearch/search/aggregations/metrics/StatsTests.java new file mode 100644 index 00000000000..b9af3c2da46 --- /dev/null +++ b/src/test/java/org/elasticsearch/search/aggregations/metrics/StatsTests.java @@ -0,0 +1,329 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.metrics; + +import org.elasticsearch.action.search.SearchResponse; +import org.elasticsearch.search.aggregations.bucket.histogram.Histogram; +import org.elasticsearch.search.aggregations.metrics.stats.Stats; +import org.junit.Test; + +import static org.elasticsearch.index.query.QueryBuilders.matchAllQuery; +import static org.elasticsearch.search.aggregations.AggregationBuilders.histogram; +import static org.elasticsearch.search.aggregations.AggregationBuilders.stats; +import static org.hamcrest.Matchers.*; + +/** + * + */ +public class StatsTests extends AbstractNumericTests { + + @Test + public void testEmptyAggregation() throws Exception { + + SearchResponse searchResponse = client().prepareSearch("empty_bucket_idx") + .setQuery(matchAllQuery()) + .addAggregation(histogram("histo").field("value").interval(1l).emptyBuckets(true).subAggregation(stats("stats"))) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(2l)); + Histogram histo = searchResponse.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + Histogram.Bucket bucket = histo.getByKey(1l); + assertThat(bucket, notNullValue()); + + Stats stats = bucket.getAggregations().get("stats"); + assertThat(stats, notNullValue()); + assertThat(stats.getName(), equalTo("stats")); + assertThat(stats.getCount(), equalTo(0l)); + assertThat(stats.getSum(), equalTo(0.0)); + assertThat(stats.getMin(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(stats.getMax(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(Double.isNaN(stats.getAvg()), is(true)); + } + + @Test + public void testUnmapped() throws Exception { + client().admin().cluster().prepareHealth("idx_unmapped").setWaitForGreenStatus().execute().actionGet(); + + SearchResponse searchResponse = client().prepareSearch("idx_unmapped") + .setQuery(matchAllQuery()) + .addAggregation(stats("stats").field("value")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(0l)); + + Stats stats = searchResponse.getAggregations().get("stats"); + assertThat(stats, notNullValue()); + assertThat(stats.getName(), equalTo("stats")); + assertThat(stats.getAvg(), equalTo(Double.NaN)); + assertThat(stats.getMin(), equalTo(Double.POSITIVE_INFINITY)); + assertThat(stats.getMax(), equalTo(Double.NEGATIVE_INFINITY)); + assertThat(stats.getSum(), equalTo(0.0)); + assertThat(stats.getCount(), equalTo(0l)); + } + + @Test + public void testSingleValuedField() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(stats("stats").field("value")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Stats stats = searchResponse.getAggregations().get("stats"); + assertThat(stats, notNullValue()); + assertThat(stats.getName(), equalTo("stats")); + assertThat(stats.getAvg(), equalTo((double) (1+2+3+4+5+6+7+8+9+10) / 10)); + assertThat(stats.getMin(), equalTo(1.0)); + assertThat(stats.getMax(), equalTo(10.0)); + assertThat(stats.getSum(), equalTo((double) 1+2+3+4+5+6+7+8+9+10)); + assertThat(stats.getCount(), equalTo(10l)); + } + + @Test + public void testSingleValuedField_PartiallyUnmapped() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx", "idx_unmapped") + .setQuery(matchAllQuery()) + .addAggregation(stats("stats").field("value")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Stats stats = searchResponse.getAggregations().get("stats"); + assertThat(stats, notNullValue()); + assertThat(stats.getName(), equalTo("stats")); + assertThat(stats.getAvg(), equalTo((double) (1+2+3+4+5+6+7+8+9+10) / 10)); + assertThat(stats.getMin(), equalTo(1.0)); + assertThat(stats.getMax(), equalTo(10.0)); + assertThat(stats.getSum(), equalTo((double) 1+2+3+4+5+6+7+8+9+10)); + assertThat(stats.getCount(), equalTo(10l)); + } + + @Test + public void testSingleValuedField_WithValueScript() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(stats("stats").field("value").script("_value + 1")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Stats stats = searchResponse.getAggregations().get("stats"); + assertThat(stats, notNullValue()); + assertThat(stats.getName(), equalTo("stats")); + assertThat(stats.getAvg(), equalTo((double) (2+3+4+5+6+7+8+9+10+11) / 10)); + assertThat(stats.getMin(), equalTo(2.0)); + assertThat(stats.getMax(), equalTo(11.0)); + assertThat(stats.getSum(), equalTo((double) 2+3+4+5+6+7+8+9+10+11)); + assertThat(stats.getCount(), equalTo(10l)); + } + + @Test + public void testSingleValuedField_WithValueScript_WithParams() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(stats("stats").field("value").script("_value + inc").param("inc", 1)) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Stats stats = searchResponse.getAggregations().get("stats"); + assertThat(stats, notNullValue()); + assertThat(stats.getName(), equalTo("stats")); + assertThat(stats.getAvg(), equalTo((double) (2+3+4+5+6+7+8+9+10+11) / 10)); + assertThat(stats.getMin(), equalTo(2.0)); + assertThat(stats.getMax(), equalTo(11.0)); + assertThat(stats.getSum(), equalTo((double) 2+3+4+5+6+7+8+9+10+11)); + assertThat(stats.getCount(), equalTo(10l)); + } + + @Test + public void testMultiValuedField() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(stats("stats").field("values")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Stats stats = searchResponse.getAggregations().get("stats"); + assertThat(stats, notNullValue()); + assertThat(stats.getName(), equalTo("stats")); + assertThat(stats.getAvg(), equalTo((double) (2+3+4+5+6+7+8+9+10+11+3+4+5+6+7+8+9+10+11+12) / 20)); + assertThat(stats.getMin(), equalTo(2.0)); + assertThat(stats.getMax(), equalTo(12.0)); + assertThat(stats.getSum(), equalTo((double) 2+3+4+5+6+7+8+9+10+11+3+4+5+6+7+8+9+10+11+12)); + assertThat(stats.getCount(), equalTo(20l)); + } + + @Test + public void testMultiValuedField_WithValueScript() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(stats("stats").field("values").script("_value - 1")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Stats stats = searchResponse.getAggregations().get("stats"); + assertThat(stats, notNullValue()); + assertThat(stats.getName(), equalTo("stats")); + assertThat(stats.getAvg(), equalTo((double) (1+2+3+4+5+6+7+8+9+10+2+3+4+5+6+7+8+9+10+11) / 20)); + assertThat(stats.getMin(), equalTo(1.0)); + assertThat(stats.getMax(), equalTo(11.0)); + assertThat(stats.getSum(), equalTo((double) 1+2+3+4+5+6+7+8+9+10+2+3+4+5+6+7+8+9+10+11)); + assertThat(stats.getCount(), equalTo(20l)); + } + + @Test + public void testMultiValuedField_WithValueScript_WithParams() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(stats("stats").field("values").script("_value - dec").param("dec", 1)) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Stats stats = searchResponse.getAggregations().get("stats"); + assertThat(stats, notNullValue()); + assertThat(stats.getName(), equalTo("stats")); + assertThat(stats.getAvg(), equalTo((double) (1+2+3+4+5+6+7+8+9+10+2+3+4+5+6+7+8+9+10+11) / 20)); + assertThat(stats.getMin(), equalTo(1.0)); + assertThat(stats.getMax(), equalTo(11.0)); + assertThat(stats.getSum(), equalTo((double) 1+2+3+4+5+6+7+8+9+10+2+3+4+5+6+7+8+9+10+11)); + assertThat(stats.getCount(), equalTo(20l)); + } + + @Test + public void testScript_SingleValued() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(stats("stats").script("doc['value'].value")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Stats stats = searchResponse.getAggregations().get("stats"); + assertThat(stats, notNullValue()); + assertThat(stats.getName(), equalTo("stats")); + assertThat(stats.getAvg(), equalTo((double) (1+2+3+4+5+6+7+8+9+10) / 10)); + assertThat(stats.getMin(), equalTo(1.0)); + assertThat(stats.getMax(), equalTo(10.0)); + assertThat(stats.getSum(), equalTo((double) 1+2+3+4+5+6+7+8+9+10)); + assertThat(stats.getCount(), equalTo(10l)); + } + + @Test + public void testScript_SingleValued_WithParams() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(stats("stats").script("doc['value'].value + inc").param("inc", 1)) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Stats stats = searchResponse.getAggregations().get("stats"); + assertThat(stats, notNullValue()); + assertThat(stats.getName(), equalTo("stats")); + assertThat(stats.getAvg(), equalTo((double) (2+3+4+5+6+7+8+9+10+11) / 10)); + assertThat(stats.getMin(), equalTo(2.0)); + assertThat(stats.getMax(), equalTo(11.0)); + assertThat(stats.getSum(), equalTo((double) 2+3+4+5+6+7+8+9+10+11)); + assertThat(stats.getCount(), equalTo(10l)); + } + + @Test + public void testScript_ExplicitSingleValued_WithParams() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(stats("stats").script("doc['value'].value + inc").param("inc", 1)) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Stats stats = searchResponse.getAggregations().get("stats"); + assertThat(stats, notNullValue()); + assertThat(stats.getName(), equalTo("stats")); + assertThat(stats.getAvg(), equalTo((double) (2+3+4+5+6+7+8+9+10+11) / 10)); + assertThat(stats.getMin(), equalTo(2.0)); + assertThat(stats.getMax(), equalTo(11.0)); + assertThat(stats.getSum(), equalTo((double) 2+3+4+5+6+7+8+9+10+11)); + assertThat(stats.getCount(), equalTo(10l)); + } + + @Test + public void testScript_MultiValued() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(stats("stats").script("doc['values'].values")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Stats stats = searchResponse.getAggregations().get("stats"); + assertThat(stats, notNullValue()); + assertThat(stats.getName(), equalTo("stats")); + assertThat(stats.getAvg(), equalTo((double) (2+3+4+5+6+7+8+9+10+11+3+4+5+6+7+8+9+10+11+12) / 20)); + assertThat(stats.getMin(), equalTo(2.0)); + assertThat(stats.getMax(), equalTo(12.0)); + assertThat(stats.getSum(), equalTo((double) 2+3+4+5+6+7+8+9+10+11+3+4+5+6+7+8+9+10+11+12)); + assertThat(stats.getCount(), equalTo(20l)); + } + + @Test + public void testScript_ExplicitMultiValued() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(stats("stats").script("doc['values'].values")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Stats stats = searchResponse.getAggregations().get("stats"); + assertThat(stats, notNullValue()); + assertThat(stats.getName(), equalTo("stats")); + assertThat(stats.getAvg(), equalTo((double) (2+3+4+5+6+7+8+9+10+11+3+4+5+6+7+8+9+10+11+12) / 20)); + assertThat(stats.getMin(), equalTo(2.0)); + assertThat(stats.getMax(), equalTo(12.0)); + assertThat(stats.getSum(), equalTo((double) 2+3+4+5+6+7+8+9+10+11+3+4+5+6+7+8+9+10+11+12)); + assertThat(stats.getCount(), equalTo(20l)); + } + + @Test + public void testScript_MultiValued_WithParams() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(stats("stats").script("new double[] { doc['value'].value, doc['value'].value - dec }").param("dec", 1)) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Stats stats = searchResponse.getAggregations().get("stats"); + assertThat(stats, notNullValue()); + assertThat(stats.getName(), equalTo("stats")); + assertThat(stats.getAvg(), equalTo((double) (1+2+3+4+5+6+7+8+9+10+0+1+2+3+4+5+6+7+8+9) / 20)); + assertThat(stats.getMin(), equalTo(0.0)); + assertThat(stats.getMax(), equalTo(10.0)); + assertThat(stats.getSum(), equalTo((double) 1+2+3+4+5+6+7+8+9+10+0+1+2+3+4+5+6+7+8+9)); + assertThat(stats.getCount(), equalTo(20l)); + } + +} \ No newline at end of file diff --git a/src/test/java/org/elasticsearch/search/aggregations/metrics/SumTests.java b/src/test/java/org/elasticsearch/search/aggregations/metrics/SumTests.java new file mode 100644 index 00000000000..f0a1216d217 --- /dev/null +++ b/src/test/java/org/elasticsearch/search/aggregations/metrics/SumTests.java @@ -0,0 +1,273 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.metrics; + +import org.elasticsearch.action.search.SearchResponse; +import org.elasticsearch.search.aggregations.bucket.histogram.Histogram; +import org.elasticsearch.search.aggregations.metrics.sum.Sum; +import org.junit.Test; + +import static org.elasticsearch.index.query.QueryBuilders.matchAllQuery; +import static org.elasticsearch.search.aggregations.AggregationBuilders.histogram; +import static org.elasticsearch.search.aggregations.AggregationBuilders.sum; +import static org.hamcrest.Matchers.equalTo; +import static org.hamcrest.Matchers.notNullValue; + +/** + * + */ +public class SumTests extends AbstractNumericTests { + + @Test + public void testEmptyAggregation() throws Exception { + + SearchResponse searchResponse = client().prepareSearch("empty_bucket_idx") + .setQuery(matchAllQuery()) + .addAggregation(histogram("histo").field("value").interval(1l).emptyBuckets(true).subAggregation(sum("sum"))) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(2l)); + Histogram histo = searchResponse.getAggregations().get("histo"); + assertThat(histo, notNullValue()); + Histogram.Bucket bucket = histo.getByKey(1l); + assertThat(bucket, notNullValue()); + + Sum sum = bucket.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + assertThat(sum.getName(), equalTo("sum")); + assertThat(sum.getValue(), equalTo(0.0)); + } + + @Test + public void testUnmapped() throws Exception { + client().admin().cluster().prepareHealth("idx_unmapped").setWaitForGreenStatus().execute().actionGet(); + + SearchResponse searchResponse = client().prepareSearch("idx_unmapped") + .setQuery(matchAllQuery()) + .addAggregation(sum("sum").field("value")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(0l)); + + Sum sum = searchResponse.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + assertThat(sum.getName(), equalTo("sum")); + assertThat(sum.getValue(), equalTo(0.0)); + } + + @Test + public void testSingleValuedField() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(sum("sum").field("value")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Sum sum = searchResponse.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + assertThat(sum.getName(), equalTo("sum")); + assertThat(sum.getValue(), equalTo((double) 1+2+3+4+5+6+7+8+9+10)); + } + + @Override + public void testSingleValuedField_PartiallyUnmapped() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx", "idx_unmapped") + .setQuery(matchAllQuery()) + .addAggregation(sum("sum").field("value")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Sum sum = searchResponse.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + assertThat(sum.getName(), equalTo("sum")); + assertThat(sum.getValue(), equalTo((double) 1+2+3+4+5+6+7+8+9+10)); + } + + @Test + public void testSingleValuedField_WithValueScript() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(sum("sum").field("value").script("_value + 1")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Sum sum = searchResponse.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + assertThat(sum.getName(), equalTo("sum")); + assertThat(sum.getValue(), equalTo((double) 2+3+4+5+6+7+8+9+10+11)); + } + + @Test + public void testSingleValuedField_WithValueScript_WithParams() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(sum("sum").field("value").script("_value + increment").param("increment", 1)) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Sum sum = searchResponse.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + assertThat(sum.getName(), equalTo("sum")); + assertThat(sum.getValue(), equalTo((double) 2+3+4+5+6+7+8+9+10+11)); + } + + @Test + public void testScript_SingleValued() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(sum("sum").script("doc['value'].value")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Sum sum = searchResponse.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + assertThat(sum.getName(), equalTo("sum")); + assertThat(sum.getValue(), equalTo((double) 1+2+3+4+5+6+7+8+9+10)); + } + + @Test + public void testScript_SingleValued_WithParams() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(sum("sum").script("doc['value'].value + inc").param("inc", 1)) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Sum sum = searchResponse.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + assertThat(sum.getName(), equalTo("sum")); + assertThat(sum.getValue(), equalTo((double) 2+3+4+5+6+7+8+9+10+11)); + } + + @Test + public void testScript_ExplicitSingleValued_WithParams() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(sum("sum").script("doc['value'].value + inc").param("inc", 1)) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Sum sum = searchResponse.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + assertThat(sum.getName(), equalTo("sum")); + assertThat(sum.getValue(), equalTo((double) 2+3+4+5+6+7+8+9+10+11)); + } + + + @Test + public void testScript_MultiValued() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(sum("sum").script("new double[] { doc['value'].value, doc['value'].value + 1 }")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Sum sum = searchResponse.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + assertThat(sum.getName(), equalTo("sum")); + assertThat(sum.getValue(), equalTo((double) 1+2+2+3+3+4+4+5+5+6+6+7+7+8+8+9+9+10+10+11)); + } + + @Test + public void testScript_ExplicitMultiValued() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(sum("sum").script("new double[] { doc['value'].value, doc['value'].value + 1 }")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Sum sum = searchResponse.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + assertThat(sum.getName(), equalTo("sum")); + assertThat(sum.getValue(), equalTo((double) 1+2+2+3+3+4+4+5+5+6+6+7+7+8+8+9+9+10+10+11)); + } + + @Test + public void testScript_MultiValued_WithParams() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(sum("sum").script("new double[] { doc['value'].value, doc['value'].value + inc }").param("inc", 1)) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Sum sum = searchResponse.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + assertThat(sum.getName(), equalTo("sum")); + assertThat(sum.getValue(), equalTo((double) 1+2+2+3+3+4+4+5+5+6+6+7+7+8+8+9+9+10+10+11)); + } + + @Test + public void testMultiValuedField() throws Exception { + + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(sum("sum").field("values")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Sum sum = searchResponse.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + assertThat(sum.getName(), equalTo("sum")); + assertThat(sum.getValue(), equalTo((double) 2+3+3+4+4+5+5+6+6+7+7+8+8+9+9+10+10+11+11+12)); + } + + @Test + public void testMultiValuedField_WithValueScript() throws Exception { + + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(sum("sum").field("values").script("_value + 1")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Sum sum = searchResponse.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + assertThat(sum.getName(), equalTo("sum")); + assertThat(sum.getValue(), equalTo((double) 3+4+4+5+5+6+6+7+7+8+8+9+9+10+10+11+11+12+12+13)); + } + + @Test + public void testMultiValuedField_WithValueScript_WithParams() throws Exception { + + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(sum("sum").field("values").script("_value + increment").param("increment", 1)) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + Sum sum = searchResponse.getAggregations().get("sum"); + assertThat(sum, notNullValue()); + assertThat(sum.getName(), equalTo("sum")); + assertThat(sum.getValue(), equalTo((double) 3+4+4+5+5+6+6+7+7+8+8+9+9+10+10+11+11+12+12+13)); + } +} \ No newline at end of file diff --git a/src/test/java/org/elasticsearch/search/aggregations/metrics/ValueCountTests.java b/src/test/java/org/elasticsearch/search/aggregations/metrics/ValueCountTests.java new file mode 100644 index 00000000000..460b28f517f --- /dev/null +++ b/src/test/java/org/elasticsearch/search/aggregations/metrics/ValueCountTests.java @@ -0,0 +1,127 @@ +/* + * Licensed to ElasticSearch and Shay Banon under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. ElasticSearch licenses this + * file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.search.aggregations.metrics; + +import org.elasticsearch.action.search.SearchResponse; +import org.elasticsearch.common.settings.ImmutableSettings; +import org.elasticsearch.common.settings.Settings; +import org.elasticsearch.search.aggregations.metrics.valuecount.ValueCount; +import org.elasticsearch.test.ElasticsearchIntegrationTest; +import org.junit.Before; +import org.junit.Test; + +import static org.elasticsearch.common.xcontent.XContentFactory.jsonBuilder; +import static org.elasticsearch.index.query.QueryBuilders.matchAllQuery; +import static org.elasticsearch.search.aggregations.AggregationBuilders.count; +import static org.hamcrest.Matchers.equalTo; +import static org.hamcrest.Matchers.notNullValue; + +/** + * + */ +public class ValueCountTests extends ElasticsearchIntegrationTest { + + @Override + public Settings indexSettings() { + return ImmutableSettings.builder() + .put("index.number_of_shards", between(1, 5)) + .put("index.number_of_replicas", between(0, 1)) + .build(); + } + + @Before + public void init() throws Exception { + createIndex("idx"); + createIndex("idx_unmapped"); + for (int i = 0; i < 10; i++) { + client().prepareIndex("idx", "type", ""+i).setSource(jsonBuilder() + .startObject() + .field("value", i+1) + .startArray("values").value(i+2).value(i+3).endArray() + .endObject()) + .execute().actionGet(); + } + client().admin().indices().prepareFlush().execute().actionGet(); + client().admin().indices().prepareRefresh().execute().actionGet(); + } + + @Test + public void unmapped() throws Exception { + + SearchResponse searchResponse = client().prepareSearch("idx_unmapped") + .setQuery(matchAllQuery()) + .addAggregation(count("count").field("value")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(0l)); + + ValueCount valueCount = searchResponse.getAggregations().get("count"); + assertThat(valueCount, notNullValue()); + assertThat(valueCount.getName(), equalTo("count")); + assertThat(valueCount.getValue(), equalTo(0l)); + } + + @Test + public void singleValuedField() throws Exception { + + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(count("count").field("value")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + ValueCount valueCount = searchResponse.getAggregations().get("count"); + assertThat(valueCount, notNullValue()); + assertThat(valueCount.getName(), equalTo("count")); + assertThat(valueCount.getValue(), equalTo(10l)); + } + + @Test + public void singleValuedField_PartiallyUnmapped() throws Exception { + SearchResponse searchResponse = client().prepareSearch("idx", "idx_unmapped") + .setQuery(matchAllQuery()) + .addAggregation(count("count").field("value")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + ValueCount valueCount = searchResponse.getAggregations().get("count"); + assertThat(valueCount, notNullValue()); + assertThat(valueCount.getName(), equalTo("count")); + assertThat(valueCount.getValue(), equalTo(10l)); + } + + @Test + public void multiValuedField() throws Exception { + + SearchResponse searchResponse = client().prepareSearch("idx") + .setQuery(matchAllQuery()) + .addAggregation(count("count").field("values")) + .execute().actionGet(); + + assertThat(searchResponse.getHits().getTotalHits(), equalTo(10l)); + + ValueCount valueCount = searchResponse.getAggregations().get("count"); + assertThat(valueCount, notNullValue()); + assertThat(valueCount.getName(), equalTo("count")); + assertThat(valueCount.getValue(), equalTo(20l)); + } +} \ No newline at end of file diff --git a/src/test/java/org/elasticsearch/test/ElasticsearchIntegrationTest.java b/src/test/java/org/elasticsearch/test/ElasticsearchIntegrationTest.java index a58b07394d5..4b5c408fd07 100644 --- a/src/test/java/org/elasticsearch/test/ElasticsearchIntegrationTest.java +++ b/src/test/java/org/elasticsearch/test/ElasticsearchIntegrationTest.java @@ -594,6 +594,11 @@ public abstract class ElasticsearchIntegrationTest extends ElasticsearchTestCase return client().admin(); } + /** Convenience method that forwards to {@link #indexRandom(boolean, List)}. */ + public void indexRandom(boolean forceRefresh, IndexRequestBuilder... builders) throws InterruptedException, ExecutionException { + indexRandom(forceRefresh, Arrays.asList(builders)); + } + /** * Indexes the given {@link IndexRequestBuilder} instances randomly. It shuffles the given builders and either * indexes they in a blocking or async fashion. This is very useful to catch problems that relate to internal document @@ -601,26 +606,25 @@ public abstract class ElasticsearchIntegrationTest extends ElasticsearchTestCase * segment or if only one document is in a segment etc. This method prevents issues like this by randomizing the index * layout. */ - public void indexRandom(boolean forceRefresh, IndexRequestBuilder... builders) throws InterruptedException, ExecutionException { - if (builders.length == 0) { + public void indexRandom(boolean forceRefresh, List builders) throws InterruptedException, ExecutionException { + if (builders.size() == 0) { return; } Random random = getRandom(); Set indicesSet = new HashSet(); - for (int i = 0; i < builders.length; i++) { - indicesSet.add(builders[i].request().index()); + for (IndexRequestBuilder builder : builders) { + indicesSet.add(builder.request().index()); } final String[] indices = indicesSet.toArray(new String[0]); - List list = Arrays.asList(builders); - Collections.shuffle(list, random); + Collections.shuffle(builders, random); final CopyOnWriteArrayList> errors = new CopyOnWriteArrayList>(); List latches = new ArrayList(); if (frequently()) { - logger.info("Index [{}] docs async: [{}] bulk: [{}]", list.size(), true, false); - final CountDownLatch latch = new CountDownLatch(list.size()); + logger.info("Index [{}] docs async: [{}] bulk: [{}]", builders.size(), true, false); + final CountDownLatch latch = new CountDownLatch(builders.size()); latches.add(latch); - for (IndexRequestBuilder indexRequestBuilder : list) { + for (IndexRequestBuilder indexRequestBuilder : builders) { indexRequestBuilder.execute(new PayloadLatchedActionListener(indexRequestBuilder, latch, errors)); if (rarely()) { if (rarely()) { @@ -634,8 +638,8 @@ public abstract class ElasticsearchIntegrationTest extends ElasticsearchTestCase } } else if (randomBoolean()) { - logger.info("Index [{}] docs async: [{}] bulk: [{}]", list.size(), false, false); - for (IndexRequestBuilder indexRequestBuilder : list) { + logger.info("Index [{}] docs async: [{}] bulk: [{}]", builders.size(), false, false); + for (IndexRequestBuilder indexRequestBuilder : builders) { indexRequestBuilder.execute().actionGet(); if (rarely()) { if (rarely()) { @@ -648,9 +652,9 @@ public abstract class ElasticsearchIntegrationTest extends ElasticsearchTestCase } } } else { - logger.info("Index [{}] docs async: [{}] bulk: [{}]", list.size(), false, true); + logger.info("Index [{}] docs async: [{}] bulk: [{}]", builders.size(), false, true); BulkRequestBuilder bulkBuilder = client().prepareBulk(); - for (IndexRequestBuilder indexRequestBuilder : list) { + for (IndexRequestBuilder indexRequestBuilder : builders) { bulkBuilder.add(indexRequestBuilder); } BulkResponse actionGet = bulkBuilder.execute().actionGet();