mirror of https://github.com/apache/druid.git
214 lines
19 KiB
Markdown
214 lines
19 KiB
Markdown
---
|
|
id: math-expr
|
|
title: "Expressions"
|
|
---
|
|
|
|
<!--
|
|
~ Licensed to the Apache Software Foundation (ASF) under one
|
|
~ or more contributor license agreements. See the NOTICE file
|
|
~ distributed with this work for additional information
|
|
~ regarding copyright ownership. The ASF licenses this file
|
|
~ to you under the Apache License, Version 2.0 (the
|
|
~ "License"); you may not use this file except in compliance
|
|
~ with the License. You may obtain a copy of the License at
|
|
~
|
|
~ http://www.apache.org/licenses/LICENSE-2.0
|
|
~
|
|
~ Unless required by applicable law or agreed to in writing,
|
|
~ software distributed under the License is distributed on an
|
|
~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
~ KIND, either express or implied. See the License for the
|
|
~ specific language governing permissions and limitations
|
|
~ under the License.
|
|
-->
|
|
|
|
> Apache Druid supports two query languages: [native queries](../querying/querying.md) and [Druid SQL](../querying/sql.md).
|
|
> This document describes the native language. For information about functions available in SQL, refer to the
|
|
> [SQL documentation](../querying/sql.md#scalar-functions).
|
|
|
|
Expressions are used in various places in the native query language, including
|
|
[virtual columns](../querying/virtual-columns.md) and [join conditions](../querying/datasource.md#join). They are
|
|
also generated by most [Druid SQL functions](../querying/sql.md#scalar-functions) during the
|
|
[query translation](../querying/sql.md#query-translation) process.
|
|
|
|
This expression language supports the following operators (listed in decreasing order of precedence).
|
|
|
|
|Operators|Description|
|
|
|---------|-----------|
|
|
|!, -|Unary NOT and Minus|
|
|
|^|Binary power op|
|
|
|*, /, %|Binary multiplicative|
|
|
|+, -|Binary additive|
|
|
|<, <=, >, >=, ==, !=|Binary Comparison|
|
|
|&&, |||Binary Logical AND, OR|
|
|
|
|
Long, double, and string data types are supported. If a number contains a dot, it is interpreted as a double, otherwise it is interpreted as a long. That means, always add a '.' to your number if you want it interpreted as a double value. String literals should be quoted by single quotation marks.
|
|
|
|
Additionally, the expression language supports long, double, and string arrays. Array literals are created by wrapping square brackets around a list of scalar literals values delimited by a comma or space character. All values in an array literal must be the same type, however null values are accepted. Typed empty arrays may be defined by prefixing with their type in angle brackets: `<STRING>[]`, `<DOUBLE>[]`, or `<LONG>[]`.
|
|
|
|
Expressions can contain variables. Variable names may contain letters, digits, '\_' and '$'. Variable names must not begin with a digit. To escape other special characters, you can quote it with double quotation marks.
|
|
|
|
For logical operators, a number is true if and only if it is positive (0 or negative value means false). For string type, it's the evaluation result of 'Boolean.valueOf(string)'.
|
|
|
|
[Multi-value string dimensions](../querying/multi-value-dimensions.html) are supported and may be treated as either scalar or array typed values. When treated as a scalar type, an expression will automatically be transformed to apply the scalar operation across all values of the multi-valued type, to mimic Druid's native behavior. Values that result in arrays will be coerced back into the native Druid string type for aggregation. Druid aggregations on multi-value string dimensions on the individual values, _not_ the 'array', behaving similar to the `UNNEST` operator available in many SQL dialects. However, by using the `array_to_string` function, aggregations may be done on a stringified version of the complete array, allowing the complete row to be preserved. Using `string_to_array` in an expression post-aggregator, allows transforming the stringified dimension back into the true native array type.
|
|
|
|
|
|
The following built-in functions are available.
|
|
|
|
## General functions
|
|
|
|
|name|description|
|
|
|----|-----------|
|
|
|cast|cast(expr,'LONG' or 'DOUBLE' or 'STRING' or 'LONG_ARRAY', or 'DOUBLE_ARRAY' or 'STRING_ARRAY') returns expr with specified type. exception can be thrown. Scalar types may be cast to array types and will take the form of a single element list (null will still be null). |
|
|
|if|if(predicate,then,else) returns 'then' if 'predicate' evaluates to a positive number, otherwise it returns 'else' |
|
|
|nvl|nvl(expr,expr-for-null) returns 'expr-for-null' if 'expr' is null (or empty string for string type) |
|
|
|like|like(expr, pattern[, escape]) is equivalent to SQL `expr LIKE pattern`|
|
|
|case_searched|case_searched(expr1, result1, \[\[expr2, result2, ...\], else-result\])|
|
|
|case_simple|case_simple(expr, value1, result1, \[\[value2, result2, ...\], else-result\])|
|
|
|bloom_filter_test|bloom_filter_test(expr, filter) tests the value of 'expr' against 'filter', a bloom filter serialized as a base64 string. See [bloom filter extension](../development/extensions-core/bloom-filter.md) documentation for additional details.|
|
|
|
|
## String functions
|
|
|
|
|name|description|
|
|
|----|-----------|
|
|
|concat|concat(expr, expr...) concatenate a list of strings|
|
|
|format|format(pattern[, args...]) returns a string formatted in the manner of Java's [String.format](https://docs.oracle.com/javase/8/docs/api/java/lang/String.html#format-java.lang.String-java.lang.Object...-).|
|
|
|like|like(expr, pattern[, escape]) is equivalent to SQL `expr LIKE pattern`|
|
|
|lookup|lookup(expr, lookup-name) looks up expr in a registered [query-time lookup](../querying/lookups.md)|
|
|
|parse_long|parse_long(string[, radix]) parses a string as a long with the given radix, or 10 (decimal) if a radix is not provided.|
|
|
|regexp_extract|regexp_extract(expr, pattern[, index]) applies a regular expression pattern and extracts a capture group index, or null if there is no match. If index is unspecified or zero, returns the substring that matched the pattern.|
|
|
|replace|replace(expr, pattern, replacement) replaces pattern with replacement|
|
|
|substring|substring(expr, index, length) behaves like java.lang.String's substring|
|
|
|right|right(expr, length) returns the rightmost length characters from a string|
|
|
|left|left(expr, length) returns the leftmost length characters from a string|
|
|
|strlen|strlen(expr) returns length of a string in UTF-16 code units|
|
|
|strpos|strpos(haystack, needle[, fromIndex]) returns the position of the needle within the haystack, with indexes starting from 0. The search will begin at fromIndex, or 0 if fromIndex is not specified. If the needle is not found then the function returns -1.|
|
|
|trim|trim(expr[, chars]) remove leading and trailing characters from `expr` if they are present in `chars`. `chars` defaults to ' ' (space) if not provided.|
|
|
|ltrim|ltrim(expr[, chars]) remove leading characters from `expr` if they are present in `chars`. `chars` defaults to ' ' (space) if not provided.|
|
|
|rtrim|rtrim(expr[, chars]) remove trailing characters from `expr` if they are present in `chars`. `chars` defaults to ' ' (space) if not provided.|
|
|
|lower|lower(expr) converts a string to lowercase|
|
|
|upper|upper(expr) converts a string to uppercase|
|
|
|reverse|reverse(expr) reverses a string|
|
|
|repeat|repeat(expr, N) repeats a string N times|
|
|
|lpad|lpad(expr, length, chars) returns a string of `length` from `expr` left-padded with `chars`. If `length` is shorter than the length of `expr`, the result is `expr` which is truncated to `length`. If either `expr` or `chars` are null, the result will be null.|
|
|
|rpad|rpad(expr, length, chars) returns a string of `length` from `expr` right-padded with `chars`. If `length` is shorter than the length of `expr`, the result is `expr` which is truncated to `length`. If either `expr` or `chars` are null, the result will be null.|
|
|
|
|
## Time functions
|
|
|
|
|name|description|
|
|
|----|-----------|
|
|
|timestamp|timestamp(expr[,format-string]) parses string expr into date then returns milliseconds from java epoch. without 'format-string' it's regarded as ISO datetime format |
|
|
|unix_timestamp|same with 'timestamp' function but returns seconds instead |
|
|
|timestamp_ceil|timestamp_ceil(expr, period, \[origin, \[timezone\]\]) rounds up a timestamp, returning it as a new timestamp. Period can be any ISO8601 period, like P3M (quarters) or PT12H (half-days). The time zone, if provided, should be a time zone name like "America/Los_Angeles" or offset like "-08:00".|
|
|
|timestamp_floor|timestamp_floor(expr, period, \[origin, [timezone\]\]) rounds down a timestamp, returning it as a new timestamp. Period can be any ISO8601 period, like P3M (quarters) or PT12H (half-days). The time zone, if provided, should be a time zone name like "America/Los_Angeles" or offset like "-08:00".|
|
|
|timestamp_shift|timestamp_shift(expr, period, step, \[timezone\]) shifts a timestamp by a period (step times), returning it as a new timestamp. Period can be any ISO8601 period. Step may be negative. The time zone, if provided, should be a time zone name like "America/Los_Angeles" or offset like "-08:00".|
|
|
|timestamp_extract|timestamp_extract(expr, unit, \[timezone\]) extracts a time part from expr, returning it as a number. Unit can be EPOCH (number of seconds since 1970-01-01 00:00:00 UTC), SECOND, MINUTE, HOUR, DAY (day of month), DOW (day of week), DOY (day of year), WEEK (week of [week year](https://en.wikipedia.org/wiki/ISO_week_date)), MONTH (1 through 12), QUARTER (1 through 4), or YEAR. The time zone, if provided, should be a time zone name like "America/Los_Angeles" or offset like "-08:00"|
|
|
|timestamp_parse|timestamp_parse(string expr, \[pattern, [timezone\]\]) parses a string into a timestamp using a given [Joda DateTimeFormat pattern](http://www.joda.org/joda-time/apidocs/org/joda/time/format/DateTimeFormat.html). If the pattern is not provided, this parses time strings in either ISO8601 or SQL format. The time zone, if provided, should be a time zone name like "America/Los_Angeles" or offset like "-08:00", and will be used as the time zone for strings that do not include a time zone offset. Pattern and time zone must be literals. Strings that cannot be parsed as timestamps will be returned as nulls.|
|
|
|timestamp_format|timestamp_format(expr, \[pattern, \[timezone\]\]) formats a timestamp as a string with a given [Joda DateTimeFormat pattern](http://www.joda.org/joda-time/apidocs/org/joda/time/format/DateTimeFormat.html), or ISO8601 if the pattern is not provided. The time zone, if provided, should be a time zone name like "America/Los_Angeles" or offset like "-08:00". Pattern and time zone must be literals.|
|
|
|
|
## Math functions
|
|
|
|
See javadoc of java.lang.Math for detailed explanation for each function.
|
|
|
|
|name|description|
|
|
|----|-----------|
|
|
|abs|abs(x) would return the absolute value of x|
|
|
|acos|acos(x) would return the arc cosine of x|
|
|
|asin|asin(x) would return the arc sine of x|
|
|
|atan|atan(x) would return the arc tangent of x|
|
|
|atan2|atan2(y, x) would return the angle theta from the conversion of rectangular coordinates (x, y) to polar * coordinates (r, theta)|
|
|
|cbrt|cbrt(x) would return the cube root of x|
|
|
|ceil|ceil(x) would return the smallest (closest to negative infinity) double value that is greater than or equal to x and is equal to a mathematical integer|
|
|
|copysign|copysign(x) would return the first floating-point argument with the sign of the second floating-point argument|
|
|
|cos|cos(x) would return the trigonometric cosine of x|
|
|
|cosh|cosh(x) would return the hyperbolic cosine of x|
|
|
|cot|cot(x) would return the trigonometric cotangent of an angle x|
|
|
|div|div(x,y) is integer division of x by y|
|
|
|exp|exp(x) would return Euler's number raised to the power of x|
|
|
|expm1|expm1(x) would return e^x-1|
|
|
|floor|floor(x) would return the largest (closest to positive infinity) double value that is less than or equal to x and is equal to a mathematical integer|
|
|
|getExponent|getExponent(x) would return the unbiased exponent used in the representation of x|
|
|
|hypot|hypot(x, y) would return sqrt(x^2+y^2) without intermediate overflow or underflow|
|
|
|log|log(x) would return the natural logarithm of x|
|
|
|log10|log10(x) would return the base 10 logarithm of x|
|
|
|log1p|log1p(x) would the natural logarithm of x + 1|
|
|
|max|max(x, y) would return the greater of two values|
|
|
|min|min(x, y) would return the smaller of two values|
|
|
|nextafter|nextafter(x, y) would return the floating-point number adjacent to the x in the direction of the y|
|
|
|nextUp|nextUp(x) would return the floating-point value adjacent to x in the direction of positive infinity|
|
|
|pi|pi would return the constant value of the π |
|
|
|pow|pow(x, y) would return the value of the x raised to the power of y|
|
|
|remainder|remainder(x, y) would return the remainder operation on two arguments as prescribed by the IEEE 754 standard|
|
|
|rint|rint(x) would return value that is closest in value to x and is equal to a mathematical integer|
|
|
|round|round(x, y) would return the value of the x rounded to the y decimal places. While x can be an integer or floating-point number, y must be an integer. The type of the return value is specified by that of x. y defaults to 0 if omitted. When y is negative, x is rounded on the left side of the y decimal points.|
|
|
|scalb|scalb(d, sf) would return d * 2^sf rounded as if performed by a single correctly rounded floating-point multiply to a member of the double value set|
|
|
|signum|signum(x) would return the signum function of the argument x|
|
|
|sin|sin(x) would return the trigonometric sine of an angle x|
|
|
|sinh|sinh(x) would return the hyperbolic sine of x|
|
|
|sqrt|sqrt(x) would return the correctly rounded positive square root of x|
|
|
|tan|tan(x) would return the trigonometric tangent of an angle x|
|
|
|tanh|tanh(x) would return the hyperbolic tangent of x|
|
|
|todegrees|todegrees(x) converts an angle measured in radians to an approximately equivalent angle measured in degrees|
|
|
|toradians|toradians(x) converts an angle measured in degrees to an approximately equivalent angle measured in radians|
|
|
|ulp|ulp(x) would return the size of an ulp of the argument x|
|
|
|
|
|
|
## Array functions
|
|
|
|
| function | description |
|
|
| --- | --- |
|
|
| array(expr1,expr ...) | constructs an array from the expression arguments, using the type of the first argument as the output array type |
|
|
| array_length(arr) | returns length of array expression |
|
|
| array_offset(arr,long) | returns the array element at the 0 based index supplied, or null for an out of range index|
|
|
| array_ordinal(arr,long) | returns the array element at the 1 based index supplied, or null for an out of range index |
|
|
| array_contains(arr,expr) | returns 1 if the array contains the element specified by expr, or contains all elements specified by expr if expr is an array, else 0 |
|
|
| array_overlap(arr1,arr2) | returns 1 if arr1 and arr2 have any elements in common, else 0 |
|
|
| array_offset_of(arr,expr) | returns the 0 based index of the first occurrence of expr in the array, or `-1` or `null` if `druid.generic.useDefaultValueForNull=false`if no matching elements exist in the array. |
|
|
| array_ordinal_of(arr,expr) | returns the 1 based index of the first occurrence of expr in the array, or `-1` or `null` if `druid.generic.useDefaultValueForNull=false` if no matching elements exist in the array. |
|
|
| array_prepend(expr,arr) | adds expr to arr at the beginning, the resulting array type determined by the type of the array |
|
|
| array_append(arr1,expr) | appends expr to arr, the resulting array type determined by the type of the first array |
|
|
| array_concat(arr1,arr2) | concatenates 2 arrays, the resulting array type determined by the type of the first array |
|
|
| array_slice(arr,start,end) | return the subarray of arr from the 0 based index start(inclusive) to end(exclusive), or `null`, if start is less than 0, greater than length of arr or less than end|
|
|
| array_to_string(arr,str) | joins all elements of arr by the delimiter specified by str |
|
|
| string_to_array(str1,str2) | splits str1 into an array on the delimiter specified by str2 |
|
|
|
|
|
|
## Apply functions
|
|
|
|
| function | description |
|
|
| --- | --- |
|
|
| map(lambda,arr) | applies a transform specified by a single argument lambda expression to all elements of arr, returning a new array |
|
|
| cartesian_map(lambda,arr1,arr2,...) | applies a transform specified by a multi argument lambda expression to all elements of the Cartesian product of all input arrays, returning a new array; the number of lambda arguments and array inputs must be the same |
|
|
| filter(lambda,arr) | filters arr by a single argument lambda, returning a new array with all matching elements, or null if no elements match |
|
|
| fold(lambda,arr) | folds a 2 argument lambda across arr. The first argument of the lambda is the array element and the second the accumulator, returning a single accumulated value. |
|
|
| cartesian_fold(lambda,arr1,arr2,...) | folds a multi argument lambda across the Cartesian product of all input arrays. The first arguments of the lambda is the array element and the last is the accumulator, returning a single accumulated value. |
|
|
| any(lambda,arr) | returns 1 if any element in the array matches the lambda expression, else 0 |
|
|
| all(lambda,arr) | returns 1 if all elements in the array matches the lambda expression, else 0 |
|
|
|
|
|
|
### Reduction functions
|
|
|
|
Reduction functions operate on zero or more expressions and return a single expression. If no expressions are passed as
|
|
arguments, then the result is `NULL`. The expressions must all be convertible to a common data type, which will be the
|
|
type of the result:
|
|
* If all arguments are `NULL`, the result is `NULL`. Otherwise, `NULL` arguments are ignored.
|
|
* If the arguments comprise a mix of numbers and strings, the arguments are interpreted as strings.
|
|
* If all arguments are integer numbers, the arguments are interpreted as longs.
|
|
* If all arguments are numbers and at least one argument is a double, the arguments are interpreted as doubles.
|
|
|
|
| function | description |
|
|
| --- | --- |
|
|
| greatest([expr1, ...]) | Evaluates zero or more expressions and returns the maximum value based on comparisons as described above. |
|
|
| least([expr1, ...]) | Evaluates zero or more expressions and returns the minimum value based on comparisons as described above. |
|
|
|
|
|
|
## IP address functions
|
|
|
|
For the IPv4 address functions, the `address` argument can either be an IPv4 dotted-decimal string (e.g., "192.168.0.1") or an IP address represented as a long (e.g., 3232235521). The `subnet` argument should be a string formatted as an IPv4 address subnet in CIDR notation (e.g., "192.168.0.0/16").
|
|
|
|
| function | description |
|
|
| --- | --- |
|
|
| ipv4_match(address, subnet) | Returns 1 if the `address` belongs to the `subnet` literal, else 0. If `address` is not a valid IPv4 address, then 0 is returned. This function is more efficient if `address` is a long instead of a string.|
|
|
| ipv4_parse(address) | Parses `address` into an IPv4 address stored as a long. If `address` is a long that is a valid IPv4 address, then it is passed through. Returns null if `address` cannot be represented as an IPv4 address. |
|
|
| ipv4_stringify(address) | Converts `address` into an IPv4 address dotted-decimal string. If `address` is a string that is a valid IPv4 address, then it is passed through. Returns null if `address` cannot be represented as an IPv4 address.|
|