Commit Graph

5452 Commits

Author SHA1 Message Date
Luca Cavanna 3e692c3f3d
HLRC: migration get assistance API (#32744)
The request and response classes have been extracted from `IndexUpgradeInfoAction` into top-level classes, and moved to the protocol jar. The `UpgradeActionRequired` enum is also moved.

Relates to #29827
2018-08-13 11:05:27 +02:00
Nik Everett f5ba801c6b Test: Only sniff host metadata for node_selectors (#32750)
Our rest testing framework has support for sniffing the host metadata on
startup and, before this change, it'd sniff that metadata before running
the first test. This prevents running these tests against
elasticsearch installations that won't support sniffing like Elastic
Cloud. This change allows tests to only sniff for metadata when they
encounter a test with a `node_selector`. These selectors are the things
that need the metadata anyway and they are super rare. Tests that use
these won't be able to run against installations that don't support
sniffing but we can just skip them. In the case of Elastic Cloud, these
tests were never going to work against Elastic Cloud anyway.
2018-08-10 13:35:47 -04:00
Jonathan Little 8a2b1a7dca Update scripted metric docs to use `state` variable (#32695)
This change brings the scripted metric agg context docs in line with the new agg state context variable.
2018-08-10 10:21:54 -07:00
Luca Cavanna 40b0a3a014
[DOCS] Adjust high-level client x-pack docs (#32747)
- Expose x-pack usage docs page which was not linked in supported-apis page
- make watcher a top-level dir outside of x-pack directory
- move x-pack info and usage pages to miscellaneous
- add new Watcher category to supported-apis (they were under miscellaneous)
- remove x-pack prefix from watcher docs titles
2018-08-10 14:12:42 +02:00
Luca Cavanna 9561a0a66e
[DOCS] Add missing link to get license API docs (#32746) 2018-08-10 10:55:25 +02:00
Jack Conradson 293c8a2b24
Painless: Add an Ingest Script Processor Example (#32302)
This commit adds two pieces. The first is a small set of documentation providing 
instructions on how to get setup to run context examples. This will require a download 
similar to how Kibana works for some of the examples. The second is an ingest processor 
example using the downloaded data. More examples will follow as ideally one per PR. 
This also adds a set of tests to individually test each script as a unit test.
2018-08-09 14:24:55 -07:00
Armin Braun 79375d35bb
Scripting: Replace Update Context (#32096)
* SCRIPTING: Move Update Scripts to their own context
* Added system property for backwards compatibility of change to `ctx.params`
2018-08-09 14:32:36 +02:00
Andy Bristol 8bfb0f3f8d
serialize suggestion responses as named writeables (#30284)
Suggestion responses were previously serialized as streamables which
made writing suggesters in plugins with custom suggestion response types
impossible. This commit makes them serialized as named writeables and
provides a facility for registering a reader for suggestion responses
when registering a suggester.

This also makes Suggestion responses abstract, requiring a suggester
implementation to provide its own types. Suggesters which do not need
anything additional to what is defined in Suggest.Suggestion should
provide a minimal subclass.

The existing plugin suggester integration tests are removed and
replaced with an equivalent implementation as an example
plugin.
2018-08-07 13:31:00 -07:00
David Roberts 733f84c49b [DOCS] Add missing docs include 2018-08-07 17:49:48 +01:00
David Roberts 2608012422
Add temporary directory cleanup workarounds (#32615)
On some Linux distributions tmpfiles.d cleans files and
directories under /tmp if they haven't been accessed for
10 days.

This can cause problems for ML as ML is currently the only
component that uses the temp directory more than a few
seconds after startup. If you didn't open an ML job for
10 days and then tried to open one then the temp directory
would have been deleted.

This commit prevents the problem occurring in the case of
Elasticsearch being managed by systemd, as systemd private
temp directories are not subject to periodic cleanup (by
default).

Additionally there are now some docs to warn people about
the risk and suggest a manual mitigation for .tar.gz users.
2018-08-07 16:59:56 +01:00
simonzheng ab81078949 [Docs] Correct a small typo (#32655) 2018-08-07 10:34:55 +02:00
DeDe Morton e01e4393a8
[Docs] Light edit to info about docker images (#32376) 2018-08-06 12:00:07 -07:00
Igor Motov e641fccfe3
Rest HL client: Add get license action (#32438)
Rest HL client: Add get license action

Continues to use String instead of a more complex License class to
hold the license text similarly to put license.

Relates #29827
2018-08-06 07:15:40 -07:00
Armin Braun 0a67cb4133
LOGGING: Upgrade to Log4J 2.11.1 (#32616)
* LOGGING: Upgrade to Log4J 2.11.1
* Upgrade to `2.11.1` to fix memory leaks in slow logger when logging large requests
   * This was caused by a bug in Log4J https://issues.apache.org/jira/browse/LOG4J2-2269 and is fixed in `2.11.1` via https://git-wip-us.apache.org/repos/asf?p=logging-log4j2.git;h=9496c0c
* Fixes #32537
* Fixes #27300
2018-08-06 14:56:21 +02:00
Jake Landis 3d4c84f7ca
ingest: doc: move Dot Expander Processor doc to correct position (#31743)
No changes to the content.
2018-08-03 07:21:05 -07:00
Shaunak Kashyap 0a83968650
Add cluster UUID to Cluster Stats API response (#32206)
* Make cluster stats response contain cluster UUID

* Updating constructor usage in Monitoring tests

* Adding cluster_uuid field to Cluster Stats API reference doc

* Adding rest api spec test for expecting cluster_uuid in cluster stats response

* Adding missing newline

* Indenting do section properly

* Missed a spot!

* Fixing the test cluster ID
2018-08-02 17:14:19 -07:00
Matthias Sieber a39f6f09f4 fixed elements in array of produced terms (#32519) 2018-08-02 11:12:15 -04:00
Mayya Sharipova 43430fe9b8
Correct minor typo in explain.asciidoc for HLRC 2018-08-01 17:58:14 -04:00
Ryan Ernst 478f6d6cf1
Scripting: Conditionally use java time api in scripting (#31441)
This commit adds a boolean system property, `es.scripting.use_java_time`,
which controls the concrete return type used by doc values within
scripts. The return type of accessing doc values for a date field is
changed to Object, essentially duck typing the type to allow
co-existence during the transition from joda time to java time.
2018-08-01 08:58:49 -07:00
Albert Zaharovits 2d87287c0d
[DOCS] Reloadable Secure Settings (#31713)
Docs on reloadable secure settings for plugins #29135 .
2018-08-01 12:07:23 +03:00
Nik Everett 22459576d7
Logging: Make node name consistent in logger (#31588)
First, some background: we have 15 different methods to get a logger in
Elasticsearch but they can be broken down into three broad categories
based on what information is provided when building the logger.

Just a class like:
```
private static final Logger logger = ESLoggerFactory.getLogger(ActionModule.class);
```
or:
```
protected final Logger logger = Loggers.getLogger(getClass());
```

The class and settings:
```
this.logger = Loggers.getLogger(getClass(), settings);
```

Or more information like:
```
Loggers.getLogger("index.store.deletes", settings, shardId)
```

The goal of the "class and settings" variant is to attach the node name
to the logger. Because we don't always have the settings available, we
often use the "just a class" variant and get loggers without node names
attached. There isn't any real consistency here. Some loggers get the
node name because it is convenient and some do not.

This change makes the node name available to all loggers all the time.
Almost. There are some caveats are testing that I'll get to. But in
*production* code the node name is node available to all loggers. This
means we can stop using the "class and settings" variants to fetch
loggers which was the real goal here, but a pleasant side effect is that
the ndoe name is now consitent on every log line and optional by editing
the logging pattern. This is all powered by setting the node name
statically on a logging formatter very early in initialization.

Now to tests: tests can't set the node name statically because
subclasses of `ESIntegTestCase` run many nodes in the same jvm, even in
the same class loader. Also, lots of tests don't run with a real node so
they don't *have* a node name at all. To support multiple nodes in the
same JVM tests suss out the node name from the thread name which works
surprisingly well and easy to test in a nice way. For those threads
that are not part of an `ESIntegTestCase` node we stick whatever useful
information we can get form the thread name in the place of the node
name. This allows us to keep the logger format consistent.
2018-07-31 10:54:24 -04:00
Colm O'Shea 97b379e0d4 fix no=>not typo (#32463)
Found a tiny typo while reading the docs
2018-07-31 13:33:23 +01:00
Yannick Welsch 11f6983d69
HLRC: Add delete watch action (#32337)
Adds the "delete watch" API to the High-Level Rest Client.

Relates #29827
2018-07-31 10:29:22 +02:00
Tal Levy 1e0fcebfe1
update rollover to leverage write-alias semantics (#32216)
Rollover should not swap aliases when `is_write_index` is set to `true`.
Instead, both the new and old indices should have the rollover alias,
with the newly created index as the new write index

Updates Rollover to leverage the ability to preserve aliases and swap which is the write index.

Historically, Rollover would swap which index had the designated alias for writing documents against. This required users to keep a separate read-alias that enabled reading against both rolled over and newly created indices, whiles the write-alias was being re-assigned at every rollover.

With the ability for aliases to designate a write index, Rollover can be a bit more flexible with its use of aliases.

Updates include:

- Rollover validates that the target alias has a write index (the index that is being rolled over). This means that the restriction that aliases only point to one index is no longer necessary.
- Rollover explicitly (and atomically) swaps which index is the write-index by explicitly assigning the existing index to have `is_write_index: false` and have the newly created index have its rollover alias as `is_write_index: true`. This is only done when `is_write_index: true` on the write index. Default behavior of removing the alias from the rolled over index stays when `is_write_index` is not explicitly set

Relevant things that are staying the same:

- Rollover is rejected if there exist any templates that match the newly-created index and configure the rollover-alias
   - I think this existed to prevent the situation where an alias pointed to two indices for a short while. Although this can technically be relaxed, the specific cases that are safe are really particular and difficult to reason, so leaving the broad restriction sounds good
2018-07-30 14:32:55 -07:00
w-bonelli 072c0be8af Update Fuzzy Query docs to clarify default behavior re max_expansions (#30819)
Stating that the Fuzzy Query generates "all possible" matching terms is misleading, given that the query's default behavior is to generate a maximum of 50 matching terms.

(cherry picked from commit 345a0071a2a41fd7f80ae9ef8a39a2cb4991aedd)
2018-07-30 13:19:26 -07:00
Jason Tedor 588db621ac
Remove reference to non-existent store type (#32418)
We removed the default_fs store type yet the docs still contain a
reference to them. This commit addresses that by removing this
reference, and changing a reference to this section of the docs to
instead refer to mmapfs.
2018-07-27 11:24:03 -04:00
Jason Tedor 3ac57f0ba3
Fix ordering of bootstrap checks in docs (#32417)
In the section of the bootstrap checks docs for the maximum map count
check, we refer to max size virtual memory check and explicitly call out
the maximum size virtual memory check as being the previous
point. However, this is not correct as the previous point is currently
the max file size check. It does make sense for these two checks to be
proximate to each other in the docs so this commit reorders the checks
so that the maximum size virtual memory check indeed comes before the
maximum map count check. This makes the sense in the maximum map count
check correct.
2018-07-27 10:40:16 -04:00
Jim Ferenczi 53ff06e621
Upgrade to Lucene-7.5.0-snapshot-608f0277b0 (#32390)
The main highlight is the removal of the reclaim_deletes_weight in the TieredMergePolicy.
The es setting index.merge.policy.reclaim_deletes_weight is deprecated in this commit and the value is ignored. The new merge policy setting setDeletesPctAllowed should be added in a follow up.
2018-07-27 08:28:51 +02:00
Sandeep Kanabar 7ad16ffd84 Docs: Correcting a typo in tophits (#32359) 2018-07-26 13:30:01 -04:00
Igor Motov 3b050194f9
Docs: Fix callouts in put license HL REST docs (#32363) 2018-07-26 09:00:35 -07:00
mzbeck 63a0436764 Update update-settings.asciidoc (#31378) 2018-07-26 17:16:53 +02:00
Jack Conradson 10bfedeb53
Painless: Fix documentation links to use existing refs (#32335)
Uses ref and xpack-ref instead of custom ones in the Painless docs for
long-term maintainability.
2018-07-25 10:59:25 -07:00
Lisa Cawley 8067250fbe
[DOCS] Adds recommendation for xpack.security.enabled (#32345) 2018-07-25 09:51:40 -07:00
debadair 32eb4d41b6
[DOCS] Removed extraneous callout number. 2018-07-24 11:51:52 -07:00
Igor Motov 29c802f88e
Rest HL client: Add put license action (#32214)
In the HL REST client we replace the License object with a string, because of 
complexity of this class. It is also not really needed on the client side since 
end-users are not interacting with the license besides passing it as a string 
to the server.

Relates #29827
2018-07-24 11:19:07 -07:00
Christoph Büscher c1cc0cef61
Add ERR to ranking evaluation documentation (#32314)
This change adds a section about the Expected Reciprocal Rank metric (ERR) to
the Ranking Evaluation documentation.
2018-07-24 19:58:34 +02:00
Daniel Mitterdorfer 73a38895fd
Add Restore Snapshot High Level REST API
With this commit we add the restore snapshot API to the Java high level
REST client.

Relates #27205
Relates #32155
2018-07-24 16:17:09 +02:00
Russ Cam e2b665c2e6
Consistent encoder names (#29492)
This commit updates encoder names to be consistent within documentation
and align with snake casing convention.
2018-07-24 09:21:43 +10:00
Zachary Tong 6ba144ae31
Add WeightedAvg metric aggregation (#31037)
Adds a new single-value metrics aggregation that computes the weighted 
average of numeric values that are extracted from the aggregated 
documents. These values can be extracted from specific numeric
fields in the documents.

When calculating a regular average, each datapoint has an equal "weight"; it
contributes equally to the final value.  In contrast, weighted averages
scale each datapoint differently.  The amount that each datapoint contributes 
to the final value is extracted from the document, or provided by a script.

As a formula, a weighted average is the `∑(value * weight) / ∑(weight)`

A regular average can be thought of as a weighted average where every value has
an implicit weight of `1`.

Closes #15731
2018-07-23 18:33:15 -04:00
Christoph Büscher fe6bb75eb4
Rename ranking evaluation `quality_level` to `metric_score` (#32168)
The notion of "quality" is an overloaded term in the search ranking evaluation 
context. Its usually used to decribe certain levels of "good" vs. "bad" of a 
seach result with respect to the users information need. We currently report the 
result of the ranking evaluation as `quality_level` which is a bit missleading.
This changes the response parameter name to `metric_score` which fits better.
2018-07-23 22:25:02 +02:00
debadair 5525cb1615
[DOCS] Clarified that you must remove X-Pack plugin when upgrading from pre-6.3. (#32016) 2018-07-20 14:17:48 -07:00
Armin Braun 7aa8a0a927
INGEST: Extend KV Processor (#31789) (#32232)
* INGEST: Extend KV Processor (#31789)

Added more capabilities supported by LS to the KV processor:
* Stripping of brackets and quotes from values (`include_brackets` in corresponding LS filter)
* Adding key prefixes
* Trimming specified chars from keys and values

Refactored the way the filter is configured to avoid conditionals during execution.
Refactored Tests a little to not have to add more redundant getters for new parameters.

Relates #31786
* Add documentation
2018-07-20 22:32:50 +02:00
Nick Peihl ac63408655
Add region ISO code to GeoIP Ingest plugin (#31669) 2018-07-20 11:23:29 -07:00
Daniel Mitterdorfer 6afb661ca5
Remove indices stats timeout from monitoring docs
With this commit we remove the documentation for the setting
`xpack.monitoring.collection.indices.stats.timeout` which has already
been removed in code.

Closes #32133
Relates #32229
2018-07-20 12:55:20 +02:00
Christoph Büscher 5cbd9ad177
Rename ranking evaluation response section (#32166)
Currently the ranking evaluation response contains a 'unknown_docs' section 
for each search use case in the evaluation set. It contains document ids for 
results in the search hits that currently don't have a quality rating.
This change renames it to `unrated_docs`, which better reflects its purpose.
2018-07-20 11:43:46 +02:00
Luca Cavanna 00a6ad0e9e
Remove aliases resolution limitations when security is enabled (#31952)
Resolving wildcards in aliases expression is challenging as we may end
up with no aliases to replace the original expression with, but if we
replace with an empty array that means _all which is quite the opposite.
Now that we support and serialize the original requested aliases,
whenever aliases are replaced we will be able to know what was
initially requested. `MetaData#findAliases` can then be updated to not
return anything in case it gets empty aliases, but the original aliases
were not empty. That means that empty aliases are interpreted as _all
only if they were originally requested that way.

Relates to #31516
2018-07-20 09:23:32 +02:00
Julie Tibshirani 0f0068b91c
Ensure that field aliases cannot be used in multi-fields. (#32219) 2018-07-20 00:18:54 -07:00
Mayya Sharipova 4c68dfe001
Handle missing values in painless (#32207)
Throw an exception for doc['field'].value
if this document is missing a value for the field.

After deprecation changes have been backported to 6.x,
make this a default behaviour in 7.0

Closes #29286
2018-07-19 17:41:06 -04:00
Vladimir Dolzhenko 7c0fc209bf
ECS Task IAM profile credentials ignored in repository-s3 plugin (#31864)
ECS Task IAM profile credentials ignored in repository-s3 plugin (#31864)

Closes #26913
2018-07-19 12:54:38 +02:00
David Turner f6d7854f76
Remove indication of future multi-homing support (#32187)
We do not support intra-cluster connections on multiple interfaces, but the
documentation indicates that we will in future. In fact there is currently no
plan to support this, so the forward-looking documentation is misleading. This
commit

- removes the misleading sentence
- fixes that a transport profile affects outbound connections, not inbound ones
- tidies up some nearby text
2018-07-19 11:33:46 +01:00
Alexander Reelsen 202894b832
Rest HL client: Add put watch action (#32026)
Relates #29827

This implementation behaves like the current transport client, that you basically cannot configure a Watch POJO representation as an argument to the put watch API, but only a bytes reference. You can use the the `WatchSourceBuilder` from the `org.elasticsearch.plugin:x-pack-core` dependency to build watches.

This commit also changes the license type to trial, so that watcher is available in high level rest client tests.

/cc @hub-cap
2018-07-19 10:40:54 +02:00
Julie Tibshirani 15ff3da653
Add support for field aliases. (#32172)
* Add basic support for field aliases in index mappings. (#31287)
* Allow for aliases when fetching stored fields. (#31411)
* Add tests around accessing field aliases in scripts. (#31417)
* Add documentation around field aliases. (#31538)
* Add validation for field alias mappings. (#31518)
* Return both concrete fields and aliases in DocumentFieldMappers#getMapper. (#31671)
* Make sure that field-level security is enforced when using field aliases. (#31807)
* Add more comprehensive tests for field aliases in queries + aggregations. (#31565)
* Remove the deprecated method DocumentFieldMappers#getFieldMapper. (#32148)
2018-07-18 09:33:09 -07:00
David Turner 380b45b965
Improve docs for search preferences (#32159)
Today it is unclear what guarantees are offered by the search preference
feature, and we claim a guarantee that is stronger than what we really offer:

> A custom value will be used to guarantee that the same shards will be used
> for the same custom value.

This commit clarifies this documentation.

Forward-port of #32098 to `master`.
2018-07-18 12:58:17 +01:00
Martijn van Groningen 1924f5d07c
Add more contexts to painless execute api (#30511)
This change adds two contexts the execute scripts against:

* SEARCH_SCRIPT: Allows to run scripts in a search script context.
This context is used in `function_score` query's script function,
script fields, script sorting and `terms_set` query.

* FILTER_SCRIPT: Allows to run scripts in a filter script context.
This context is used in the `script` query.

In both contexts a index name needs to be specified and a sample document.
The document is needed to create an in-memory index that the script can
access via the `doc[...]` and other notations. The index name is needed
because a mapping is needed to index the document.

Examples:

```
POST /_scripts/painless/_execute
{
  "script": {
    "source": "doc['field'].value.length()"
  },
  "context" : {
    "search_script": {
      "document": {
        "field": "four"
      },
      "index": "my-index"
    }
  }
}
```

Returns:

```
{
  "result": 4
}
```

POST /_scripts/painless/_execute
{
  "script": {
    "source": "doc['field'].value.length() <= params.max_length",
    "params": {
      "max_length": 4
    }
  },
  "context" : {
    "filter_script": {
      "document": {
        "field": "four"
      },
      "index": "my-index"
    }
  }
}

Returns:

```
{
  "result": true
}
```

Also changed PainlessExecuteAction.TransportAction to use TransportSingleShardAction
instead of HandledAction, because now in case score or filter contexts are used
the request needs to be redirected to a node that has an active IndexService
for the index being referenced (a node with a shard copy for that index).
2018-07-18 12:42:07 +02:00
Christoph Büscher 36165265ce
Fix put mappings java API documentation (#31955)
The current docs of the put-mapping Java API is currently broken. It its current
form, it creates an index and uses the whole mapping definition given as a JSON
string as the type name. Since we didn't check the index created in the
IndicesDocumentationIT so far this went unnoticed.

This change adds test to catch this error to the documentation test, changes the
documentation so it works correctly now and adds an input validation to
PutMappingRequest#buildFromSimplifiedDef() which was used internally to reject
calls where no mapping definition is given.

Closes #31906
2018-07-17 09:09:03 +02:00
Christoph Büscher 61486680a2
Add exclusion option to `keep_types` token filter (#32012)
Currently the `keep_types` token filter includes all token types specified using
its `types` parameter. Lucenes TypeTokenFilter also provides a second mode where
instead of keeping the specified tokens (include) they are filtered out
(exclude). This change exposes this option as a new `mode` parameter that can
either take the values `include` (the default, if not specified) or `exclude`.

Closes #29277
2018-07-17 09:04:41 +02:00
ahmedakef 5b65e2ccbf DOCS: put LIMIT 10 to the SQL query (#32065)
Provides a more precise equivalent SQL query for the aggregation example in the getting started guide.
2018-07-16 16:09:56 -04:00
debadair ff8260c234
Tweaked Elasticsearch Service links for SEO 2018-07-16 12:58:08 -07:00
debadair cccc3f7a64
Tweaked Elasticsearch Service links for SEO 2018-07-16 12:47:57 -07:00
DeDe Morton ecd05d5be4 Use correct formatting for links (#29460) 2018-07-16 21:11:24 +02:00
Jack Conradson 42ca520377
Clean Up Snapshot Create Rest API (#31779)
Make SnapshotInfo and CreateSnapshotResponse parsers lenient for backwards compatibility.  Remove extraneous fields from CreateSnapshotRequest toXContent.
2018-07-13 13:07:26 -07:00
Paul Sanwald feb07559aa fix typo 2018-07-13 14:59:11 -04:00
Colin Goodheart-Smithe 0edb096eb4 Adds a new auto-interval date histogram (#28993)
* Adds a new auto-interval date histogram

This change adds a new type of histogram aggregation called `auto_date_histogram` where you can specify the target number of buckets you require and it will find an appropriate interval for the returned buckets. The aggregation works by first collecting documents in buckets at second interval, when it has created more than the target number of buckets it merges these buckets into minute interval bucket and continues collecting until it reaches the target number of buckets again. It will keep merging buckets when it exceeds the target until either collection is finished or the highest interval (currently years) is reached. A similar process happens at reduce time.

This aggregation intentionally does not support min_doc_count, offest and extended_bounds to keep the already complex logic from becoming more complex. The aggregation accepts sub-aggregations but will always operate in `breadth_first` mode deferring the computation of sub-aggregations until the final buckets from the shard are known. min_doc_count is effectively hard-coded to zero meaning that we will insert empty buckets where necessary.

Closes #9572

* Adds documentation

* Added sub aggregator test

* Fixes failing docs test

* Brings branch up to date with master changes

* trying to get tests to pass again

* Fixes multiBucketConsumer accounting

* Collects more buckets than needed on shards

This gives us more options at reduce time in terms of how we do the
final merge of the buckeets to produce the final result

* Revert "Collects more buckets than needed on shards"

This reverts commit 993c782d117892af9a3c86a51921cdee630a3ac5.

* Adds ability to merge within a rounding

* Fixes nonn-timezone doc test failure

* Fix time zone tests

* iterates on tests

* Adds test case and documentation changes

Added some notes in the documentation about the intervals that can bbe
returned.

Also added a test case that utilises the merging of conseecutive buckets

* Fixes performance bug

The bug meant that getAppropriate rounding look a huge amount of time
if the range of the data was large but also sparsely populated. In
these situations the rounding would be very low so iterating through
the rounding values from the min key to the max keey look a long time
(~120 seconds in one test).

The solution is to add a rough estimate first which chooses the
rounding based just on the long values of the min and max keeys alone
but selects the rounding one lower than the one it thinks is
appropriate so the accurate method can choose the final rounding taking
into account the fact that intervals are not always fixed length.

Thee commit also adds more tests

* Changes to only do complex reduction on final reduce

* merge latest with master

* correct tests and add a new test case for 10k buckets

* refactor to perform bucket number check in innerBuild

* correctly derive bucket setting, update tests to increase bucket threshold

* fix checkstyle

* address code review comments

* add documentation for default buckets

* fix typo
2018-07-13 13:08:35 -04:00
Ryan Ernst 2c3ea43f45
HLRC: Add xpack usage api (#31975)
This commit adds the _xpack/usage api to the high level rest client.
Currently in the transport api, the usage data is exposed in a limited
fashion, at most giving one level of helper methods for the inner keys
of data, but then exposing thos subobjects as maps of objects. Rather
than making parsers for every set of usage data from each feature, this
PR exposes the entire set of usage data as a map of maps.
2018-07-13 09:33:27 -07:00
Mayya Sharipova 80492cacfc
Add second level of field collapsing (#31808)
* Put second level collapse under inner_hits

Closes #24855
2018-07-13 11:40:03 -04:00
Clinton Gormley bc1284eb28 Docs: Restyled cloud link in getting started 2018-07-13 15:48:14 +02:00
Clinton Gormley 9a928756e9 Docs: Change formatting of Cloud options 2018-07-13 15:40:38 +02:00
Alan Woodward a01e26a39b
Correct spelling of AnalysisPlugin#requriesAnalysisSettings (#32025)
Because this is a static method on a public API, and one that we encourage
plugin authors to use, the method with the typo is deprecated in 6.x
rather than just renamed.
2018-07-13 13:13:21 +01:00
Daniel Mitterdorfer f174f72fee
Circuit-break based on real memory usage
With this commit we introduce a new circuit-breaking strategy to the parent
circuit breaker. Contrary to the current implementation which only accounts for
memory reserved via child circuit breakers, the new strategy measures real heap
memory usage at the time of reservation. This allows us to be much more
aggressive with the circuit breaker limit so we bump it to 95% by default. The
new strategy is turned on by default and can be controlled  with the new cluster
setting `indices.breaker.total.userealmemory`.

Note that we turn it off for all integration tests with an internal test cluster
because it leads to spurious test failures which are of no value (we cannot
fully control heap memory usage in tests). All REST tests, however, will make
use of the real memory circuit breaker.

Relates #31767
2018-07-13 10:08:28 +02:00
Nik Everett 9822a6f911 Docs: Explain closing the high level client
It looks like we weren't clear on when and why you should close the high
level client and folks were closing it after every request which is not
efficient. This explains why you should close the client and when so
this shouldn't be as common.

Closes #32001
2018-07-12 16:22:42 -04:00
Jimi Ford e955ffc38d Docs: fix typo in datehistogram (#31972) 2018-07-11 15:04:57 -04:00
Clinton Gormley aedbfc63cd Docs: Added note about cloud service to installation and getting started 2018-07-11 20:17:18 +02:00
Lisa Cawley efcfd0d827
[DOCS] Removes alternative docker pull example (#31934) 2018-07-11 09:08:32 -07:00
James Baiera 5bcdff73d7
Add Snapshots Status API to High Level Rest Client (#31515)
This PR adds the Snapshots Status API to the Snapshot Client, as 
well as additional documentation for the status api.
2018-07-11 12:07:31 -04:00
Sohaib Iftikhar 88c270d844 Added lenient flag for synonym token filter (#31484)
* Added lenient flag for synonym-tokenfilter.

Relates to #30968

* added docs for synonym-graph-tokenfilter

-- Also made lenient final
-- changed from !lenient to lenient == false

* Changes after review (1)

-- Renamed to ElasticsearchSynonymParser
-- Added explanation for ElasticsearchSynonymParser::add method
-- Changed ElasticsearchSynonymParser::logger instance to static

* Added lenient option for WordnetSynonymParser

-- also added more documentation

* Added additional documentation

* Improved documentation
2018-07-10 17:11:50 -04:00
lcawl de273651ae [DOCS] Fix broken link in painless example 2018-07-09 10:47:42 -07:00
Mayya Sharipova 5481fbc249
Handle missing values in painless (#30975)
* Handle missing values in painless

Throw an exception for `doc['field'].value`
if this document is missing a value for the `field`.

For 7.0:
This is the default behaviour from 7.0

For 6.x:
To enable this behavior from 6.x, a user can set a jvm.option:
 `-Des.script.exception_for_missing_value=true` on a node.
If a user does not enable this behavior, a deprecation warning is logged on start up.

Closes #29286
2018-07-09 11:59:49 -04:00
Jim Ferenczi 584fa261cc
Remove the ability to index or query context suggestions without context (#31007)
This is a follow up of #30712 that removes the ability to index or query
and context enabled completion field without context.

Relates #30712
2018-07-09 16:01:01 +02:00
Armin Braun e46ed73379
Ingest: Add ignore_missing option to RemoveProc (#31693)
Added `ignore_missing` setting to the RemoveProcessor to fix #23086
2018-07-09 10:24:34 +02:00
Russ Cam 0dac73c4fb Remove link to oss-MSI (#31844)
This commit removes the link to an oss-MSI; there is only one version of the MSI, which includes X-Pack.

(cherry picked from commit d2e5db8a806ec8a25162f79db5209aceed4f30f7)
2018-07-09 11:31:38 +10:00
Nik Everett fb27f3e7f0
HLREST: Add x-pack-info API (#31870)
This is the first x-pack API we're adding to the high level REST client
so there is a lot to talk about here!

= Open source

The *client* for these APIs is open source. We're taking the previously
Elastic licensed files used for the `Request` and `Response` objects and
relicensing them under the Apache 2 license.

The implementation of these features is staying under the Elastic
license. This lines up with how the rest of the Elasticsearch language
clients work.

= Location of the new files

We're moving all of the `Request` and `Response` objects that we're
relicensing to the `x-pack/protocol` directory. We're adding a copy of
the Apache 2 license to the root fo the `x-pack/protocol` directory to
line up with the language in the root `LICENSE.txt` file. All files in
this directory will have the Apache 2 license header as well. We don't
want there to be any confusion. Even though the files are under the
`x-pack` directory, they are Apache 2 licensed.

We chose this particular directory layout because it keeps the X-Pack
stuff together and easier to think about.

= Location of the API in the REST client

We've been following the layout of the rest-api-spec files for other
APIs and we plan to do this for the X-Pack APIs with one exception:
we're dropping the `xpack` from the name of most of the APIs. So
`xpack.graph.explore` will become `graph().explore()` and
`xpack.license.get` will become `license().get()`.

`xpack.info` and `xpack.usage` are special here though because they
don't belong to any proper category. For now I'm just calling
`xpack.info` `xPackInfo()` and intend to call usage `xPackUsage` though
I'm not convinced that this is the final name for them. But it does get
us started.

= Jars, jars everywhere!

This change makes the `xpack:protocol` project a `compile` scoped
dependency of the `x-pack:plugin:core` and `client:rest-high-level`
projects. I intend to keep it a compile scoped dependency of
`x-pack:plugin:core` but I intend to bundle the contents of the protocol
jar into the `client:rest-high-level` jar in a follow up. This change
has grown large enough at this point.

In that followup I'll address javadoc issues as well.

= Breaking-Java

This breaks that transport client by a few classes around. We've
traditionally been ok with doing this to the transport client.
2018-07-08 11:03:56 -04:00
Costin Leau 9ffb26ab02
SQL: Remove restriction for single column grouping (#31818)
For historical reasons SQL restricts GROUP BY to only one field.
This commit removes the restriction and improves the test suite with
multi group by tests.

Close #31793
2018-07-06 20:55:27 +03:00
Piotr Prądzyński 99030e7af5 Docs: Inconsistency between description and example (#31858) 2018-07-06 12:44:20 -04:00
Christoph Büscher 3c11c7c261
[Docs] Add clarification to analysis example (#31826)
There have been at least two PRs trying to fix the spelling of "lazi" because it
isn't very clear from the example that the english analyzer will stem each token
in the example. This adds a short description of the analysis process to make
this clearer.

Relates to #31797
2018-07-06 14:36:58 +02:00
Christoph Büscher 450a450b2c
[Docs] Clarify accepted sort case (#31605)
Rescore only works with an explicite "sort" element if it is on descending
"_score". Even using "order" : "asc" will throw an error.
2018-07-06 10:11:36 +02:00
Alan Woodward d3a8094d35
Remove obsolete parameters from analyze rest spec (#31795)
This commit also fixes a typo in the analyze high-level client
documentation.
2018-07-06 09:05:34 +01:00
Jim Ferenczi 573613e7ca
[Docs] Fix wrong link in Korean analyzer docs (#31815) 2018-07-06 09:30:48 +02:00
Tim Brooks a5f5ea8422
Get snapshot rest client cleanups (#31740)
This is a followup to #31537. It makes a number of changes requested by
a review that came after the PR was merged. These are mostly cleanups
and doc improvements.
2018-07-05 17:19:46 -06:00
Nik Everett c0b2ef55b8 Docs: Explain _bulk?refresh shard targeting
Only the shards that receive the bulk request will be affected by
`refresh`. Imagine a `_bulk?refresh=wait_for` request with three
documents in it that happen to be routed to different shards in an index
with five shards. The request will only wait for those three shards to
refresh. The other two shards of that make up the index do not
participate in the `_bulk` request at all.

Relates to #31819
2018-07-05 16:24:03 -04:00
Sohaib Iftikhar 09e8ac8167 REST high-level client: add get index API (#31703)
Also added master_timeout parameter for the indices.get spec

Relates to #27205
2018-07-05 13:52:25 -04:00
Sohaib Iftikhar 40b822c878 Scripting: Remove support for deprecated StoredScript contexts (#31394)
Removes support for storing scripts without the usual json around the
script. So You can no longer do:
```
POST _scripts/<templatename>
{
    "query": {
        "match": {
            "title": "{{query_string}}"
        }
    }
}
```

and must instead do:
```
POST _scripts/<templatename>
{
    "script": {
        "lang": "mustache",
        "source": {
            "query": {
                "match": {
                    "title": "{{query_string}}"
                }
            }
        }
    }
}
```

This improves error reporting when you attempt to store a script but don't
quite get the syntax right. Before, there was a good chance that we'd
think of it as a "raw" template and just store it. Now we won't do that.
Nice.
2018-07-05 09:30:08 -04:00
Alpar Torok cf2295b408
Add JDK11 support and enable in CI (#31644)
* Upgrade bouncycastle

Required to fix
`bcprov-jdk15on-1.55.jar; invalid manifest format `
on jdk 11

* Downgrade bouncycastle to avoid invalid manifest

* Add checksum for new jars

* Update tika permissions for jdk 11

* Mute test failing on jdk 11

* Add JDK11 to CI

* Thread#stop(Throwable) was removed

http://mail.openjdk.java.net/pipermail/core-libs-dev/2018-June/053536.html

* Disable failing tests #31456

* Temprorarily disable doc tests

To see if there are other failures on JDK11

* Only blacklist specific doc tests

* Disable only failing tests in ingest attachment plugin

* Mute failing HDFS tests #31498

* Mute failing lang-painless tests #31500

* Fix backwards compatability builds

Fix JAVA version to 10 for ES 6.3

* Add 6.x to bwx -> java10

* Prefix out and err from buildBwcVersion for readability

```
> Task :distribution:bwc:next-bugfix-snapshot:buildBwcVersion
  [bwc] :buildSrc:compileJava
  [bwc] WARNING: An illegal reflective access operation has occurred
  [bwc] WARNING: Illegal reflective access by org.codehaus.groovy.reflection.CachedClass (file:/home/alpar/.gradle/wrapper/dists/gradle-4.5-all/cg9lyzfg3iwv6fa00os9gcgj4/gradle-4.5/lib/groovy-all-2.4.12.jar) to method java.lang.Object.finalize()
  [bwc] WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.reflection.CachedClass
  [bwc] WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
  [bwc] WARNING: All illegal access operations will be denied in a future release
  [bwc] :buildSrc:compileGroovy
  [bwc] :buildSrc:writeVersionProperties
  [bwc] :buildSrc:processResources
  [bwc] :buildSrc:classes
  [bwc] :buildSrc:jar

```

* Also set RUNTIME_JAVA_HOME for bwcBuild

So that we can make sure it's not too new for the build to understand.

* Align bouncycastle dependency

* fix painles array tets

closes #31500

* Update jar checksums

* Keep 8/10 runtime/compile untill consensus builds on 11

* Only skip failing tests if running on Java 11

* Failures are dependent of compile java version not runtime

* Condition doc test exceptions on compiler java version as well

* Disable hdfs tests based on runtime java

* Set runtime java to minimum supported for bwc

* PR review

* Add comment with ticket for forbidden apis
2018-07-05 03:24:01 +00:00
Christoph Büscher 5f87a84bef
[Docs] Correct default window_size (#31582) 2018-07-04 14:07:20 +02:00
Lee Hinman 42ff8aa2ef
[DOCS] Add missing get mappings docs to HLRC (#31765)
This commit adds the high-level rest client docs for the get mappings API that
was added in #30889
2018-07-03 17:10:52 -06:00
Lisa Cawley ac7fadd336
[DOCS] Starting Elasticsearch (#31701) 2018-07-03 13:40:37 -07:00
Jake Landis c0056cddd8
ingest: Introduction of a bytes processor (#31733)
ingest: Introduction of a bytes processor

This processor allows for human readable byte values (e.g. 1kb) to be converted to value in bytes (e.g. 1024). Internally this processor re-uses "ByteSizeValue.parseBytesSizeValue" which supports conversions up to Long.MAX_VALUE and the following units: "b", "kb", "mb", "gb", "tb", pb".

This change also introduces a generic return type for the AbstractStringProcessor to allow for code reuse while supporting a String -> T conversion. (String -> Long in this case).
2018-07-03 10:40:56 -05:00
Alan Woodward 1d114071da
Add analyze API to high-level rest client (#31577) 2018-07-03 15:57:02 +01:00
Costin Leau 093ea037b4 [DOCS] Typos 2018-07-03 17:19:48 +03:00
Costin Leau de9e56aa01
DOC: Add examples to the SQL docs (#31633)
Significantly improve the example snippets in the documentation.
The examples are part of the test suite and checked nightly.
To help readability, the existing dataset was extended (test_emp renamed
to emp plus library).
Improve output of JDBC tests to be consistent with the CLI
Add lenient flag to JDBC asserts to allow type widening (a long is
equivalent to a integer as long as the value is the same).
2018-07-03 16:56:31 +03:00
David Turner 4108722052
Add support for AWS session tokens (#30414)
AWS supports the creation and use of credentials that are only valid for a
fixed period of time. These credentials comprise three parts: the usual access
key and secret key, together with a session token. This commit adds support for
these three-part credentials to the EC2 discovery plugin and the S3 repository
plugin.

Note that session tokens are only valid for a limited period of time and yet
there is no mechanism for refreshing or rotating them when they expire without
restarting Elasticsearch.  Nonetheless, this feature is already useful for
nodes that need only run for a few days, such as for training, testing or
evaluation. #29135 tracks the work towards allowing these credentials to be
refreshed at runtime.

Resolves #16428
2018-07-03 14:12:07 +01:00