Commit Graph

28727 Commits

Author SHA1 Message Date
Boaz Leskes ffc9999567 fix StartRecoveryRequestTests.testSerialization 2017-09-14 23:20:55 +03:00
Michael Basnight f385e0cf26 Add bad_request to the rest-api-spec catch params (#26539)
This adds another request to the catch params. It also makes sure that
the generic request param does not allow 400 either.
2017-09-14 14:24:03 -05:00
Boaz Leskes 1ca0b5e9e4 Introduce a History UUID as a requirement for ops based recovery (#26577)
The new ops based recovery, introduce as part of  #10708, is based on the assumption that all operations below the global checkpoint known to the replica do not need to be synced with the primary. This is based on the guarantee that all ops below it are available on primary and they are equal. Under normal operations this guarantee holds. Sadly, it can be violated when a primary is restored from an old snapshot. At the point the restore primary can miss operations below the replica's global checkpoint, or even worse may have total different operations at the same spot. This PR introduces the notion of a history uuid to be able to capture the difference with the restored primary (in a follow up PR).

The History UUID is generated by a primary when it is first created and is synced to the replicas which are recovered via a file based recovery. The PR adds a requirement to ops based recovery to make sure that the history uuid of the source and the target are equal. Under normal operations, all shard copies will stay with that history uuid for the rest of the index lifetime and thus this is a noop. However, it gives us a place to guarantee we fall back to file base syncing in special events like a restore from snapshot (to be done as a follow up) and when someone calls the truncate translog command which can go wrong when combined with primary recovery (this is done in this PR).

We considered in the past to use the translog uuid for this function (i.e., sync it across copies) and thus avoid adding an extra identifier. This idea was rejected as it removes the ability to verify that a specific translog really belongs to a specific lucene index. We also feel that having a history uuid will serve us well in the future.
2017-09-14 21:25:02 +03:00
Michael Basnight e69c39a60f Add missing catch arguments to the rest api spec (#26536) 2017-09-14 11:05:46 -05:00
Bernd 59600dfe2d [Docs] Correct typo in removal_of_types.asciidoc (#26646) 2017-09-14 15:34:07 +02:00
Christoph Büscher c7c6443b10 [Docs] "The the" is a great band, but ... (#26644)
Removing several occurrences of this typo in the docs and javadocs, seems to be
a common mistake. Corrections turn up once in a while in PRs, better to correct
some of this in one sweep.
2017-09-14 15:08:20 +02:00
David Pilato b6c6effa2a Move all repository-azure classes under one single package (#26624)
As we did for S3, we can collapse all packages within one single `org.elasticsearch.repositories.azure` package name.

Follow up for https://github.com/elastic/elasticsearch/pull/23518#issuecomment-328903585
2017-09-14 11:56:02 +02:00
Daniel A. Ochoa 914416e9f4 [Docs] Update link in removal_of_types.asciidoc (#26614)
Fix link to [parent-child relationship].
2017-09-14 10:11:03 +02:00
Jim Ferenczi 401f4ba2ce Fix percolator highlight sub fetch phase to not highlight query twice (#26622)
* Fix percolator highlight sub fetch phase to not highlight query twice

The PercolatorHighlightSubFetchPhase does not override hitExecute and since it extends HighlightPhase the search hits
are highlighted twice (by the highlight phase and then by the percolator). This does not alter the results, the second highlighting
just overrides the first one but this slow down the request because it duplicates the work.
2017-09-14 09:31:14 +02:00
Jason Tedor ca6bce75da Refactor bootstrap check results and error messages
This commit refactors the bootstrap checks into a single result object
that encapsulates whether or not the check passed, and a failure message
if the check failed. This simpifies the checks, and enables the messages
to more easily be based on the state used to discern whether or not the
check passed.

Relates #26637
2017-09-13 21:30:27 -04:00
Simon Willnauer b4de2a6f28 Add BootstrapContext to expose settings and recovered state to bootstrap checks (#26628)
This exposes the node settings and the persistent part of the cluster state to the
bootstrap checks to allow plugins to enforce certain preconditions based on the
recovered state.
2017-09-13 22:14:17 +02:00
Christoph Büscher 2eaf7534f3 [Tests] Removing skipping tests in search rest tests
After backporting the script_field soft limit to the 6.x branches, this test can
now also run in a mixed cluster.

Relates to #26598

 enter the commit message for your changes. Lines starting
2017-09-13 18:21:15 +02:00
Jason Tedor 7be5ee5f28 Initialize checkpoint tracker with allocation ID
This commit pushes the allocation ID down through to the global
checkpoint tracker at construction rather than when activated as a
primary.

Relates #26630
2017-09-13 12:15:15 -04:00
Adrien Grand 93da7720ff Move non-core mappers to a module. (#26549)
Today we have all non-plugin mappers in core. I'd like to start moving those
that neither map to json datatypes nor are very frequently used like `date` or
`ip` to a module.

This commit creates a new module called `mappers-extra` and moves the
`scaled_float` and `token_count` mappers to it. I'd like to eventually move
`range` fields there but it's more complicated due to their intimate
relationship with range queries.

Relates #10368
2017-09-13 17:58:53 +02:00
Tanguy Leroux 7404221b55 [Docs] Clarify size parameter in Completion Suggester doc (#26617) 2017-09-13 17:28:31 +02:00
Christoph Büscher 027c555c9b Add soft limit on allowed number of script fields in request (#26598)
Requesting to many script_fields in a search request can be costly
because of script execution. This change introduces a soft limit on the number
of script fields that are allowed per request. The setting can be
changed per index using the index.max_script_fields setting.

Relates to #26390
2017-09-13 17:22:16 +02:00
Adrien Grand 64770b3fbd Remove MapperService#dynamic. (#26603)
We ignore it as of 6.0 and forbid it as of 7.0.
2017-09-13 17:00:52 +02:00
Jim Ferenczi c709b8d6ac Fix incomplete sentences in parent-join docs (#26623)
* Fix incomplete sentences in parent-join docs

Closes #26590
2017-09-13 16:09:00 +02:00
Adrien Grand 454cfc2cea More efficient encoding of range fields. (#26470)
This PR removes the vInt that precedes every value in order to know how long
they are. Instead the query takes an enum that tells how to compute the length
of values: for fixed-length data (ip addresses, double, float) the length is a
constant while longs and integers use a variable-length representation that
allows the length to be computed from the encoded values.

Also the encoding of ints/longs was made a bit more efficient in order not to
waste 3 bits in the header. As a consequence, values between -8 and 7 can now
be encoded on 1 byte and values between -2048 and 2047 can now be encoded on 2
bytes or less.

Closes #26443
2017-09-13 15:26:33 +02:00
Jason Tedor 670849894f Ensure module is bundled before installing in tests
This commit adds a dependency to the install module task on the task
that builds the module. This is needed for standalone integration
tests that require other modules to be installed. Without this, we do
not have a guarantee that the module is bundled.
2017-09-13 08:17:16 -04:00
Ivan Brusic 9e05b3260b Add boolean similarity to built in similarity types (#26613) 2017-09-13 13:58:30 +02:00
Christoph Büscher d2cfad6187 [Tests] Remove skip tests in search/30_limits.yml
After backporting the related change to the 6.x branches, this test can now also
be run in a mixed cluster.

Relates to #26574
2017-09-13 13:15:31 +02:00
Jason Tedor b3e7e85cf1 Let search phases override max concurrent requests
If the query coordinating node is also a data node that holds all the
shards for a search request, we can end up recursing through the can
match phase (because we send a local request and on response in the
listener move to the next shard and do this again, without ever having
returned from previous shards). This recursion can lead to stack
overflow for even a reasonable number of indices (daily indices over a
sixty days with five shards per day is enough to trigger the stack
overflow). Moreover, all this execution would be happening on a network
thread (the thread that initially received the query). With this commit,
we allow search phases to override max concurrent requests. This allows
the can match phase to avoid recursing through the shards towards a
stack overflow.

Relates #26484
2017-09-13 06:16:27 -04:00
Christoph Büscher e00db235bc Add a soft limit for the number of requested doc-value fields (#26574)
Requesting to many docvalue_fields in a search request can potentially be costly
because it might incur a per-field per-document seek. This change introduces a
soft limit on the number of fields that can be retrieved. The setting can be
changed per index using the `index.max_docvalue_fields_search` setting.

Relates to #26390
2017-09-13 11:57:06 +02:00
David Pilato a34db4e09f Support for accessing Azure repositories through a proxy (#23518)
You can define a proxy using the following settings:

```yml
azure.client.default.proxy.host: proxy.host
azure.client.default.proxy.port: 8888
azure.client.default.proxy.type: http
```

Supported values for `proxy.type` are `direct`, `http` or `socks`. Defaults to `direct` (no proxy).

Closes #23506

BTW I changed a test `testGetSelectedClientBackoffPolicyNbRetries` as it was using an old setting name `cloud.azure.storage.azure.max_retries` instead of `azure.client.azure1.max_retries`.
2017-09-13 11:51:55 +02:00
Russ Cam 62a7205577 Add beta tag to MSI Windows Installer (#26616) 2017-09-13 13:23:12 +10:00
Adrien Grand 04b24c7780 Fix Lucene version of 5.6.1. 2017-09-12 17:54:50 +02:00
David Pilato b01b1c2a58 Remove azure deprecated settings (#26099)
Follow up for #23405.

We remove azure deprecated settings in 7.0:

* The legacy azure settings which where starting with `cloud.azure.storage.` prefix have been removed.
This includes `account`, `key`, `default` and `timeout`.
You need to use settings which are starting with `azure.client.` prefix instead.

* Global timeout setting `cloud.azure.storage.timeout` has been removed.
You must set it per azure client instead. Like `azure.client.default.timeout: 10s` for example.
2017-09-12 16:51:44 +02:00
Michael Basnight 0e57a416f1 Handle the 5.6.0 release 2017-09-12 09:48:09 -05:00
Simon Willnauer 42f3129d7b Allow plugins to validate cluster-state on join (#26595)
Today we don't have a pluggable way to validate if the cluster state
is compatible with the node that joins. We already apply some checks for index
compatibility that prevents nodes to join a cluster with indices it doesn't support
but for plugins this isn't possible. This change adds a cluster state validator that
allows plugins to prevent a join if the cluster-state is incompatible.
2017-09-12 15:32:33 +02:00
Yu 3d4e28aee1 Remove index mapper dynamic settings (#25734)
Remove "index.mapper.dynamic" setting for 6.0 (and after) indices, but
still keep working for 5.x (and before) indices. Remove two index
dynamic disable test cases as the disability of index.mapper.dynamic is
already removed for current version. Add a new test class for version
test.
2017-09-12 14:29:10 +02:00
mohit 06150d40a2 update AWS SDK for ECS Task IAM support in discovery-ec2 (#26479)
This commit contains:

* update AWS SDK for ECS Task IAM support
* ignore dependencies not essential to `discovery-ec2`:
  * jmespath seems to be used for `waiters`
  * amazon ion is a protocol not used by EC2 or IAM
2017-09-12 10:34:12 +02:00
etiennecarriere 706067211a Azure repository: Accelerate the listing of files (used in delete snapshot) (#25710)
This commit reworks the azure listing of snapshot files to do a single listing, instead of once per blob.

closes #25424
2017-09-11 16:09:27 -07:00
Ryan Ernst 95d40758c2 Build: Remove norelease from forbidden patterns (#26592)
closes #26547
2017-09-11 15:55:02 -07:00
Ryan Ernst 9834081254 Fix reference to painless inside expression engine (#26528)
This was a simple copy/paste bug in an earlier refactoring.
2017-09-11 15:44:23 -07:00
Ryan Ernst 8ba4ff3be0 Build: Move javadoc linking to root build.gradle (#26529)
Javadoc linking between projects currently relies on
projectSubstitutions. However, that is an extension variable that is not
part of BuildPlugin. This commit moves the javadoc linking into the root
build.gradle, alongside where projectSubstitutions are defined.
2017-09-11 15:43:34 -07:00
Ryan Ernst 5c35bff1c3 Test: Remove leftover static bwc test case (#26584)
This test case was leftover from the static bwc tests. There was still
one use for checking we do not load old indices, but this PR moves the
legacy code needed for that directly into the test. I also opened a
follow up issue to completely remove the unsupported test: #26583.
2017-09-11 15:38:30 -07:00
Ryan Ernst c0c5d5488f Docs: Remove remaining references to file and native scripts (#26580)
relates #25690
2017-09-11 11:39:29 -07:00
Jason Tedor b2e4bfa0a7 Snapshot fallback should consider build.snapshot
When determining if a build is a snapshot build, we look for a field in
the JAR manifest. However, when running tests, we are not running with a
compiled core Elasticsearch JAR, we are running with the compiled core
classes on the classpath. We have a fallback for this, we always assume
such a situation is a snapshot build. However, when running builds with
-Dbuild.snapshot=false, this is not the case. As such, we need to
fallback to the value of build.snapshot. However, there are cases where
we are not running with a compiled core Elasticsearch JAR (e.g., when
the transport client is embedded in a web container) so we should only
do this fallback if we are in tests. To verify we are in tests, we check
if randomized runner is on the classpath.

Relates #26554
2017-09-11 07:42:11 -04:00
Jim Ferenczi c62b0192d0 #26496: Set the correct bwc version after backport to 6.x 2017-09-11 13:09:44 +02:00
Adrien Grand 1adee8b5a8 Fix the MapperFieldType.rangeQuery API. (#26552)
RangeQueryBuilder needs to perform too many `instanceof` checks in order to
check for `date` or `range` fields in order to know what it should do with the
shape relation, time zone and date format.

This commit adds those 3 parameters to the `rangeQuery` factory method so that
those instanceof checks are not necessary anymore.
2017-09-11 11:02:05 +02:00
Adrien Grand 2bc3eeccde Deduplicate `_field_names`. (#26550)
This is a minor optimization that should save some utf8 conversions and indexing.
2017-09-11 10:57:08 +02:00
jiangzhuolin a6d57f50f6 [Docs] Update method setSource(byte[] source) (#26561)
The method setSource(byte[]) from the type IndexRequestBuilder is deprecated,
use the method setSource(byte[] source, XContentType xContentType).
2017-09-11 10:20:20 +02:00
Md.Abdulla-Al-Sun d00d18a36d [Docs] Fix typo in javadocs (#26556) 2017-09-09 22:25:31 +02:00
Jason Tedor e16cb0e4db Allow multiple digits in Vagrant 2.x minor versions
This commit allows the minor version in Vagrant 2.x versions to have two
digits when we perform the Vagrant version check.
2017-09-09 15:28:26 -04:00
Kevin Risden f476c10825 Support Vagrant 2.x
This commit adds support for Vagrant 2.x by allowing these versions to
pass the Vagrant version check.

Relates #26563
2017-09-09 15:27:21 -04:00
Lee Hinman 2702918780 Limit the number of expanded fields it query_string and simple_query_string (#26541)
* Limit the number of expanded fields it query_string and simple_query_string

This limits the number of automatically expanded fields for the "all fields"
mode (`"default_field": "*"`) for the `query_string` and `simple_query_string`
queries to 1024 fields.

Resolves #25105

* Add blurb about limit to the docs
2017-09-08 13:37:55 -06:00
Lee Hinman dd90cf1bbb Throw a better error message for empty field names (#26543)
* Throw a better error message for empty field names

When a document is parsed with a `""` for a field name, we currently throw a
confusing error about `.` being present in the field. This changes the error
message to be clearer about what's causing the problem.

Resolves #23348

* Fix exception message in test
2017-09-08 13:30:17 -06:00
Martijn van Groningen b391425da1
Added support to the percolate query to percolate multiple documents
The percolator will add a `_percolator_document_slot` field to all percolator
hits to indicate with what document it has matched. This number matches with
the order in which the documents have been specified in the percolate query.

Also improved the support for multiple percolate queries in a search request.
2017-09-08 17:28:39 +02:00
Lee Hinman 4e43aac0f8 Expand "NO" decision message in NodeVersionAllocationDecider (#26542)
This explains the `NO` Decision a little more.

Resolves #10403
2017-09-08 09:18:34 -06:00