Commit Graph

961 Commits

Author SHA1 Message Date
Martijn van Groningen 8e1ef0cff9
Rewrite shard follow node task logic (#31581)
The current shard follow mechanism is complex and does not give us easy ways the have visibility into the system (e.g. why we are falling behind).
The main reason why it is complex is because the current design is highly asynchronous. Also in the current model it is hard to apply backpressure
other than reducing the concurrent reads from the leader shard.

This PR has the following changes:
* Rewrote the shard follow task to coordinate the shard follow mechanism between a leader and follow shard in a single threaded manner.
  This allows for better unit testing and makes it easier to add stats.
* All write operations read from the shard changes api should be added to a buffer instead of directly sending it to the bulk shard operations api.
  This allows to apply backpressure. In this PR there is a limit that controls how many write ops are allowed in the buffer after which no new reads
  will be performed until the number of ops is below that limit.
* The shard changes api includes the current global checkpoint on the leader shard copy. This allows reading to be a more self sufficient process;
  instead of relying on a background thread to fetch the leader shard's global checkpoint.
* Reading write operations from the leader shard (via shard changes api) is a separate step then writing the write operations (via bulk shards operations api).
  Whereas before a read would immediately result into a write.
* The bulk shard operations api returns the local checkpoint on the follow primary shard, to keep the shard follow task up to date with what has been written.
* Moved the shard follow logic that was previously in ShardFollowTasksExecutor to ShardFollowNodeTask.
* Moved over the changes from #31242 to make shard follow mechanism resilient from node and shard failures.

Relates to #30086
2018-07-10 16:00:55 +02:00
Nhat Nguyen 5875399738
Upgrade to Lucene-7.5.0-snapshot-b9e064b935 (#31839)
This snapshot includes LUCENE-8384 which blocks work in Lucene rollback.
2018-07-08 15:05:08 -04:00
Nhat Nguyen 4be6b0e2ae Merge branch 'master' into ccr
* master:
  [ML] Fix master node deadlock during ML daily maintenance (#31836)
  Build: Switch integ-test-zip to OSS-only (#31866)
  SQL: Remove restriction for single column grouping (#31818)
  Build: Fix detection of Eclipse Compiler Server (#31838)
  Docs: Inconsistency between description and example (#31858)
  Re-enable bwc tests now that #29538 has been backported and 6.x intake build succeeded.
  QA: build improvements related to SQL projects (#31862)
  [Docs] Add clarification to analysis example (#31826)
  Check timeZone() argument in AbstractSqlQueryRequest (#31822)
  SQL: Fix incorrect HAVING equality (#31820)
  Smaller aesthetic fixes to InternalTestCluster (#31831)
  [Docs] Clarify accepted sort case (#31605)
  Temporarily disable bwc test in order to backport #29538
  Remove obsolete parameters from analyze rest spec (#31795)
  [Docs] Fix wrong link in Korean analyzer docs (#31815)
  Fix profiling of ordered terms aggs (#31814)
  Properly mute test involving JDK11 closes #31739
  Do not return all indices if a specific alias is requested via get aliases api. (#29538)
  Get snapshot rest client cleanups (#31740)
  Docs: Explain _bulk?refresh shard targeting
  Fix handling of points_only with term strategy in geo_shape (#31766)
2018-07-07 09:12:39 -04:00
Martijn van Groningen c6e7df3ac2
Re-enable bwc tests now that #29538 has been backported and 6.x intake
build succeeded.
2018-07-06 18:03:25 +02:00
Jim Ferenczi bf9ae5f0c8
Fix profiling of ordered terms aggs (#31814)
This change fixes the profiling of `terms` aggregation ordered
by a sub aggregation.

Closes #22123
2018-07-06 09:00:52 +02:00
Martijn van Groningen 4761a1fa29
Do not return all indices if a specific alias is requested via get aliases api. (#29538)
If a get alias api call requests a specific alias pattern then
indices not having any matching aliases should not be included in the response.

This is a second attempt to fix this (first attempt was #28294).
The reason that the first attempt was reverted is because when xpack
security is enabled then index expression (like * or _all) are resolved
prior to when a request is processed in the get aliases transport action,
then `MetaData#findAliases` can't know whether requested all where
requested since it was already expanded in concrete alias names. This
change replaces aliases(...) replaceAliases(...) method on AliasesRequests
class and leave the aliases(...) method on subclasses. So there is a distinction
between when xpack security replaces aliases and a user setting aliases via
the transport or high level http client.

Closes #27763
2018-07-06 08:54:32 +02:00
Tim Brooks a5f5ea8422
Get snapshot rest client cleanups (#31740)
This is a followup to #31537. It makes a number of changes requested by
a review that came after the PR was merged. These are mostly cleanups
and doc improvements.
2018-07-05 17:19:46 -06:00
Igor Motov 94d3ddd1d1
Fix handling of points_only with term strategy in geo_shape (#31766)
Fixes 2 issues that together cause errors during index creation
with geo_shapes that use the term strategy. The term strategy changes
the default for points_only parameter, but this wasn't taken into
account during serialization. So, setting the term strategy would add
`"points_only": true` to serialization. At the same time if the term
strategy would also cause the `points_only` setting to be not marked as
a processed element during parsing, which would cause index creation to
fail with the error: `Mapping definition for [location] has unsupported`
`parameters:  [points_only : true]`.

Fixes #31707
2018-07-05 11:40:33 -07:00
Nhat Nguyen 6dd3434519 Merge branch 'master' into ccr
* master:
  REST high-level client: add get index API (#31703)
  SQL: Allow long literals (#31777)
  SQL: Fix incorrect message for aliases (#31792)
  Test: Do not remove xpack templates when cleaning (#31642)
  Reduce more raw types warnings (#31780)
  Add unreleased version 6.3.2
  Scripting: Remove support for deprecated StoredScript contexts (#31394)
  [ML][TEST] Use java 11 valid time format in DataDescriptionTests (#31817)
  [ML] Don't treat stale FAILED jobs as OPENING in job allocation (#31800)
  [ML] Fix calendar and filter updates from non-master nodes (#31804)
  Fix license header generation on Windows (#31790)
  mark RollupIT.testTwoJobsStartStopDeleteOne as AwaitsFix
  mark SearchAsyncActionTests.testFanOutAndCollect as AwaitsFix
  Correct exclusion of test on JDK 11
  Fix doclint jdk 11
  Add JDK11 support and enable in CI (#31644)
  Watcher: Fix check for currently executed watches (#31137)
  Watcher: Ensure correct method is used to read secure settings (#31753)
  SQL: Update CLI logo
2018-07-05 14:07:06 -04:00
Christoph Büscher bd1c513422
Reduce more raw types warnings (#31780)
Similar to #31523.
2018-07-05 15:38:06 +02:00
Mayya Sharipova ca5822eae1 Add unreleased version 6.3.2 2018-07-05 09:36:40 -04:00
Sohaib Iftikhar 40b822c878 Scripting: Remove support for deprecated StoredScript contexts (#31394)
Removes support for storing scripts without the usual json around the
script. So You can no longer do:
```
POST _scripts/<templatename>
{
    "query": {
        "match": {
            "title": "{{query_string}}"
        }
    }
}
```

and must instead do:
```
POST _scripts/<templatename>
{
    "script": {
        "lang": "mustache",
        "source": {
            "query": {
                "match": {
                    "title": "{{query_string}}"
                }
            }
        }
    }
}
```

This improves error reporting when you attempt to store a script but don't
quite get the syntax right. Before, there was a good chance that we'd
think of it as a "raw" template and just store it. Now we won't do that.
Nice.
2018-07-05 09:30:08 -04:00
Vladimir Dolzhenko dabf2b41e0 mark SearchAsyncActionTests.testFanOutAndCollect as AwaitsFix 2018-07-05 09:39:54 +02:00
Nhat Nguyen f736205fd5 Merge branch 'master' into ccr
* master:
  [ML] Rate limit established model memory updates (#31768)
  [Docs] Correct default window_size (#31582)
  S3 fixture should report 404 on unknown bucket (#31782)
  Detach Transport from TransportService (#31727)
  [ML] Limit ML filter items to 10K (#31731)
  [ML] Return statistics about forecasts as part of the jobsstats and usage API (#31647)
  Fixture for Minio testing (#31688)
  [DOCS] Add missing get mappings docs to HLRC (#31765)
  [DOCS] Starting Elasticsearch (#31701)
  Painless: Complete Removal of Painless Type (#31699)
  Fix not waiting for Netty ThreadDeathWatcher in IT (#31758)
  Consolidate watcher setting update registration (#31762)
  Build: re-enabled bwc (#31769)
  ingest: Introduction of a bytes processor (#31733)
  Fix coerce validation_method in GeoBoundingBoxQueryBuilder (#31747)
  Add analyze API to high-level rest client (#31577)
  [DOCS] Typos
  DOC: Add examples to the SQL docs (#31633)
  Add support for AWS session tokens (#30414)
  Watcher: Reenable start/stop yaml tests (#31754)
  Implemented XContent serialisation for GetIndexResponse (#31675)
  JDBC: Fix stackoverflow on getObject and timestamp conversion (#31735)
  resolveHasher defaults to NOOP (#31723)
  Account for XContent overhead in in-flight breaker
  Split CircuitBreaker-related tests (#31659)
  Add write*Blob option to replace existing blob (#31729)
  Painless: Add Context Docs (#31190)
  Watcher: Fix chain input toXcontent serialization (#31721)
  Docs: Match the examples in the description (#31710)
  rest-high-level: added get cluster settings (#31706)
  [Docs] Correct typos (#31720)
  Clean up double semicolon code typos (#31687)
  [DOCS] Check for Windows and *nix file paths (#31648)
  [ML] Validate ML filter_id (#31535)
  Revert long lines
  Fix TransportChangePasswordActionTests
2018-07-04 09:18:27 -04:00
Simon Willnauer 3f2a241b7f
Detach Transport from TransportService (#31727)
Today TransportService is tightly coupled with Transport since it
requires an instance of TransportService in order to receive responses
and send requests. This is mainly due to the Request and Response handlers
being maintained in TransportService but also because of the lack of a proper 
callback interface.

This change moves request handler registry and response handler registration into
Transport and adds all necessary methods to `TransportConnectionListener` in order
to remove the `TransportService` dependency from `Transport`
Transport now accepts one or more `TransportConnectionListener` instances that are
executed sequentially in a blocking fashion.
2018-07-04 11:32:35 +02:00
Igor Motov 396c578066
Fix coerce validation_method in GeoBoundingBoxQueryBuilder (#31747)
The Rectangle constructor validates bounds before coerce has a chance
to normalize coordinates so it cannot be used as intermittent storage.
This commit removes the Rectangle as an intermittent storage for the
bounding box coordinates.

Fixes #31718
2018-07-03 08:08:40 -07:00
Alan Woodward 1d114071da
Add analyze API to high-level rest client (#31577) 2018-07-03 15:57:02 +01:00
David Turner 4108722052
Add support for AWS session tokens (#30414)
AWS supports the creation and use of credentials that are only valid for a
fixed period of time. These credentials comprise three parts: the usual access
key and secret key, together with a session token. This commit adds support for
these three-part credentials to the EC2 discovery plugin and the S3 repository
plugin.

Note that session tokens are only valid for a limited period of time and yet
there is no mechanism for refreshing or rotating them when they expire without
restarting Elasticsearch.  Nonetheless, this feature is already useful for
nodes that need only run for a few days, such as for training, testing or
evaluation. #29135 tracks the work towards allowing these credentials to be
refreshed at runtime.

Resolves #16428
2018-07-03 14:12:07 +01:00
Sohaib Iftikhar a5fd4a7709 Implemented XContent serialisation for GetIndexResponse (#31675)
This PR does the server side work for adding the Get Index API to the REST
high-level-client, namely moving resolving default settings to the
transport action. A follow up would be the client side changes.
2018-07-03 08:08:50 -04:00
Daniel Mitterdorfer 3d53daeb2f
Account for XContent overhead in in-flight breaker
So far the in-flight request circuit breaker has only accounted for the
on-the-wire representation of a request. However, we convert the raw
request into XContent internally which increases the overhead.
Therefore, we increase the value of the corresponding setting
`network.breaker.inflight_requests.overhead` from one to two. While this
value is still rather conservative (we assume that the representation as
structured objects has no overhead compared to the byte[]), it is closer
to reality than the current value.

Relates #31613
2018-07-03 09:17:16 +02:00
Daniel Mitterdorfer ee4dbc8ded
Split CircuitBreaker-related tests (#31659)
`MemoryCircuitBreakerTests` conflates two test aspects: It tests
individual circuit breakers as well as the circuit breaker hierarchy.
With this commit we split those two aspects into two test classes:

* Tests for individual circuit breakers stay in the current class
* Other tests are moved to `HierarchyCircuitBreakerServiceTests`
2018-07-03 09:16:26 +02:00
Yannick Welsch 2bb4f38371
Add write*Blob option to replace existing blob (#31729)
Adds a new parameter to the BlobContainer#write*Blob methods to specify whether the existing file
should be overridden or not. For some metadata files in the repository, we actually want to replace
the current file. This is currently implemented through an explicit blob delete and then a fresh write.
In case of using a cloud provider (S3, GCS, Azure), this results in 2 API requests instead of just 1.
This change will therefore allow us to achieve the same functionality using less API requests.
2018-07-03 09:13:50 +02:00
Sohaib Iftikhar c55d11f8b5 rest-high-level: added get cluster settings (#31706)
Relates to #27205
2018-07-02 13:25:17 -04:00
Christoph Büscher 31aabe4bf9
Clean up double semicolon code typos (#31687) 2018-07-02 15:14:44 +02:00
Nhat Nguyen 540da7399c
TEST: Randomize soft-deletes settings (#31585)
This change allows us to test our system with/without soft-deletes enabled.
2018-06-29 17:35:21 -04:00
Nhat Nguyen 52d9012d31 Merge branch 'master' into ccr
* master:
  Do not check for object existence when deleting repository index files (#31680)
  Remove extra check for object existence in repository-gcs read object (#31661)
  Support multiple system store types (#31650)
  [Test] Clean up some repository-s3 tests (#31601)
  [Docs] Use capital letters in section headings (#31678)
  [DOCS] Add PQL language Plugin (#31237)
  Merge AzureStorageService and AzureStorageServiceImpl and clean up tests (#31607)
  TEST: Fix test task invocation (#31657)
  Revert "[TEST] Mute failing tests in NativeRealmInteg and ReservedRealmInteg"
  Fix RealmInteg test failures
  Extend allowed characters for grok field names (#21745) (#31653)
  [DOCS] Fix licensing API details (#31667)
  [TEST] Mute failing tests in NativeRealmInteg and ReservedRealmInteg
  Fix CreateSnapshotRequestTests Failure (#31630)
  Configurable password hashing algorithm/cost (#31234)
  [TEST] Mute failing NamingConventionsTaskIT tests
  [DOCS] Replace CONFIG_DIR with ES_PATH_CONF (#31635)
  Core: Require all actions have a Task (#31627)
2018-06-29 10:12:03 -04:00
Tanguy Leroux 8fa06294ab
Do not check for object existence when deleting repository index files (#31680)
Before deleting a repository index generation file, BlobStoreRepository
checks for the existence of the file and then deletes it. We can save
a request here by using BlobContainer.deleteBlobIgnoringIfNotExists()
 which ignores error when deleting a file that does not exist.

Since there is no way with S3 to know if a non versioned file existed
before being deleted, this pull request also changes S3BlobContainer so
that it now implements deleteBlobIgnoringIfNotExists(). It will now save
 one more request (blobExist?) when appropriate. The tests and fixture
 have been modified to conform the S3 API that always returns a 204/NO
 CONTENT HTTP response on deletions.
2018-06-29 14:15:34 +02:00
Nhat Nguyen 573df2dd2e
Relax TranslogWriter#assertNoSeqNumberConflict (#31569)
If the recovery and indexing are concurrently happening, it's possible
for a replica to receive the same operation twice: one from the
replication, and the other from the recovery. However, these operations
are not identical because we don't store the versionType of operations
in the Lucene index.

The TranslogWriter#assertNoSeqNumberConflict assertion has been tripped
several times in the CCR branch since we use Lucene in peer-recovery.
This commit relaxes that assertion by excluding the versionType from the
check.
2018-06-28 17:42:14 -04:00
Jack Conradson 02c01cb4cf
Fix CreateSnapshotRequestTests Failure (#31630)
Original test failure found here in issue (#31625). Had to rework the tests to only include options available externally for create snapshot requests.
2018-06-28 12:12:55 -07:00
Ioannis Kakavas db6b33978e
Configurable password hashing algorithm/cost (#31234)
Make password hashing algorithm/cost configurable for the 
stored passwords of users for the realms that this applies
(native, reserved). Replaces predefined choice of bcrypt with
cost factor 10.
This also introduces PBKDF2 with configurable cost
(number of iterations) as an algorithm option for password hashing
both for storing passwords and for the user cache.
Password hash validation algorithm selection takes into
consideration the stored hash prefix and only a specific number
of algorithnm and cost factor options for brypt and pbkdf2 are 
whitelisted and can be selected in the relevant setting.
2018-06-28 19:39:39 +03:00
Ryan Ernst f924835265
Core: Require all actions have a Task (#31627)
The TaskManager and TaskAwareRequest could return null when registering
a task according to their javadocs, but no implementations ever actually
did that. This commit removes that wording from the javadocs and ensures
null is no longer allowed.
2018-06-28 08:24:03 -07:00
Nhat Nguyen 2c10cd3e1d Merge branch 'master' into ccr
* master:
  Docs: Remove duplicate test setup
  Print output when the name checker IT fails (#31660)
  Fix syntax errors in get-snapshots docs (#31656)
  Docs: Fix description of percentile ranks example example (#31652)
  Add MultiSearchTemplate support to High Level Rest client (#30836)
  Add test for low-level client round-robin behaviour (#31616)
  SQL: Refactor package names of sql-proto and sql-shared-proto projects (#31622)
  Remove deprecation warnings to prepare for Gradle 5 (sourceSets.main.output.classesDirs) (#30389)
  Correct integTest enable logic (#31646)
  Fix missing get-snapshots docs reference #31645
  Do not check for Azure container existence (#31617)
  Merge AwsS3Service and InternalAwsS3Service in a S3Service class (#31580)
  Upgrade gradle wrapper to 4.8 (#31525)
  Only set vm.max_map_count if greater than default (#31512)
  Add Get Snapshots High Level REST API (#31537)
  QA: Merge query-builder-bwc to restart test (#30979)
  Update reindex.asciidoc (#31626)
  Docs: Skip xpack snippet tests if no xpack (#31619)
  mute CreateSnapshotRequestTests
  HLRest: Fix test for explain API
  [TEST] Fix RemoteClusterConnectionTests
  Add Create Snapshot to High-Level Rest Client (#31215)
  Remove legacy MetaDataStateFormat (#31603)
  Add explain API to high-level REST client (#31387)
  Preserve thread context when connecting to remote cluster (#31574)
  Unify headers for full text queries
  Remove redundant 'minimum_should_match'
  JDBC driver prepared statement set* methods  (#31494)
  [TEST] call yaml client close method from test suite (#31591)
2018-06-28 11:13:01 -04:00
Nhat Nguyen e4315cc710
Use external version type for history ops from Lucene (#31568)
Today we use INTERNAL version type for the history operations from
Lucene index. However, this does not cover the case in which the
original operation has the version type EXTERNAL and version number 0.
Semantically all operations from translog or Lucene should always use
EXTERNAL version type.
2018-06-28 09:51:40 -04:00
markharwood 09dd19a403
Add MultiSearchTemplate support to High Level Rest client (#30836)
Add MultiSearchTemplate support to High Level Rest client.
Addresses part of #27205
2018-06-28 14:05:26 +01:00
Alpar Torok 0afec8f31c
Remove deprecation warnings to prepare for Gradle 5 (sourceSets.main.output.classesDirs) (#30389)
* Remove deprecation warnings to prepare for Gradle 5

Gradle replaced `project.sourceSets.main.output.classesDir` of type
`File` with `project.sourceSets.main.output.classesDirs` of type
`FileCollection`
(see [SourceSetOutput](https://github.com/gradle/gradle/blob/master/subprojects/plugins/src/main/java/org/gradle/api/tasks/SourceSetOutput.java))
Build output is now stored on a per language folder.

There are a few places where we use that, here's these and how it's
fixed:

- Randomized Test execution
    - look in all test folders ( pass the multi dir configuration to the
    ant runner )
    - DRY the task configuration by introducing `basedOn` for
      `RandomizedTestingTask` DSL
- Extend the naming convention test to support passing in multiple
  directories
- Fix the standalon test plugin, the dires were not passed trough,
  checked with a debuger and the statement had no affect due to a
  missing `=`.

Closes #30354

* Only check Java tests, PR feedback

- Name checker was ran for Groovy tests that don't adhere to the same
  convections causing the check to fail
- implement PR feedback

* Replace `add` with `addAll`

This worked because the list is passed to `project.files` that does the
right thing.

* Revert "Only check Java tests, PR feedback"

This reverts commit 9bd9389875d8b88aadb50df57a45cd0d2b073241.

* Remove `basedOn` helper

* Bring some changes back

Previus revert accidentally reverted too much

* Fix negation

* add back public

* revert name check changes

* Revert "revert name check changes"

This reverts commit a2800c0b363168339ea65e2a79ec8256e5883e6d.

* Pass all dirs to name check

Only run on Java for build-tools, this is safe because it's a self test.
It needs more work before we could pass in the Groovy classes as well as
these inherit from `GroovyTestCase`

* remove self tests from name check

The self complicates the task setup and disable real checks on
build-tools.
With this change there are no more self tests, and the build-tools tests
adhere to the conventions.
The self test will be replaced by gradle test kit, thus the addition of
the Gradle plugin builder plugin.

* First test to run a Gradle build

* Add tests that replace the name check self test

* Clean up integ test base class

* Always run tests

* Align with test naming conventions

* Make integ. test case inherit from unit test case

The check requires this

* Remove `import static org.junit.Assert.*`
2018-06-28 15:14:34 +03:00
Alpar Torok 8557bbab28
Upgrade gradle wrapper to 4.8 (#31525)
* Move to Gradle 4.8 RC1

* Use latest version of plugin

The current does not work with Gradle 4.8 RC1

* Switch to Gradle GA

* Add and configure build compare plugin

* add work-around for https://github.com/gradle/gradle/issues/5692

* work around https://github.com/gradle/gradle/issues/5696

* Make use of Gradle build compare with reference project

* Make the manifest more compare friendly

* Clear the manifest in compare friendly mode

* Remove animalsniffer from buildscript classpath

* Fix javadoc errors

* Fix doc issues

* reference Gradle issues in comments

* Conditionally configure build compare

* Fix some more doclint issues

* fix typo in build script

* Add sanity check to make sure the test task was replaced

Relates to #31324. It seems like Gradle has an inconsistent behavior and
the taks is not always replaced.

* Include number of non conforming tasks in the exception.

* No longer replace test task, create implicit instead

Closes #31324. The issue has full context in comments.

With this change the `test` task becomes nothing more than an alias for `utest`.
Some of the stand alone tests that had a `test` task now have `integTest`, and a
few of them that used to have `integTest` to run multiple tests now only
have `check`.
This will also help separarate unit/micro tests from integration tests.

* Revert "No longer replace test task, create implicit instead"

This reverts commit f1ebaf7d93e4a0a19e751109bf620477dc35023c.

* Fix replacement of the test task

Based on information from gradle/gradle#5730 replace the task taking
into account the task providres.
Closes #31324.

* Only apply build comapare plugin if needed

* Make sure test runs before integTest

* Fix doclint aftter merge

* PR review comments

* Switch to Gradle 4.8.1 and remove workaround

* PR review comments

* Consolidate task ordering
2018-06-28 08:13:21 +03:00
Tim Brooks 9ac81a1322
Add Get Snapshots High Level REST API (#31537)
With this commit we add the get snapshots API to the Java high level
REST client.

Relates #27205
2018-06-27 18:04:34 -06:00
Vladimir Dolzhenko a368195965 mute CreateSnapshotRequestTests 2018-06-27 21:36:28 +02:00
Yannick Welsch 7520025d7d [TEST] Fix RemoteClusterConnectionTests
With PR #31574 we now ensure that connections are established under the proper
thread context. There is a test in RemoteClusterConnectionTests, however,
that shuts down the service while connecting. With the above change, a new kind
of exception can occur that the test is unaware of.
2018-06-27 18:57:15 +02:00
Jack Conradson 61eefc84f3
Add Create Snapshot to High-Level Rest Client (#31215)
Added support to the high-level rest client for the create snapshot API call. This required 
several changes to toXContent which may need to be cleaned up in a later PR. Also 
added several parsers for fromXContent to be able to retrieve appropriate responses 
along with tests.
2018-06-27 09:30:10 -07:00
Yannick Welsch 01623f66de
Remove legacy MetaDataStateFormat (#31603)
Removes the legacy (pre-1.5) legacy MetaDataStateFormat.
2018-06-27 17:36:58 +02:00
Yu 616703b880 Add explain API to high-level REST client (#31387)
Relates to #27205
2018-06-27 15:20:56 +02:00
Yannick Welsch b7246199db
Preserve thread context when connecting to remote cluster (#31574)
Establishing remote cluster connections uses a queue to coordinate multiple concurrent connect
attempts. Connect attempts can be initiated by user triggered searches as well as by system events 
(e.g. when nodes disconnect). Multiple such concurrent events can lead to the connectListener of
one event to be called under the thread context of another connect attempt. This can lead to the
situation as seen in #31462 where the connect listener is executed under the system context, which 
breaks when fetching the search shards from the remote cluster.

Closes #31462
2018-06-27 10:46:42 +02:00
Nhat Nguyen a55f614b85 Merge branch 'master' into ccr
* master:
  ingest: Add ignore_missing property to foreach filter (#22147) (#31578)
  Fix a formatting issue in the docvalue_fields documentation. (#31563)
  reduce log level at gradle configuration time
  [TEST] Close additional clients created while running yaml tests (#31575)
  Docs: Clarify sensitive fields watcher encryption (#31551)
  Watcher: Remove never executed code (#31135)
  Add support for switching distribution for all integration tests (#30874)
  Improve robustness of geo shape parser for malformed shapes (#31449)
  QA: Create xpack yaml features (#31403)
  Improve test times for tests using `RandomObjects::addFields` (#31556)
  [Test] Add full cluster restart test for Rollup (#31533)
  Enhance thread context uniqueness assertion
  [DOCS] Fix heading format errors (#31483)
  fix writeIndex evaluation for aliases (#31562)
  Add x-opaque-id to search slow logs (#31539)
  Watcher: Fix put watch action (#31524)
  Add package pre-install check for java binary (#31343)
  Reduce number of raw types warnings (#31523)
  Migrate scripted metric aggregation scripts to ScriptContext design (#30111)
  turn GetFieldMappingsResponse to ToXContentObject (#31544)
  Close xcontent parsers (partial) (#31513)
  Ingest Attachment: Upgrade Tika to 1.18 (#31252)
  TEST: Correct the assertion arguments order (#31540)
2018-06-26 16:25:32 -04:00
Igor Motov 544e473908
Improve robustness of geo shape parser for malformed shapes (#31449)
Ensures that malformed geoshapes are more reliably ignored if
"ignore_malformed" is set to true instead of failing the entire
document.

Closes #31428
2018-06-26 06:44:10 -07:00
Sohaib Iftikhar ca4c857a90 Improve test times for tests using `RandomObjects::addFields` (#31556)
Currently RandomObjects::addFields can potentially generate a large number of fields This commit decreases the chances that a new object or array is added as a new branch of an object, which lowers the probability of ending up with very big documents generated. It also reduces the number of documents generated for the SimulatePipelineResponseTests from 10 to 5 to reduce the testing time required for parsing.
2018-06-26 12:39:53 +02:00
Jason Tedor 424be02728
Enhance thread context uniqueness assertion
This commit enhances the assertion message when de-duplicating values on
a thread context header so that if this assertion trips we can see the
values and their correpsonding unique values to understand why this
assertion might trip.
2018-06-26 00:01:45 -04:00
Tal Levy c6012ed575
fix writeIndex evaluation for aliases (#31562)
AliasOrIndex.Alias#writeIndex was returning a write index when
an alias was pointing to only one index, regardless whether `is_write_index` was
set to `false`. This fixes that so that there is no write index in such a case
that an alias points to only one index with `is_write_index=false`.
2018-06-25 17:17:16 -07:00
Igor Motov 237650e9c0
Add x-opaque-id to search slow logs (#31539)
Add x-opaque-id to search slow logs only. Indexing slow log and audit
logs will be handled as separate PRs.

Relates #31521
2018-06-25 12:20:27 -07:00
Christoph Büscher 86ab3a2d1a
Reduce number of raw types warnings (#31523)
A first attempt to reduce the number of raw type warnings, 
most of the time by using the unbounded wildcard.
2018-06-25 15:59:03 +02:00