Commit Graph

45265 Commits

Author SHA1 Message Date
Andy Bristol 23395a9b9f
search as you type fieldmapper (#35600)
Adds the search_as_you_type field type that acts like a text field optimized
for as-you-type search completion. It creates a couple subfields that analyze
the indexed terms as shingles, against which full terms are queried, and a
prefix subfield that analyze terms as the largest shingle size used and
edge-ngrams, against which partial terms are queried

Adds a match_bool_prefix query type that creates a boolean clause of a term
query for each term except the last, for which a boolean clause with a prefix
query is created.

The match_bool_prefix query is the recommended way of querying a search as you
type field, which will boil down to term queries for each shingle of the input
text on the appropriate shingle field, and the final (possibly partial) term
as a term query on the prefix field. This field type also supports phrase and
phrase prefix queries however
2019-03-27 13:29:13 -07:00
Andy Bristol c0c6d702a2
ignore 409 conflict in reindex responses (#39543)
The reindex family of APIs (reindex, update-by-query, delete-by-query) can
sometimes return responses that have an error status code (409 Conflict in this
case) but still have a body in the usual BulkByScrollResponse format. When the
HLRC tries to handle such responses, it blows up because it tris to parse it
expecting the error format that errors in general use. This change prompts the
HLRC to parse the response using the expected BulkByScrollResponse format.
2019-03-27 13:27:17 -07:00
Benjamin Trent 6563dc7ed9
Muting test for #40553 (#40555) 2019-03-27 14:52:12 -05:00
Tim Brooks ab44f5fd5d
Add InboundHandler for inbound message handling (#40430)
This commit adds an InboundHandler to handle inbound message processing.
With this commit, this code is moved out of the TcpTransport.
Additionally, finer grained unit tests are added to ensure that the
inbound processing works as expected
2019-03-27 12:33:26 -06:00
Julie Tibshirani 419cf1c02f Fix an off-by-one error in the vector field dimension limit. (#40489)
Previously only vectors up to 499 dimensions were accepted, whereas the stated
limit is 500.
2019-03-27 11:17:58 -07:00
Yannick Welsch 64b31f44af No mapper service and index caches for replicated closed indices (#40423)
Replicated closed indices can't be indexed into or searched, and therefore don't need a shard with
full indexing and search capabilities allocated. We can save on a lot of heap memory for those
indices by not allocating a mapper service and caching infrastructure (which preallocates a constant
amount per instance). Before this change, a 1GB ES instance could host 250 replicated closed
metricbeat indices (each index with one shard). After this change, the same instance can host 7300
replicated closed metricbeat instances (not that this would be a recommended configuration). Most
of the remaining memory is in the cluster state and the IndexSettings object.
2019-03-27 19:04:24 +01:00
Yannick Welsch 8f7c5732f1 Use default discovery implementation for single-node discovery (#40036)
Switches "discovery.type: single-node" from using a separate implementation for single-node discovery to using the existing standard discovery implementation, with two small adaptions:

-  auto-bootstrapping, but requiring initial_master_nodes not to be set.
- not actively pinging other nodes using the Peerfinder
- not allowing other nodes to join its single-node cluster (if they have e.g. been set up using regular discovery and connect to the single-disco node).
2019-03-27 19:04:24 +01:00
Tim Brooks 3860ddd1a4
Move outbound message handling to OutboundHandler (#40336)
Currently there are some components of message serializer and sending
that still occur in TcpTransport. This commit makes it possible to
send a message without the TcpTransport by moving all of the remaining
application logic to the OutboundHandler. Additionally, it adds unit
tests to ensure that this logic works as expected.
2019-03-27 11:47:36 -06:00
David Kyle 13d4d73ce3 Mute DataFrameTaskFailedStateIT.testFailureStateInteraction (#40544) 2019-03-27 17:39:44 +00:00
Benjamin Trent 95a0c524a1
Muting test #40368 (#40542) 2019-03-27 12:11:10 -05:00
Benjamin Trent be67752c34
Muting test related to #40537 (#40539) 2019-03-27 11:47:09 -05:00
Marios Trivyzas 8e049c5f58
SQL: Fix getTime() methods in JDBC (#40484)
Previously, `getTime(colIdx/colLabel)` and
`getObject(colIdx/colLabel, java.sql.Time.class)` methods were computing
the time from a `ZonedDateTime` by applying day in millis modulo on the epoch millis
of the `ZonedDateTime` object. This is wrong as we need to keep the time
related fields at the timezone of the `ZonedDateTime` object and just
set the date info to the epoch date (01/01/1970).

Additionally fixes a testing issue as the original timezone id is converted
to an offset string when parsing the response from the server.
2019-03-27 17:14:08 +01:00
David Turner 707d40ce06 Stabilise testStaleMasterNotHijackingMajority (#40253)
This test inadvertently asserts that the election occurs after a master failure
is clean. However, messy elections are a fact of life so we should not fail on
a messy election.

This change moves this test away from an `AbstractDisruptionTestCase` since it
does not need the fault detector to be so enthusiastic, and weakens the
assertions to merely say that we ignore states published by the old master
without saying anything about the cleanliness of the election.

Closes #36556
2019-03-27 16:00:14 +00:00
David Turner f1eca21081
Mention legacy discovery settings in migration doc (#40528)
Today the upgrade assistant identifies that discovery configuration is required
in production mode, but links to docs saying to add one of these settings:

- `discovery.seed_hosts`
- `discovery.seed_providers`
- `cluster.initial_master_nodes`

However these settings do not exist in 6.7 so this is unhelpful advice. This
commit adjusts the docs to give more useful advice to users arriving at this
page from the upgrade assistant.

Relates https://discuss.elastic.co/t/174102
2019-03-27 15:56:02 +00:00
James Rodewig 9a110c793b Merge remote-tracking branch 'jrodewig/7.x' into 7.x 2019-03-27 11:39:22 -04:00
Andrei Stefan 99dca30197 SQL: MATCH and QUERY documentation; one list of functions (#40494)
* Document MATCH and QUERY function predicates.
* Polish the functions pages and add a list of functions to the main Functions & Operators page.

(cherry picked from commit 4cec0ae1b962ec7ea011a290aec72740386eb808)
2019-03-27 17:37:10 +02:00
Tim Brooks 760cfffe4b
Move TransportMessageListener to TransportService (#40474)
Currently the TransportMessageListener is applied and used in the
Transport class. However, local requests and responses never make it to
this class. This PR moves the listener add/remove methods to the
TransportService. After this change the Transport can only have one
listener set with it. This one listener is the TransportService, which
will then propogate the events to the external listeners.

Additionally this commit back ports #40237

Remove Tracer from MockTransportService

Currently the TransportMessageListener is applied and used in the
Transport class. However, local requests and responses never make it to
this class. This PR moves the listener add/remove methods to the
TransportService. After this change the Transport can only have one
listener set with it. This one listener is the TransportService, which
will then propogate the events to the external listeners.
2019-03-27 09:24:20 -06:00
Sébastien Loix 59c98fbb66 [Docs] Fix command to save SMTP password for email account (#40444) 2019-03-27 10:36:53 -04:00
James Rodewig 03aaeb35cc Deprecate elasticsearch.yml as supported Slack config method (#40410) 2019-03-27 10:32:25 -04:00
Like d57b7a2d5b Migrate Streamable to Writeable for WatchStatus (#37390) 2019-03-27 07:18:35 -07:00
Benjamin Trent 22e688a709
Muting watcher tests for issue #35503 (#40527)
* Muting watcher tests for issue #35503

* blocking the two troublsome suites
2019-03-27 09:16:06 -05:00
James Rodewig cfe85c73bf Fix typo in rollup_index definition (#40520) 2019-03-27 10:04:49 -04:00
Bogdan Pintea 6182b464a3 SQL: ODBC: document extra connection string parameters (#40476)
* document ODBC's extra connection string parameters

Document the connection string parameters that are currently not
configurable from the GUI.

Add a note about SmartScreen possible warning on driver MSI
installation.

Add a note about the TLS certificate file not being supported as bundled
or password-protected.

* rephrasing and restructuring the list of params

- addressing PR review notes

* rephrasings and reference linking

- addressing PR review notes

(cherry picked from commit ca90190226c0a8c63f51c2603b27df299ce8d503)
2019-03-27 14:20:23 +01:00
Boaz Leskes 6efe748e2e Only recommend disabling allocation of replicas in a rolling upgrade (#40299)
The current documentation recommends disabling allocations of all shards. This prevents shards of
new indices from being allocated as well. That means that during a rolling upgrade, operations like
roll over and index shrinking will operate correctly. This, in turn, can cause issues for data ingestion
and ILM.
2019-03-27 14:18:17 +01:00
David Kyle 61845dd38b [ML] Fix serialisation of Start Data Frame request (#40483) 2019-03-27 12:56:34 +00:00
Przemyslaw Gomulka 65f01277ed
Parse composite patterns using ClassicFormat.parseObject backport(#40100) (#40501)
Java-time fails parsing composite patterns when first pattern matches only the prefix of the input. It expects pattern in longest to shortest order. Because of this constructing just one DateTimeFormatter with appendOptional is not sufficient. Parsers have to be iterated and if the parsing fails, the next one in order should be used. In order to not degrade performance parsing should not be throw exceptions on failure. Format.parseObject was used as it only returns null when parsing failed and allows to check if full input was read.
 
closes #39916
backport #40100
2019-03-27 13:51:44 +01:00
David Kyle c990b30019
[ML] Data Frame HLRC Get API (#40509) 2019-03-27 12:40:39 +00:00
Yannick Welsch b4b17e16e0 Remove TransportSingleItemBulkWriteAction as replication action (#40424)
The implementation of TransportIndexAction and TransportDeleteAction as
TransportReplicationAction existed for interoperability with older 5.x nodes, as these older nodes
coordinated single index / deletes as replication requests. This BWC layer is no longer needed in 7.x,
where these single actions are now mapped to bulk requests. Completely removing the deprecated
transport actions is not possible yet if we want to keep BWC with a 6.x transport client. The best
way here is to wait for the transport client to go away and then just remove the actions.
2019-03-27 13:16:58 +01:00
Dimitrios Liappis 856789f791
Wrap Dockerfile yum operations in retries (#40460)
While yum does retry retrieving files 10 times by default [1], slow
network fetches, governed by `minrate` cause immediate aborts without
getting retried.

Wrap yum commands in a 10 iteration retry loop.

[1] http://man7.org/linux/man-pages/man5/yum.conf.5.html

Backport of #40349
2019-03-27 14:14:47 +02:00
Benjamin Trent 12943c5d2c
[ML] Add data frame task state object and field (#40169) (#40490)
* [ML] Add data frame task state object and field

* A new state item is added so that the overall task state can be
accoutned for
* A new FAILED state and reason have been added as well so that failures
can be shown to the user for optional correction

* Addressing PR comments

* adjusting after master merge

* addressing pr comment

* Adjusting auditor usage with failure state

* Refactor, renamed state items to task_state and indexer_state

* Adding todo and removing redundant auditor call

* Address HLRC changes and PR comment

* adjusting hlrc IT test
2019-03-27 06:53:58 -05:00
Costin Leau 38dc005b8d SQL: Update JDBC class name in client screenshots
(cherry picked from commit 79a3b93deb858961ecbe4516ed9d8f209016666f)
2019-03-27 12:21:53 +02:00
Costin Leau 33737b6b21 SQL: Polish parsing of CAST expression (#40428)
(cherry picked from commit 9d291aa300bbb827eeae606e7d3e55eeef7cce00)
2019-03-27 12:20:58 +02:00
Jim Ferenczi fe05a4d511 Fix random failures in SearchResponseMerger#testMergeSearchHits (#40223)
This commit fixes the expectation in the test when the search hits are empty.

Closes #40214
2019-03-27 11:17:10 +01:00
Hendrik Muhs f4e56118c2 [ML] generate unique doc ids for data frame (#40382)
create and use unique, deterministic document ids based on the grouping values.

This is a pre-requisite for updating documents as well as preventing duplicates after a hard failure during indexing.
2019-03-27 08:27:05 +01:00
Alpar Torok 524e0273ae Testclusters: convert plugin repository-s3 (#40399)
* Add support for setting and keystore settings
* system properties and env var config
* use testclusters for repository-s3
* Some cleanup of the build.gradle file for plugin-s3
* add runner {} to rest integ test task
2019-03-27 08:40:16 +02:00
Ryan Ernst aa24669e95
Resolve JAVA_HOME at windows service install time (#39714)
On windows, JAVA_HOME is currently resolved when the windows service is started. However, this is contrary to what our documentation states. This commit moves resolution to service install. This has the side effect of making java existence checking optional in elasticsearch-env.bat, since the rest of the service commands do not require java.

closes #30720
2019-03-26 19:48:02 -07:00
Ryan Ernst f82f9423b7 Remove platform test command (#40280)
The platformTest gradle task was a packaging test meant to ensure unit
tests run on the various supported operating systems without relying on
CI to maintain a full matrix of platforms. Howevever, it never really
worked out as intended and is now additional code in our vagrant setup
to maintain. This commit removes the platformTest task.
2019-03-26 19:47:33 -07:00
lcawl 81510746d6 [DOCS] Adds anchors and x-pack icons for frozen indices 2019-03-26 15:53:18 -07:00
Jason Tedor cbdb20faba
Add more Java 12 distributions to testing (#40482)
This commit adds the Oracle JDK 12 and the Zulu JDK 12 distributions to
testing.
2019-03-26 18:48:43 -04:00
Julie Tibshirani 25954a8dd3 Stop clearing all watches in watcher integration tests. (#39724) 2019-03-26 13:14:33 -07:00
Ryan Ernst 562ebaefa6 Delete jdk extract dir before extracting (#40471)
This commit first deletes the directory we extract the bundled jdk into in order to make gradle's copy task happy.
2019-03-26 11:41:37 -07:00
Ryan Ernst e49ba6a9d9 Remove unused variable (#40468)
This was leftover from previous iterations of test clusters refactorings
2019-03-26 11:26:31 -07:00
alex101101 fb8ad0cf30 Add a soft limit to the field name length (#40309)
Adds an optional limit to the length of field names, throws an IllegalArgumentException if the limit is breached. 
Closes #33651
2019-03-26 17:58:32 +01:00
Daniel Mitterdorfer f2b5960f90 Add version 6.7.1 2019-03-26 17:38:14 +01:00
debadair 324a5400a4 [DOCS] Switched release state back to unreleased. Closes #40259 (#40470) 2019-03-26 09:20:41 -07:00
Yannick Welsch bf7b167bba Remove timeout task after completing cluster state publication (#40411)
Each cluster state publication schedules a cancellation task with the provided publication timeout
(30s by default). This scheduled cancellation keeps a reference to the publication, and therefore the
full cluster state that was published. In case of frequently updating a large cluster state, this results
in a large number of cancellation tasks keeping references to all previously published cluster states.
2019-03-26 17:13:57 +01:00
Jay Modi 9bd8600c2e
Use ephemeral ports for idp-fixture (#40333)
This change removes the use of hardcoded port values for the
idp-fixture in favor of the mapped ephemeral ports. This should prevent
failures due to port conflicts in CI.
2019-03-26 08:44:53 -06:00
Henning Andersen bf444b9f02 Store Pending Deletions Fix (#40345)
FilterDirectory.getPendingDeletions does not delegate, fixed
temporarily by overriding in StoreDirectory.

This in turn caused duplicate file name use after a trimUnsafeCommits
had been done, since a new IndexWriter would not consider the pending
deletes in IndexFileDeleter. This should only happen on windows (AFAIK).

Reenabled doing index updates for all tests using
IndexShardTests.indexOnReplicaWithGaps (which could fail due to above
when using mocked WindowsFS).

Added getPendingDeletions delegation to all elasticsearch
FilterDirectory subclasses that were not trivial test-only overrides to
minimize the risk of hitting this issue in another case.
2019-03-26 15:30:44 +01:00
Alpar Torok b5a8de9a7f Revert tmpdir (#40312)
* Revert "Configure TMP for test nodes on Windows (#39959)"
This reverts commit 97562a874fcb1f29fb05272ab860a0307e97d1aa.

* Configure a tmp dir without spaces
* Pass on TMP instead of changing it
2019-03-26 13:31:18 +02:00
Alpar Torok d50305bbee Upgrade to Gradle 5.3! (#40346)
Here are the highlights of this release:
 - Feature variants AKA "optional dependencies"
 - Type-safe accessors in Kotlin precompiled script plugins
 - Gradle Module Metadata 1.0

For more details see https://docs.gradle.org/5.3/release-notes.html
2019-03-26 13:23:40 +02:00