Commit Graph

2069 Commits

Author SHA1 Message Date
Julie Tibshirani fba710469a
Refactor the REST actions to clarify what endpoints are deprecated. (#36869) 2018-12-20 18:06:41 -08:00
Tim Brooks d9b2ed6135
Send clear session as routable remote request (#36805)
This commit adds a RemoteClusterAwareRequest interface that allows a
request to specify which remote node it should be routed to. The remote
cluster aware client will attempt to route the request directly to this
node. Otherwise it will send it as a proxy action to eventually end up
on the requested node.

It implements the ccr clean_session action with this client.
2018-12-20 17:43:12 -07:00
David Turner d6d5134890
Fix name of SettingsBasedHostsProviderIT (#36778)
It is not a _host_ provider because it provides more than one host.
2018-12-20 17:11:12 +00:00
David Turner 52d34e45e7
[Zen2] Minor logging improvements (#36818)
* Adds term number and greppable phrase 'coordinator becoming' to Coordinator
  mode changes
* Adds term and version to messages from the ClusterApplier about master
  changes
* Reduces some LeaderChecker messages to TRACE level
2018-12-20 15:31:52 +00:00
Andrey Ershov ca92d74e7e
[Zen2] Change unsafe bootstrap nodes count to nodes list in tests (#36559)
This commit modifies ESSingleNodeTestCase and ESIntegTestCase and
several concrete test classes to use node names when bootstrapping the
cluster.

Today ClusterBootstrapService.INITIAL_MASTER_NODE_COUNT_SETTING
setting is used to bootstrap clusters in tests. Instead, we want to use
ClusterBootrstapService.INITIAL_MASTER_NODES_SETTING and get rid of
the former setting eventually.

There were two main problems when refactoring InternalTestCluster:

1. Nodes are created one-by-one in buildNode method. And node.name
is created in this method as well. It's not suitable for bootstrapping,
because we need to have the names of all master eligible nodes in
advance, before creating the node with bootstrapping configuration set.
We address this issue by separating buildNode into two methods:
getNodeSettings and buildNode. We first iterate over all nodes to
get nodes settings, then change the setting for the bootstrapping node
and then proceed with building the node.
2. If autoManageMinMasterNodes = false, there is no way for the test to
set the list of bootstrapping nodes because node names are not known in
advance. This problem is solved by adding updateNodesSettings method
to NodeConfigurationSource and ESIntegTestCase (which could be
overridden by concrete integration test class). Once we have the list
of settings for all nodes, the integration test class is allowed to
update it. In our case, we update the
ClusterBootrstapService.INITIAL_MASTER_NODES_SETTING setting.
2018-12-20 15:20:33 +01:00
Dimitris Athanasiou 08bcd83757
[ML] Reduce persistent tasks periodic reassignment interval in ... (#36845)
... MlDistributedFailureIT.testLoseDedicatedMasterNode.

An intermittent failure has been observed in
`MlDistributedFailureIT. testLoseDedicatedMasterNode`.
The test launches a cluster comprised by a dedicated master node
and a data and ML node. It creates a job and datafeed and starts them.
It then shuts down and restarts the master node. Finally, the test asserts
that the two tasks have been reassigned within 10s.

The intermittent failure is due to the assertions that the tasks have been
reassigned failing. Investigating the failure revealed that the `assertBusy`
that performs that assertion times out. Furthermore, it appears that the
job task is not reassigned because the memory tracking info is stale.

Memory tracking info is refreshed asynchronously when a job is attempted
to be reassigned. Tasks are attempted to be reassigned either due to a relevant
cluster state change or periodically. The periodic interval is controlled by a cluster
setting called `cluster.persistent_tasks.allocation.recheck_interval` and defaults to 30s.

What seems to be happening in this test is that if all cluster state changes after the
master node is restarted come through before the async memory info refresh completes,
then the job might take up to 30s until it is attempted to reassigned. Thus the `assertBusy`
times out.

This commit changes the test to reduce the periodic check that reassigns persistent
tasks to `200ms`. If the above theory is correct, this should eradicate those failures.

Closes #36760
2018-12-20 14:53:36 +02:00
Ryan Ernst cfc0a47232
Core: Deprecate negative epoch timestamps (#36793)
Negative timestamps are currently supported in joda time. These are
dates before epoch. However, it doesn't really make sense to have a
negative timestamp, since this is a modern format. Any dates before
epoch can be represented with normal date formats, like ISO8601.
Additionally, implementing negative epoch timestamp parsing in java time
has an edge case which would more than double the code required. This
commit deprecates use of negative epoch timestamps.
2018-12-20 00:17:06 -08:00
Tim Brooks a7f344cc7f
Use ByteBuffer#slice in BytesReference wrapper (#36862)
Currently the ByteBufferReference does not duplicate the buffer.
This means that any changes to the buffer's limit or position will
impact the reference. This can lead to unexpected behavior. This commit
uses the ByteBuffer#slice method to ensure that the reference retains
its own ByteBuffer.
2018-12-19 16:58:43 -07:00
Julie Tibshirani ecb822c666
Deprecate the document create endpoint. (#36863) 2018-12-19 15:20:20 -08:00
Mayya Sharipova 9c1e47d434 Add 6.5.5 version constants 2018-12-19 17:02:14 -05:00
Julie Tibshirani 32ef80f3d4
Avoid duplicate types deprecation messages in search-related APIs. (#36802) 2018-12-19 12:59:25 -08:00
David Causse 3412627efe QueryRescorer should keep the window size when rewriting (#36836)
This attribute being controlled by the parent class it's easy to miss
it during rewrites.
2018-12-19 21:42:18 +01:00
Ryan Ernst c85c16bd94
Core: Revert back to joda's multi date formatters (#36814)
This commit partially reverts #36447 by using the ability of Joda time's
DateTimeFormatterBuilder to append multiple parsers instead of using the
MergedDateFormatter. The MergedDateFormatter will be removed in a future
change, as it is not as performant due to creating potentially many
exceptions during heavy date parsing. This change is a stop-gap until
that followup is ready.

closes #36602
2018-12-19 11:20:35 -08:00
Christoph Büscher 9058698d9d
Removing some deprecated methods (#36829)
Changes:
* Removed deprecated method in InnerHitBuilder
* Removed fields() from SearchRequestBuilder
* Removed deprecated GeoDistanceSortBuilder#geohashes
2018-12-19 18:45:43 +01:00
Christoph Büscher de052ae533
Bump laster 6.x version in VersionTests (#36851) 2018-12-19 18:19:16 +01:00
Nhat Nguyen b63f9b967c
Fix translog bwc serialization (#36676)
Serializing of a Translog#Index from v7.0.0 to 6.x is broken since
#29224 where we removed the _parent field.

Relates #29224
2018-12-19 11:19:25 -05:00
Ignacio Vera a34a3532ce
[Javadoc]: Remove lucene tags (#36834)
Remove lucene tags as they break gradle javadoc task

Relates #36794
2018-12-19 15:29:09 +01:00
Jason Tedor f2a5373495
Add the 6.7.0 constant to the master branch
Now that the 6.x branch has been bumped to the 6.7.0 version, this
commit adds knowledge of the 6.7.0 version to the master branch.
2018-12-19 08:24:10 -05:00
Yannick Welsch 8f141b8a41
Fix ClusterInfoServiceIT timeouts (#36758)
The test testClusterInfoServiceInformationClearOnError relies on timing behavior. It sets
InternalClusterInfoService.INTERNAL_CLUSTER_INFO_TIMEOUT_SETTING to 1s and relies on the
fact that the stats request completes within that timeframe (which our ever-so-slow CI seems to
violate at times). Unfortunately the logging has been misimplemented in InternalClusterInfoService,
so the corresponding log messages showing that the requests have timed out are missing for this.
The issue can be locally reproduced by reducing the timeout to something lower.

Closes #36554
2018-12-19 13:59:58 +01:00
David Turner 3f8f907606
Extend time allowed to detect disconnection (#36827)
If the master sends both its follower checks just before disconnection then
neither will receive a response, meaning it must wait for the checks to time
out and send another in order to detect the disconnection and stand down.

Closes #36788
2018-12-19 12:51:14 +00:00
Yannick Welsch 487a1c4f71
Fix cluster state persistence for single-node discovery (#36825)
Single-node discovery is not persisting cluster states, which was caused by a recent 7.0-only
refactoring. This commit ensures that the cluster state is properly persisted when using single-node
discovery and adds a corresponding test.
2018-12-19 13:26:04 +01:00
Boaz Leskes 216b154107
Replace 0L with an UNASSIGNED_PRIMARY_TERM constant (#36819)
* Replace 0L with an UNASSIGNED_PRIMARY_TERM constant

0 is an illegal value for a primary term that is often used to indicate
the primary term isn't assigned or is irrelevant. This PR replaces the
usage of 0 with a constant, to improve readability and so it can be
tracked and if needed, replaced.

* feedback
2018-12-19 13:15:05 +01:00
Armin Braun 978713a67c
SNAPSHOTS+TESTS: Correctly Wait for Clean State (#36801)
* Test must wait until there is no in-progress deletion as well here since the master failover leads to a snapshot deletion
* Closes #36779
2018-12-19 13:14:36 +01:00
Alan Woodward 344917efab
Add script filter to intervals (#36776)
This commit adds the ability to filter out intervals based on their start and end position, and internal
gaps:
```
POST _search
{
  "query": {
    "intervals" : {
      "my_text" : {
        "match" : {
          "query" : "hot porridge",
          "filter" : {
            "script" : {
              "source" : "interval.start > 10 && interval.end < 20 && interval.gaps == 0"
            }
          }
        }
      }
    }
  }
}
```
2018-12-19 11:12:18 +00:00
Przemyslaw Gomulka 1345dff507
Fix line length in org.elasticsearch.snapshots (#36646)
Remove the line length suppression for this package and fix offending
lines in both main and test

relates #34884
2018-12-19 11:29:21 +01:00
Luca Cavanna d2ce576c8c
Use SearchRequest copy constructor in ExpandSearchPhase (#36772)
Relates to #36641
2018-12-19 10:47:17 +01:00
Alan Woodward dd540ef618
Use index-prefix fields for terms of length min_chars - 1 (#36703)
The default index_prefix settings will index prefixes of between 2 and 5 characters in length. 
Currently, if a prefix search falls outside of this range at either end we fall back to a standard prefix 
expansion, which is still very expensive for single character prefixes. However, we have an option 
here to use a wildcard expansion rather than a prefix expansion, so that a query of a* gets remapped 
to a? against the _index_prefix field - likely to be a very small set of terms, and certain to be much
smaller than a* against the whole index.

This commit adds this extra level of mapping for any prefix term whose length is one less than
the min_chars parameter of the index_prefixes field.
2018-12-19 08:55:05 +00:00
Nhat Nguyen 1e9d0bb01e AwaitsFix testRebuildLocalCheckpointTracker
Tracked at #36470
2018-12-19 03:12:34 -05:00
Simon Willnauer 8e5db90eec
Never corrupt fully deleted segments in tests (#36741)
Today we might corrupt a fully deleted segment which is then pruned
once a snapshot is taken. This causes random test failures in CorruptedFileIT.
This change hardens the selection of files to corrupt and removes some fragile
code preventing fully deleted segments to be taken into account.

Closes #36526
2018-12-19 07:39:10 +01:00
Alpar Torok e9ef5bdce8
Converting randomized testing to create a separate unitTest task instead of replacing the builtin test task (#36311)
- Create a separate unitTest task instead of Gradle's built in 
- convert all configuration to use the new task 
- the  built in task is now disabled
2018-12-19 08:25:20 +02:00
Alpar Torok 8015560bd4
Remove line length supressions from plugins (#36708) 2018-12-19 08:06:48 +02:00
Tim Brooks aaf466ff5e
Revert transport.port change for tests (#36809)
Commit #36786 updated docs and strings to reference transport.port instead of
transport.tcp.port. However, this breaks backwards compatibility tests
as the tests rely on string configurations and transport.port does not
exist prior to 6.6. This commit reverts the places were we reference
transport.tcp.port for tests. This work will need to be reintroduced in
a backwards compatible way.
2018-12-18 19:01:13 -07:00
Nick Knize 20b58f0b0f
[GEO] Fork Lucene's LatLonShape Classes to local lucene package (#36794)
Lucene 7.6 uses a smaller encoding for LatLonShape. This commit forks the LatLonShape classes to Elasticsearch's local lucene package. These classes will be removed on the release of Lucene 7.6.
2018-12-18 16:48:30 -06:00
Tim Brooks 47a9a8de49
Update transport docs and settings for changes (#36786)
This is related to #36652. In 7.0 we plan to deprecate a number of
settings that make reference to the concept of a tcp transport. We
mostly just have a single transport type now (based on tcp). Settings
should only reference tcp if they are referring to socket options. This
commit updates the settings in the docs. And removes string usages of
the old settings. Additionally it adds a missing remote compress setting
to the docs.
2018-12-18 13:09:58 -07:00
Boaz Leskes 10f6408814 enable bwc tests and switch transport serialization version to 6.6.0 for CAS features
Relates to #36148
2018-12-18 20:03:07 +01:00
Yannick Welsch e35de2ea2c
Synchronize WriteReplicaResult callbacks (#36770)
TransportWriteAction.WriteReplicaResult is not properly synchronized, which can lead to a data race
between the thread that calls respond and the AsyncAfterWriteAction that calls either onSuccess or
onFailure. This data race results in the response listener not being called, which ultimately results in
a stuck replication task on the replica.
2018-12-18 19:23:35 +01:00
Tim Brooks 1fa105658e
Add CcrRestoreSourceService to track sessions (#36578)
This commit is related to #36127. It adds a CcrRestoreSourceService to
track Engine.IndexCommitRef need for in-process file restores. When a
follower starts restoring a shard through the CcrRepository it opens a
session with the leader through the PutCcrRestoreSessionAction. The
leader responds to the request by telling the follower what files it
needs to fetch for a restore. This is not yet implemented.

Once, the restore is complete, the follower closes the session with the
DeleteCcrRestoreSessionAction action.
2018-12-18 11:23:13 -07:00
Ryan Ernst 8ec8342a52
Internal: Remove originalSettings from Node (#36569)
This commit removes the originalSettings member from Node. It was only
needed to allows test clusters to recreate the node in certain
situations. Instead, the test cluster now keeps track of these settings.
2018-12-18 10:05:27 -08:00
Ryan Ernst 0b22ca3a0f
Core: Deprecate use of scientific notation in epoch time parsing (#36691)
The joda epoch parsing code currently supports passing epoch time as a
number in scientific notation. However, no systems appear to exist which
output timestamps in scientific notation. In java time, it is
particularly complex to implement scientific notation timestamp parsing
within a DateTimeFormatter. This commit adds a deprecation warning when
the epoch time parsers in joda parse scientific notation, so that it can
be removed when switching to java time.
joda are passed a time in scientific notation.
2018-12-18 09:51:03 -08:00
Alexander Reelsen ea9b08dee1 Tests: Mute SnapshotDisruptionIT.testDisruptionOnSnapshotInitialization
Relates #36779
2018-12-18 18:25:06 +01:00
Luca Cavanna bd12e00b53 Update versions in SearchSortValues transport serialization
Now that #36617 is backported to 6.x, the version in the transport serialization conditionals for the search sort values can be updated to 6.6.0
2018-12-18 17:03:00 +01:00
Luca Cavanna 1aad08c0c0 Update version in SearchHits transport serialization
Now that #36555 is backported to 6.x, the version in the transport serialization conditionals for sort and collapse info can be updated to 6.6.0
2018-12-18 17:03:00 +01:00
Nick Knize ec0dc2c0e9
[Geo] Integrate Lucene's LatLonShape (BKD Backed GeoShapes) as default `geo_shape` indexing approach (#36751)
* [Geo] Expose BKDBackedGeoShapes as new VECTOR strategy

This commit exposes lucene's LatLonShape field as a new
strategy in GeoShapeFieldMapper. To use the new indexing
approach, strategy should be set to "vector" in the
geo_shape field mapper. If the tree parameter is set
the mapper will throw an IAE. Note the following:

When using vector strategy:

* geo_shape query does not support querying by POINT,
MULTIPOINT, or GEOMETRYCOLLECTION.
* LINESTRING and MULTILINESTRING queries do not support
WITHIN relation.
* CONTAINS relation is not supported.
* The tree, precision, tree_levels, distance_error_pct,
and points_only parameters will not throw an exception
but they have no effect and will be marked as
deprecated..

All other features are supported.

* revert change to PercolatorFieldMapper

* fix ExistsQuery for geo_shape vector strategy

* add deprecation logging for tree, precision, tree_levels, distance_error_pct, and points_only

* initial update to geoshape docs, including mapping migration updates

* initial support for GeoCollection queries

* fix docs and javadoc errors

* clean up geocollection queries

* set deprecated mapping tests to NOTCONSOLE

* fix geo-shape mapper asciidoc mapping and test warnings

* add support for point queries using LatLonShapeBoundingBoxQuery

* update GeoShapeQueryBuilderTests to include POINT queries for VECTOR strategy. Other comment cleanups

* add lucene geometry build testing to ShapeBuilder tests

* remove deprecated prefix tree mapping from geo-shape.asciidoc

* refactor GeoShapeFieldMapper into LegacyGeoShapeFieldMapper and GeoShapeFieldMapper

Both classes derive from BaseGeoShapeFieldMapper that provides shared parameters:
coerce, ignoreMalformed, ignore_z_value, orientation.

* update docs to remove vector strategy

* fix GeometryCollectionBuilder#buildLucene to return the object created by the shape builder

* fix LineLength failure in GeoJsonShapeParserTests

* ShapeMapper refactor changes from PR feedback

* fix typo in geo-shape.asciidoc

* ignore circle test in docs

* update indexing-approach ref to geoshape-indexing-approach

* add warnings check for LegacyGeoShapeFieldMapper to AbstractBuilderTestCase

* fix deprecatedParameters setup

* update indexing approach

* fixing unexpected warnings failures

* move orientation back to field type

* remove if in LegacyGeoShapeFieldMapper#doXContent. Fix GeoShapeFieldMapper to work with double array as a point

* fix indexing-approach link in circle section of geoshape docs

* add strategy to deprecation warnings check

* fix test failures

* fix typo in QueryStringQueryBuilderTests

* fix total hits to totalHits().value

* fix version number

* add version check to BaseGeoShapeFieldMapper

* fix line length!

* revert version check in BaseGeoShapeFieldMapper

* Fix serialization of mappings of legacy shapes.
2018-12-18 09:54:56 -06:00
Mayya Sharipova f884b2b1cd
Deprecate types in index API (#36575)
* Deprecate types in index API

- deprecate type-based constructors of IndexRequest
- update tests to use typeless IndexRequest constructors
- no yaml tests as they have been already added in #35790

Relates to #35190
2018-12-18 08:53:49 -05:00
Boaz Leskes 5f76f39386
Rename seq# powered optimistic concurrency control parameters to ifSeqNo/ifPrimaryTerm (#36757)
This PR renames the parameters previously introduce to the following:

### URL Parameters
```
PUT twitter/_doc/1?if_seq_no=501&if_primary_term=1
{
    "user" : "kimchy",
    "post_date" : "2009-11-15T14:12:12",
    "message" : "trying out Elasticsearch"
}

DELETE twitter/_doc/1?if_seq_no=501&if_primary_term=1
```

### Bulk API
```
POST _bulk
{ "index" : { "_index" : "test", "_type" : "_doc", "_id" : "1", "if_seq_no": 501, "if_primary_term": 1 } }
{ "field1" : "value1" }
{ "delete" : { "_index" : "test", "_type" : "_doc", "_id" : "2", "if_seq_no": 501, "if_primary_term": 1 } }
```

### Java API
```
IndexRequest.ifSeqNo(long seqNo)
IndexRequest.ifPrimaryTerm(long primaryTerm)
DeleteRequest.ifSeqNo(long seqNo)
DeleteRequest.ifPrimaryTerm(long primaryTerm)
```

Relates #36148
Relates #10708
2018-12-18 14:35:18 +01:00
Boaz Leskes 9087c98a5a
Expose Sequence Number based Optimistic Concurrency Control in the rest layer (#36721)
Relates #36148 
Relates #10708
2018-12-18 10:56:02 +01:00
Luca Cavanna 8a412c6a26
Ensure MapperService#getAllMetaFields elements order is deterministic (#36739)
MapperService#getAllMetaFields returns an array, which is created out of
an `ObjectHashSet`. Such set does not guarantee deterministic hash
ordering. The array returned by its toArray may be sorted differently
at each run. This caused some repeatability issues in our tests (see #29080)
as we pick random fields from the array of possible metadata fields,
but that won't be repeatable if the input array is sorted differently at
every run. Once setting the tests seed, hppc picks that up and the sorting is
deterministic, but failures don't repeat with the seed that gets printed out
originally (as a seed was not originally set).
See also https://issues.carrot2.org/projects/HPPC/issues/HPPC-173.

With this commit, we simply create a static sorted array that is used for
`getAllMetaFields`. The change is in production code but really affects
only testing as the only production usage of this method was to iterate
through all values when parsing fields in the high-level REST client code.
Anyways, this seems like a good change as returning an array would imply
that it's deterministically sorted.
2018-12-18 10:43:14 +01:00
Luca Cavanna b57e12aa44
Add raw sort values to SearchSortValues transport serialization (#36617)
In order for CCS alternate execution mode (see #32125) to be able to do the final reduction step on the CCS coordinating node, we need to serialize additional info in the transport layer as part of each `SearchHit`. Sort values are already present but they are formatted according to the provided `DocValueFormat` provided. The CCS node needs to be able to reconstruct the lucene `FieldDoc` to include in the `TopFieldDocs` and `CollapseTopFieldDocs` which will feed the `mergeTopDocs` method used to reduce multiple search responses (one per cluster) into one.

This commit adds such information to the `SearchSortValues` and exposes it through a new getter method added to `SearchHit` for retrieval. This info is only serialized at transport and never printed out at REST.
2018-12-18 09:20:51 +01:00
Nicholas Knize 96d279ed83 Revert "[Geo] Integrate Lucene's LatLonShape (BKD Backed GeoShapes) as default `geo_shape` indexing approach (#35320)"
This reverts commit 5bc7822562.
2018-12-17 20:09:46 -06:00
Christoph Büscher 2f5300e3a6
Deprecate types in get_source and exist_source (#36426)
This change adds a new untyped endpoint `{index}/_source/{id}` for both the
GET and the HEAD methods to get the source of a document or check for its
existance. It also adds deprecation warnings to RestGetSourceAction that emit
a warning when the old deprecated "type" parameter is still used. Also updating
documentation and tests where appropriate.

Relates to #35190
2018-12-18 00:57:42 +01:00