Commit Graph

47388 Commits

Author SHA1 Message Date
Nhat Nguyen cf9a73b5ac Call afterWriteOperation after trim translog in peer recovery (#45182)
testShouldFlushAfterPeerRecovery was added #28350 to make sure the
flushing loop triggered by afterWriteOperation eventually terminates.
This test relies on the fact that we call afterWriteOperation after
making changes in translog. In #44756, we roll a new generation in
RecoveryTarget#finalizeRecovery but do not call afterWriteOperation.

Relates #28350
Relates #45073
2019-08-10 22:59:02 -04:00
Nhat Nguyen 25c6102101 Trim local translog in peer recovery (#44756)
Today, if an operation-based peer recovery occurs, we won't trim
translog but leave it as is. Some unacknowledged operations existing in
translog of that replica might suddenly reappear when it gets promoted.
With this change, we ensure trimming translog above the starting
sequence number of phase 2. This change can allow us to read translog
forward.
2019-08-10 22:59:02 -04:00
Armin Braun 1cd464d675
Isolate Request in Call-Chain for REST Request Handling (#45130) (#45417)
* Follow up to #44949
* Stop using a special code path for multi-line JSON and instead handle its detection like that of other XContent types when creating the request
* Only leave a single path that holds a reference to the full REST request
   * In the next step we can move the copying of request content to happen before the actual request handling and make it conditional on the handler in question to stop copying bulk requests as suggested in #44564
2019-08-10 10:21:01 +02:00
Mark Vieira 491880edde
Fix build cache misses caused by embedded reaper jar (#45404)
(cherry picked from commit 788feced760bc2a5f453ebb07f1dbb288c6232b2)
2019-08-09 21:10:48 -07:00
Mark Vieira b5c3c9767c
Publish CI build scans to Gradle Enterprise (#45249)
(cherry picked from commit 957a232afc3ecd43326509043ab73cc6c2a26411)
2019-08-09 15:53:50 -07:00
Benjamin Trent fac1a6f8e8
[ML][Data Frame] have DataFrameTransformConfigUpdate#apply set Version (#45391) (#45400) 2019-08-09 14:32:49 -05:00
Hendrik Muhs bf4da6c6ad
[ML-DataFrame] fix starting a batch data frame after stopping at runtime (#45340) (#45381)
fix loading of next checkpoint after data frame transform has been stopped/started within one run

closes #45339
2019-08-09 20:30:11 +02:00
Dimitris Athanasiou 27497ff75f
[7.x][ML] Add regression analysis to DF analytics (#45292) (#45388)
This commit adds a first draft of a regression analysis
to data frame analytics. There is high probability that
the exact syntax might change.

This commit adds the new analysis type and its parameters as
well as appropriate validation. It also modifies the extractor
and the fields detector to be able to handle categorical fields
as regression analysis supports them.
2019-08-09 19:31:13 +03:00
Armin Braun d1ed9bdbfd
Use StepListener to Simplify SnapshotResiliencyTests (#45233) (#45386)
* Reduces complicated callback relations in `testSuccessfulSnapshotAndRestore` to flat steps of sequential actions
* Will refactor the other tests in this suit as a follow up
   * This format certainly makes it easier to create more complicated tests that involve multiple subsequent snapshots as it would allow adding loops
2019-08-09 18:19:48 +02:00
Yannick Welsch 9e6d874a41
Show BWC version in ClusterFormationFailureHelper (#45352)
When having a cluster state from 6.x, display the metadata version as the cluster state version.
Avoids confusion where a cluster state from 6.x is displayed as version 0 even if has some actual
content.
2019-08-09 16:23:38 +02:00
James Rodewig 846928a52a [DOCS] Reformats interval query (#45350) 2019-08-09 08:53:47 -04:00
James Rodewig 1506d4436b [DOCS] Reformats cat plugins API (#45344) 2019-08-09 08:49:00 -04:00
James Rodewig eec87ffab8 [DOCS] Reformats simple query string query (#45343) 2019-08-09 08:33:05 -04:00
Alpar Torok 634a070430 Restrict which tasks can use testclusters (#45198)
* Restrict which tasks can use testclusters

This PR fixes a problem between the interaction of test-clusters and
build cache.
Before this any task could have used a cluster without tracking it as
input.
With this change a new interface is introduced to track the tasks that
can use clusters and we do consider the cluster as input for all of
them.
2019-08-09 13:38:01 +03:00
Yannick Welsch 5ddeb488a6 Allow _update on write alias (#45318)
Using the document update API on aliases with a write index does not work.

Follow-up to #31520
2019-08-09 11:44:24 +02:00
Hendrik Muhs 7d0aff0ed5 [ML-DataFrame] fix test failure in checkpoint retrieval (#45297)
gracefully handle if index response returns null, increase and assert timeout

closes #45238
2019-08-09 09:04:53 +02:00
Armin Braun a501d68f23
Upgrade to Netty 4.1.38 (#45132) (#45364)
* A number of fixes to buffer handling in the .37 and .38 -> we should stay up to date
2019-08-09 03:38:14 +02:00
Tal Levy 2a99eaa7c2 Revert "removes the CellIdSource abstraction from geo-grid aggs (#45307) (#45353)"
This reverts commit 7b0a8040de.
2019-08-08 17:40:03 -07:00
Ryan Ernst 1794718e8e Make git revision loading lazy (#45358)
This commit makes the gitRevision property a lazy loaded value by
returning an Object implementing toString(). The Dockerfile template is
also changed to use groovy templates instead of the mavenfilter hack, so
converting to String will not happen until runtime.
2019-08-08 17:08:07 -07:00
Armin Braun 12ed6dc999
Only retain reasonable history for peer recoveries (#45208) (#45355)
Today if a shard is not fully allocated we maintain a retention lease for a
lost peer for up to 12 hours, retaining all operations that occur in that time
period so that we can recover this replica using an operations-based recovery
if it returns. However it is not always reasonable to perform an
operations-based recovery on such a replica: if the replica is a very long way
behind the rest of the replication group then it can be much quicker to perform
a file-based recovery instead.

This commit introduces a notion of "reasonable" recoveries. If an
operations-based recovery would involve copying only a small number of
operations, but the index is large, then an operations-based recovery is
reasonable; on the other hand if there are many operations to copy across and
the index itself is relatively small then it makes more sense to perform a
file-based recovery. We measure the size of the index by computing its number
of documents (including deleted documents) in all segments belonging to the
current safe commit, and compare this to the number of operations a lease is
retaining below the local checkpoint of the safe commit. We consider an
operations-based recovery to be reasonable iff it would involve replaying at
most 10% of the documents in the index.

The mechanism for this feature is to expire peer-recovery retention leases
early if they are retaining so much history that an operations-based recovery
using that lease would be unreasonable.

Relates #41536
2019-08-09 01:56:32 +02:00
Tal Levy 7b0a8040de
removes the CellIdSource abstraction from geo-grid aggs (#45307) (#45353)
CellIdSource is a helper ValuesSource that encodes GeoPoint
into a long-encoded representation of the grid bucket the point
is associated with. This complicates thing as usage evolves to
support shapes that are associated with more than one bucket ordinal.
2019-08-08 16:33:16 -07:00
Hendrik Muhs 68f9102550 [ML-DataFrame] audit changes in the source index (#45282)
add audits when the set of source indexes changes and in a special case runs empty
2019-08-08 23:31:55 +02:00
Andrei Stefan 740d58fd46
SQL: Uniquely named inner_hits sections for each nested field condition (#45341)
* Name each inner_hits section of nested queries differently and extract and combine the multiple values it generates into a single list.
This also introduces a limitation (its origin it's with Elasticsearch
though) on the sorting capabilities when the sorting is based on the
nested fields filtered: only one of the conditions applied to nested
documents will be used in the nested sorting.

(cherry picked from commit cfc5cf68f6e83b07bb9006986d0903d6be418ec6)
2019-08-09 00:22:49 +03:00
Tim Brooks af908efa41
Disable netty direct buffer pooling by default (#44837)
Elasticsearch does not grant Netty reflection access to get Unsafe. The
only mechanism that currently exists to free direct buffers in a timely
manner is to use Unsafe. This leads to the occasional scenario, under
heavy network load, that direct byte buffers can slowly build up without
being freed.

This commit disables Netty direct buffer pooling and moves to a strategy
of using a single thread-local direct buffer for interfacing with sockets.
This will reduce the memory usage from networking. Elasticsearch
currently derives very little value from direct buffer usage (TLS,
compression, Lucene, Elasticsearch handling, etc all use heap bytes). So
this seems like the correct trade-off until that changes.
2019-08-08 15:10:31 -06:00
Armin Braun b19de55095
Add missing wait to testAutomaticReleaseOfIndexBlock (#45342) (#45351)
Today the test waits for one of the shards to be blocked, but this does not
mean that the block has been applied on all nodes, so a subsequent indexing
operation may still go through.

Fixes #45338
2019-08-08 22:39:22 +02:00
Henning Andersen d139896b66
Reindex share retry between hit sources (#44203) (#45348)
The client and remote hit sources had each their own retry mechanism,
which would do the same. Supporting resiliency we would have to expand
on the retry mechanisms and as a preparation for that, the retry
mechanism is now shared such that each sub class is only responsible for
sending requests and converting responses/failures to common format.

Part of #42612
2019-08-08 22:01:29 +02:00
Mark Vieira 214cbb28df
Fix for build runtime classpath instability (#45347)
(cherry picked from commit dee4ee2f0d4190ab54d0a4f0aa251d8c03e9db6d)
2019-08-08 12:41:17 -07:00
Christoph Büscher a552b33276 Fix occasional SuggestSearchIT failure (#45330)
Refreshes happening during indexing can result differen segment counts and
slightly skewed term statistics, which in turn has the potential to change
suggestion output slightly. In order to prevent this, disable refresh for the
affected tests.

Closes #43261
2019-08-08 21:06:32 +02:00
Mark Vieira 0ae103c40f
Avoid unnecessary eager creation of Gradle tasks (#45098) (#45310) 2019-08-08 10:50:09 -07:00
Jack Conradson b716b840d3 Remove loop counter from Reserved in Painless AST. (#45298)
This change adds a compiler pass to give each node the chance to store 
settings necessary for analysis and writing. This removes the need to pass 
this in a somewhat convoluted way through an additional class called 
Reserved, and also removes the need to have the Walker set values for 
settings on reserved. This is next step in decoupling the Painless grammar 
from the Painless AST.
2019-08-08 09:34:51 -07:00
David Roberts 14545f8958
[ML-DataFrame] Combine task_state and indexer_state in _stats (#45324)
This commit replaces task_state and indexer_state in the
data frame _stats output with a single top level state
that combines the two. It is defined as:

- failed if what's currently reported as task_state is failed
- stopped if there is no persistent task
- Otherwise what's currently reported as indexer_state

Backport of #45276
2019-08-08 16:24:26 +01:00
Dimitris Athanasiou e53bb050db Mute testAutomaticReleaseOfIndexBlock
Relates #45338
2019-08-08 17:56:41 +03:00
Andrey Ershov 07c656fba9 Mute testCustomDataPaths on Windows
See #45333

(cherry picked from commit 671e1ad1068aee4b593ad0c8ab13ff60b4f125b8)
2019-08-08 16:26:56 +02:00
Mayya Sharipova f0f2294695 Add filters in examples of vector functions (#45327) 2019-08-08 09:44:59 -04:00
Zachary Tong 86d6597890 Use newIndexSearcher() instead of newSearcher() (#45248)
`newSearcher()` from lucene can randomly choose index readers which
are not compatible with our tests, like ParallelCompositeReader.
The `newIndexSearcher()` method on AggregatorTestCase is a wrapper
similar to newSearcher but compatible with our tests
2019-08-08 09:34:38 -04:00
István Zoltán Szabó 4e32470827 [DOCS] Reformats cluster reroute API. (#45328) 2019-08-08 15:27:54 +02:00
István Zoltán Szabó 4d96c83854 [DOCS] Reformats cluster pending tasks API (#45280)
Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2019-08-08 14:49:32 +02:00
James Rodewig e72bda1703 [DOCS] Reformats cat nodes API (#45285) 2019-08-08 08:38:02 -04:00
James Rodewig b806e6edde [DOCS] Reformats cat pending tasks API (#45287) 2019-08-08 08:32:04 -04:00
István Zoltán Szabó 276e9c6697 [DOCS] Adds supported time units ref to the ML and DF API params. (#45322) 2019-08-08 14:26:19 +02:00
Martijn van Groningen e066133016
Change the ingest simulate api to not include dropped documents (#44161)
If documents are dropped by the `drop` processor then
these documents are returned as a `null` value in the response.

=== Example

Create pipeline:

```
PUT _ingest/pipeline/droppipeline
{
    "processors": [
        {
            "set": {
                "field": "bla",
                "value": "val"
            }
        },
        {
            "drop": {}
        }
    ]
}
```

Simulate request:

POST _ingest/pipeline/droppipeline/_simulate
{
    "docs": [
        {
            "_source": {
                "message": "text"
            }
        }
    ]
}

Response:

```
{
    "docs": [
        null
    ]
}
```

Response if verbose is enabled:

```
{
    "docs": [
        {
            "processor_results": [
                {
                    "doc": {
                        "_index": "_index",
                        "_type": "_doc",
                        "_id": "_id",
                        "_source": {
                            "message": "text",
                            "bla": "val"
                        },
                        "_ingest": {
                            "timestamp": "2019-07-10T11:07:10.758315Z"
                        }
                    }
                },
                null
            ]
        }
    ]
}
```

Closes #36150

* Abort pipeline simulation in verbose mode when document has been dropped
by drop processor
2019-08-08 13:04:33 +02:00
Ioannis Kakavas 99ddb8b3d8 Allow empty token endpoint for implicit flow (#45038)
When using the implicit flow in OpenID Connect, the
op.token_endpoint_url should not be mandatory as there is no need
to contact the token endpoint of the OP.
2019-08-08 12:50:53 +03:00
David Turner ddcc38cf1c
More read-only-allow-delete docs (#45320)
Adds to the `index.blocks.read_only_allow_delete` docs the information that
this block may be added or removed automatically, and rewords the
breaking-changes docs to mention the blocks explicitly and to recommend using a
different block.

Relates #42559
2019-08-08 09:58:23 +01:00
István Zoltán Szabó 9f62c04637 [DOCS] Reformats cluster health and cluster state APIs (#45206)
Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2019-08-08 10:25:05 +02:00
Martijn van Groningen fb959d188c
Backport: Add description to force-merge tasks (#41365) (#45191)
* Add description to force-merge tasks (#41365)

This is static information that is part of the force merge request.

Relates to #15975
2019-08-08 08:15:09 +02:00
Mark Vieira fca458f1c8
Use system properties for build cache configuration (#45295) 2019-08-07 13:14:54 -07:00
Gordon Brown e3599fded7
Add warning about versions to Deprecation API docs (#36624)
Add a note that the Deprecation API may not be up to date with all
breaking changes until the last minor version in a major version series.
2019-08-07 14:11:05 -06:00
Michael Basnight 89861d0884 Add ingest processor existence helper method (#45156)
This commit adds a helper method to the ingest service allowing it to
inspect a pipeline by id and verify the existence of a processor in the
pipeline. This work exposed a potential bug in that some processors
contain inner processors that are passed in at instantiation. These
processors needed a common way to expose their inner processors, so the
WrappingProcessor was created in order to expose the inner processor.
2019-08-07 11:19:04 -05:00
Mark Vieira 341ab48ec0
Improve SCM info in build scans (#45264) 2019-08-07 09:06:11 -07:00
Benjamin Trent 5db9982f71
[7.x] [ML][Data Frame] Add update transform api endpoint (#45154) (#45279)
* [ML][Data Frame] Add update transform api endpoint (#45154)

This adds the ability to `_update` stored data frame transforms. All mutable fields are applied when the next checkpoint starts. The exception being `description`.

This PR contains all that is necessary for this addition:
* HLRC
* Docs
* Server side
2019-08-07 10:37:35 -05:00