Commit Graph

48124 Commits

Author SHA1 Message Date
Yannick Welsch 9638ca20b0 Allow dropping documents with auto-generated ID (#46773)
When using auto-generated IDs + the ingest drop processor (which looks to be used by filebeat
as well) + coordinating nodes that do not have the ingest processor functionality, this can lead
to a NullPointerException.

The issue is that markCurrentItemAsDropped() is creating an UpdateResponse with no id when
the request contains auto-generated IDs. The response serialization is lenient for our
REST/XContent format (i.e. we will send "id" : null) but the internal transport format (used for
communication between nodes) assumes for this field to be non-null, which means that it can't
be serialized between nodes. Bulk requests with ingest functionality are processed on the
coordinating node if the node has the ingest capability, and only otherwise sent to a different
node. This means that, in order to reproduce this, one needs two nodes, with the coordinating
node not having the ingest functionality.

Closes #46678
2019-09-19 16:46:33 +02:00
Takumasa Ochi 7a3054c5dc Fix typos in `match` in profile API (#46723)
* Replace `matches` with correct `match`
* Use present tense consistently
* Replace `metric` with correct `match`
2019-09-19 16:07:52 +02:00
Armin Braun b32031ee03
Fix snapshot_restore_exception in Docs Tests (#46852) (#46870)
We should only snapshot the index we're going to
restore in the next step. Otherwise, we will
potentially not get the correct response or
fail restoring outright due to internal indices
getting created concurrently when running against
the x-pack distribution.

Closes #46844
2019-09-19 16:05:51 +02:00
Jason Tedor 4a9943a906
Bundle AdoptOpenJDK 13 (#46860)
This commit switches to use AdoptOpenJDK 13 for bundling the JDK,
instead of Oracle OpenJDK 13.
2019-09-19 08:56:17 -04:00
Russ Cam e467dddd4c Use snake casing for document field (#45432)
This commit updates the search example to use snake casing for the document field. This is consistent with other examples
2019-09-19 14:27:00 +02:00
Martijn Laarman f1de8c0193 Document PUT as the prefered HTTP method for index API (#45622)
Follow up from #42346. Since the `methods` array is in order of
preference when calling the index API with an `{id}` we prefer to use
the `PUT` http method.
2019-09-19 14:24:22 +02:00
James Rodewig b73a9604c1
[DOCS] Separate and reformat synced flush API docs (#46634) (#46839) 2019-09-19 08:22:37 -04:00
LHearen e555ef2e86 [DOCS] make bulk page more readable (#46200)
comma and semi-colon required to make it more readable
2019-09-19 14:20:41 +02:00
maidoo 0f2e72b7f8 [DOCS] Term dict memory usage is a 7.3.0 enhancement, not 7.2.0 (#46662)
Closes #46635
2019-09-19 14:15:12 +02:00
Toby McLaughlin 7795e21ea3 [DOCS] Fix typo in GCS snapshot repo doc (#46048) 2019-09-19 14:06:37 +02:00
Armin Braun 6b09c2cdbb
Limit Netty Workers in NativeRealmIntegTestCase (#46816) (#46850)
The fact that this test randomly uses a relatively large number
of nodes and hence Netty worker threads created a problem with
running out of direct memory on CI.
Tests run with 512M heap (and hence 512M direct memory) by default.
On a CI worker with 16 cores, this means Netty will by default set
up 32 transport workers. If we get unlucky and a lot of them
actually do work (and thus instantiate a `CopyBytesSocketChannel`
which costs 1M per thread for the thread-local IO buffer) we
would run out of memory.

This specific failure was only seen with `NativeRealmIntegTests` so I
only added the constraint on the Netty worker count here.
We can add it to other tests (or `SecurityIntegTestCase`) if need be
but for now it doesn't seem necessary so I opted for least impact.

Closes #46803
2019-09-19 13:07:42 +02:00
Armin Braun a087553009
Rearrange BlobStoreRepository to Prepare #46250 (#46824) (#46853)
In #46250 we will have to move to two different delete
paths for BwC. Both paths will share the initial
reads from the repository though so I extracted/separated the
delete logic away from the initial reads to significantly simplify
the diff against #46250 here.
Also, I added some JavaDoc from #46250 here as well which makes the
code a little easier to follow even ignoring #46250 I think.
2019-09-19 13:07:00 +02:00
Luca Cavanna a6c9f8ffae [DOCS] mention automatic cancellation in search docs (#46706)
Add a small note around automatic search cancellation which was missed as part of #43332

Relates to #43332
2019-09-19 09:35:30 +02:00
István Zoltán Szabó 3be51fbdf7 [DOCS] Adds regression analytics resources and examples to the data frame analytics APIs and the evaluation API (#46176)
* [DOCS] Adds regression analytics resources and examples to the data frame analytics APIs.
Co-Authored-By: Benjamin Trent <ben.w.trent@gmail.com>
Co-Authored-By: Tom Veasey <tveasey@users.noreply.github.com>
2019-09-19 09:23:18 +02:00
István Zoltán Szabó 65fffcc9c1 [DOCS]Changes failed transforms limitation. (#46754) 2019-09-19 09:04:41 +02:00
Alpar Torok 86d216cc9c Update .ci/bwcVersions 2019-09-19 09:35:01 +03:00
Lisa Cawley a32fe1618c [DOCS] Updates transform landing page (#46689) 2019-09-18 14:00:43 -07:00
Mark Vieira 507d879b70
Disable bwc distribution caching in master branch (#46686)
This commit disables caching of BWC snapshot distributions in the "trunk" (aka master) branch.
Since the previous major release branches move quickly we rarely get cache hits for these 
tasks, and the artifacts themselves are very large. This means the overhead here is high and
savings basically zero. We conditionally disable task output caching in this scenario in CI to 
avoid excessive build cache overhead as well as causing too much turn in the cache itself which
would lead to lots of cache entry evictions.
2019-09-18 11:52:17 -07:00
Dimitris Athanasiou 02a5e153dc
[7.x][ML] Parse and index data frame analytics state (#46804) (#46820)
This commit reuses the same state processor that is used for autodetect
to parse state output from data frame analytics jobs. We then index the
state document into the state index.

Backport of #46804
2019-09-18 20:37:40 +03:00
James Rodewig b08b3741db [DOCS] Use versions file from elastic/docs (#46719) 2019-09-18 11:18:14 -04:00
James Rodewig d00bad9ab2 [DOCS] Reformat force merge API docs (#46685) 2019-09-18 10:10:51 -04:00
Benjamin Trent 9cf9c64ec2
[7.x] [ML][Transforms] remove `force` flag from _start (#46414) (#46748)
* [ML][Transforms] remove `force` flag from _start (#46414)

* [ML][Transforms] remove `force` flag from _start

* fixing expected error message

* adjusting bwc version
2019-09-18 10:06:05 -04:00
Turaç Kangal d1f47cf00e
[HLRC] Send terminate_after as query string parameter to the count API (#46474)
Prior to this commit terminate_after was sent as request body parameter
(via SearchSourceBuilder), which is not possible in the count api.

Closes #46446
2019-09-18 15:54:43 +02:00
James Rodewig 6863b90272 [DOCS] Remove leveloffset for CCR docs (#46818) 2019-09-18 09:44:43 -04:00
Tanguy Leroux 3ae51f25dd Move testSnapshotWithLargeSegmentFiles to ESMockAPIBasedRepositoryIntegTestCase (#46802)
This commit moves the common test testSnapshotWithLargeSegmentFiles 
to the ESMockAPIBasedRepositoryIntegTestCase base class.
2019-09-18 15:41:30 +02:00
Dimitris Athanasiou cebe8da617
[7.x][ML] MlMemoryTracker should ignore analytics tasks without config (#46789) (#46811)
It is possible for a running analytics job that its config is removed
from the '.ml-config' index (perhaps the user deleted the entire index,
etc.). In that case the task remains without a matching config. I have
raised #46781 to discuss how to deal with this issue.

This commit focuses on `MlMemoryTracker` and changes it so that when
we get the configs for the running tasks we leniently ignore missing ones.
This at least means memory tracking will keep working for other jobs
if one or more are missing.

In addition, this commit makes the cleanup code for native analytics
tests more robust by explicitly stopping all jobs and force-stopping
if an error occurs. This helps so that a single failing test does
not cause other tests fail due to pending tasks.

Backport of #46789
2019-09-18 16:35:25 +03:00
Christos Soulios 0076083b35
Implement rounding optimization for fixed offset timezones (#46809)
Fixes #45702 with date_histogram aggregation when using fixed_interval.
Optimization has been implemented for both fixed and calendar intervals
2019-09-18 15:56:34 +03:00
István Zoltán Szabó e59be0354a [DOCS] Reformats validate API (#46389)
* [DOCS] Reformats validate API.
Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2019-09-18 14:31:17 +02:00
Armin Braun 142b10604e
Fix testHistoryRetention (#46799) (#46805)
Suppress the reasonable-history check in this test to guarantee we're always getting ops based recovery even after a background sync.

Closes #45953

Co-Authored-By: David Turner <david.turner@elastic.co>
2019-09-18 13:22:55 +02:00
Armin Braun f983b67fdc
Add Assertion About Leaking index-N to Repo Tests (#46774) (#46801)
This adds an assert to make sure we're not leaking
index-N blobs on the shard level to the repo consistency checks.
It is ok to have a single redundant index-N blob in a failure scenario
but additional index-N should always be cleaned up before adding more.
2019-09-18 13:15:56 +02:00
Alpar Torok f3e67bdd17 Add resolution rule to allow resolving all deps (#46768)
Since the `resolveAllDependencies` task resolves all the congfigurations
it can find, this was not caught by our testing, but it's required to be
configuraed specifically.

We should probably cut-over to the new configurations at some point to
avoid problems like this.

Closes elastic/infra#14580
2019-09-18 11:09:43 +03:00
Tanguy Leroux 799f7def9f Add block support to AzureBlobStoreRepositoryTests (#46664)
This commit adds support for Put Block API to the internal HTTP server
used in Azure repository integration tests. This allows to test the
behavior of the Azure SDK client when the Azure Storage service
returns errors when uploading Blob in multiple blocks or when
downloading a blob using ranged downloads.
2019-09-18 09:43:08 +02:00
Tanguy Leroux fd42358a6d Add support for Multipart upload to S3 repository integration tests (#46704)
This commit adds support for Multipart upload to the internal HTTP 
server used in S3 repository integration tests.
2019-09-18 09:40:25 +02:00
Tanguy Leroux 4db37801d0 Add resumable uploads support to GCS repository integration tests (#46562)
This commit adds support for resumable uploads to the internal HTTP 
server used in GoogleCloudStorageBlobStoreRepositoryTests. This 
way we can also test the behavior of the Google's client when the 
service returns server errors in response to resumable upload requests.

The BlobStore implementation for GCS has the choice between 2 
methods to upload a blob: resumable and multipart. In the current 
implementation, the client executes a resumable upload if the blob 
size is larger than LARGE_BLOB_THRESHOLD_BYTE_SIZE, 
otherwise it executes a multipart upload. This commit makes this 
logic overridable in tests, allowing to randomize the decision of 
using one method or the other.

The commit add support for single request resumable uploads 
and chunked resumable uploads (the blob is uploaded into multiple 
2Mb chunks; each chunk being a resumable upload). For this last 
case, this PR also adds a test testSnapshotWithLargeSegmentFiles 
which makes it more probable that a chunked resumable upload is 
executed.
2019-09-18 09:33:05 +02:00
István Zoltán Szabó d931665f91 [DOCS] Adds force parameter to Stop transforms API docs. (#46770) 2019-09-18 09:16:07 +02:00
Martijn van Groningen ac4e990924
Add ingest cluster state listeners (#46650)
In the case that an ingest processor factory relies on other configuration
in the cluster state in order to construct a processor instance then
it is currently undetermined if a processor facotry can be notified about
a change if multiple cluster state updates are bundled together and
if a processor implement `ClusterStateApplier` interface.
(IngestService implements this interface too)

The idea with ingest cluster state listener is that it is guaranteed to
update the processor factory first before the ingest service creates
a pipeline with their respective processor instances.

Currently this concept is used in the enrich branch:
https://github.com/elastic/elasticsearch/blob/enrich/x-pack/plugin/enrich/src/main/java/org/elasticsearch/xpack/enrich/EnrichProcessorFactory.java#L21

In this case it a processor factory is interested in enrich indices' _meta
mapping fields.

This is the third PR that merges changes made to server module from
the enrich branch (see #32789) into the master branch.

Changes to the server module are merged separately from the pr that will
merge enrich into master, so that these changes can be reviewed in isolation.
2019-09-18 09:13:16 +02:00
Jason Tedor 7e71b2dbb3
Update bundled JDK to JDK 13 (#46795)
With the next minor release of Elasticsearch we will drop support for
JDK 12 and bump to JDK 13. While we want to use AdoptOpenJDK as the
bundled JDK, we are waiting for a release there. This commit moves to
OpenJDK 13 for now, and we will move to AdoptOpenJDK 13 as soon as its
available. Since macOS Catalina is delayed until October, we have some
time to update this.
2019-09-17 17:51:49 -07:00
Lee Hinman b85468d6ea
Add node setting for disabling SLM (#46794) (#46796)
This adds the `xpack.slm.enabled` setting to allow disabling of SLM
functionality as well as its HTTP API endpoints.

Relates to #38461
2019-09-17 17:39:41 -06:00
Oliver Gupte cbd58d3b78
Give kibana user privileges to create APM agent config index (#46765) (#46792)
* Give kibana user reserved role privileges on .apm-* to create APM agent configuration index.

* fixed test to include checking all .apm-* permissions

* changed pattern from ".apm-*" to the more specific ".apm-agent-configuration"
2019-09-17 15:01:42 -07:00
Jason Tedor cd71d4a83b
Use AdoptOpenJDK as the bundled JDK (#46470) (#46785)
This commit teaches the build how to bundle AdoptOpenJDK with our
artifacts, and switches to AdoptOpenJDK as the bundled JDK. We keep the
functionality to also bundle Oracle OpenJDK distributions.
2019-09-17 13:40:35 -07:00
Lisa Cawley 00c604a5a6 [DOCS] Augments ecommerce example (#46788) 2019-09-17 11:47:48 -07:00
James Rodewig 9dcf8a53a2 [DOCS] Add CCR docs to the Elasticsearch Reference Guide (#45704)
Previously, cross-cluster replication (CCR) documentation was located in
the Stack Overview:
https://www.elastic.co/guide/en/elastic-stack-overview/master/xpack-ccr.html

This adds CCR documentation to the Elasticsearch Reference Guide with a
level offset for headings.

The level offset and CCR Stack Overview docs will be removed in later
commits.
2019-09-17 14:26:20 -04:00
Armin Braun 371c355bca
Retry GCS Resumable Upload on Error 410 (#45963) (#46783)
A resumable upload session can fail on with a 410 error and should
be retried in that case. I added retrying twice using resetting of
the given `InputStream` as the retry mechanism since the same
approach is used by the AWS S3 SDK already as well and relied upon
by the S3 repository implementation.

Related GCS documentation:
https://cloud.google.com/storage/docs/json_api/v1/status-codes#410_Gone
2019-09-17 19:06:43 +02:00
Armin Braun 2c70d403fc
Reenable+Fix testMasterShutdownDuringFailedSnapshot (#46303) (#46747)
Reenable this test since it was fixed by #45689 in production
code (specifically, the fact that we write the `snap-` blobs
without overwrite checks now).
Only required adding the assumed blocking on index file writes
to test code to properly work again.

* Closes #25281
2019-09-17 18:09:48 +02:00
Armin Braun b00de8edf3
Ensure SAS Tokens in Test Use Minimal Permissions (#46112) (#46628)
There were some issues with the Azure implementation requiring
permissions to list all containers ue to a container exists
check. This was caught in CI this time, but going forward we
should ensure that CI is executed using a token that does not
allow listing containers.

Relates #43288
2019-09-17 15:40:11 +02:00
Costin Leau 92e518e789 SQL: Properly handle indices with no/empty mapping (#46775)
When encountering only indices with empty mapping, the IndexResolver
throws an exception as it expects to find at least one entry.
This commit fixes this case so that an empty mapping is returned.

Fix #46757

(cherry picked from commit 5f4f5807acb93b5fab36718c092c328977a396b6)
2019-09-17 16:01:22 +03:00
David Turner 65dc888623 Resume partial download from S3 on connection drop (#46589)
Today if the connection to S3 times out or drops after starting to download an
object then the SDK does not attempt to recover or resume the download, causing
the restore of the whole shard to fail and retry. This commit allows
Elasticsearch to detect such a mid-stream failure and to resume the download
from where it failed.
2019-09-17 13:11:36 +01:00
Armin Braun b0f09b279f
Make Snapshot Logic Write Metadata after Segments (#45689) (#46764)
* Write metadata during snapshot finalization after segment files to prevent outdated metadata in case of dynamic mapping updates as explained in #41581
* Keep the old behavior of writing the metadata beforehand in the case of mixed version clusters for BwC reasons
   * Still overwrite the metadata in the end, so even a mixed version cluster is fixed by this change if a newer version master does the finalization
* Fixes #41581
2019-09-17 13:09:39 +02:00
Tomas Della Vedova e1cf103980
Fixes for API specification (#46522) (#46736)
Follow-up of #42346
2019-09-17 11:49:24 +02:00
Costin Leau 683b5fdeca SQL: Support queries with HAVING over SELECT (#46709)
Handle queries with implicit GROUP BY where the aggregation is not in
the projection/SELECT but inside the filter/HAVING such as:

SELECT 1 FROM x HAVING COUNT(*) > 0

The engine now properly identifies the case and handles it accordingly.

Fix #37051

(cherry picked from commit fa53ca05d8219c27079b50b4a5b7aeb220c7cde2)
2019-09-17 11:14:39 +03:00