Commit Graph

945 Commits

Author SHA1 Message Date
Jim Ferenczi dbd857341f
Upgrade to 7.4.0-snapshot-1ed95c097b (#30357)
Upgrade to lucene-7.4.0-snapshot-1ed95c097b

This version contains:
* An Analyzer for Korean
* An IntervalQuery and IntervalsSource that retrieve minimum intervals of positional queries.
* A new API to retrieve matches (offsets and positions) of a query for a single document.
* Support for soft deletes in the index writer.
* A fixed shingle filter that handles index time synonyms.
* Support for emoji sequence in ICUTokenizer (with an upgrade to icu 61.1)
2018-05-04 11:44:22 +02:00
Alexander Reelsen 0e6cbbd811
Watcher: Ensure trigger service pauses execution (#30363)
When the watcher service pauses execution due to a cluster state update,
the trigger service and its engines also need to pause properly instead
of keeping going. This is also important when the .watches index is 
deleted, so that watches don't stay in a triggered mode.
2018-05-04 09:02:04 +02:00
Jay Modi aa0d7c73f8
Security: reduce garbage during index resolution (#30180)
The IndexAndAliasesResolver resolves the indices and aliases for each
request and also handles local and remote indices. The current
implementation uses the ResolvedIndices class to hold the resolved
indices and aliases. While evaluating the indices and aliases against
the user's permissions, the final value for ResolvedIndices is
constructed. Prior to this change, this was done by creating a
ResolvedIndices for the first set of indices and for each additional
addition, a new ResolvedIndices object is created and merged with
the existing one. With a small number of indices and aliases this does
not pose a large problem; however as the number of indices/aliases
grows more list allocations and array copies are needed resulting in a
large amount of garbage and severely impacted performance.

This change introduces a builder for ResolvedIndices that appends to
mutable lists until the final value has been constructed, which will
ultimately reduce the amount of garbage generated by this code.
2018-05-03 12:48:23 -06:00
jaymode 9573492a9e
Test: use trial license in qa tests with security
QA tests that use security need to use a trial license instead of a
basic license. Basic licenses do not enable security so these tests are
not running in the expected configuration. This can also lead to issues
that seem completely unrelated such as authentication failures right
after cluster formation.

The authentication failure right after cluster formation happens since
a request makes it past the authentication license checks and the
request starts the authentication process. During authentication the
cluster forms and the license is updated to a basic license, which
disables all realms. The request that is being authenticated then tries
to iterate over the realms, but the realms are empty and the request
cannot be authenticated. This results in a authentication failure even
though the credentials provided are correct.

Closes #30306
2018-05-03 10:33:08 -06:00
Dimitris Athanasiou a1e23feba2
[ML] Add integration test for model plots (#30359)
Relates #30004
2018-05-03 17:02:45 +01:00
Costin Leau 65dbc17510
SQL: Fix bug caused by empty composites (#30343)
When dealing with filtering, a composite aggregation might return empty
buckets (which have been filtered) which gets sent as is to the client.
Unfortunately this interprets the response as no more data instead of
retrying.

This now has changed and the listener keeps retrying until either the
query has ended or data passes the filter.

Fix #30292
2018-05-03 17:13:32 +03:00
Dimitris Athanasiou 3b260dcfc1
[ML] Account for gaps in data counts after job is reopened (#30294)
This commit fixes an issue with the data diagnostics were
empty buckets are not reported even though they should. Once
a job is reopened, the diagnostics do not get initialized from
the current data counts (especially the latest record timestamp).
The result is that if the data that is sent have a time gap compared
to the previous ones, that gap is not accounted for in the empty bucket
count.

This commit fixes that by initializing the diagnostics with the current
data counts.

Closes #30080
2018-05-03 15:08:24 +01:00
Alexander Reelsen 18394540ab
Tests: Use different watch ids per test in smoke test (#30331)
Each test now has its own watch id that is being used.
This ensures there are no old history entries, which can potentially
lead to broken test assertions.
2018-05-03 11:41:06 +02:00
Alexander Reelsen 2c38d12e23
Watcher: Make start/stop cycle more predictable and synchronous (#30118)
The current implementation starts/stops watcher using an executor. This
can result in our of order operations.

This commit reduces those executor calls to an absolute minimum in order
to be able to do state changes within the cluster state listener method,
which runs in sequence.

When a state change occurs that forces the watcher service to pause
(like no watcher index, no master node, no local shards), the service is
now in a paused state.

Pausing is a super lightweight operation, which marks the
ExecutionService as paused and waits for the currently executing watches
to finish in the background via an executor. The same applies for
stopping, the potentially long running operation is outsourced in to an
executor, as waiting for executed watches is decoupled from the current
state.

The only other long running operation is starting, where watches need to
be loaded. This is also done via an executor, but has an additional
protection by checking the cluster state version it was started with. If
another cluster state version was trying to load the watches, then this
loading will not take effect.

This PR also cleans up some unused states, like the a simple boolean in
the HistoryStore/TriggeredWatchStore marking it as started or stopped,
as this can now be caught in the execution service.

Another advantage of this approach is the fact, that now only triggered
watches are not getting executed, while watches that are run via the
Execute Watch API will still be executed regardless if watcher is
stopped or not.

Lastly the TickerScheduleTriggerEngine thread now only starts on data nodes.
2018-05-03 09:47:12 +02:00
Andy Bristol 824e648662 [test] add debug logging for packaging test 2018-05-02 18:09:44 -07:00
lcawley b3516786ec [DOCS] Removed X-Pack Breaking Changes 2018-05-02 15:52:14 -07:00
lcawley 427d6912ea [DOCS] Fixes link to TLS LDAP info 2018-05-02 15:30:49 -07:00
Tim Brooks 226b45e509
Update versions for start_trial after backport (#30218)
This commit is a follow up to #30135. It updates the stream
compatibility versions in the start_trial requests and responses to
reflect that fact that this work has been backported to 6.3.
2018-05-02 15:51:59 -06:00
Lisa Cawley e697299103
[DOCS] Fixes broken links to bootstrap user (#30349) 2018-05-02 13:09:25 -07:00
Ioannis Kakavas cca1a2a7cf
Make licensing FIPS-140 compliant (#30251)
Necessary changes so that the licensing functionality can be
used in a JVM in FIPS 140 approved mode.
* Uses adequate salt length in encryption
* Changes key derivation to PBKDF2WithHmacSHA512 from a custom
  approach with SHA512 and manual key stretching
* Removes redundant manual padding

Other relevant changes:
* Uses the SAH512 hash instead of the encrypted key bytes as the
  key fingerprint to be included in the license specification
* Removes the explicit verification check of the encryption key
  as this is implicitly checked in signature verification.
2018-05-02 22:19:43 +03:00
Lisa Cawley 3e9fe3c9cd
[DOCS] Reorganizes authentication details in Stack Overview (#30280) 2018-05-02 12:08:02 -07:00
Ryan Ernst fb0aa562a5
Network: Remove http.enabled setting (#29601)
This commit removes the http.enabled setting. While all real nodes (started with bin/elasticsearch) will always have an http binding, there are many tests that rely on the quickness of not actually needing to bind to 2 ports. For this case, the MockHttpTransport.TestPlugin provides a dummy http transport implementation which is used by default in ESIntegTestCase.

closes #12792
2018-05-02 11:42:05 -07:00
Lisa Cawley 383856a175
[DOCS] Adds LDAP realm configuration details (#30214) 2018-05-02 11:22:32 -07:00
Lisa Cawley 5064ff6ad4
[DOCS] Adds native realm configuration details (#30215) 2018-05-02 10:56:31 -07:00
Lisa Cawley 0d7ac9a74c
[DOCS] Enables edit links for X-Pack pages (#30278) 2018-05-02 10:13:42 -07:00
Costin Leau 7790cb5fa9
SQL: Reduce number of ranges generated for comparisons (#30267)
* SQL: Reduce number of ranges generated for comparisons

Rewrote optimization rule for combining ranges by improving the
detection of binary comparisons in a tree to better combine
them in a range, regardless of their place inside an expression.
Additionally, improve the comparisons of Numbers of different types
Also, improve reassembly of conjunction/disjunction into balanced
trees.
Do not promote BinaryComparisons to Ranges since it introduces NULL
boundaries and thus a corner-case that needs too much handling
Compare BinaryComparisons directly between themselves and to Ranges

Fix #30017
2018-05-02 19:35:01 +03:00
Nhat Nguyen d621fc7a00
Add tombstone document into Lucene for Noop (#30226)
This commit adds a tombstone document into Lucene for every No-op. 
With this change, Lucene index is expected to have a complete history 
of operations like Translog. In fact, this guarantee is subjected to the
soft-deletes retention merge policy.

Relates #29530
2018-05-02 09:08:29 -04:00
Adrien Grand 231a63fdf8
Remove useless version checks in REST tests. (#30165)
Many tests are added with a version check so that they do not run against a
version that doesn't have the feature yet. Master is 7.0, so all tests that
do not run against 6.0+ can be removed and the version check can be removed
on all tests that always run on 6.0+.
2018-05-02 11:34:15 +02:00
Jason Tedor 217d090dcd Merge branch 'master' into ccr
* master:
  Fix message content in users tool (#30293)
  [DOCS] Fixes links to breaking changes
  [DOCS] Adds new installation package details (#29590)
  Revert "Build: Move gradle wrapper jar to a dot dir (#30146)"
2018-05-01 23:49:21 -04:00
Nhat Nguyen eb4281edef CCR side #30244
Relates #30244
2018-05-01 21:08:24 -04:00
Nhat Nguyen d52ca33bd9 Merge branch 'master' into ccr
* master: (68 commits)
  [DOCS] Removes X-Pack Elasticsearch release notes (#30272)
  Correct an example in the top-level suggester documentation. (#30224)
  [DOCS] Removes broken link
  [DOCS] Adds file realm configuration details (#30221)
  [DOCS] Adds PKI realm configuration details (#30225)
  Fix a reference to match_phrase_prefix in the match query docs. (#30282)
  Fix failure for validate API on a terms query (#29483)
  [DOCS] Fix 6.4-specific link in changelog (#30314)
  Remove RepositoriesMetaData variadic constructor (#29569)
  Test: increase authentication logging for debugging
  [DOCS] Removes redundant SAML realm settings (#30196)
  REST Client: Add Request object flavored methods (#29623)
  [DOCS] Adds changelog to Elasticsearch Reference (#30271)
  [DOCS] Fixes section error
  SQL: Teach the CLI to ignore empty commands (#30265)
  [DOCS] Adds Active Directory realm configuration details (#30223)
  [DOCS] Removes redundant file realm settings (#30192)
  [DOCS] Fixes users command name (#30275)
  Build: Move gradle wrapper jar to a dot dir (#30146)
  Build: Log a warning if disabling reindex-from-old (#30304)
2018-05-01 21:07:54 -04:00
Tim Vernum 66daaaa1cc
Fix message content in users tool (#30293)
The elasticsearch-users utility had various messages that were
outdated or incorrect. This commit updates the output from this
command to reflect current terminology and configuration.
2018-05-02 11:04:28 +10:00
lcawley 42cc122a9b [DOCS] Fixes links to breaking changes 2018-05-01 17:21:28 -07:00
Lisa Cawley 092dd6cb89
[DOCS] Removes X-Pack Elasticsearch release notes (#30272) 2018-05-01 16:05:23 -07:00
lcawley 6a55eead94 [DOCS] Removes broken link 2018-05-01 14:42:29 -07:00
Lisa Cawley db44a4cddb
[DOCS] Adds file realm configuration details (#30221) 2018-05-01 14:11:38 -07:00
Lisa Cawley 86addc0c8b
[DOCS] Adds PKI realm configuration details (#30225) 2018-05-01 13:47:24 -07:00
Michael Basnight 62a9b8909e
Remove RepositoriesMetaData variadic constructor (#29569)
The variadic constructor was only used in a few places and the
RepositoriesMetaData class is backed by a List anyway, so just using a
List will make it simpler to instantiate it.
2018-05-01 15:02:06 -05:00
jaymode cdae8ff5a0
Test: increase authentication logging for debugging
This commit increases the logging for authentication in the x-pack
multi-node qa test project. This is needed to assist in debugging HTTP
authorization failures in waiting for the second node in these tests.

See #30306
2018-05-01 13:15:59 -06:00
Lisa Cawley 1a0838bd0b
[DOCS] Removes redundant SAML realm settings (#30196) 2018-05-01 11:53:11 -07:00
Lisa Cawley 5b5c98c96b
[DOCS] Adds changelog to Elasticsearch Reference (#30271) 2018-05-01 10:34:26 -07:00
lcawley 4dec3c36ee Merge remote-tracking branch 'upstream/master' 2018-05-01 10:22:05 -07:00
lcawley 0f688e0867 [DOCS] Fixes section error 2018-05-01 10:21:39 -07:00
Nik Everett abe797bd42
SQL: Teach the CLI to ignore empty commands (#30265)
Cause the CLI to ignore commands that are empty or consist only of
newlines. This is a fairly standard thing for SQL CLIs to do.

It looks like:
```
sql> ;
sql>
   |
   | ;
sql> exit;
Bye!
```

I think I *could* have implemented this with a `CliCommand` that throws
out empty string but it felt simpler to bake it in to the `CliRepl`.

Closes #30000
2018-05-01 13:17:24 -04:00
Lisa Cawley 7933f5e28e
[DOCS] Adds Active Directory realm configuration details (#30223) 2018-05-01 09:15:13 -07:00
Lisa Cawley 846783c2b6
[DOCS] Removes redundant file realm settings (#30192) 2018-05-01 08:55:35 -07:00
Lisa Cawley b34e5cf216
[DOCS] Fixes users command name (#30275) 2018-05-01 08:52:22 -07:00
Nik Everett 5e9e6fed90
HTML5ify Javadoc for xpack core (#30277)
xpack core contains a fork of `Cron` from quartz who's javadoc has a
`<table>` with non-html5 compatible stuff. This html5ifies the table and
switches the `:x-pack:plugin:core` project to building javadoc with
HTML5.
2018-05-01 08:32:58 -04:00
David Turner b9e1860f36 Mark JdbcSqlSpecIT as @AwaitsFix
Relates #30292
2018-05-01 10:03:18 +01:00
Dimitris Athanasiou 057cdffed5
[ML] Refactor DataStreamDiagnostics to use array (#30129)
This commit refactors the DataStreamDiagnostics class
achieving the following advantages:

- simpler code; by encapsulating the moving bucket histogram
into its own class
- better performance; by using an array to store the buckets
instead of a map
- explicit handling of gap buckets; in preparation of fixing #30080
2018-05-01 09:50:32 +01:00
Martijn van Groningen 8a2df6c3b9
[CCR] Only normalize -1 seqno in shard changes request. (#30238)
Prior to this change a -1 seqno would be normalized earlier, which
caused a leader shard containing a single operation to be ignored.

Closes #30227
2018-05-01 08:40:23 +02:00
Boaz Leskes 4a537ef03c
Bulk operation fail to replicate operations when a mapping update times out (#30244)
Starting with the refactoring in https://github.com/elastic/elasticsearch/pull/22778 (released in 5.3) we may fail to properly replicate operation when a mapping update on master fails. If a bulk
operations needs a mapping update half way, it will send a request to the master before continuing 
to index the operations. If that request times out or isn't acked (i.e., even one node in the cluster 
didn't process it within 30s), we end up throwing the exception and aborting the entire bulk. This is 
a problem because all operations that were processed so far are not replicated any more to the 
replicas.  Although these operations were never "acked" to the user (we threw an error) it cause the 
local checkpoint on the replicas to lag (on 6.x) and the primary and replica to diverge. 

This PR does a couple of things:
1) Most importantly, treat *any* mapping update failure as a document level failure, meaning only 
    the relevant indexing operation will fail.
2) Removes the mapping update callbacks from `IndexShard.applyIndexOperationOnPrimary` and 
    similar methods for simpler execution. We don't use exceptions any more when a mapping 
    update was successful.

I think we need to do more work here (the fact that a single slow node can prevent those mappings 
updates from being acked and thus fail operations is bad), but I want to keep this as small as I can 
(it is already too big).
2018-05-01 08:15:02 +02:00
lcawley 51902238f3 [DOCS] Fixes syskeygen command name 2018-04-30 13:20:22 -07:00
David Roberts 225f7093a9
[ML] Include 3rd party C++ component notices (#30132)
The overall NOTICE file for the ML X-Pack module should
include the notices from the 3rd party C++ components as
well as the 3rd party Java components.
2018-04-30 20:05:27 +01:00
Lisa Cawley e95a7aa6f0
[DOCS] Removes redundant Active Directory realm settings (#30190) 2018-04-30 08:52:21 -07:00
Lisa Cawley 05160e6cd8
[DOCS] Removes redundant LDAP realm settings (#30193) 2018-04-30 08:04:15 -07:00
Chris Earle 725a5af2c6
_cluster/state should always return cluster_uuid (#30143)
Currently, the only way to get the REST response for the `/_cluster/state`
call to return the `cluster_uuid` is to request the `metadata` metrics,
which is one of the most expensive response structures. However, external
monitoring agents will likely want the `cluster_uuid` to correlate the
response with other API responses whether or not they want cluster
metadata.
2018-04-30 10:16:11 -04:00
Lisa Cawley 7eaec6031d
[DOCS] Fixes broken links (#30219) 2018-04-27 13:24:15 -07:00
Lisa Cawley 962c965812
[DOCS] Adds native realm security settings (#30186) 2018-04-27 12:40:05 -07:00
Nik Everett f4ed902698
CCS: Drop http address from remote cluster info (#29568)
They are expensive to fetch and no longer needed by Kibana so they
*shouldn't* be needed by anyone else either.

Closes #29207
2018-04-27 14:19:00 -04:00
Zachary Tong fee000a37f
[TEST] Redirect links to new locations (#30179)
We had a number of awaitsFix links that weren't updated after the xpack
merge.

Where possible I changed the links to the new locations, but in some
circumstances the original ticket was closed (suggesting the awaitsfix
should be removed) or was otherwise unclear the status.
2018-04-27 09:24:46 -07:00
Tanguy Leroux 7ae3b3b155
Move repository-s3 fixture tests to QA test project (#29372)
This commit moves the repository-s3 fixture test added in #29296 in a
new `repository-s3/qa/amazon-s3` project. This new project allows the
REST integration tests to be executed using the real S3 service when
all the required environment variables are provided. When no env var
is provided, then the tests are executed using the fixture added
in #29296.

The REST tests located at the `repository-s3`plugin  project now only 
verify that the plugin is correctly loaded.

The REST tests have been adapted to allow a bucket name and a base 
path to be specified as env vars. This way it is possible to run the tests
with different base paths (could be anything, like a CI job name or a
branch name) without multiplicating buckets.

Related to #29349
2018-04-27 16:49:06 +02:00
Nhat Nguyen 112b5f1744 Merge branch 'master' into ccr
* master: (24 commits)
  Watcher: Ensure mail message ids are unique per watch action (#30112)
  REST: Remove GET support for clear cache indices (#29525)
  SQL: Correct error message (#30138)
  Require acknowledgement to start_trial license (#30135)
  Fix a bug in FieldCapabilitiesRequest#equals and hashCode. (#30181)
  SQL: Add BinaryMathProcessor to named writeables list (#30127)
  Tests: Use buildDir as base for generated-resources (#30191)
  Fix SliceBuilderTests#testRandom failures
  Build: Fix deb version to use tilde with prerelease versions (#29000)
  Fix edge cases in CompositeKeyExtractorTests (#30175)
  Document time unit limitations for date histograms (#30177)
  Add support for field capabilities to the high-level REST client. (#29664)
  Remove licenses missed by the migration (#30128)
  [DOCS] Updates docker installation package details (#30110)
  Fix TermsSetQueryBuilder.doEquals() method (#29629)
  [Monitoring] Remove unhelpful Monitoring tests (#30144)
  [Test] Fix RenameProcessorTests.testRenameExistingFieldNullValue() (#29655)
  add copyright/scope configuration for intellij to Contributing Guide (#29688)
  [test] include oss tar in packaging tests (#30155)
  TEST: Update settings should go through cluster state (#29682)
  ...
2018-04-27 09:23:31 -04:00
Alexander Reelsen 707ba28d48
Watcher: Ensure mail message ids are unique per watch action (#30112)
Email message IDs are supposed to be unique. In order to guarantee this,
we need to take the action id of a watch action into account as well,
not just the watch id from the watch execution context. This prevents
that two actions from the same watch execution end up with the same
message id.
2018-04-27 08:55:25 +02:00
Costin Leau e0b8893645
SQL: Correct error message (#30138)
* SQL: Correct error message

Error messages had placeholders that were not replaced; this PR fixes
that

Fix #30016
2018-04-27 09:24:25 +03:00
Tim Brooks 592481e4ed
Require acknowledgement to start_trial license (#30135)
This is related to #30134. It modifies the start_trial action to require
an acknowledgement parameter in the rest request to actually start the
trial license. There are backwards compatibility issues as prior ES
versions did not support this parameter. To handle this, it is assumed
that a request coming from a node prior to 6.3 is acknowledged. And
attempts to write a non-acknowledged request to a prior to 6.3 node will
throw an exception.

Additionally this PR adds messages about the trial license the user is
generating.
2018-04-26 21:42:44 -04:00
Costin Leau 804c38303f
SQL: Add BinaryMathProcessor to named writeables list (#30127)
BinaryMathProcessor was missing from the list of register  named
writeables causing deserialization errors

Fix #30014
2018-04-27 01:43:44 +03:00
Ryan Ernst 55e1b1e8b5
Tests: Use buildDir as base for generated-resources (#30191)
This commit moves the generated-resources directory created by many qa
projects into the build directory, so it is not seen as unknown files to
git.
2018-04-26 15:36:14 -07:00
Christoph Büscher d0416c76fe
Fix edge cases in CompositeKeyExtractorTests (#30175)
Currently the test picks random java.util.TimeZone ids in some places.
Internally we still need to convert back to joda DateTimeZone by id
occassionally (e.g. when serializing to pre 6.3 versions). There are
some deprecated "SystemV/*" time zones that Jodas DateTimeZone refuses
to convert. This change excludes those rare cases from the set of
allowed random time zones. It would be quiet odd for them to appear in
practice.

Closes #30156
2018-04-26 20:06:47 +02:00
Michael Basnight cb7e3ffd75
Remove licenses missed by the migration (#30128)
A few of the old style license got kept around because their comment
string did not start with a space. This caused the license check to not
see it as a license and skip it. This commit cleans it up.
2018-04-26 11:37:04 -05:00
Lisa Cawley eebcdce7f9
[DOCS] Updates docker installation package details (#30110) 2018-04-26 08:54:29 -07:00
Chris Earle 1d81ec1562
[Monitoring] Remove unhelpful Monitoring tests (#30144)
This removes some monitoring tests that have been silenced for a long
time. These tests don't really provide any value for the upgrade suite and
they just create noise due to their occasional timing-related failures.
2018-04-26 11:32:46 -04:00
Andy Bristol 67c0cf1dbf
[test] include oss tar in packaging tests (#30155)
Add the oss tar distribution to the packaging test plugin. Test the oss
tar distribution in the core packaging tests, and the non-oss tar
distribution in the x-pack packaging tests.
2018-04-26 06:58:41 -07:00
Martijn van Groningen 4bcecd34a1
Merge remote-tracking branch 'es/master' into ccr
* es/master:
  Watcher: Fold two smoke test projects into smoke-test-watcher (#30137)
  In the field capabilities API, deprecate support for providing fields in the request body. (#30157)
  Set JAVA_HOME before forking setup commands (#29647)
  Remove animal sniffer from low-level REST client (#29646)
  Cleanup .gitignore (#30145)
  Do not add noop from local translog to translog again (#29637)
  Build: Assert jar LICENSE and NOTICE files match
  Correct transport compression algorithm in docs (#29645)
  [Test] Fix docs check for DEB package in packaging tests (#30126)
  Painless: Docs Clean Up (#29592)
  Fixes Eclipse build for sql jdbc project (#30114)
  Remove reference to `not_analyzed`.
  [Docs] Add community analysis plugin (#29612)
2018-04-26 09:25:19 +02:00
Alexander Reelsen fadcce8367
Watcher: Fold two smoke test projects into smoke-test-watcher (#30137)
In order to have less qa/ projects, this commit removes the
watcher-smoke-test-mustache and the watcher-smoke-test-painless projects
and moves the tests of both into the watcher-smoke-test projects.

This results in less projects to parse when calling gradle and a shorter test
time due to being able to run everything within one elasticsearch
instance.
2018-04-26 09:23:54 +02:00
Nik Everett a8f40b3e04 Build: Assert jar LICENSE and NOTICE files match
Adds tasks that check that the all jars that we build have LICENSE.txt
and NOTICE.txt files and that the files are correct. Sets check to
depend on these task.

This is mostly there for extra parnoia because we automatically
configure all Jar tasks to include the LICENSE.txt and NOTICE.txt
files anyway. But it is quite possible to add configuration to those
tasks that would override either file.

This causes check to depend on several more things than it used to.
Take, for example, javadoc:

check depends on the new verifyJavadocJarNotice which depends on
extractJavadocJar which depends on javadocJar which depends on
javadoc, this check now depends on javadoc.
2018-04-25 19:53:24 -04:00
Colin Goodheart-Smithe c02b895653
Fixes Eclipse build for sql jdbc project (#30114)
The bundled configuration isn't recognised by eclipse so these
dependencies are missed when it imports the `x-pack:plugin:sql:jdbc`
project. This change makes these dependencies compile dependencies if
the build is running for Eclipse.
2018-04-25 16:10:32 +01:00
Martijn van Groningen b126ffa1cb
Updated sha files 2018-04-25 16:00:48 +02:00
Martijn van Groningen 596fab9eb7
fixed compile error after merge 2018-04-25 15:59:22 +02:00
Martijn van Groningen e6b88fa5a0
removed duplicated license 2018-04-25 12:18:02 +02:00
Martijn van Groningen 831e7e0777
Fixed test failure after migrating xpack ccr code to elasticsearch repository. 2018-04-25 10:10:06 +02:00
Martijn van Groningen 5a67a0f78f
Applying changes required for ccr after moving ccr code to elasticsearch 2018-04-25 08:03:29 +02:00
Martijn van Groningen cbd38983f4
Opened x-pack ccr code 2018-04-25 08:03:28 +02:00
Michael Basnight 72f57c8e72
Add comments inadvertently removed during migrate
A few files had their first comment removed even though it did not
contain a license. This re-adds those comments.
2018-04-24 13:41:09 -05:00
David Kyle cfc66a1fd5 [ML] Wait for updates to established memory usage
Tests need to wait for changes to the job's established memory usage to
propagate and an over enthusiastic optimisation meant jobs were updated
from stale state causing recent change to be lost.
2018-04-24 13:46:58 -04:00
Jason Tedor ab101976d6 Fix SQL CLI on Windows
This commit fixes the classpath for the SQL CLI tool on Windows. As the
x-pack bin folder was collapsed into the distribution bin folder, the
location of the classpath here needed to no longer contain the old
plugins directory.
2018-04-20 15:34:01 -07:00
Jason Tedor d99d0fa669 Add distribution type to startup scripts
This commit adds the distribution type to the startup scripts so that we
can discern from log output and the main response the type of the
distribution (deb/rpm/tar/zip).
2018-04-20 15:34:01 -07:00
Jason Tedor c7f6b35485 Fix classpath for X-Pack scripts on Windows
With the move of X-Pack to a module, the classpath for the scripts needs
to be adjusted. This was done on Unix, but not for Windows. This commit
addresses Windows.
2018-04-20 15:34:01 -07:00
Ryan Ernst d8636d0fe3 Rename syskeygen
This commit renames syskeygen to elasticsearch-syskeygen
2018-04-20 15:34:01 -07:00
Jason Tedor c7c0e330b8 Rename users
This commit renames users to elasticsearch-users.
2018-04-20 15:34:01 -07:00
Jason Tedor b05c8bf781 Rename migrate
This commit renames migrate to elasticsearch-migrate.
2018-04-20 15:34:00 -07:00
Jason Tedor 2d3fac307a Rename saml-metadata
This commit renames saml-metadata to elasticsearch-saml-metadata.
2018-04-20 15:34:00 -07:00
Jason Tedor 8a8a1dfc2e Rename sql-cli
This commit renames sql-cli to elasticsearch-sql-cli.
2018-04-20 15:34:00 -07:00
Ryan Ernst 6dbb7be067 Rename croneval
This commit renames croneval to elasticsearch-croneval
2018-04-20 15:34:00 -07:00
Jason Tedor 7ba2defdbf Rename setup-passwords
This commit renames setup-passwords to elasticsearch-setup-passwords.
2018-04-20 15:34:00 -07:00
Jason Tedor 03ffd16921 Rename certutil
This commit renames certutil to elasticsearch-certutil.
2018-04-20 15:33:59 -07:00
Jason Tedor a6b4682711 Rename certgen
This commit renames certgen to elasticsearch-certgen.
2018-04-20 15:33:59 -07:00
Ryan Ernst 0d8aa7527e Reorganize license files
This commit moves the apache and elastic license files into a new
root level `licenses` directory and rewrites the top level LICENSE.txt
to clarify the repository has a mix of apache and elastic licensed code.
2018-04-20 15:33:59 -07:00
Ryan Ernst 9a45662309 Convert license header check to use the new Elastic License
This commit adapts the license headers check to no longer look for the
ealsticsearch confidential license, but instead to look for the new
Elastic License header.
2018-04-20 15:33:59 -07:00
Jason Tedor e64e6d8996 Add distribution flavor to startup scripts
This commit adds the distribution flavor (default versus oss) to the
build process which is passed through the startup scripts to
Elasticsearch. This change will be used to customize the message on
attempting to install/remove x-pack based on the distribution flavor.
2018-04-20 15:33:58 -07:00
Ryan Ernst fab5e21e7d Build: Split distributions into oss and default
This commit makes x-pack a module and adds it to the default
distrubtion. It also creates distributions for zip, tar, deb and rpm
which contain only oss code.
2018-04-20 15:33:57 -07:00
Ryan Ernst cb56bf49d5 Fixup build after opening x-pack 2018-04-20 15:33:46 -07:00
Ryan Ernst 2efd22454a Migrate x-pack-elasticsearch source to elasticsearch 2018-04-20 15:29:54 -07:00
Martijn van Groningen 9b9d0f9057 Enabled licence header check and fixed unchecked casts. (#4408) 2018-04-20 11:15:52 +02:00
Martijn van Groningen cfd7847628 fixed issues after merging in master 2018-04-20 07:59:13 +02:00
Nhat Nguyen f97aec7b8b Sibling of enforce access to translog via engine
Since elastic/elasticsearch#29542, we no longer expose translog instance
but only provide creating translog snapshot method. This commit adapts
that change in CCR branch.

Relates elastic/elasticsearch#29542
2018-04-18 11:54:00 -04:00
Martijn van Groningen 56ca59a513 Add the ability to the follow index to follow an index in a remote cluster.
The follow index api completely reuses CCS infrastructure that was exposed via:
https://github.com/elastic/elasticsearch/pull/29495

This means that the leader index parameter support the same ccs index
to indicate that an index resides in a different cluster.

I also added a qa module that smoke tests the cross cluster nature of ccr.
The idea is that this test just verifies that ccr can read data from a
remote leader index and that is it, no crazy randomization or indirectly
testing other features.
2018-04-17 07:36:40 +02:00
Martijn van Groningen c0d42e9cd1 Fixed test 2018-04-16 10:48:46 +02:00
Martijn van Groningen a94b38b88e Fixed compile errors and test failures after merging master into ccr. 2018-04-13 16:35:09 +02:00
Martijn van Groningen d77f756f5c ccr: use indices stats api to fetch global checkpoint of the follower shards and
keep track of shard follow stats inside shard follow stats' node task instead of persistent task status.

By maintaining the shard follow stats inside its node task the stats update is quicker as
no cluster state update is required. The stats are now transient; meaning if the task
is going to run a different node then the stats are gone too. Currently only the processed
global checkpoint is being tracked and this is being restored when a shard follow node task
starts via the indices stats api (the reason of the first change of this change). Other stats
that we may add in the future (like fetch_time, see: https://gist.github.com/s1monw/dba13daf8493bf48431b72365e110717)
it is ok if we start from zero in case a shard follow task moves to another node.
2018-04-05 14:52:20 +02:00
Martijn van Groningen d976fa44e7 Removed LocalCheckpointTracker usage. 2018-03-29 07:41:23 +02:00
Martijn van Groningen a22a7d079d ccr: Added maximum translog limit that a single shard changes response can return.
This limit is based on the number of estimate bytes in each translog
operation that fall between the minimum and maximum request sequence number.

If this limit is met then the shard follow task executor will make sure
that a subsequent shard changes request will be performed to fetch the
remaining translog operations.

This limit is needed in order to protect against returning too many
translog operations in a single shard changes response.

Relates to #2436
2018-03-28 15:49:57 +02:00
Martijn van Groningen 282740610b Fixed test after merging in master branch. 2018-03-28 09:54:41 +02:00
Nhat Nguyen 51111a8106 CCR: Stop FollowExistingIndexAction after report failure (#4111)
We check for the existence of both leader and follower index, then properly 
report to the caller. However, we do not return after reporting failure. This
causes the caller receive exception twice: IllegalArgumentException then
NullPointerException. This commit makes sure to stop the action after reporting
failure.
2018-03-26 13:56:47 -04:00
Martijn van Groningen 9e4c68c389 Fixed compile and test errors after merging in master 2018-03-16 17:47:10 +01:00
Martijn van Groningen 10cfa21a68 required changes after merge master branch into ccr branch. 2018-02-22 15:03:33 +01:00
Martijn van Groningen 1a9a7ffe97 removed hack 2018-02-07 17:54:28 +01:00
Martijn van Groningen c442d14f1d Several changes that were required after merging master into the ccr branch. 2018-02-05 13:25:58 +01:00
Martijn van Groningen 4e818254ad re-enabled java integration tests 2018-01-25 14:18:34 +01:00
Martijn van Groningen 05d3d2e49c fix packages after merge 2018-01-24 09:28:42 +01:00
Jason Tedor 9b6bb2c635 Enable run task for CCR
This commit enables the run task for ccr by specifying that the ccr
project not be evaluated until after core is evaluated. This is
important since ccr is alphabetically before core and thus Gradle
evaluates it first.

Relates #3665
2018-01-22 15:07:20 -05:00
Martijn van Groningen 83a82d83d0 Moved ccr source code to its own gradle module after xpack split. 2018-01-22 11:09:04 +01:00
Martijn van Groningen 7a4860452a Update the persistent task status in the background.
Follow up for #3256
2018-01-16 10:48:55 +01:00
Jason Tedor bad5135d35 Fix checkpoints in assertion
The checkpoints in the assertion message that the follower checkpoint is
less than the leader checkpoint are backwards. This commit fixes this
message.
2018-01-15 11:50:12 -05:00
Martijn van Groningen 18788531c8 Fixed test failure.
Relates to #3256
2018-01-12 14:04:26 +01:00
Martijn van Groningen a56b0479e1 Changed the chunk coordinator to process the chunks concurrently. (#3256)
The shard follow task executor determines the range of translog operations
between the leader shard's global checkpoint and the last know processed
seqno by the current shard follow task that are missing.

Then the chunks coordinator can then chunk this range up in smaller ranges
if the requested range is above the configured max chunk size. If it is
smaller than the entire range then the chunk coordinator has just one
chuck to coordinate.

Each chunk is added to a queue and is processed by the ChunkProcessor,
that reads the translog ops from the leader shard and then indexes
these translog ops into the follow shard. After that a new chuck is polled
from the queue and the ChunkProcessor performs the same actions until
there are no more chunks in the queue to process. After that the shard
follow task executor will determine a new range of translog operations
to process.

This change changes the chunk coordinator to start polling from the chunk
queue with multiple threads at the same time to handle dealing with a higher
indexing load on the leader side better.
2018-01-10 15:29:11 +01:00
Martijn van Groningen 38aea9588b Fixed compile errors after merge. 2018-01-04 18:18:46 +01:00
Martijn van Groningen dbdefc5689 Fix issue after merging 2017-12-21 14:53:12 +01:00
Martijn van Groningen 973f126c24 Merge branch 'master' into ccr 2017-12-18 16:52:09 +01:00
Martijn van Groningen 6c06aacb57 Fixed NumberFormatException in the rest layer by only setting batch_size if it has been specified. 2017-12-07 09:34:25 +01:00
Martijn van Groningen 0dc5a89bc5 fixed compile error 2017-12-01 14:14:17 +01:00
Martijn van Groningen cc5665bce6 pass down batch size correctly 2017-12-01 14:14:17 +01:00
Martijn van Groningen 3cd5eed91e fixed code violations 2017-12-01 14:14:17 +01:00
Martijn van Groningen c3f7d4f580 * Made ChunkCoordinator unittest testable
* Fixed a small issue where each batch would fetch / index the previous batch last operation
* Made batch size a request param on the follow existing index api request.
  This makes is easy to tune this param when running tests from scripts.
* Changed default batch size from 256 to 1024.
2017-12-01 14:14:17 +01:00
Martijn van Groningen 2e382bf7f3 Disallow dynamic mapping updated in follow shards
I forgot to configure a mapping in the follow shard shard, which caused
a dynamic update (due to type auto creation), but this was ignored.

Subsequent searches in follow index then failed due to a mapping missing.
(The _id couldn't be fetched during fetch phase, because the mapping was missing)

We should at a later stage investigate how to best solve this, but for
know to avoid confusion just fail if a dynamic update happens in a
follow shard.
2017-12-01 09:33:06 +01:00
Jason Tedor be5f83a6bd Implement translog operation bulk action
This commit adds a bulk action for apply translog operations in bulk to
an index. This action is then used in the persistent task for CCR to
apply shard changes from a leader shard.

Relates #3147
2017-11-29 07:29:56 -05:00
Jason Tedor 11aa83011c Fix get operations between test
This test was broken by an upstream change that no longer guarantees we
see the operations from the upstream translog in the order they appear
in that translog. As such, the assertions in this test were too strong
so this commit relaxes them.

Relates #3153
2017-11-29 07:16:52 -05:00
Jason Tedor abd263fca9 Merge branch 'master' into ccr
* master:
  [DOCS] Fixed cross cluster search docs issue (#3113)
  Watcher: Add transform input for chained input (#2861)
  [DOCS] Fix cleaning up ML resources after tests
  [ML] Specify ML_ORIGIN when calling the job update endpoint internally (#3110)
  Refactor CLI commands as logging-aware commands
  [ML] Stop datafeed when job fails (#3107)
2017-11-27 14:03:10 -05:00
Martijn van Groningen e6ad1c3e9d ccr: Added apis and persistent tasks for following an index and log changes instead of indexing. 2017-11-22 09:04:59 +01:00
Jason Tedor 8e0b34b507 Override indexing plans
Operations from a leader shard will be indexed into the engine with the
origin set to primary. The problem is here is that then we have primary
semantics in the engine such as assertions about sequence numbers being
unassigned, and we do not have correct semantics for out-of-order
delivery of operations (as we should on a following engine, whether or
not it is primary since the ordering is determined from the
leader). This commit handles this by always using the replica plan for
indexing into a following engine, whether or not the engine is for a
primary shard.

Relates #3000
2017-11-15 13:38:46 -05:00
Jason Tedor 913936f2b0 Maintain order of operations semantics on follower
A following engine even for a primary shard needs to maintain order of
operations semantics as if it were behaving like a replica. That is,
rather than assuming that the order of operations presented to the
engine is the de facto order of operations as is the case for a leader
engine for a primary shard, a following engine must behave like all
replicas behave which is that they resolve order of operations based on
sequence numbers. This commit causes this to be the case for following
engines.

Relates #2931
2017-11-09 15:18:54 -05:00
Martijn van Groningen bf4e18fdfc fixed compile error 2017-11-02 15:12:35 +01:00
Jason Tedor 769349a9ab Add following engine implementation
This commit is a first step towards a following engine
implementation. Future work will build on this by using this engine to
execute operations on a following engine from another engine (typically
a remote leader engine) that has already assigned sequence numbers to
such operations.

Relates #2776
2017-10-30 13:38:02 -04:00
Martijn van Groningen 41c3dc91c1 Changed the shard changes api to use get threadpool instead of management threadpool. 2017-10-17 10:02:28 +02:00
Martijn van Groningen 2663b5ef39 remove unused import 2017-10-16 10:36:22 +02:00
Jason Tedor 85c1935d9d Increase generations in operations between test
This commit sets an index setting for the size of a translog generation
and increases the number of documents indexed to increase the chance of
multiple generations being present when testing getting operations
between two sequence numbers.
2017-10-16 08:59:18 +02:00
Jason Tedor bcd61bfca5 Fix off-by-one error in shard changes action test
This commit fixes an off-by-one error in the shard changes action test
for getting operations between two sequence numbers. The off-by-one
error arises because sequence numbers are indexed from zero, so if N
documents are indexed then the maximum sequence number starting from
zero would be N - 1.
2017-10-16 08:54:04 +02:00
Martijn van Groningen 1f495f59a1 [CCR] Add an internal api to read translog operations between a sequence number range (#2563)
* xdcr: Add an internal api to read translog operations between a sequence number range.

This api will be used later by the persistent task for the following index to pull data from the leader index.

The persistent task can fetch the global checkpoint from the shard stats for each primary shard of the leader index.
Based on the global checkpoint of the primary shards of the following index, the persistent task can send several
calls to the internal api added in this commit to replicate changes from follow index to leader index in a batched manner.
2017-10-16 07:38:03 +02:00
Jason Tedor 3033aba67b Introduce pluggable engine factory for CCR
This commit utilizes the pluggable engine factory feature in core to
introduce a pluggable engine factory for XDCR. For now this is only a
skeleton implementation to proof out the pluggable engine factory
concept. Future work will implement a genuine following engine for XDCR.

Relates #2655
2017-10-12 10:52:33 -04:00
Jason Tedor 063a76c3db Introduce CCR container class
This commit introduces the container class for CCR functionality. Future
work will expose more specific CCR functionality to the X-Pack plugin
through this class.

Relates #2704
2017-10-06 14:46:39 -04:00