Commit Graph

90 Commits

Author SHA1 Message Date
Nhat Nguyen d52ca33bd9 Merge branch 'master' into ccr
* master: (68 commits)
  [DOCS] Removes X-Pack Elasticsearch release notes (#30272)
  Correct an example in the top-level suggester documentation. (#30224)
  [DOCS] Removes broken link
  [DOCS] Adds file realm configuration details (#30221)
  [DOCS] Adds PKI realm configuration details (#30225)
  Fix a reference to match_phrase_prefix in the match query docs. (#30282)
  Fix failure for validate API on a terms query (#29483)
  [DOCS] Fix 6.4-specific link in changelog (#30314)
  Remove RepositoriesMetaData variadic constructor (#29569)
  Test: increase authentication logging for debugging
  [DOCS] Removes redundant SAML realm settings (#30196)
  REST Client: Add Request object flavored methods (#29623)
  [DOCS] Adds changelog to Elasticsearch Reference (#30271)
  [DOCS] Fixes section error
  SQL: Teach the CLI to ignore empty commands (#30265)
  [DOCS] Adds Active Directory realm configuration details (#30223)
  [DOCS] Removes redundant file realm settings (#30192)
  [DOCS] Fixes users command name (#30275)
  Build: Move gradle wrapper jar to a dot dir (#30146)
  Build: Log a warning if disabling reindex-from-old (#30304)
2018-05-01 21:07:54 -04:00
Michael Basnight 62a9b8909e
Remove RepositoriesMetaData variadic constructor (#29569)
The variadic constructor was only used in a few places and the
RepositoriesMetaData class is backed by a List anyway, so just using a
List will make it simpler to instantiate it.
2018-05-01 15:02:06 -05:00
Nik Everett abe797bd42
SQL: Teach the CLI to ignore empty commands (#30265)
Cause the CLI to ignore commands that are empty or consist only of
newlines. This is a fairly standard thing for SQL CLIs to do.

It looks like:
```
sql> ;
sql>
   |
   | ;
sql> exit;
Bye!
```

I think I *could* have implemented this with a `CliCommand` that throws
out empty string but it felt simpler to bake it in to the `CliRepl`.

Closes #30000
2018-05-01 13:17:24 -04:00
Nik Everett 5e9e6fed90
HTML5ify Javadoc for xpack core (#30277)
xpack core contains a fork of `Cron` from quartz who's javadoc has a
`<table>` with non-html5 compatible stuff. This html5ifies the table and
switches the `:x-pack:plugin:core` project to building javadoc with
HTML5.
2018-05-01 08:32:58 -04:00
Dimitris Athanasiou 057cdffed5
[ML] Refactor DataStreamDiagnostics to use array (#30129)
This commit refactors the DataStreamDiagnostics class
achieving the following advantages:

- simpler code; by encapsulating the moving bucket histogram
into its own class
- better performance; by using an array to store the buckets
instead of a map
- explicit handling of gap buckets; in preparation of fixing #30080
2018-05-01 09:50:32 +01:00
Martijn van Groningen 8a2df6c3b9
[CCR] Only normalize -1 seqno in shard changes request. (#30238)
Prior to this change a -1 seqno would be normalized earlier, which
caused a leader shard containing a single operation to be ignored.

Closes #30227
2018-05-01 08:40:23 +02:00
Boaz Leskes 4a537ef03c
Bulk operation fail to replicate operations when a mapping update times out (#30244)
Starting with the refactoring in https://github.com/elastic/elasticsearch/pull/22778 (released in 5.3) we may fail to properly replicate operation when a mapping update on master fails. If a bulk
operations needs a mapping update half way, it will send a request to the master before continuing 
to index the operations. If that request times out or isn't acked (i.e., even one node in the cluster 
didn't process it within 30s), we end up throwing the exception and aborting the entire bulk. This is 
a problem because all operations that were processed so far are not replicated any more to the 
replicas.  Although these operations were never "acked" to the user (we threw an error) it cause the 
local checkpoint on the replicas to lag (on 6.x) and the primary and replica to diverge. 

This PR does a couple of things:
1) Most importantly, treat *any* mapping update failure as a document level failure, meaning only 
    the relevant indexing operation will fail.
2) Removes the mapping update callbacks from `IndexShard.applyIndexOperationOnPrimary` and 
    similar methods for simpler execution. We don't use exceptions any more when a mapping 
    update was successful.

I think we need to do more work here (the fact that a single slow node can prevent those mappings 
updates from being acked and thus fail operations is bad), but I want to keep this as small as I can 
(it is already too big).
2018-05-01 08:15:02 +02:00
David Roberts 225f7093a9
[ML] Include 3rd party C++ component notices (#30132)
The overall NOTICE file for the ML X-Pack module should
include the notices from the 3rd party C++ components as
well as the 3rd party Java components.
2018-04-30 20:05:27 +01:00
Chris Earle 725a5af2c6
_cluster/state should always return cluster_uuid (#30143)
Currently, the only way to get the REST response for the `/_cluster/state`
call to return the `cluster_uuid` is to request the `metadata` metrics,
which is one of the most expensive response structures. However, external
monitoring agents will likely want the `cluster_uuid` to correlate the
response with other API responses whether or not they want cluster
metadata.
2018-04-30 10:16:11 -04:00
Zachary Tong fee000a37f
[TEST] Redirect links to new locations (#30179)
We had a number of awaitsFix links that weren't updated after the xpack
merge.

Where possible I changed the links to the new locations, but in some
circumstances the original ticket was closed (suggesting the awaitsfix
should be removed) or was otherwise unclear the status.
2018-04-27 09:24:46 -07:00
Nhat Nguyen 112b5f1744 Merge branch 'master' into ccr
* master: (24 commits)
  Watcher: Ensure mail message ids are unique per watch action (#30112)
  REST: Remove GET support for clear cache indices (#29525)
  SQL: Correct error message (#30138)
  Require acknowledgement to start_trial license (#30135)
  Fix a bug in FieldCapabilitiesRequest#equals and hashCode. (#30181)
  SQL: Add BinaryMathProcessor to named writeables list (#30127)
  Tests: Use buildDir as base for generated-resources (#30191)
  Fix SliceBuilderTests#testRandom failures
  Build: Fix deb version to use tilde with prerelease versions (#29000)
  Fix edge cases in CompositeKeyExtractorTests (#30175)
  Document time unit limitations for date histograms (#30177)
  Add support for field capabilities to the high-level REST client. (#29664)
  Remove licenses missed by the migration (#30128)
  [DOCS] Updates docker installation package details (#30110)
  Fix TermsSetQueryBuilder.doEquals() method (#29629)
  [Monitoring] Remove unhelpful Monitoring tests (#30144)
  [Test] Fix RenameProcessorTests.testRenameExistingFieldNullValue() (#29655)
  add copyright/scope configuration for intellij to Contributing Guide (#29688)
  [test] include oss tar in packaging tests (#30155)
  TEST: Update settings should go through cluster state (#29682)
  ...
2018-04-27 09:23:31 -04:00
Alexander Reelsen 707ba28d48
Watcher: Ensure mail message ids are unique per watch action (#30112)
Email message IDs are supposed to be unique. In order to guarantee this,
we need to take the action id of a watch action into account as well,
not just the watch id from the watch execution context. This prevents
that two actions from the same watch execution end up with the same
message id.
2018-04-27 08:55:25 +02:00
Costin Leau e0b8893645
SQL: Correct error message (#30138)
* SQL: Correct error message

Error messages had placeholders that were not replaced; this PR fixes
that

Fix #30016
2018-04-27 09:24:25 +03:00
Tim Brooks 592481e4ed
Require acknowledgement to start_trial license (#30135)
This is related to #30134. It modifies the start_trial action to require
an acknowledgement parameter in the rest request to actually start the
trial license. There are backwards compatibility issues as prior ES
versions did not support this parameter. To handle this, it is assumed
that a request coming from a node prior to 6.3 is acknowledged. And
attempts to write a non-acknowledged request to a prior to 6.3 node will
throw an exception.

Additionally this PR adds messages about the trial license the user is
generating.
2018-04-26 21:42:44 -04:00
Costin Leau 804c38303f
SQL: Add BinaryMathProcessor to named writeables list (#30127)
BinaryMathProcessor was missing from the list of register  named
writeables causing deserialization errors

Fix #30014
2018-04-27 01:43:44 +03:00
Christoph Büscher d0416c76fe
Fix edge cases in CompositeKeyExtractorTests (#30175)
Currently the test picks random java.util.TimeZone ids in some places.
Internally we still need to convert back to joda DateTimeZone by id
occassionally (e.g. when serializing to pre 6.3 versions). There are
some deprecated "SystemV/*" time zones that Jodas DateTimeZone refuses
to convert. This change excludes those rare cases from the set of
allowed random time zones. It would be quiet odd for them to appear in
practice.

Closes #30156
2018-04-26 20:06:47 +02:00
Michael Basnight cb7e3ffd75
Remove licenses missed by the migration (#30128)
A few of the old style license got kept around because their comment
string did not start with a space. This caused the license check to not
see it as a license and skip it. This commit cleans it up.
2018-04-26 11:37:04 -05:00
Martijn van Groningen 4bcecd34a1
Merge remote-tracking branch 'es/master' into ccr
* es/master:
  Watcher: Fold two smoke test projects into smoke-test-watcher (#30137)
  In the field capabilities API, deprecate support for providing fields in the request body. (#30157)
  Set JAVA_HOME before forking setup commands (#29647)
  Remove animal sniffer from low-level REST client (#29646)
  Cleanup .gitignore (#30145)
  Do not add noop from local translog to translog again (#29637)
  Build: Assert jar LICENSE and NOTICE files match
  Correct transport compression algorithm in docs (#29645)
  [Test] Fix docs check for DEB package in packaging tests (#30126)
  Painless: Docs Clean Up (#29592)
  Fixes Eclipse build for sql jdbc project (#30114)
  Remove reference to `not_analyzed`.
  [Docs] Add community analysis plugin (#29612)
2018-04-26 09:25:19 +02:00
Nik Everett a8f40b3e04 Build: Assert jar LICENSE and NOTICE files match
Adds tasks that check that the all jars that we build have LICENSE.txt
and NOTICE.txt files and that the files are correct. Sets check to
depend on these task.

This is mostly there for extra parnoia because we automatically
configure all Jar tasks to include the LICENSE.txt and NOTICE.txt
files anyway. But it is quite possible to add configuration to those
tasks that would override either file.

This causes check to depend on several more things than it used to.
Take, for example, javadoc:

check depends on the new verifyJavadocJarNotice which depends on
extractJavadocJar which depends on javadocJar which depends on
javadoc, this check now depends on javadoc.
2018-04-25 19:53:24 -04:00
Colin Goodheart-Smithe c02b895653
Fixes Eclipse build for sql jdbc project (#30114)
The bundled configuration isn't recognised by eclipse so these
dependencies are missed when it imports the `x-pack:plugin:sql:jdbc`
project. This change makes these dependencies compile dependencies if
the build is running for Eclipse.
2018-04-25 16:10:32 +01:00
Martijn van Groningen b126ffa1cb
Updated sha files 2018-04-25 16:00:48 +02:00
Martijn van Groningen 596fab9eb7
fixed compile error after merge 2018-04-25 15:59:22 +02:00
Martijn van Groningen e6b88fa5a0
removed duplicated license 2018-04-25 12:18:02 +02:00
Martijn van Groningen 831e7e0777
Fixed test failure after migrating xpack ccr code to elasticsearch repository. 2018-04-25 10:10:06 +02:00
Martijn van Groningen 5a67a0f78f
Applying changes required for ccr after moving ccr code to elasticsearch 2018-04-25 08:03:29 +02:00
Martijn van Groningen cbd38983f4
Opened x-pack ccr code 2018-04-25 08:03:28 +02:00
Michael Basnight 72f57c8e72
Add comments inadvertently removed during migrate
A few files had their first comment removed even though it did not
contain a license. This re-adds those comments.
2018-04-24 13:41:09 -05:00
David Kyle cfc66a1fd5 [ML] Wait for updates to established memory usage
Tests need to wait for changes to the job's established memory usage to
propagate and an over enthusiastic optimisation meant jobs were updated
from stale state causing recent change to be lost.
2018-04-24 13:46:58 -04:00
Jason Tedor ab101976d6 Fix SQL CLI on Windows
This commit fixes the classpath for the SQL CLI tool on Windows. As the
x-pack bin folder was collapsed into the distribution bin folder, the
location of the classpath here needed to no longer contain the old
plugins directory.
2018-04-20 15:34:01 -07:00
Jason Tedor d99d0fa669 Add distribution type to startup scripts
This commit adds the distribution type to the startup scripts so that we
can discern from log output and the main response the type of the
distribution (deb/rpm/tar/zip).
2018-04-20 15:34:01 -07:00
Jason Tedor c7f6b35485 Fix classpath for X-Pack scripts on Windows
With the move of X-Pack to a module, the classpath for the scripts needs
to be adjusted. This was done on Unix, but not for Windows. This commit
addresses Windows.
2018-04-20 15:34:01 -07:00
Ryan Ernst d8636d0fe3 Rename syskeygen
This commit renames syskeygen to elasticsearch-syskeygen
2018-04-20 15:34:01 -07:00
Jason Tedor c7c0e330b8 Rename users
This commit renames users to elasticsearch-users.
2018-04-20 15:34:01 -07:00
Jason Tedor b05c8bf781 Rename migrate
This commit renames migrate to elasticsearch-migrate.
2018-04-20 15:34:00 -07:00
Jason Tedor 2d3fac307a Rename saml-metadata
This commit renames saml-metadata to elasticsearch-saml-metadata.
2018-04-20 15:34:00 -07:00
Jason Tedor 8a8a1dfc2e Rename sql-cli
This commit renames sql-cli to elasticsearch-sql-cli.
2018-04-20 15:34:00 -07:00
Ryan Ernst 6dbb7be067 Rename croneval
This commit renames croneval to elasticsearch-croneval
2018-04-20 15:34:00 -07:00
Jason Tedor 7ba2defdbf Rename setup-passwords
This commit renames setup-passwords to elasticsearch-setup-passwords.
2018-04-20 15:34:00 -07:00
Jason Tedor 03ffd16921 Rename certutil
This commit renames certutil to elasticsearch-certutil.
2018-04-20 15:33:59 -07:00
Jason Tedor a6b4682711 Rename certgen
This commit renames certgen to elasticsearch-certgen.
2018-04-20 15:33:59 -07:00
Jason Tedor e64e6d8996 Add distribution flavor to startup scripts
This commit adds the distribution flavor (default versus oss) to the
build process which is passed through the startup scripts to
Elasticsearch. This change will be used to customize the message on
attempting to install/remove x-pack based on the distribution flavor.
2018-04-20 15:33:58 -07:00
Ryan Ernst fab5e21e7d Build: Split distributions into oss and default
This commit makes x-pack a module and adds it to the default
distrubtion. It also creates distributions for zip, tar, deb and rpm
which contain only oss code.
2018-04-20 15:33:57 -07:00
Ryan Ernst 2efd22454a Migrate x-pack-elasticsearch source to elasticsearch 2018-04-20 15:29:54 -07:00
Martijn van Groningen 9b9d0f9057 Enabled licence header check and fixed unchecked casts. (#4408) 2018-04-20 11:15:52 +02:00
Martijn van Groningen cfd7847628 fixed issues after merging in master 2018-04-20 07:59:13 +02:00
Nhat Nguyen f97aec7b8b Sibling of enforce access to translog via engine
Since elastic/elasticsearch#29542, we no longer expose translog instance
but only provide creating translog snapshot method. This commit adapts
that change in CCR branch.

Relates elastic/elasticsearch#29542
2018-04-18 11:54:00 -04:00
Martijn van Groningen 56ca59a513 Add the ability to the follow index to follow an index in a remote cluster.
The follow index api completely reuses CCS infrastructure that was exposed via:
https://github.com/elastic/elasticsearch/pull/29495

This means that the leader index parameter support the same ccs index
to indicate that an index resides in a different cluster.

I also added a qa module that smoke tests the cross cluster nature of ccr.
The idea is that this test just verifies that ccr can read data from a
remote leader index and that is it, no crazy randomization or indirectly
testing other features.
2018-04-17 07:36:40 +02:00
Martijn van Groningen c0d42e9cd1 Fixed test 2018-04-16 10:48:46 +02:00
Martijn van Groningen a94b38b88e Fixed compile errors and test failures after merging master into ccr. 2018-04-13 16:35:09 +02:00
Martijn van Groningen d77f756f5c ccr: use indices stats api to fetch global checkpoint of the follower shards and
keep track of shard follow stats inside shard follow stats' node task instead of persistent task status.

By maintaining the shard follow stats inside its node task the stats update is quicker as
no cluster state update is required. The stats are now transient; meaning if the task
is going to run a different node then the stats are gone too. Currently only the processed
global checkpoint is being tracked and this is being restored when a shard follow node task
starts via the indices stats api (the reason of the first change of this change). Other stats
that we may add in the future (like fetch_time, see: https://gist.github.com/s1monw/dba13daf8493bf48431b72365e110717)
it is ok if we start from zero in case a shard follow task moves to another node.
2018-04-05 14:52:20 +02:00