Commit Graph

1201 Commits

Author SHA1 Message Date
Alpar Torok a38f509284 Testclusters: convert left-overs from checkPart1 ()
* Testclusters: convert left-overs from checkPart1
2019-06-25 19:14:45 +03:00
Alpar Torok 2beb193311 Disable testing conventions on Windows ()
Tests are disabled on Windows. Conventions also need to be disabled.
2019-06-25 18:49:08 +03:00
Przemyslaw Gomulka 67a67ac3ce
Fix DieWithDignity logs assertion backport()
This test is likely to kill the server in the middle of writing logs. This means that we can end up with logs with partially written json log lines and standard json parsers would fail on this.
This fix is to use regular expressions on json logs.(just like the previous approach on plain text logs)
closes 
2019-06-25 14:35:24 +02:00
Andrei Stefan 84d3de1fee
Mute DieWithDignityIT test. () 2019-06-20 15:10:21 +03:00
Mark Vieira e44b8b1e2e
[Backport] Remove dependency substitutions 7.x ()
* Remove unnecessary usage of Gradle dependency substitution rules ()

(cherry picked from commit 12d583dbf6f7d44f00aa365e34fc7e937c3c61f7)
2019-06-04 13:50:23 -07:00
Przemyslaw Gomulka d5061a151a
Remove suppresions for "unchecked" for hamcrest varargs methods Backport(41528)
In hamcrest 2.1 warnings for unchecked varargs were fixed by hamcrest using @SafeVarargs for those matchers where this warning occurred.
This PR is aimed to remove these annotations when Matchers.contains ,Matchers.containsInAnyOrder or Matchers.hasItems was used
backport 
2019-05-31 13:58:49 +02:00
Jason Tedor 371cb9a8ce
Remove Log4j 1.2 API as a dependency ()
We had this as a dependency for legacy dependencies that still needed
the Log4j 1.2 API. This appears to no longer be necessary, so this
commit removes this artifact as a dependency.

To remove this dependency, we had to fix a few places where we were
accidentally relying on Log4j 1.2 instead of Log4j 2 (easy to do, since
both APIs were on the compile-time classpath).

Finally, we can remove our custom Netty logger factory. This was needed
when we were on Log4j 1.2 and handled logging in our own unique
way. When we migrated to Log4j 2 we could have dropped this
dependency. However, even then Netty would still pick up Log4j 1.2 since
it was on the classpath, thus the advantage to removing this as a
dependency now.
2019-05-30 16:08:07 -04:00
Nhat Nguyen ab832c4f17 Use doc instead of _doc in FullClusterRestartIT
ES does not accept doc type starting with underscore until 6.2.0. We
have to use "doc" instead of "_doc" in FullClusterRestartIT if we are
upgrading from a 6.2.0- cluster.

Closes 
2019-05-27 21:35:56 -04:00
Nhat Nguyen 4123ade2b6 Add test ensure we can execute update requests in mixed cluster
Relates 
2019-05-27 18:27:50 -04:00
Ignacio Vera 5d3e381648
mute test testClosedIndices () 2019-05-27 12:05:17 +02:00
Nhat Nguyen 5d2fcc53e4 Unmute FullClusterRestartIT#testClosedIndices
Fixed in 
Closes 
2019-05-26 11:20:04 -04:00
Nhat Nguyen 84df48ccb3 Recovery with syncId should verify seqno infos ()
This change verifies and aborts recovery if source and target have the
same syncId but different sequenceId. This commit also adds an upgrade
test to ensure that we always utilize syncId.
2019-05-21 22:44:17 -04:00
Zachary Tong 6ae6f57d39
[7.x Backport] Force selection of calendar or fixed intervals ()
The date_histogram accepts an interval which can be either a calendar
interval (DST-aware, leap seconds, arbitrary length of months, etc) or
fixed interval (strict multiples of SI units). Unfortunately this is inferred
by first trying to parse as a calendar interval, then falling back to fixed
if that fails.

This leads to confusing arrangement where `1d` == calendar, but
`2d` == fixed.  And if you want a day of fixed time, you have to
specify `24h` (e.g. the next smallest unit).  This arrangement is very
error-prone for users.

This PR adds `calendar_interval` and `fixed_interval` parameters to any
code that uses intervals (date_histogram, rollup, composite, datafeed, etc).
Calendar only accepts calendar intervals, fixed accepts any combination of
units (meaning `1d` can be used to specify `24h` in fixed time), and both
are mutually exclusive.

The old interval behavior is deprecated and will throw a deprecation warning.
It is also mutually exclusive with the two new parameters. In the future the
old dual-purpose interval will be removed.

The change applies to both REST and java clients.
2019-05-20 12:07:29 -04:00
Ryan Ernst 69824ed908 Cleanup plugin bin directories ()
This commit adds deletion of the bin directory to postrm cleanup. While
the package's bin files are cleaned up by the package manager, plugins
may have created subdirectories under bin. We already cleanup plugins,
but not the extra bin dirs their installation created.

closes 
2019-05-10 11:00:41 -07:00
Ryan Ernst 6343ec3d3e
Update lintian overrides () ()
The deb package has been updated several times in the past to contain
overrides in order to pass lintian inspection. However, there have never
been any tests to ensure we do not fallback to failure. This commit
updates the overrides file given things that have changed since 2.x like
adding ML and bundling the jdk.

closes 
2019-05-08 12:09:02 -07:00
Jason Tedor d7fd51a84e
Provide names for all artifact repositories ()
This commit adds a name for each Maven and Ivy repository used in the
build.
2019-05-07 06:35:28 -04:00
Jason Tedor ff317d0d7f
Set metadata sources for Ivy repositories ()
We have faked some Ivy repositories on a few artifact locations. Today
when Gradle attempts to resolve these artifacts, it follows its default
strategy to search for Gradle metadata, then Maven POM files, then Ivy
descriptors, and finally will fallback to looking directly for the
artifact. This wastes times on remote network calls that will 404 anyway
since these metadata resources will not exist for these fake Ivy
repositories. This commit overrides the Gradle strategy to look directly
for artifacts.
2019-05-05 11:16:52 -04:00
Jason Tedor 241c4ef97a
Use https for artifact locations
This commit switches to using https for some artifact locations.
2019-05-03 16:15:48 -04:00
Jason Tedor 03c959f188
Upgrade keystore on package install ()
When Elasticsearch is run from a package installation, the running
process does not have permissions to write to the keystore. This is
because of the root:root ownership of /etc/elasticsearch. This is why we
create the keystore if it does not exist during package installation. If
the keystore needs to be upgraded, that is currently done by the running
Elasticsearch process. Yet, as just mentioned, the Elasticsearch process
would not have permissions to do that during runtime. Instead, this
needs to be done during package upgrade. This commit adds an upgrade
command to the keystore CLI for this purpose, and that is invoked during
package upgrade if the keystore already exists. This ensures that we are
always on the latest keystore format before the Elasticsearch process is
invoked, and therefore no upgrade would be needed then. While this bug
has always existed, we have not heard of reports of it in practice. Yet,
this bug becomes a lot more likely with a recent change to the format of
the keystore to remove the distinction between file and string entries.
2019-05-03 10:34:30 -04:00
Jason Tedor 7f3ab4524f
Bump 7.x branch to version 7.2.0
This commit adds the 7.2.0 version constant to the 7.x branch, and bumps
BWC logic accordingly.
2019-05-01 13:38:57 -04:00
Jim Ferenczi a8c3c5cd78
Restore rolling upgrade test for the _all field ()
This commit adapts the rolling upgrade test introduced in 
to ignore runs that upgrade from a version on or after 7.0.

Closes 
2019-04-27 18:37:15 +02:00
Ryan Ernst 7e3875d781 Upgrade hamcrest to 2.1 ()
hamcrest has some improvements in newer versions, like FileMatchers
that make assertions regarding file exists cleaner. This commit upgrades
to the latest version of hamcrest so we can start using new and improved
matchers.
2019-04-24 23:40:03 -07:00
Ryan Ernst cc867e8a5a
Remove /var/run/elasticsearch from packages ()
The pid dir for both systemd and init.d is already managed by those
respective systems (tmpfiles.d and the init script, respectively). Since
the /var/run dir is often mounted as tmpfs, it does not make sense to
have the elasticsearch pid dir added by the package installation. This
commit removes that empty dir from deb and rpm.
2019-04-22 09:57:51 -07:00
Mark Vieira 1287c7d91f
[Backport] Replace usages RandomizedTestingTask with built-in Gradle Test () ()
* Replace usages RandomizedTestingTask with built-in Gradle Test ()

This commit replaces the existing RandomizedTestingTask and supporting code with Gradle's built-in JUnit support via the Test task type. Additionally, the previous workaround to disable all tasks named "test" and create new unit testing tasks named "unitTest" has been removed such that the "test" task now runs unit tests as per the normal Gradle Java plugin conventions.

(cherry picked from commit 323f312bbc829a63056a79ebe45adced5099f6e6)

* Fix forking JVM runner

* Don't bump shadow plugin version
2019-04-09 11:52:50 -07:00
Nhat Nguyen a9416708d0 Ensure relocation occur in testRelocationWithConcurrentIndexing ()
If the relocation is throttled, the subsequent search request on the
target node (i.e., with preference _only_nodes=target_node) will fail
because some shards have not moved to that node yet. With this change,
we will wait for the relocation happens by busily checking the routing
table of the testing index on the target node.

Closes 
2019-04-08 22:47:20 -04:00
Mark Vieira 2569fb60de Avoid sharing source directories as it breaks intellij ()
* Avoid sharing source directories as it breaks intellij
* Subprojects share main project output classes directory
* Fix jar hell
* Fix sql security with ssl integ tests
* Relax dependency ordering rule so we don't explode on cycles
2019-04-08 17:26:46 +03:00
Alpar Torok 21b99a3aeb Remove unneded cluster config from test ()
This configuration doesn't influence the logger test.
Should be removed to avoid confusion
2019-04-08 17:26:46 +03:00
Ioannis Kakavas aea4e6596f
Source additional files correctly in elasticsearch-cli ()
Since we only source additional sources from the same dir as our
cli scripts, resolve the path relevant to $ES_HOME
2019-04-05 17:42:34 -04:00
Przemyslaw Gomulka 65d25186d3
Bat scripts to work with JAVA_HOME with parentheses Backports ()
backports :
Bat scripts to work with JAVA_HOME with parentheses ()
Link to SYSTEM_JAVA_HOME on windows ()
2019-04-04 18:42:05 +02:00
Przemyslaw Gomulka a6faf85f26
Migrate systemd packaging tests from bats to java backport() ()
Migrating systemd bats tests from bats to java dsl. This also covers
partially the sysv, but more must be added

relates 
backport 
2019-04-04 08:41:28 +02:00
Yannick Welsch 6ae7d593ea Avoid background sync on relocated primary ()
There were some test failures caused by the background retention lease sync running on a relocated
primary. This commit fixes the situation that triggered the assertion and reactivates the failing test.

Closes 
2019-04-03 20:28:48 +02:00
Jason Tedor df65e46d10
Deprecate versions of Java prior to Java 11 ()
This commit deprecates versions of Java prior to Java 11. This commit
will cause a warning to be printed to standard error when any command
line tool is invoked, or when Elasticsearch is started. Additionally, we
log a deprecation message when Elasticsearch is started.
2019-04-03 06:39:40 -04:00
Dimitrios Liappis a3f53cfd05
[TEST] Mute RecoveryWithConcurrentIndexing test ()
Due to https://github.com/elastic/elasticsearch/issues/40731

Backport of https://github.com/elastic/elasticsearch/pull/40733
2019-04-02 16:23:22 +03:00
Luca Cavanna f2ca45c210 Add integration tests to verify CCS output ()
We recently introduced the option to minimize network roundtrips when
executing cross-cluster search requests. All the changes made around
that are separately unit tested, and there are some yaml tests that
exercise the new code-path which involves multiple coordination steps.
This commit adds new integration tests that compare the output given by
CCS when running the same queries using the two different execution
modes available.

Relates to 
2019-03-29 17:01:14 +01:00
Ryan Ernst aa24669e95
Resolve JAVA_HOME at windows service install time ()
On windows, JAVA_HOME is currently resolved when the windows service is started. However, this is contrary to what our documentation states. This commit moves resolution to service install. This has the side effect of making java existence checking optional in elasticsearch-env.bat, since the rest of the service commands do not require java.

closes 
2019-03-26 19:48:02 -07:00
Nhat Nguyen 606411bce8 Mute testRelocationWithConcurrentIndexing
Tracked at 
2019-03-20 12:46:36 -04:00
Nhat Nguyen d58864745c Dump recovery if fail to get doc count with preference ()
With this change, we will dump the recovery state if we fail to get doc
count for a given index with a preference in rolling upgrade tests. We
should have more information to look into why the provided preference is
not valid. I also unmuted `testRelocationWithConcurrentIndexing` in this
change.

Relates 
2019-03-19 13:09:04 -04:00
Henning Andersen dde41cc2dd Node repurpose tool ()
When a node is repurposed to master/no-data or no-master/no-data, v7.x
will not start (see  and ). The `elasticsearch repurpose`
tool can fix this by cleaning up the problematic data.
2019-03-19 11:52:02 +01:00
Andrey Ershov d35d9d1886 cd ES_HOME in elasticsearch-env ()
This commit adds cd $ES_HOME to elasticsearch-env and removes it from
elasticsearch. This way, both elasticsearch and elasticsearch-cli are
executed with the working directory set to $ES_HOME. The need for the
fix arose from the following bug:
1. Explicitly set path.data to relative to ES_HOME path in
elasticsearch.yml.
2. Run elasticsearch from any directory. Elasticsearch is able to
correctly start.
3. Stop elasticsearch.
4. Run elasticsearch-node unsafe-bootstrap, not from ES_HOME directory.
It will fail with an exception.
This commit fixes the issue and adds a new test.

This PR fixes the issue and adds a new test.
Also tests >=100 are renamed because alphabetic order does not work for
them.

(cherry picked from commit 2ffc29306ff7366efc598e7b4dd2ce528895cd3a
with fixes by  and )
2019-03-18 16:26:27 +01:00
Jason Tedor 86d1d03c37
Remove cluster state size ()
This commit removes the cluster state size field from the cluster state
response, and drops the backwards compatibility layer added in 6.7.0 to
continue to support this field. As calculation of this field was
expensive and had dubious value, we have elected to remove this field.
2019-03-15 17:16:25 -04:00
Ryan Ernst 8f09c77777 Add no-jdk distributions ()
This commit adds a variant for every official distribution that omits
the bundled jdk. The "no-jdk" naming is conveyed through the package
classifier, alongside the platform. Package tests are also added for
each new distribution.
2019-03-15 00:55:57 -07:00
Jason Tedor 9181668edf
Stop returning cluster state size by default ()
Computing the compressed size of the cluster state on every invocation
of cluster:monitor/state action is expensive, and the value of this
field is dubious anyway. Therefore we want to remove computing this
field. As a first step, we stop computing and return this field by
default. To avoid breaking users, we will give them a system property to
use to tide them over until the next major release when we will actually
remove this field. This comes with a deprecation warning too, and the
backport to the appropriate minor will also include a note in the
migration guide. There will be a follow-up to remove this field in the
next major version.
2019-03-14 08:57:55 -04:00
Yannick Welsch 4f941c6963 Do not swallow exceptions in TimedRunnable ()
Executors of type fixed_auto_queue_size (i.e. search / search_throttled) wrap runnables into
TimedRunnable, which is an AbstractRunnable. This is dangerous as it might silently swallow
exceptions, and possibly miss calling a response listener. While this has not triggered any failures in
the tests I have run so far, it might help uncover future problems.

Follow-up to 
2019-03-11 19:03:12 +01:00
Ryan Ernst 465343f12a
Bundle java in distributions ()
* Bundle java in distributions

Setting up a jdk is currently a required external step when installing
elasticsearch. This is particularly problematic for the rpm/deb packages
as installing a jdk in the same package installation command does not
guarantee any order, so must be done in separate steps. Additionally,
JAVA_HOME must be set and often causes problems in selecting a correct
jdk when, for example, the system java is an older unsupported version.

This commit bundles platform specific openjdks into each distribution.
In addition to eliminating the issues above, it also presents future
possible improvements like using jlink to build jdk images only
containing modules that elasticsearch uses.

closes 
2019-03-08 11:04:18 -08:00
Przemyslaw Gomulka c23ff0803e
Fix JsonLoggerTests.testJsonInStacktraceMessageIsSplitted backport#39830
a test was splitting a json log message with a wrong line separator.
Using System.lineSeparator consistently prevents this problem
closes 
backport#39830
2019-03-08 16:34:17 +01:00
Alpar Torok 0f89427eb6
Back port build changes from same version bwc tests ()
* Back port build changes from 

This back-ports how versions are determined and bwc test are set up from
  without enabling the bwc from current version tests so it's
 easier/possible to backmerge future buld changes.
It's expected that the tets are lacking many of the required fixes in
this version to enable them.
2019-03-07 17:25:09 +02:00
Nhat Nguyen 15ab0e23e0 Unmute testHistoryUUIDIsGenerated
Fixed in 
Relates to 
2019-03-04 10:21:23 -05:00
David Turner f5fb93afdf Mute FullClusterRestartIT#testClosedIndices ()
Relates 
2019-03-02 12:26:45 +00:00
Tanguy Leroux e005eeb0b3
Backport support for replicating closed indices to 7.x ()()
Backport support for replicating closed indices ()
    
    Before this change, closed indexes were simply not replicated. It was therefore
    possible to close an index and then decommission a data node without knowing
    that this data node contained shards of the closed index, potentially leading to
    data loss. Shards of closed indices were not completely taken into account when
    balancing the shards within the cluster, or automatically replicated through shard
    copies, and they were not easily movable from node A to node B using APIs like
    Cluster Reroute without being fully reopened and closed again.
    
    This commit changes the logic executed when closing an index, so that its shards
    are not just removed and forgotten but are instead reinitialized and reallocated on
    data nodes using an engine implementation which does not allow searching or
     indexing, which has a low memory overhead (compared with searchable/indexable
    opened shards) and which allows shards to be recovered from peer or promoted
    as primaries when needed.
    
    This new closing logic is built on top of the new Close Index API introduced in
    6.7.0 (). Some pre-closing sanity checks are executed on the shards before
    closing them, and closing an index on a 8.0 cluster will reinitialize the index shards
    and therefore impact the cluster health.
    
    Some APIs have been adapted to make them work with closed indices:
    - Cluster Health API
    - Cluster Reroute API
    - Cluster Allocation Explain API
    - Recovery API
    - Cat Indices
    - Cat Shards
    - Cat Health
    - Cat Recovery
    
    This commit contains all the following changes (most recent first):
    * c6c42a1 Adapt NoOpEngineTests after 
    * 3f9993d Wait for shards to be active after closing indices ()
    * 5e7a428 Adapt the Cluster Health API to closed indices ()
    * 3e61939 Adapt CloseFollowerIndexIT for replicated closed indices ()
    * 71f5c34 Recover closed indices after a full cluster restart ()
    * 4db7fd9 Adapt the Recovery API for closed indices ()
    * 4fd1bb2 Adapt more tests suites to closed indices ()
    * 0519016 Add replica to primary promotion test for closed indices ()
    * b756f6c Test the Cluster Shard Allocation Explain API with closed indices ()
    * c484c66 Remove index routing table of closed indices in mixed versions clusters ()
    * 00f1828 Mute CloseFollowerIndexIT.testCloseAndReopenFollowerIndex()
    * e845b0a Do not schedule Refresh/Translog/GlobalCheckpoint tasks for closed indices ()
    * cf9a015 Adapt testIndexCanChangeCustomDataPath for replicated closed indices ()
    * b9becdd Adapt testPendingTasks() for replicated closed indices ()
    * 02cc730 Allow shards of closed indices to be replicated as regular shards ()
    * e53a9be Fix compilation error in IndexShardIT after merge with master
    * cae4155 Relax NoOpEngine constraints ()
    * 54d110b [RCI] Adapt NoOpEngine to latest FrozenEngine changes
    * c63fd69 [RCI] Add NoOpEngine for closed indices ()
    
    Relates to 
2019-03-01 14:48:26 +01:00
Ryan Ernst 6e7b643775
Deprecate fallback to java on PATH ()
Finding java on the path is sometimes confusing for users and
unexpected, as well as leading to a different java being used than a
user expects.  This commit adds warning messages when starting
elasticsearch (or any tools like the plugin cli) and using java found
on the PATH instead of via JAVA_HOME.
2019-02-20 17:07:11 -08:00