A new engine now can have more than one empty translog since #28676.
This cause #testShouldPeriodicallyFlush failed because in the test we
asssume an engine should have one empty translog. This commit takes into
account the extra translog size of a new engine.
In the past the Low Level REST Client was super careful not to wrap
any exceptions that it throws from synchronous calls so that callers can
catch the exceptions and work with them. The trouble with that is that
the exceptions are originally thrown on the async thread pool and then
transfered back into calling thread. That means that the stack trace of
the exception doesn't have the calling method which is *super* *ultra*
confusing.
This change always wraps exceptions transferred from the async thread
pool so that the stack trace of the thrown exception contains the
caller's stack. It tries to preserve the type of the throw exception but
this is quite a fiddly thing to get right. We have to catch every type
of exception that we want to preserve, wrap with the same type and
rethrow. I've preserved the types of all exceptions that we had tests
mentioning but no other exceptions. The other exceptions are either
wrapped in `IOException` or `RuntimeException`.
Closes#28399
Looks like sometimes it catches a throttled watch.
```
18:23:19 FAILURE 7.93s | SmokeTestWatcherWithSecurityIT.testSearchTransformHasPermissions <<< FAILURES!
18:23:19 > Throwable elastic/x-pack-elasticsearch#1: java.lang.AssertionError:
18:23:19 > Expected: is "executed"
18:23:19 > but: was "throttled"
18:23:19 > at __randomizedtesting.SeedInfo.seed([C40A591DC5A7785E:ED7ABE031CA86AB3]:0)
18:23:19 > at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
18:23:19 > at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchTransformHasPermissions(SmokeTestWatcherWithSecurityIT.java:189)
18:23:19 > at java.lang.Thread.run(Thread.java:748)
```
Original commit: elastic/x-pack-elasticsearch@b96b6438d3
The serialization changes for rejected execution exceptions has been
backported to 6.x with the intention to appear in all versions since
6.3.0. Therefore, this BWC layer is no longer needed in master since
master would never speak to a node that does not speak the same
serialization.
The rejected execution handler API says that rejectedExecution(Runnable,
ThreadPoolExecutor) throws a RejectedExecutionException if the task must
be rejected due to capacity on the executor. We do throw something that
smells like a RejectedExecutionException (it is named
EsRejectedExecutionException) yet we violate the API because
EsRejectedExecutionException is not a RejectedExecutionException. This
has caused problems before where we try to catch RejectedExecution when
invoking rejectedExecution but this causes EsRejectedExecutionException
to go uncaught. This commit addresses this by modifying
EsRejectedExecutionException to extend
RejectedExecutionException.
Adds support for triple quoted strings to the documentation test
generator. Kibana's CONSOLE tool has supported them for a year but we
were unable to use them in Elasticsearch's docs because the process that
converts example snippets into tests couldn't handle this. This change
adds code to convert them into standard JSON so we can pass them to
Elasticsearch.
This commit adds a Samba4 test fixture that acts as a domain controller
and has the same contents as the cloud active directory instance that
we previously used for tests.
The tests also support reading information from environment variables
so that they can be run against a real active directory instance in our
CI builds.
In addition, this commit also fixes a few issues that surfaced when
making this change. The first is a change in the base DN that is
searched when performing down-level authentication. The base DN is
now the configuration object instead of the domain DN. This change was
required due to the original producing unnecessary referrals, which we
cannot easily follow when running against this test figure. Referrals
cannot easily be followed as they are returned by the ldap server with
an unresolvable DNS name unless the host points to the samba4 instance
for DNS. The port returned in the referral url is the one samba is bound
to, which differs from the port that is forwarded to the host by the
test fixture.
The other issue that is resolved by this change is the addition of
settings that allow specifying non-standard ports for active directory.
This is needed for down-level authentication as we may need to query
the regular port of active directory instead of the global catalog
port as the configuration object is not replicated to the global
catalog.
relates elastic/x-pack-elasticsearch#185
Relates elastic/x-pack-elasticsearch#3800
Original commit: elastic/x-pack-elasticsearch@883c742fba
I have seen this question a couple times already, most recently at
https://twitter.com/dimosr7/status/973872744965332993
I tried to keep the explanation as simple as I could, which is not always easy
as this is a matter of trade-offs.
This stops providing the "hkey" (hash) of the license now that the
Monitoring UI no longer uses it (6.3+).
Original commit: elastic/x-pack-elasticsearch@235402fa92
This adds an indicator to Monitoring's portion of X-Pack usage whether or
not collection is actually enabled. It's no longer enough to have an
exporter defined by default to know if monitoring is actually running.
Original commit: elastic/x-pack-elasticsearch@b2eb881d61
The settings `indices.recovery.concurrent_streams` and
`indices.recovery.concurrent_small_file_streams` were removed in
f5e4cd46164630e09f308ed78c512eea8bda8a05. This commit removes their last traces
from the codebase.
Today the synced-flush always issues a new sync-id even though all
shards haven't been changed since the last seal. This causes active
shards to have different a sync-id from offline shards even though all
were sealed and no writes since then.
This commit adjusts not to renew sync-id if all active shards are sealed
with the same sync-id.
Closes#27838
This commit (which will be reverted soon) adds logging on the output of
starting Wildfly. This is needed to debug an issue with Wildfly not
starting in CI.
The index prefix field is normally indexed as docs-only, given that it cannot
be used in phrases. However, in the case that the parent field has been indexed
with offsets, or has term-vector offsets, we should also store this in the index
prefix field for highlighting.
Note that this commit does not implement highlighting on prefix fields, but
rather ensures that future work can implement this without a backwards-break
in index data.
Closes#28994
The cd command on Windows has an oddity regarding changing
directories. If the drive of the current directory is a different drive
than than of the directory that was passed to the cd command, cd acts in
query mode and does not change the current directory. Instead, a flag is
needed to put the cd command into set mode so that the directory
actually changes. This causes a problem when starting Elasticsearch from
a directory different than the one where it is installed and this commit
fixes the issue.
The method `PersistentTasksClusterService.finishTask()` has been
modified since it was added and does not use any `removeOncompletion`
flag anymore. Its behavior is now similar to `removeTask()` and can be
replaced by this one. When a non existing task is removed, the cluster
state update task will fail and its `source` will still indicate
`finish persistent task`/`remove persistent task`.
Additionally:
* Included the existing update by query java api docs in java-api docs.
(for some reason it was never included, it needed some tweaking and
then it was good to go)
* moved delete-by-query / update-by-query code samples to java file so
that we can verify that these samples at least compile.
Closes#24203
Today we allow any other method of starting Elastisearch to override
jvm.options via ES_JAVA_OPTS. Yet, for some settings in the Windows
service, we do not allow this. This commit removes this in favor of
being consistent with other packaging choices.
This commit enhances the error messages reported when JAVA_HOME and
RUNTIME_JAVA_HOME are not correctly set to point towards the minimum
compiler and minimum runtime JDKs that are expected by the builds. The
previous error message would say:
Java 1.9 or above is required to build Elasticsearch
which is confusing if the user does have a JDK 9 installation and is
even the version that they have on their path yet they have JAVA_HOME
pointing to another JDK installation. The error message reported after
this change is:
the environment variable JAVA_HOME must be set to a JDK installation directory for Java 1.9 but is [/usr/java/jdk-8] corresponding to [1.8]
* CLI Command: MultiCommand must close subcommands to release resources properly
- Changes are done to override the close method and call close on subcommands using IOUtils#close
- Unit Test
Closes#28953
This is the last YAML test, that waits for a watch execution by
specifying some timeout value. This one also gets replaced with a java
test that uses `assertBusy()` and thus is much more likely to succeed.
relates elastic/x-pack-elasticsearch#1513
Original commit: elastic/x-pack-elasticsearch@c2ab8777f4
Currently ESIndexLevelReplicationTestCase executes write operations
without acquiring index shard permit. This may prevent the primary term
on replica from being updated or cause a race between resync and
indexing on primary. This commit ensures that write operations are
always executed under shard permit like the production code.
This adds a minimum compatible version to the model snapshot.
Nodes with a version earlier than that version cannot read
that model snapshot. Thus, such jobs are not assigned to
incompatible nodes.
relates elastic/x-pack-elasticsearch#4077
Original commit: elastic/x-pack-elasticsearch@2ffa6adce0
This adds back usage stats by pickybacking on the watcher stats, which
are already running distributed in order to collect and merge watcher
statistics.
In order to be able to track statistics, we need to add information for
each watch in an in-memory data structure that is processed whenever a
usage request is coming in. This processing creates a number of counters
for each node, which then are merged together in the usage stats.
relates elastic/x-pack-elasticsearch#4071
Original commit: elastic/x-pack-elasticsearch@c8bfed288f
Changes made in #28972 seems to have changed some assumptions about how
SMILE and CBOR write byte[] values and how this is tested. This changes
the generation of the randomized DocumentField values back to BytesArray
while expecting the JSON and YAML deserialisation to produce Base64
encoded strings and SMILE and CBOR to parse back BytesArray instances.
Closes#29080
I also had to make the test more lenient. This is due to the fact that
Lucene's RamUsageTester was changed in order not to reflect `java.*`
classes and the way that it estimates ram usage of maps is by assuming
it has similar memory usage to an `Object[]` array that stores all keys
and values. The implementation in `LiveVersionMap` tries to be slightly
more realistic by taking the load factor and linked lists into account,
so it usually gives a higher estimate which happens to be closer to
reality.
Closes#22548
Provide more actionable error message when installing an offline plugin
in the plugins directory, and the `plugins` directory for the node
contains plugin distribution.
Closes#27401
The toXContent serialization of the transform input was broken, which
could lead to the bad case that a watch could be stored with an invalid
toXContent serialization, that could not be read again, when the watch
should either be executed or even just returned by the Get watch API.
relates elastic/x-pack-elasticsearch#4049
Original commit: elastic/x-pack-elasticsearch@f06ad768b8
When parsing GetResponse it was possible that the equality check failed because
items in the map were in a different order (in the `.equals` implementation).
