We have a bootstrap check for the maximum size of the virtual memory
address space for the Elasticsearch process. We can set this in the
service file for Elasticsearch when installed as a service on
systemd-based systems for a better user experience than them fumbling
through thinking they should set this via /etc/security/limits.d (as a
lot of pages on the Internet would tell them) not realizing that systemd
completely ignores these for services and then trying to figure out how
to add a unit file for the Elasticsearch service.
Relates #25975
The systemd service file that ships with Elasticsearch installs on
systemd-based systems contains a suggestion for setting LimitMEMLOCK if
the user wants to enable bootstrap.memory_lock. However, this setting
this in the installed service file goes against best practices for
working with systemd, and goes against our existing documentation for
how to set this. Therefore, we should not have this suggestion in the
service file otherwise users might be led to think they should edit it
there.
Relates #25979
On non-Windows platforms, we ignore the environment variable
JAVA_TOOL_OPTIONS (this is an environment variable that the JVM respects
by default for picking up extra JVM options). The primary reason that we
ignore this because of the Jayatana agent on Ubuntu; a secondary reason
is that it produces an annoying "Picked up JAVA_TOOL_OPTIONS: ..."
output message. When the elasticsearch-env batch script was introduced
for Windows, ignoring this environment variable was deliberately not
carried over as the primary reason does not apply on Windows. However,
after additional thinking, it seems that we should simply be consistent
to the extent possible here (and also avoid that annoying "Picked up
JAVA_TOOL_OPTIONS: ..." on Windows too). This commit causes the Windows
version of elasticsearch-env to also ignore JAVA_TOOL_OPTIONS.
Relates #25968
This commit adds a bootstrap check for the maximum file size, and
ensures the limit is set correctly when Elasticsearch is installed as a
service on systemd-based systems.
Relates #25974
When invoking the elasticsearch-env.bat batch script on Windows, if the
script exits due to an error (e.g., Java can not be found, or the wrong
version of Java is found), then the script exits. Sadly, on Windows,
this does not also terminate the caller, instead returning control. This
means we have to explicitly exit so that is what we do in this commit.
Relates #25959
Today we strip some ignored JVM options before starting the main Java
process (e.g., we unset JAVA_TOOL_OPTIONS, and we ignore
JAVA_OPTS). However, there is another Java process that we start before
starting the main process: the Java version checker. We are currently
starting this before ignoring the undesired JVM options so the Java
version checker will pick up JAVA_TOOL_OPTIONS and it will silently
ignore JAVA_OPTS. Instead, we should ignore JAVA_TOOL_OPTIONS here too,
and not silently ignore JAVA_OPTS but instead warn before doing so (as
we already do for the main Java process). This commit rearranges the
execution of these steps so that we do the right thing here.
Relates #25969
This commit removes a legacy check against running bin/elasticsearch
that is not produced from a distribution. This check exists for legacy
reasons, namely when bin/elasticsearch previously sat in the root of the
Elasticsearch repository. In this old scenario, someone might clone the
repository, see the bin folder and try to run bin/elasticsearch without
first production a distribution. Today, this is unlikely since
bin/elasticsearch now sits in
distribution/src/main/resources/bin/elasticsearch so first, bin is no
longer in the root of the repository, and second, the src indicates this
is source and not already for production. Moreover, our README in the
root of the repository provides clear instructions for getting started:
either download a distribution or build one from source. In the name of
simplicity, we therefore remove this legacy check.
Relates #25960
This commit cleans up a few items with the script packaging:
- remove the now dead elasticsearch.in.sh script
- add assertions for the existence elasticsearch-env and
elasticsearch-keystore
This commit addresses a change in core Elasticsearch where the
command-line flag --path.conf is no longer respected. Instead, the
configuration path must be passed through the system property
es.path.conf. We adapt the Windows batch file and the service for this
change.
This commit fixes tests for environment-aware commands. A previous
change added a check that es.path.conf is not null. The problem is that
this system property is not being set in tests so this check trips every
single time. To fix this, we move the check into a method that can be
overridden, and then override this method in relevant places in tests to
avoid having to set the property in tests. We also add a test that this
check works as expected.
A previous change enabled it so that users could configure the
configuration path via a command-line option --path.conf. However, a
subsequent change has made it so that we expect users to set the
configuration path via the environment variable CONF_DIR. To enable
this, we now pass the value of CONF_DIR as the value for the
command-line option --path.conf. This has two problems:
- the presence of --path.conf always being on the command line breaks
other flags like --help for multi-commands
- the scripts for which --help is not broken say that you can pass
--path.conf but this is a lie since passing it will make it appear
twice in the command-line arguments breaking the script
Since --path.conf is no longer the way that we want users to set the
configuration path, we should remove the --path.conf option. However, we
still need a way to get the configuration path from the scripts to the
running Java process. To do this, we now pass the configuration path as
a system property. This keeps it off the script command line fixing the
above problems.
The only remaining question (that I can see) is whether or not to
respect -Des.path.conf=<some path> if the user sets this in their
jvm.options or via ES_JAVA_OPTS. I think that we should not do this (as
has been our tradition), es.path.home and es.path.conf are special,
should be set by our scripts only so users should not be setting them at
all so we should not take any effort to respect these flags if the user
tries to otherwise use them.
Relates #25943
When running a script that depends on elasticsearch-env, the
elasticsearch-env script seeks backwards from the directory containing
the script to find Elasticsearch home. This is done by seeking backwards
in the path to find bin, and then going one directory above
that. Unfortunately, if the script is started relatively from the bin
directory, then bin will appear in the path since it is a relative
path. This commit fixes this by making the starting path absolute before
attempting to seek backwards.
This commit removes all external dependencies from the rest client jar
and shades them in an 'org.elasticsearch.client' package within the jar
using shadowJar gradle plugin. All projects that depended on the
existing jar have been converted to using the 'org.elasticsearch.client'
package prefixes to interact with the rest client.
Closes#25208
The quoting for the ExecStart entry is broken as quotes must wrap an
entire argument, and arguments are separated by spaces. It turns out
that any quoting is unnecessary here, systemd will handle it correctly
either way.
This commit introduces the elasticsearch-env script. The purpose of this
script is threefold:
- vastly simplify the various scripts used in Elasticsearch
- provide a script that can be included in other scripts in the
Elasticsearch ecosystem (e.g., plugins)
- correctly establish the environment for all scripts (e.g., so that
users can run `elasticsearch-keystore` from a package distribution
without having to worry about setting `CONF_DIR` first, otherwise the
keystore would be created in the wrong location)
Relates #25815
This commit fixes the elasticsearch-keystore script handling of
path.conf; the problem here is that the script is setting a system
property that is completely unobserved. Instead, we use the path.conf
command line flag.
Relates #25811
This commit removes legacy checks for unsupported an environment
variable and unsupported system properties. This environment variable
and these system properties have not been supported since 1.x so it is
safe to stop checking for the existence of these settings.
Relates #25809
Today we explicitly export the HOSTNAME variable from scripts. This is
probably a relic from the days when the scripts were not run on bash but
instead assume a POSIX-compliant shell only where HOSTNAME is not
guaranteed to exist. Yet, bash guarantees that HOSTNAME is set so we do
not need to set it in scripts. This commit removes this legacy.
Relates #25807
Today we enable users to customize the environment through the use of
ES_INCLUDE. This made sense for legacy reasons when we did not have
nicities like jvm.options (so dumped JVM options in the default include
script) and somewhat duplicates some of the functionality that we will
need from a dedicated environment script. This commit removes support
for ES_INCLUDE as a first step towards a dedicated include script.
Relates #25804
The problem here is simple: when using direct buffers as in NIO, the JDK
relies on explict GC invocataions to trigger cleaning up direct buffers;
if such GCs do not occur and the direct buffer limit is reached, the JVM
will throw an out of memory exception. With explicit GCs disabled, the
JVM is neutered from explicitly cleaning up direct buffers in the act of
reserving a new direct buffer and instead relies on a GC occurring for
another reason. If such a GC never occurs, the JVM will OOM. This commit
removes disabling of explicit GCs. Note that these explicit GCs only
occur as a last ditch effort before going OOM when the JVM is trying to
reserve more direct memory. This is a known issue, see for example:
JDK-8142537.
Relates #25759
This commit changes the default heap size to 1 GB. Experimenting with
elasticsearch is often done on laptops, and 1 GB is much friendlier to
laptop memory. It does put more pressure on the gc, but the tradeoff is
a smaller default footprint. Users running in production can (and
should) adjust the heap size as necessary for their usecase.
This commit enables management of the main Elasticsearch log files
out-of-the-box by the following changes:
- compress rolled logs
- roll logs every 128 MB
- maintain a sliding window of logs
- remove the oldest logs maintaining no more than 2 GB of compressed
logs on disk
Relates #25660
This commit removes the environment variable ES_JVM_OPTIONS that allows
the jvm.options file to sit separately from the rest of the config
directory. Instead, we use the CONF_DIR environment variable for custom
configuration location just as we do for the other configuration files.
Relates #25679
On Debian-based systems the install scripts are run with set -e meaning
that if there is an error in executing one of these scripts then the
script fails. If systemd-sysctl is masked then trying to restart the
systemd-sysctl service to pick up the changes to vm.max_map_count will
fail leading to the post-install script failing. Instead, we should
account for the possbility of failure here by not letting the command to
restart this service exit with non-zero status code. This commit does
this, and adds a test for this situation.
Relates #25657
This is a regression introduced in #25510, which removed the explicit fetching of upstream. Sadly this doesn't work if you don't have any local branch referring to `upstream` as an upstream branch.
Some times we need a fix / change to have two parts in two different branches (corresponding to two different ES releases). In order to be able to test these cases you need to run the BWC tests against a local branch rather than then using a branch from `github.com/elastic/elasticsearch`.
This commit adds a system property called `tests.bwc.refspec` that allows you to do it. Note that I've chosen to go with the simplest code change for now, at the expense of some user friendliness.
We previously tried to maintain (while not formally supporting) 32-bit
support, although we never tested this anywhere in CI. Since we do not
formally support this, and 32-bit usage is very low, we have elected to
no longer maintain 32-bit support. This commit removes any implication
of 32-bit support.
Relates #25435
This commit removes the default path settings for data and logs. With
this change, we now ship the packages with these settings set in the
elasticsearch.yml configuration file rather than going through the
default.path.data and default.path.logs dance that we went through in
the past.
Relates #25408
This commit removes path.conf as a valid setting and replaces it with a
command-line flag for specifying a non-default path for configuration.
Relates #25392
When JAVA_HOME is not set we try to detect the location of Java. If its
location contains a space, due to a lack of quoting we will be
unsuccessful in invoking Java. This commit adds the necessary quoting to
handle this case.
Relates #23822
Removes the `assemble` task from the `build` task when we have
removed `assemble` from the project. We removed `assemble` from
projects that aren't published so our releases will be faster. But
That broke CI because CI builds with `gradle precommit build` and,
it turns out, that `build` includes `check` and `assemble`. With
this change CI will only run `check` for projects without an
`assemble`.
Removes the `assemble` task from projects that are not published.
This should speed up `gradle assemble` by skipping projects that
don't need to be built. Which is useful because `gradle assemble`
is how we cut releases.
This prevents possible race conditions between the Elasticsearch JVM and
plugin native controller processes that can cause the Elasticsearch shutdown
to hang. The problem can happen when the JVM and the controller process
receive a SIGTERM at almost the same time.
(There's an assumption here that Elasticsearch will continue to use other
mechanisms to kill native controller processes.)
This commit changes the task type of the checkoutBwcBranch task to Exec
from LoggedExec so that the output of the checkout command is
shown. This enables us to see the SHA used for the checkout which can be
useful when debugging a BWC break.
Relates #25166
Commit bf007e8d93 was a forward port of logic needed in 5.x to get
the correct bwc branch. However, other changes on master meant that this forward port was not
needed and actually broke the bwc tests. This change removes the incorrect if statement.
Relates #25134
When testing against the previous 5.x release, the bwc project incorrectly would checkout the 5.x
branch instead of the 5.5 branch as it still had the logic that applies for major versions bwc. This change adds
a check to compare the major version when making the decision on the branch to use.
During package install on systemd-based systems, some sysctl settings
should be set (e.g. vm.max_map_count).
In some environments, changing sysctl settings plainly does not work;
previously a global environment variable named
ES_SKIP_SET_KERNEL_PARAMETERS was introduced to skip calling sysctl, but
this causes trouble for:
- configuration management systems, which usually cannot apply an env
var when running a package manager
- package upgrades, which will not have the env var set any more, and
thus leaving the package management system in a bad state (possibly
half-way upgraded, can be very hard to recover)
This removes the env var again and instead of calling systemd-sysctl
manually, tells systemd to restart the wrapper unit - which itself can
be masked by system administrators or management tools if it is known
that sysctl does not work in a given environment.
The restart is not silent on systems in their default configuration, but
is ignored if the unit is masked.
Relates #24234
* Adds nodes usage API to monitor usages of actions
The nodes usage API has 2 main endpoints
/_nodes/usage and /_nodes/{nodeIds}/usage return the usage statistics
for all nodes and the specified node(s) respectively.
At the moment only one type of usage statistics is available, the REST
actions usage. This records the number of times each REST action class is
called and when the nodes usage api is called will return a map of rest
action class name to long representing the number of times each of the action
classes has been called.
Still to do:
* [x] Create usage service to store usage statistics
* [x] Record usage in REST layer
* [x] Add Transport Actions
* [x] Add REST Actions
* [x] Tests
* [x] Documentation
* Rafactors UsageService so counts are done by the handlers
* Fixing up docs tests
* Adds a name to all rest actions
* Addresses review comments
By default, the remove plugin CLI command preserves configuration
files. This is so that if a user is upgrading the plugin (which is done
by first removing the old version and then installing the new version)
they do not lose their configuration file. Yet, there are circumstances
where preserving the configuration file is not desired. This commit adds
a purge option to the remove plugin CLI command.
Relates #24981
Removes the `distribution:bwc` project in favor of
`distribution:bwc-release-snapshot` and
`distribution:bwc-stable-snapshot`.
`distribution:bwc-release-snapshot` builds a snapshot of the
latest release branch (5.4 now) if needed for backwards
compatibility. `distribution:bwc-stable-snapshot` builds a
snapshot of the latest stable branch (5.x now) if needed for
backwards compatibility.
Some packaging tests depend on snapshot versions of packaging
distributions yet the build does not use a repository that includes such
distributions. While we could add such a repository, a better strategy
is to follow our approach for other BWC tests where we depend on a
locally-compiled archive distribution. This commit adds a local
compilation of packaging artifacts and substitutes these anywhere that
we would otherwise depend on a snapshot of these artifacts.
Relates #24861
This commit expands the logic for version extraction from Version.java
to include a list of all versions for backcompat purposes. The tests
using bwcVersion are converted to use this list, but those tests
(rolling upgrade and backwards-5.0) are still not randomized; that will
happen in another followup.
Jason Tedor
Michael Basnight
Ryan Ernst
Boaz Leskes
Nik Everett
David Roberts
jaymode
Christian Hofstaedtler
Colin Goodheart-Smithe