Commit Graph

462 Commits

Author SHA1 Message Date
Jason Tedor 8639bf4a1a Pass config path as a system property
A previous change enabled it so that users could configure the
configuration path via a command-line option --path.conf. However, a
subsequent change has made it so that we expect users to set the
configuration path via the environment variable CONF_DIR. To enable
this, we now pass the value of CONF_DIR as the value for the
command-line option --path.conf. This has two problems:
 - the presence of --path.conf always being on the command line breaks
   other flags like --help for multi-commands
 - the scripts for which --help is not broken say that you can pass
   --path.conf but this is a lie since passing it will make it appear
   twice in the command-line arguments breaking the script

Since --path.conf is no longer the way that we want users to set the
configuration path, we should remove the --path.conf option. However, we
still need a way to get the configuration path from the scripts to the
running Java process. To do this, we now pass the configuration path as
a system property. This keeps it off the script command line fixing the
above problems.

The only remaining question (that I can see) is whether or not to
respect -Des.path.conf=<some path> if the user sets this in their
jvm.options or via ES_JAVA_OPTS. I think that we should not do this (as
has been our tradition), es.path.home and es.path.conf are special,
should be set by our scripts only so users should not be setting them at
all so we should not take any effort to respect these flags if the user
tries to otherwise use them.

Relates #25943
2017-07-28 12:15:22 +09:00
Jason Tedor 6c650874c9 Fix scripts when launching from bin
When running a script that depends on elasticsearch-env, the
elasticsearch-env script seeks backwards from the directory containing
the script to find Elasticsearch home. This is done by seeking backwards
in the path to find bin, and then going one directory above
that. Unfortunately, if the script is started relatively from the bin
directory, then bin will appear in the path since it is a relative
path. This commit fixes this by making the starting path absolute before
attempting to seek backwards.
2017-07-28 10:21:34 +09:00
Michael Basnight e816ef89a2 Shade external dependencies in the rest client jar
This commit removes all external dependencies from the rest client jar
and shades them in an 'org.elasticsearch.client' package within the jar
using shadowJar gradle plugin. All projects that depended on the
existing jar have been converted to using the 'org.elasticsearch.client'
package prefixes to interact with the rest client.

Closes #25208
2017-07-24 12:55:43 -05:00
Jason Tedor 46d75a3552 Fix broken quotes in systemd unit file
The quoting for the ExecStart entry is broken as quotes must wrap an
entire argument, and arguments are separated by spaces. It turns out
that any quoting is unnecessary here, systemd will handle it correctly
either way.
2017-07-21 17:04:49 +09:00
Jason Tedor 0310a6a947 Introduce elasticsearch-env
This commit introduces the elasticsearch-env script. The purpose of this
script is threefold:
 - vastly simplify the various scripts used in Elasticsearch
 - provide a script that can be included in other scripts in the
   Elasticsearch ecosystem (e.g., plugins)
 - correctly establish the environment for all scripts (e.g., so that
   users can run `elasticsearch-keystore` from a package distribution
   without having to worry about setting `CONF_DIR` first, otherwise the
   keystore would be created in the wrong location)

Relates #25815
2017-07-21 09:38:49 +09:00
Jason Tedor 137ab70d58 Fix elasticsearch-keystore handling of path.conf
This commit fixes the elasticsearch-keystore script handling of
path.conf; the problem here is that the script is setting a system
property that is completely unobserved. Instead, we use the path.conf
command line flag.

Relates #25811
2017-07-20 23:01:57 +09:00
Jason Tedor 9d8f11dc27 Remove legacy checks for config file settings
This commit removes legacy checks for unsupported an environment
variable and unsupported system properties. This environment variable
and these system properties have not been supported since 1.x so it is
safe to stop checking for the existence of these settings.

Relates #25809
2017-07-20 22:42:39 +09:00
Jason Tedor 3042b5dc7d Stop exporting HOSTNAME from scripts
Today we explicitly export the HOSTNAME variable from scripts. This is
probably a relic from the days when the scripts were not run on bash but
instead assume a POSIX-compliant shell only where HOSTNAME is not
guaranteed to exist. Yet, bash guarantees that HOSTNAME is set so we do
not need to set it in scripts. This commit removes this legacy.

Relates #25807
2017-07-20 22:27:47 +09:00
Jason Tedor 67a4288c9a Remove support for ES_INCLUDE
Today we enable users to customize the environment through the use of
ES_INCLUDE. This made sense for legacy reasons when we did not have
nicities like jvm.options (so dumped JVM options in the default include
script) and somewhat duplicates some of the functionality that we will
need from a dedicated environment script. This commit removes support
for ES_INCLUDE as a first step towards a dedicated include script.

Relates #25804
2017-07-20 15:41:59 +09:00
Jason Tedor c63b7f8b0b Stop disabling explicit GC
The problem here is simple: when using direct buffers as in NIO, the JDK
relies on explict GC invocataions to trigger cleaning up direct buffers;
if such GCs do not occur and the direct buffer limit is reached, the JVM
will throw an out of memory exception. With explicit GCs disabled, the
JVM is neutered from explicitly cleaning up direct buffers in the act of
reserving a new direct buffer and instead relies on a GC occurring for
another reason. If such a GC never occurs, the JVM will OOM. This commit
removes disabling of explicit GCs. Note that these explicit GCs only
occur as a last ditch effort before going OOM when the JVM is trying to
reserve more direct memory. This is a known issue, see for example:
JDK-8142537.

Relates #25759
2017-07-18 15:16:52 +09:00
Ryan Ernst b1762d69b5 Setup: Change default heap to 1G (#25695)
This commit changes the default heap size to 1 GB. Experimenting with
elasticsearch is often done on laptops, and 1 GB is much friendlier to
laptop memory. It does put more pressure on the gc, but the tradeoff is
a smaller default footprint. Users running in production can (and
should) adjust the heap size as necessary for their usecase.
2017-07-14 09:38:08 -07:00
Jason Tedor 86e9438d3c Prevent excessive disk consumption by log files
This commit enables management of the main Elasticsearch log files
out-of-the-box by the following changes:
 - compress rolled logs
 - roll logs every 128 MB
 - maintain a sliding window of logs
 - remove the oldest logs maintaining no more than 2 GB of compressed
   logs on disk

Relates #25660
2017-07-12 15:52:00 -04:00
Jason Tedor 5a416b9922 Use config directory to find jvm.options
This commit removes the environment variable ES_JVM_OPTIONS that allows
the jvm.options file to sit separately from the rest of the config
directory. Instead, we use the CONF_DIR environment variable for custom
configuration location just as we do for the other configuration files.

Relates #25679
2017-07-12 15:29:13 -04:00
Jason Tedor 3a827827c1 Avoid failing install if system-sysctl is masked
On Debian-based systems the install scripts are run with set -e meaning
that if there is an error in executing one of these scripts then the
script fails. If systemd-sysctl is masked then trying to restart the
systemd-sysctl service to pick up the changes to vm.max_map_count will
fail leading to the post-install script failing. Instead, we should
account for the possbility of failure here by not letting the command to
restart this service exit with non-zero status code. This commit does
this, and adds a test for this situation.

Relates #25657
2017-07-11 17:38:50 -04:00
Boaz Leskes 17a587e709 bwc checkout should fetch from all remotes
This is a regression introduced in #25510, which removed the explicit fetching of upstream. Sadly this doesn't work if you don't have any local branch referring to `upstream` as an upstream branch.
2017-07-07 13:39:24 +02:00
Boaz Leskes f79c2cb8c0 Allow BWC Testing against a specific branch (#25510)
Some times we need a fix / change to have two parts in two different branches (corresponding to two different ES releases). In order to be able to test these cases you need to run the BWC tests against a local branch rather than then using a branch from `github.com/elastic/elasticsearch`.

This commit adds a system property called `tests.bwc.refspec` that allows you to do it. Note that I've chosen to go with the simplest code change for now, at the expense of some user friendliness.
2017-07-07 11:18:03 +02:00
Jason Tedor be906628d5 Remove implicit 32-bit support
We previously tried to maintain (while not formally supporting) 32-bit
support, although we never tested this anywhere in CI. Since we do not
formally support this, and 32-bit usage is very low, we have elected to
no longer maintain 32-bit support. This commit removes any implication
of 32-bit support.

Relates #25435
2017-06-28 08:24:33 -04:00
Jason Tedor dfd241e0a6 Remove default path settings
This commit removes the default path settings for data and logs. With
this change, we now ship the packages with these settings set in the
elasticsearch.yml configuration file rather than going through the
default.path.data and default.path.logs dance that we went through in
the past.

Relates #25408
2017-06-26 21:43:20 -04:00
Jason Tedor 5a9fc8aa2a Remove path.conf setting
This commit removes path.conf as a valid setting and replaces it with a
command-line flag for specifying a non-default path for configuration.

Relates #25392
2017-06-26 15:18:29 -04:00
Jason Tedor dc9b67461f Fix use of spaces on Windows if JAVA_HOME not set
When JAVA_HOME is not set we try to detect the location of Java. If its
location contains a space, due to a lack of quoting we will be
unsuccessful in invoking Java. This commit adds the necessary quoting to
handle this case.

Relates #23822
2017-06-22 21:55:31 -04:00
Nik Everett 929194ef05 Fix artifact location
Fix the location of the rpm and deb used for the packaging upgrade
tests when upgrading from -SNAPSHOT version.
2017-06-19 14:20:30 -04:00
Nik Everett 21b1db2965 Remove assemble from build task when assemble removed
Removes the `assemble` task from the `build` task when we have
removed `assemble` from the project. We removed `assemble` from
projects that aren't published so our releases will be faster. But
That broke CI because CI builds with `gradle precommit build` and,
it turns out, that `build` includes `check` and `assemble`. With
this change CI will only run `check` for projects without an
`assemble`.
2017-06-16 17:19:14 -04:00
Nik Everett 7b358190d6 Remove assemble task when not used for publishing (#25228)
Removes the `assemble` task from projects that are not published.
This should speed up `gradle assemble` by skipping projects that
don't need to be built. Which is useful because `gradle assemble`
is how we cut releases.
2017-06-16 11:46:34 -04:00
David Roberts a5658c0fea When stopping via systemd only kill the JVM, not its control group (#25195)
This prevents possible race conditions between the Elasticsearch JVM and
plugin native controller processes that can cause the Elasticsearch shutdown
to hang.  The problem can happen when the JVM and the controller process
receive a SIGTERM at almost the same time.

(There's an assumption here that Elasticsearch will continue to use other
mechanisms to kill native controller processes.)
2017-06-14 09:23:41 +01:00
Jason Tedor a7a3af6f48 Log checkout so SHA is known
This commit changes the task type of the checkoutBwcBranch task to Exec
from LoggedExec so that the output of the checkout command is
shown. This enables us to see the SHA used for the checkout which can be
useful when debugging a BWC break.

Relates #25166
2017-06-09 22:06:51 -04:00
jaymode 5997e4a39d
Remove incorrect bwc branch logic from master
Commit bf007e8d93 was a forward port of logic needed in 5.x to get
the correct bwc branch. However, other changes on master meant that this forward port was not
needed and actually broke the bwc tests. This change removes the incorrect if statement.

Relates #25134
2017-06-09 09:47:06 -06:00
jaymode bf007e8d93
Fix branch logic for bwc tests in the same major version (#25134)
When testing against the previous 5.x release, the bwc project incorrectly would checkout the 5.x
branch instead of the 5.5 branch as it still had the logic that applies for major versions bwc. This change adds
a check to compare the major version when making the decision on the branch to use.
2017-06-08 11:39:29 -06:00
Christian Hofstaedtler c3ec6a1714 Honor masking of systemd-sysctl.service
During package install on systemd-based systems, some sysctl settings
should be set (e.g. vm.max_map_count).

In some environments, changing sysctl settings plainly does not work;
previously a global environment variable named
ES_SKIP_SET_KERNEL_PARAMETERS was introduced to skip calling sysctl, but
this causes trouble for:
 - configuration management systems, which usually cannot apply an env
   var when running a package manager
 - package upgrades, which will not have the env var set any more, and
   thus leaving the package management system in a bad state (possibly
   half-way upgraded, can be very hard to recover)

This removes the env var again and instead of calling systemd-sysctl
manually, tells systemd to restart the wrapper unit - which itself can
be masked by system administrators or management tools if it is known
that sysctl does not work in a given environment.

The restart is not silent on systems in their default configuration, but
is ignored if the unit is masked.

Relates #24234
2017-06-06 10:44:41 -04:00
Colin Goodheart-Smithe 779fb9a1c0 Adds nodes usage API to monitor usages of actions (#24169)
* Adds nodes usage API to monitor usages of actions

The nodes usage API has 2 main endpoints

/_nodes/usage and /_nodes/{nodeIds}/usage return the usage statistics
for all nodes and the specified node(s) respectively.

At the moment only one type of usage statistics is available, the REST
actions usage. This records the number of times each REST action class is
called and when the nodes usage api is called will return a map of rest
action class name to long representing the number of times each of the action
classes has been called.

Still to do:

* [x] Create usage service to store usage statistics
* [x] Record usage in REST layer
* [x] Add Transport Actions
* [x] Add REST Actions
* [x] Tests
* [x] Documentation

* Rafactors UsageService so counts are done by the handlers

* Fixing up docs tests

* Adds a name to all rest actions

* Addresses review comments
2017-06-02 08:46:38 +01:00
Jason Tedor 5961e1d6ab Remove import from InstallPluginCommandTests
This commit removes an unused import from
InstallPluginCommandTests.java.
2017-06-01 08:56:25 -04:00
Jason Tedor 9b4a189147 Add purge option to remove plugin CLI
By default, the remove plugin CLI command preserves configuration
files. This is so that if a user is upgrading the plugin (which is done
by first removing the old version and then installing the new version)
they do not lose their configuration file. Yet, there are circumstances
where preserving the configuration file is not desired. This commit adds
a purge option to the remove plugin CLI command.

Relates #24981
2017-06-01 08:53:39 -04:00
Jason Tedor 9b6b4ffe8e Set number of processes in systemd unit file
This commit sets the number of processes in the systemd unit file for
Elasticsearch to meet the bootstrap checks.

Relates #24970
2017-05-30 15:34:20 -04:00
Nik Everett bc4df47a76 Rework bwc snapshot projects to build up to two bwc versions (#24870)
Removes the `distribution:bwc` project in favor of
`distribution:bwc-release-snapshot` and
`distribution:bwc-stable-snapshot`.
`distribution:bwc-release-snapshot` builds a snapshot of the
latest release branch (5.4 now) if needed for backwards
compatibility. `distribution:bwc-stable-snapshot` builds a
snapshot of the latest stable branch (5.x now) if needed for
backwards compatibility.
2017-05-29 10:22:32 -04:00
Jason Tedor 95ebdbaddb Add BWC packaging distributions
Some packaging tests depend on snapshot versions of packaging
distributions yet the build does not use a repository that includes such
distributions. While we could add such a repository, a better strategy
is to follow our approach for other BWC tests where we depend on a
locally-compiled archive distribution. This commit adds a local
compilation of packaging artifacts and substitutes these anywhere that
we would otherwise depend on a snapshot of these artifacts.

Relates #24861
2017-05-24 11:55:32 -04:00
Colin Goodheart-Smithe 779cf3d468 remove remaining references to scripts directory (#24771)
The packaging scripts still referenced the scripts directory which is now removed.
2017-05-18 11:48:15 +01:00
Ryan Ernst 463fe2f4d4 Scripting: Remove file scripts (#24627)
This commit removes file scripts, which were deprecated in 5.5.

closes #21798
2017-05-17 14:42:25 -07:00
Ryan Ernst ff34434bba Build: Extract all ES versions into gradle properties (#24748)
This commit expands the logic for version extraction from Version.java
to include a list of all versions for backcompat purposes. The tests
using bwcVersion are converted to use this list, but those tests
(rolling upgrade and backwards-5.0) are still not randomized; that will
happen in another followup.
2017-05-17 12:58:37 -07:00
Russ Cam 026e94e6d9 Handle parentheses in path
The variable assignment needs to be quoted to correctly handle scenario
where the path contains parentheses.

Realtes #24731
2017-05-17 07:22:05 -04:00
Ryan Ernst 2a65bed243 Tests: Change rest test extension from .yaml to .yml (#24659)
This commit renames all rest test files to use the .yml extension
instead of .yaml. This way the extension used within all of
elasticsearch for yaml is consistent.
2017-05-16 17:24:35 -07:00
Jason Tedor 92ba969804 Remove Jython hacks
We had a hack in setting up permissions for tests to support testing
the lang-python plugin. We also had a hack to prevent Log4j from
loading a shaded version of Jansi provided by Jython. This plugin has
been removed so these hacks are no longer necessary.

Relates #24681
2017-05-15 12:40:24 -04:00
Jason Tedor 3f54ae6264 Fix handling of spaces in plugin script on Windows
A previous fix for handling spaces in the Windows scripts should also be
applied to the plugin script. This commit does that.

Closes #24750
2017-05-09 15:37:43 -04:00
Jason Tedor 3e485c2ca5 Fix plugin installation permissions
When installing plugin permissions, we try to set the permissions on all
installed files ourselves because a umask from the user could violate
everything needed to get the permissions right. Sadly, directories were
not handled correctly at all and so we were still left with broken
installations with umasks like 0077. This commit fixes this issue, adds
a thorough unit test for the situation, and most importantly, adds a
test that sets the umask before installing the plugin.

Relates #24527
2017-05-06 13:25:10 -04:00
Nik Everett 3b47355e56 Try not to lose stacktraces (#24426)
This adds `-XX:-OmitStackTraceInFastThrow` to the JVM arguments
which *should* prevent the JVM from omitting stack traces on
common exception sites. Even though these sites are common, we'd
still like the exceptions to debug them.

This also adds the flag when running tests and adapts some tests
that had workarounds for the absense of the flag.

Closes #24376
2017-05-02 11:34:12 -04:00
Ryan Ernst cdcc75dd2a Plugins: Add support for platform specific plugins (#24265)
This commit adds support for plugins having a platform specific variant.
It also adds unit tests for all official and maven urls.
2017-04-27 11:27:29 -07:00
Jason Tedor 08f52d1a18 Remove extra newline in RemovePluginCommand.java
This commit removes an extra newline in RemovePluginCommand.java because
it drives the OCD among us insane.
2017-04-21 22:18:33 -04:00
Jason Tedor 9912650641 Fix delete of plugin directory on remove plugin
This commit fixes an issue when deleting the plugin directory while
executing the remove plugin command. Namely, we take out a file
descriptor on the plugin directory to traverse its contents to obtain
the list of files to delete. We leaked this file descriptor. On
Unix-based filesystems, this is not a problem, deleting the plugin
directory deletes the plugin directory. On Windows though, a delete is
not executed until the last file descriptor is closed. Since we leaked
this file descriptor, the plugin was not actually deleted. This led to
test failures that tried to cleanup left behind temporary directories
but these test failures were just exposing this bug. This commit fixes
this issue by ensuring that we close the file descriptor to the plugin
directory when we are finished with it.

Relates #24266
2017-04-21 22:15:29 -04:00
Jason Tedor fe91c72151 Use a marker file when removing a plugin
Today when removing a plugin, we attempt to move the plugin directory to
a temporary directory and then delete that directory from the
filesystem. We do this to avoid a plugin being in a half-removed
state. We previously tried an atomic move, and fell back to a non-atomic
move if that failed. Atomic moves can fail on union filesystems when the
plugin directory is not in the top layer of the
filesystem. Interestingly, the regular move can fail as well. This is
because when the JDK is executing such a move, it first tries to rename
the source directory to the target directory and if this fails with
EXDEV (as in the case of an atomic move failing), it falls back to
copying the source to the target, and then attempts to rmdir the
source. The bug here is that the JDK never deleted the contents of the
source so the rmdir will always fail (except in the case of an empty
directory).

Given all this silliness, we were inspired to find a different
strategy. The strategy is simple. We will add a marker file to the
plugin directory that indicates the plugin is in a state of
removal. This file will be the last file out the door during removal. If
this file exists during startup, we fail startup.

Relates #24252
2017-04-21 15:50:44 -04:00
Ryan Ernst ba48674695 Build: Move plugin cli and tests to distribution tool (#24220)
The plugin cli currently resides inside the elasticsearch jar. This
commit moves it into a plugin-cli jar. This is change alone is a no-op;
it does not change anything about what is loaded at runtime. But it will
allow easier testing (with fixtures in the future to test ES or maven
installation), as well as eventually not loading these classes when
starting elasticsearch.
2017-04-21 09:25:58 -07:00
Ryan Ernst 212f24aa27 Tests: Clean up rest test file handling (#21392)
This change simplifies how the rest test runner finds test files and
removes all leniency.  Previously multiple prefixes and suffixes would
be tried, and tests could exist inside or outside of the classpath,
although outside of the classpath never quite worked. Now only classpath
tests are supported, and only one resource prefix is supported,
`/rest-api-spec/tests`.

closes #20240
2017-04-18 15:07:08 -07:00
Ryan Ernst a8017ff020 Tests: Move cluster dependencies from runner to cluster (#24142)
After splitting integ tests into cluster configuration and the test
runner task, we still have dependencies of the test runner added as deps
of the cluster. This commit adds dependencies directly to the cluster,
so that the runner can have other dependencies independent of what is
needed for the cluster.
2017-04-17 16:02:46 -07:00
Jason Tedor 51db6ab7dc Revert "Ensure output for createClone task"
This reverts commit 2620200ff7.
2017-04-15 13:54:01 -04:00
Lee Hinman 162ce85ff2 Remove customization of ES_USER and ES_GROUP
This removes the ability to configure Elasticsearch to use custom username
and/or group when it is run.

Resolves #23848
2017-04-14 13:24:21 -06:00
Ryan Ernst b0003edcf4 Build: Support offline build of bwc zip (#24037)
The bwc checkout for backcompat tests currently always tries to fetch
the latest from the upstream remote. This change makes fetching from
upstream conditional on not running an offline build.
2017-04-10 20:04:18 -07:00
David Roberts 37aadb2adf Add the ability to include extra notices in a plugin's NOTICES file (#23898)
Adds the option for a plugin to specify extra directories containing notices
and licenses files to be incorporated into the overall notices file that is
generated for the plugin.

This can be useful, for example, where the plugin has a non-Java dependency
that itself incorporates many 3rd party components.
2017-04-10 12:37:42 +01:00
Jason Tedor 2620200ff7 Ensure output for createClone task
This commit changes the task type from LoggedExec to Exec for the BWC
createClone task to ensure that we can get some debug output.
2017-04-06 08:37:04 -04:00
Ryan Ernst c48b60dcf1 Tests: Extend rest test timeout to 30 minutes for windows (#23912)
Windows rest tests consistenly fail because the filesystem appears to be
an order of magnitude slower than that of *nix, at least in the context
of our rest tests. This commit overrides the suite timeout to 30 mins
for windows. From past failures, it appears this should be enough, as
the tests seem to fail when they are almost complete. The default suite
timeout for ESTestCase is 20 mins, so this leaves ample buffer for
windows shenanigans.
2017-04-05 12:11:03 -07:00
Ryan Ernst a903aabbd6 Build: Ensure upstream check works even when using info logging (#23804)
The LoggedExec task does not capture output when info logging is
enabled. This commit changes the upstream check to use Exec directly,
so as not to break when info logging is enabled.
2017-03-29 13:42:47 -07:00
Ryan Ernst 5a414cbe5c Build: Use filesystem agnostic printing of bwc dir path (#23803)
This will use File.toString() for the `git clone` command, which will
automatically be correct for whatever system the build is running on.

closes #23784
2017-03-29 13:13:09 -07:00
Ryan Ernst cc1addeac2 Build: Find bwc version during build (#23801)
We currently have the last minor version of the previous major hardcoded
in tests like rolling upgrade. This change programatically finds this
during gradle initialization by parsing versions from Version.java.
2017-03-29 12:11:38 -07:00
Ryan Ernst d159920073 Build: Use GradleBuild task for invoking 5.x checkout build (#23770)
This commit switches from executing gradle when building the bwc testing
zip through Exec, to using GradleBuild. In addition to not depending on
gradle being in the PATH, it also has the added benefit of much better
logging while the bwc build is going on (the actual tasks show up as
tasks of a subproject within the current build).
2017-03-27 15:02:18 -07:00
Ryan Ernst 8822332355 Fetch all upstream refs for bwc checkout 2017-03-24 02:31:10 -07:00
Ryan Ernst a26321e03b Fix bwc checkout upstream remote check 2017-03-24 02:22:59 -07:00
Ryan Ernst 8c53555b28 Tests: Use local clone build of 5.x with bwc tests (#22946)
The current rest backcompat tests, which run against a mixed cluster of
5.x and 6.0 nodes, depend on snapshot builds of 5.x. However, this has
the potential for inconsistency that results in CI failures, and happens
quite often, whenever some backcompat logic is added to 5.x, but the bwc
test on master fails because the 5.x code has not yet been published as
a snapshot.

This change creates a git clone of the 5.x branch,
builds the zip distribution, and ties that into gradle substitutions for
the 5.x version.
2017-03-23 22:32:13 -07:00
Ryan Ernst f8453aca57 Packaging: Remove classpath ordering hack (#23596)
After the removal of the joda time hack we used to have, we can cleanup
the codebase handling in security, jarhell and plugins to be more picky
about uniqueness. This was originally in #18959 which was never merged.

closes #18959
2017-03-21 12:12:16 -07:00
Ryan Ernst 175bda64a0 Build: Rework integ test setup and shutdown to ensure stop runs when desired (#23304)
Gradle's finalizedBy on tasks only ensures one task runs after another,
but not immediately after. This is problematic for our integration tests
since it allows multiple project's integ test clusters to be
simultaneously. While this has not been a problem thus far (gradle 2.13
happened to keep the finalizedBy tasks close enough that no clusters
were running in parallel), with gradle 3.3 the task graph generation has
changed, and numerous clusters may be running simultaneously, causing
memory pressure, and thus generally slower tests, or even failure if the
system has a limited amount of memory (eg in a vagrant host).

This commit reworks how integ tests are configured. It adds an
`integTestCluster` extension to gradle which is equivalent to the current
`integTest.cluster` and moves the rest test runner task to
`integTestRunner`.  The `integTest` task is then just a dummy task,
which depends on the cluster runner task, as well as the cluster stop
task. This means running `integTest` in one project will both run the
rest tests, and shut down the cluster, before running `integTest` in
another project.
2017-02-22 12:43:15 -08:00
Jay Modi b234644035 Enforce Content-Type requirement on the rest layer and remove deprecated methods (#23146)
This commit enforces the requirement of Content-Type for the REST layer and removes the deprecated methods in transport
requests and their usages.

While doing this, it turns out that there are many places where *Entity classes are used from the apache http client
libraries and many of these usages did not specify the content type. The methods that do not specify a content type
explicitly have been added to forbidden apis to prevent more of these from entering our code base.

Relates #19388
2017-02-17 14:45:41 -05:00
Ryan Ernst 164079bb93 Disable javadoc for java vesion checker 2017-02-16 09:56:11 -08:00
Jason Tedor c9cde11a5e Introduce Java version checker
Today when users start Elasticsearch with their Java configuration
pointing to a pre-Java 8 install, they encounter a cryptic message:

    Exception in thread "main" java.lang.UnsupportedClassVersionError:
    org/elasticsearch/bootstrap/Elasticsearch : Unsupported major.minor
    version 52.0

They often think that they have Java 8 installed but if their JAVA_HOME
or other configuration is causing them to start with a pre-Java 8
install, this error message does not help them.

We introduce a Java version checker that runs on Java 6 as part of the
startup scripts. If the Java version is pre-Java 8, we can display a
helpful error message to the user informing them of the Java version
that the runtime was started with. Otherwise, Elasticsearch starts as it
does today.
2017-02-16 09:06:49 -05:00
Ryan Ernst 60b823c756 Add version checker tool to distributions 2017-02-16 09:06:49 -05:00
Ryan Ernst 8453cf0622 Build: Add notice file generation (#23170)
This change improves the notice file present in our distributions to
include notice and license files from each included dependency.

closes #22546
2017-02-15 09:40:16 -08:00
Jay Modi 7520a107be Optionally require a valid content type for all rest requests with content (#22691)
This change adds a strict mode for xcontent parsing on the rest layer. The strict mode will be off by default for 5.x and in a separate commit will be enabled by default for 6.0. The strict mode, which can be enabled by setting `http.content_type.required: true` in 5.x, will require that all incoming rest requests have a valid and supported content type header before the request is dispatched. In the non-strict mode, the Content-Type header will be inspected and if it is not present or not valid, we will continue with auto detection of content like we have done previously.

The content type header is parsed to the matching XContentType value with the only exception being for plain text requests. This value is then passed on with the content bytes so that we can reduce the number of places where we need to auto-detect the content type.

As part of this, many transport requests and builders were updated to provide methods that
accepted the XContentType along with the bytes and the methods that would rely on auto-detection have been deprecated.

In the non-strict mode, deprecation warnings are issued whenever a request with body doesn't provide the Content-Type header.

See #19388
2017-02-02 14:07:13 -05:00
Daniel Mitterdorfer 4073349267 Add example configuration for rotating GC log (#22715)
By default, the JVM GC log file grows without
limitation. This is inconvenient for a long running
process like Elasticsearch.

With this commit we add an example configuration
for a rotating GC log in `conig/jvm.options`.
2017-01-20 13:23:04 +01:00
Jason Tedor fc3280b3cf Expose logs base path
For certain situations, end-users need the base path for Elasticsearch
logs. Exposing this as a property is better than hard-coding the path
into the logging configuration file as otherwise the logging
configuration file could easily diverge from the Elasticsearch
configuration file. Additionally, Elasticsearch will only have
permissions to write to the log directory configured in the
Elasticsearch configuration file. This commit adds a property that
exposes this base path.

One use-case for this is configuring a rollover strategy to retain logs
for a certain period of time. As such, we add an example of this to the
documentation.

Additionally, we expose the property es.logs.cluster_name as this is
used as the name of the log files in the default configuration.

Finally, we expose es.logs.node_name in cases where node.name is
explicitly set in case users want to include the node name as part of
the name of the log files.

Relates #22625
2017-01-16 07:39:37 -05:00
Jason Tedor a6fb10826b Remove doc links from config template
The config template that ships with Elasticsearch distributions contains
links to various pieces of documentation. Links go out of date and get
broken. This commit removes such links from the config template.

Relates #22553
2017-01-11 11:23:07 -05:00
Jason Tedor f1ee2243b7 Remove disabling Netty pooled allocator
This commit reverts switching to the unpooled allocator (for now) to let
some benchmarks run to see if this is the source of an increase in GC
times.

Relates #22452
2017-01-10 14:30:15 -05:00
Nik Everett 12923ef896 Close and flush refresh listeners on shard close
Right now closing a shard looks like it strands refresh listeners,
causing tests like
`delete/50_refresh/refresh=wait_for waits until changes are visible in search`
to fail. Here is a build that fails:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+multi_cluster_search+multijob-darwin-compatibility/4/console

This attempts to fix the problem by implements `Closeable` on
`RefreshListeners` and rejecting listeners when closed. More importantly
the act of closing the instance flushes all pending listeners
so we shouldn't have any stranded listeners on close.

Because it was needed for testing, this also adds the number of
pending listeners to the `CommonStats` object and all API to which
that flows: `_cat/nodes`, `_cat/indices`, `_cat/shards`, and
`_nodes/stats`.
2017-01-06 20:03:32 -05:00
Ryan Ernst cd6e3f4cea Merge branch 'master' into keystore 2017-01-06 09:32:08 -08:00
Jason Tedor 9219d667e8 Disable the Netty recycler and pooled allocator
Netty plays a lot of games with recycling byte buffers in thread local
caches, and using a pooled byte buffer allocator to reduce pressure on
the garbage collector.

The recycler in particular appears to be fraught with peril. It appears
that there are circumstances where the recycler does not recycle quickly
enough and can exceed its capacity leading to heap exhaustion and out of
memory errors. If you spend a few minutes reading the history of the
recycler on the Netty GitHub issues, it appears it has been nothing but
a source of trouble, and the project itself has an open issue that
proposes disabling by default and possibly even removing the recycler.

The pooled byte buffer allocator has problems itself. It sizes the pool
based on the number of runtime processors and can indeed grab a very
large percentage of the heap (in some cases 50% or more). Additionally,
the Netty project continues to struggle with leaks here.

We are seeing users struggle with issues in 5.x that I think are largely
driven by some of the problems here with Netty.

This change proposes to disable the recycler, and to disable the pooled
byte buffer allocator. I think that disabling these features will return
some of the stablity that these features appear to be losing us.

I have done performance testing on my workstation with disabling these
and I do not see a difference in performance. I propose that we make
this change in master and let some nightly benchmarks run to confirm
that there is not a difference in performance. If we are comfortable
with the performance changes, I propose backporting this to all active
branches.

Relates #22452
2017-01-06 06:37:49 -05:00
javanna ccc4e414ff remove double slash from license header 2017-01-05 17:15:34 +01:00
Nik Everett 232af512f4 Switch from standalone-test to standalone-rest-test
standalone-rest-test doesn't configure unit tests and for these
integ test only tests, that is what we want.
2017-01-05 10:55:47 +01:00
Nik Everett 812f63e5ef Require either BuildPlugin or StandaloneTestBasePlugin to use RestTestPlugin
It used to be that RestTestPlugin "came with" StandaloneTestBasePlugin
but we'd like to use it with BuildPlugin for the high level rest client.
2017-01-05 10:55:47 +01:00
Ryan Ernst fb690ef748 Settings: Add infrastructure for elasticsearch keystore
This change is the first towards providing the ability to store
sensitive settings in elasticsearch. It adds the
`elasticsearch-keystore` tool, which allows managing a java keystore.
The keystore is loaded upon node startup in Elasticsearch, and used by
the Setting infrastructure when a setting is configured as secure.

There are a lot of caveats to this PR. The most important is it only
provides the tool and setting infrastructure for secure strings. It does
not yet provide for keystore passwords, keypairs, certificates, or even
convert any existing string settings to secure string settings. Those
will all come in follow up PRs. But this PR was already too big, so this
at least gets a basic version of the infrastructure in.

The two main things to look at.  The first is the `SecureSetting` class,
which extends `Setting`, but removes the assumption for the raw value of the
setting to be a string. SecureSetting provides, for now, a single
helper, `stringSetting()` to create a SecureSetting which will return a
SecureString (which is like String, but is closeable, so that the
underlying character array can be cleared). The second is the
`KeyStoreWrapper` class, which wraps the java `KeyStore` to provide a
simpler api (we do not need the entire keystore api) and also extend
the serialized format to add metadata needed for loading the keystore
with no assumptions about keystore type (so that we can change this in
the future) as well as whether the keystore has a password (so that we
can know whether prompting is necessary when we add support for keystore
passwords).
2016-12-22 16:28:34 -08:00
Nik Everett f5f2149ff2 Remove much ceremony from parsing client yaml test suites (#22311)
* Remove a checked exception, replacing it with `ParsingException`.
* Remove all Parser classes for the yaml sections, replacing them with static methods.
* Remove `ClientYamlTestFragmentParser`. Isn't used any more.
* Remove `ClientYamlTestSuiteParseContext`, replacing it with some static utility methods.

I did not rewrite the parsers using `ObjectParser` because I don't think it is worth it right now.
2016-12-22 11:00:34 -05:00
David Pilato 2ec5deef08 Link for setting page in elasticsearch.yml is outdated
Closes #22299.
2016-12-21 11:28:07 +01:00
Jason Tedor 5e68b632f8 Another fix for handling of paths on Windows
A previous fix for the handling of paths on Windows related to paths
containing multiple spaces introduced a issue where if JAVA_HOME ends
with a backslash, then Elasticsearch will refuse to start. This is not a
critical bug as a workaround exists (remove the trailing backslash), but
should be fixed nevertheless. This commit addresses this situation while
not regressing the previous fix.

Relates #22132
2016-12-20 21:08:36 -05:00
Jason Tedor 53b9ff820d Fix handling of spaces in Windows paths
This commit fixes the handling of spaces in Windows paths. The current
mechanism works fine in a path that contains a single space, but fails
on a path that contains multiple spaces. With this commit, that is no
longer the case.

Relates #21921
2016-12-01 19:36:41 -05:00
Jason Tedor b9df2e2287 Improve the out-of-the-box experience
Elasticsearch can be run in a few different ways:
 - from the command line on Linux and Windows
 - as a service on Linux and Windows

on both 32-bit client and 64-bit server VMs. We strive for a great
out-of-the-box experience any of these combinations but today it is
lacking on 32-bit client JVMs and on the Windows service. There are two
deficiencies that arise:
 - on any 32-bit client JVM we fail to start out of the box because we
   force the server JVM in jvm.options
 - when installing the Windows service, the thread stack size must be
   specified in jvm.options

This commit attempts to address these deficiencies.

We should continue to force the server JVM because there are systems
where the server JVM is not active by default (e.g., the 32-bit JDK on
Windows). This does mean that if a user tries to run with a client JVM
they will see a failure message at startup but this is the best that we
can do if we want to continue to force the server JVM. Thus, this commit
at least documents this situation.

To improve the situation with installing the Windows service, this
commit adds a default setting for the thread stack size. This default is
chosen based on the default thread stack size across all 64-bit server
JVMs. This means that if a user tries to run with a 32-bit JVM they
could otherwise see significantly higher memory usage (this situation is
complicated, it's really only on Windows where the extra memory usage is
egregious, but cutting into the 32-bit address space on any system is
bad). So this commit makes it so that the out-of-the-box experience is
improved for the Windows service on 64-bit server JVMs and we document
the need to adjust this setting on 32-bit JVMs.

Again, we are focusing on the out-of-the-box experience here and this
means optimizing for the best experience on any 64-bit server JVM as
this covers the vast majority of the user base. The users that are on
32-bit JVMs will suffer a little bit but at least now any user on any
64-bit server JVM can start Elasticsearch out of the box.

Finally, we fix some references to the jvm.options documentation.

Relates #21920
2016-12-01 17:26:29 -05:00
Jason Tedor 32df032c59 Add option to skip kernel parameters on install
During package install on systemd-based systems, we try to set
vm.max_map_count. On some systems (e.g., containers), users do not have
the ability to tune these parameters from within the container. This
commit provides an option for these users to skip setting such kernel
parameters.

Relates #21899
2016-12-01 17:23:51 -05:00
Jason Tedor b6ba4ae34b Add version 5.0.3
This commit adds version 5.0.3 and the BWC indices for version 5.0.2.

Relates #21867
2016-11-29 18:34:55 -05:00
Jason Tedor d3b444ad8a Truncate log messages from the end
Our default pattern layout truncates log messages. This is to avoid
blowing disk space from excessively log messages, which can happen if a
message contains a mapping or an large query. Yet, we trunacte from the
beginning which is probably where the most germane information is. This
commit modifies the default pattern layout to trunacte from the end.

Relates #21609
2016-11-16 21:45:14 -05:00
Nik Everett e66261eee9 Handle release of 5.0.1
Adds a version constant for it, bwc indices, and a vagrant upgrade-from
version. Also bumps the "upgrade from" version for the backwards-5.0
test and adds `skip`s for tests that don't fail against 5.0 so we skip
them during the backwards testing.

Finally, this skips the "Shrink index via API" test because it fails
consistently for me. Inconsistently for CI, but consistently for me.
I'll work on making it consistent tomorrow.
2016-11-15 19:31:28 -05:00
Yannick Welsch 6f69f54216 Use pre-JDK9 style FilePermissions on JDK9 when running from start scripts 2016-11-14 19:52:35 +01:00
Jason Tedor 404b9afeca Account for different paths for sysctl utilities
On some systems these utilities are in /usr/lib/systemd/systemd-sysctl
and /usr/sbin/sysctl, and on others the /usr is dropped. This commit
accounts for that fact.
2016-11-12 07:54:17 -05:00
Jason Tedor 1fde26347a Set vm.max_map_count on systemd package install
Our docs claim that we set vm.max_map_count automatically. This is not
quite the case. The story is that on SysV init we set vm.max_map_count
each time the service starts, which is good. On systemd, we create a
sysctl.d conf file that sets vm.map_max_count, but this is only
meaningful if the system is rebooted after package install. This commit
modifies the post-install script so that we run systemd-sysctl so that
the vm.max_map_count change occurs after package install without a
reboot.

Relates #21507
2016-11-11 16:41:54 -05:00
Luca Cavanna 06aabd9ecd Remove max_local_storage_nodes from elasticsearch.yml (#21467)
Given that the default is now 1, the comment in the config file was outdated. Also considering that the default value is production ready, we shouldn't list it among the values that need attention when going to production.

Relates to #19964
2016-11-10 16:52:38 +01:00
Jason Tedor b43ed8821f Export ES_JVM_OPTIONS for SysV init
The environment variable ES_JVM_OPTIONS allows end-users to specify a
custom location for the jvm.options file. Unfortunately, this
environment variable is not exported from the SysV init scripts. This
commit addresses this issue, and includes a test that ES_JVM_OPTIONS and
ES_JAVA_OPTS work for the SysV init packages.

Relates #21445
2016-11-09 17:52:37 -05:00
Ryan Ernst 562a30d3c6 Move licenses for core jar to core directory (#21383)
All plugins currently have their own licenses dir for the
dependencyLicenses task, but core disables this and has the check inside
distribution. This may have been better for maven, but for
gradle it makes more sense to just use the dependencyLicenses task that
automatically exists inside :core, and remove the hacked up version that
is inside distribution.
2016-11-07 15:29:35 -08:00
Ryan Ernst 7a2c984bcc Test: Remove multi process support from rest test runner (#21391)
At one point in the past when moving out the rest tests from core to
their own subproject, we had multiple test classes which evenly split up
the tests to run. However, we simplified this and went back to a single
test runner to have better reproduceability in tests. This change
removes the remnants of that multiplexing support.
2016-11-07 15:07:34 -08:00
Alexander Reelsen 5413efc570 Packaging Deb: configure start-stop-daemon to not go into background (#21343)
On ubuntu 14.04, which uses upstart, where as our debian package uses
sysvinit, there is no stdout/stderr message printed when starting up,
because the start-stop-daemon swallows it.

As Elasticsearch is started to daemonize, we can remove the background
flag from the start-stop-daemon and thus see, if the system does not have
enough memory for starting up - something that happens often on VMs, since
Elasticsearch 5.0 uses 2gb by default instead of one.

Relates #21300
Relates #12716
2016-11-07 13:59:11 +01:00
Adrien Grand 2a70f6e7b1 Upgrade to lucene-6.3.0-snapshot-a66a445. (#21309)
This addresses a bug that was introduced with https://issues.apache.org/jira/browse/LUCENE-7501.
2016-11-04 10:34:04 +01:00