Commit Graph

897 Commits

Author SHA1 Message Date
Boaz Leskes f79c2cb8c0 Allow BWC Testing against a specific branch (#25510)
Some times we need a fix / change to have two parts in two different branches (corresponding to two different ES releases). In order to be able to test these cases you need to run the BWC tests against a local branch rather than then using a branch from `github.com/elastic/elasticsearch`.

This commit adds a system property called `tests.bwc.refspec` that allows you to do it. Note that I've chosen to go with the simplest code change for now, at the expense of some user friendliness.
2017-07-07 11:18:03 +02:00
Jason Tedor be906628d5 Remove implicit 32-bit support
We previously tried to maintain (while not formally supporting) 32-bit
support, although we never tested this anywhere in CI. Since we do not
formally support this, and 32-bit usage is very low, we have elected to
no longer maintain 32-bit support. This commit removes any implication
of 32-bit support.

Relates #25435
2017-06-28 08:24:33 -04:00
Jason Tedor dfd241e0a6 Remove default path settings
This commit removes the default path settings for data and logs. With
this change, we now ship the packages with these settings set in the
elasticsearch.yml configuration file rather than going through the
default.path.data and default.path.logs dance that we went through in
the past.

Relates #25408
2017-06-26 21:43:20 -04:00
Jason Tedor 5a9fc8aa2a Remove path.conf setting
This commit removes path.conf as a valid setting and replaces it with a
command-line flag for specifying a non-default path for configuration.

Relates #25392
2017-06-26 15:18:29 -04:00
Jason Tedor dc9b67461f Fix use of spaces on Windows if JAVA_HOME not set
When JAVA_HOME is not set we try to detect the location of Java. If its
location contains a space, due to a lack of quoting we will be
unsuccessful in invoking Java. This commit adds the necessary quoting to
handle this case.

Relates #23822
2017-06-22 21:55:31 -04:00
Nik Everett 929194ef05 Fix artifact location
Fix the location of the rpm and deb used for the packaging upgrade
tests when upgrading from -SNAPSHOT version.
2017-06-19 14:20:30 -04:00
Nik Everett 21b1db2965 Remove assemble from build task when assemble removed
Removes the `assemble` task from the `build` task when we have
removed `assemble` from the project. We removed `assemble` from
projects that aren't published so our releases will be faster. But
That broke CI because CI builds with `gradle precommit build` and,
it turns out, that `build` includes `check` and `assemble`. With
this change CI will only run `check` for projects without an
`assemble`.
2017-06-16 17:19:14 -04:00
Nik Everett 7b358190d6 Remove assemble task when not used for publishing (#25228)
Removes the `assemble` task from projects that are not published.
This should speed up `gradle assemble` by skipping projects that
don't need to be built. Which is useful because `gradle assemble`
is how we cut releases.
2017-06-16 11:46:34 -04:00
David Roberts a5658c0fea When stopping via systemd only kill the JVM, not its control group (#25195)
This prevents possible race conditions between the Elasticsearch JVM and
plugin native controller processes that can cause the Elasticsearch shutdown
to hang.  The problem can happen when the JVM and the controller process
receive a SIGTERM at almost the same time.

(There's an assumption here that Elasticsearch will continue to use other
mechanisms to kill native controller processes.)
2017-06-14 09:23:41 +01:00
Jason Tedor a7a3af6f48 Log checkout so SHA is known
This commit changes the task type of the checkoutBwcBranch task to Exec
from LoggedExec so that the output of the checkout command is
shown. This enables us to see the SHA used for the checkout which can be
useful when debugging a BWC break.

Relates #25166
2017-06-09 22:06:51 -04:00
jaymode 5997e4a39d
Remove incorrect bwc branch logic from master
Commit bf007e8d93 was a forward port of logic needed in 5.x to get
the correct bwc branch. However, other changes on master meant that this forward port was not
needed and actually broke the bwc tests. This change removes the incorrect if statement.

Relates #25134
2017-06-09 09:47:06 -06:00
jaymode bf007e8d93
Fix branch logic for bwc tests in the same major version (#25134)
When testing against the previous 5.x release, the bwc project incorrectly would checkout the 5.x
branch instead of the 5.5 branch as it still had the logic that applies for major versions bwc. This change adds
a check to compare the major version when making the decision on the branch to use.
2017-06-08 11:39:29 -06:00
Christian Hofstaedtler c3ec6a1714 Honor masking of systemd-sysctl.service
During package install on systemd-based systems, some sysctl settings
should be set (e.g. vm.max_map_count).

In some environments, changing sysctl settings plainly does not work;
previously a global environment variable named
ES_SKIP_SET_KERNEL_PARAMETERS was introduced to skip calling sysctl, but
this causes trouble for:
 - configuration management systems, which usually cannot apply an env
   var when running a package manager
 - package upgrades, which will not have the env var set any more, and
   thus leaving the package management system in a bad state (possibly
   half-way upgraded, can be very hard to recover)

This removes the env var again and instead of calling systemd-sysctl
manually, tells systemd to restart the wrapper unit - which itself can
be masked by system administrators or management tools if it is known
that sysctl does not work in a given environment.

The restart is not silent on systems in their default configuration, but
is ignored if the unit is masked.

Relates #24234
2017-06-06 10:44:41 -04:00
Colin Goodheart-Smithe 779fb9a1c0 Adds nodes usage API to monitor usages of actions (#24169)
* Adds nodes usage API to monitor usages of actions

The nodes usage API has 2 main endpoints

/_nodes/usage and /_nodes/{nodeIds}/usage return the usage statistics
for all nodes and the specified node(s) respectively.

At the moment only one type of usage statistics is available, the REST
actions usage. This records the number of times each REST action class is
called and when the nodes usage api is called will return a map of rest
action class name to long representing the number of times each of the action
classes has been called.

Still to do:

* [x] Create usage service to store usage statistics
* [x] Record usage in REST layer
* [x] Add Transport Actions
* [x] Add REST Actions
* [x] Tests
* [x] Documentation

* Rafactors UsageService so counts are done by the handlers

* Fixing up docs tests

* Adds a name to all rest actions

* Addresses review comments
2017-06-02 08:46:38 +01:00
Jason Tedor 5961e1d6ab Remove import from InstallPluginCommandTests
This commit removes an unused import from
InstallPluginCommandTests.java.
2017-06-01 08:56:25 -04:00
Jason Tedor 9b4a189147 Add purge option to remove plugin CLI
By default, the remove plugin CLI command preserves configuration
files. This is so that if a user is upgrading the plugin (which is done
by first removing the old version and then installing the new version)
they do not lose their configuration file. Yet, there are circumstances
where preserving the configuration file is not desired. This commit adds
a purge option to the remove plugin CLI command.

Relates #24981
2017-06-01 08:53:39 -04:00
Jason Tedor 9b6b4ffe8e Set number of processes in systemd unit file
This commit sets the number of processes in the systemd unit file for
Elasticsearch to meet the bootstrap checks.

Relates #24970
2017-05-30 15:34:20 -04:00
Nik Everett bc4df47a76 Rework bwc snapshot projects to build up to two bwc versions (#24870)
Removes the `distribution:bwc` project in favor of
`distribution:bwc-release-snapshot` and
`distribution:bwc-stable-snapshot`.
`distribution:bwc-release-snapshot` builds a snapshot of the
latest release branch (5.4 now) if needed for backwards
compatibility. `distribution:bwc-stable-snapshot` builds a
snapshot of the latest stable branch (5.x now) if needed for
backwards compatibility.
2017-05-29 10:22:32 -04:00
Jason Tedor 95ebdbaddb Add BWC packaging distributions
Some packaging tests depend on snapshot versions of packaging
distributions yet the build does not use a repository that includes such
distributions. While we could add such a repository, a better strategy
is to follow our approach for other BWC tests where we depend on a
locally-compiled archive distribution. This commit adds a local
compilation of packaging artifacts and substitutes these anywhere that
we would otherwise depend on a snapshot of these artifacts.

Relates #24861
2017-05-24 11:55:32 -04:00
Colin Goodheart-Smithe 779cf3d468 remove remaining references to scripts directory (#24771)
The packaging scripts still referenced the scripts directory which is now removed.
2017-05-18 11:48:15 +01:00
Ryan Ernst 463fe2f4d4 Scripting: Remove file scripts (#24627)
This commit removes file scripts, which were deprecated in 5.5.

closes #21798
2017-05-17 14:42:25 -07:00
Ryan Ernst ff34434bba Build: Extract all ES versions into gradle properties (#24748)
This commit expands the logic for version extraction from Version.java
to include a list of all versions for backcompat purposes. The tests
using bwcVersion are converted to use this list, but those tests
(rolling upgrade and backwards-5.0) are still not randomized; that will
happen in another followup.
2017-05-17 12:58:37 -07:00
Russ Cam 026e94e6d9 Handle parentheses in path
The variable assignment needs to be quoted to correctly handle scenario
where the path contains parentheses.

Realtes #24731
2017-05-17 07:22:05 -04:00
Ryan Ernst 2a65bed243 Tests: Change rest test extension from .yaml to .yml (#24659)
This commit renames all rest test files to use the .yml extension
instead of .yaml. This way the extension used within all of
elasticsearch for yaml is consistent.
2017-05-16 17:24:35 -07:00
Jason Tedor 92ba969804 Remove Jython hacks
We had a hack in setting up permissions for tests to support testing
the lang-python plugin. We also had a hack to prevent Log4j from
loading a shaded version of Jansi provided by Jython. This plugin has
been removed so these hacks are no longer necessary.

Relates #24681
2017-05-15 12:40:24 -04:00
Jason Tedor 3f54ae6264 Fix handling of spaces in plugin script on Windows
A previous fix for handling spaces in the Windows scripts should also be
applied to the plugin script. This commit does that.

Closes #24750
2017-05-09 15:37:43 -04:00
Jason Tedor 3e485c2ca5 Fix plugin installation permissions
When installing plugin permissions, we try to set the permissions on all
installed files ourselves because a umask from the user could violate
everything needed to get the permissions right. Sadly, directories were
not handled correctly at all and so we were still left with broken
installations with umasks like 0077. This commit fixes this issue, adds
a thorough unit test for the situation, and most importantly, adds a
test that sets the umask before installing the plugin.

Relates #24527
2017-05-06 13:25:10 -04:00
Nik Everett 3b47355e56 Try not to lose stacktraces (#24426)
This adds `-XX:-OmitStackTraceInFastThrow` to the JVM arguments
which *should* prevent the JVM from omitting stack traces on
common exception sites. Even though these sites are common, we'd
still like the exceptions to debug them.

This also adds the flag when running tests and adapts some tests
that had workarounds for the absense of the flag.

Closes #24376
2017-05-02 11:34:12 -04:00
Ryan Ernst cdcc75dd2a Plugins: Add support for platform specific plugins (#24265)
This commit adds support for plugins having a platform specific variant.
It also adds unit tests for all official and maven urls.
2017-04-27 11:27:29 -07:00
Jason Tedor 08f52d1a18 Remove extra newline in RemovePluginCommand.java
This commit removes an extra newline in RemovePluginCommand.java because
it drives the OCD among us insane.
2017-04-21 22:18:33 -04:00
Jason Tedor 9912650641 Fix delete of plugin directory on remove plugin
This commit fixes an issue when deleting the plugin directory while
executing the remove plugin command. Namely, we take out a file
descriptor on the plugin directory to traverse its contents to obtain
the list of files to delete. We leaked this file descriptor. On
Unix-based filesystems, this is not a problem, deleting the plugin
directory deletes the plugin directory. On Windows though, a delete is
not executed until the last file descriptor is closed. Since we leaked
this file descriptor, the plugin was not actually deleted. This led to
test failures that tried to cleanup left behind temporary directories
but these test failures were just exposing this bug. This commit fixes
this issue by ensuring that we close the file descriptor to the plugin
directory when we are finished with it.

Relates #24266
2017-04-21 22:15:29 -04:00
Jason Tedor fe91c72151 Use a marker file when removing a plugin
Today when removing a plugin, we attempt to move the plugin directory to
a temporary directory and then delete that directory from the
filesystem. We do this to avoid a plugin being in a half-removed
state. We previously tried an atomic move, and fell back to a non-atomic
move if that failed. Atomic moves can fail on union filesystems when the
plugin directory is not in the top layer of the
filesystem. Interestingly, the regular move can fail as well. This is
because when the JDK is executing such a move, it first tries to rename
the source directory to the target directory and if this fails with
EXDEV (as in the case of an atomic move failing), it falls back to
copying the source to the target, and then attempts to rmdir the
source. The bug here is that the JDK never deleted the contents of the
source so the rmdir will always fail (except in the case of an empty
directory).

Given all this silliness, we were inspired to find a different
strategy. The strategy is simple. We will add a marker file to the
plugin directory that indicates the plugin is in a state of
removal. This file will be the last file out the door during removal. If
this file exists during startup, we fail startup.

Relates #24252
2017-04-21 15:50:44 -04:00
Ryan Ernst ba48674695 Build: Move plugin cli and tests to distribution tool (#24220)
The plugin cli currently resides inside the elasticsearch jar. This
commit moves it into a plugin-cli jar. This is change alone is a no-op;
it does not change anything about what is loaded at runtime. But it will
allow easier testing (with fixtures in the future to test ES or maven
installation), as well as eventually not loading these classes when
starting elasticsearch.
2017-04-21 09:25:58 -07:00
Ryan Ernst 212f24aa27 Tests: Clean up rest test file handling (#21392)
This change simplifies how the rest test runner finds test files and
removes all leniency.  Previously multiple prefixes and suffixes would
be tried, and tests could exist inside or outside of the classpath,
although outside of the classpath never quite worked. Now only classpath
tests are supported, and only one resource prefix is supported,
`/rest-api-spec/tests`.

closes #20240
2017-04-18 15:07:08 -07:00
Ryan Ernst a8017ff020 Tests: Move cluster dependencies from runner to cluster (#24142)
After splitting integ tests into cluster configuration and the test
runner task, we still have dependencies of the test runner added as deps
of the cluster. This commit adds dependencies directly to the cluster,
so that the runner can have other dependencies independent of what is
needed for the cluster.
2017-04-17 16:02:46 -07:00
Jason Tedor 51db6ab7dc Revert "Ensure output for createClone task"
This reverts commit 2620200ff7.
2017-04-15 13:54:01 -04:00
Lee Hinman 162ce85ff2 Remove customization of ES_USER and ES_GROUP
This removes the ability to configure Elasticsearch to use custom username
and/or group when it is run.

Resolves #23848
2017-04-14 13:24:21 -06:00
Ryan Ernst b0003edcf4 Build: Support offline build of bwc zip (#24037)
The bwc checkout for backcompat tests currently always tries to fetch
the latest from the upstream remote. This change makes fetching from
upstream conditional on not running an offline build.
2017-04-10 20:04:18 -07:00
David Roberts 37aadb2adf Add the ability to include extra notices in a plugin's NOTICES file (#23898)
Adds the option for a plugin to specify extra directories containing notices
and licenses files to be incorporated into the overall notices file that is
generated for the plugin.

This can be useful, for example, where the plugin has a non-Java dependency
that itself incorporates many 3rd party components.
2017-04-10 12:37:42 +01:00
Jason Tedor 2620200ff7 Ensure output for createClone task
This commit changes the task type from LoggedExec to Exec for the BWC
createClone task to ensure that we can get some debug output.
2017-04-06 08:37:04 -04:00
Ryan Ernst c48b60dcf1 Tests: Extend rest test timeout to 30 minutes for windows (#23912)
Windows rest tests consistenly fail because the filesystem appears to be
an order of magnitude slower than that of *nix, at least in the context
of our rest tests. This commit overrides the suite timeout to 30 mins
for windows. From past failures, it appears this should be enough, as
the tests seem to fail when they are almost complete. The default suite
timeout for ESTestCase is 20 mins, so this leaves ample buffer for
windows shenanigans.
2017-04-05 12:11:03 -07:00
Ryan Ernst a903aabbd6 Build: Ensure upstream check works even when using info logging (#23804)
The LoggedExec task does not capture output when info logging is
enabled. This commit changes the upstream check to use Exec directly,
so as not to break when info logging is enabled.
2017-03-29 13:42:47 -07:00
Ryan Ernst 5a414cbe5c Build: Use filesystem agnostic printing of bwc dir path (#23803)
This will use File.toString() for the `git clone` command, which will
automatically be correct for whatever system the build is running on.

closes #23784
2017-03-29 13:13:09 -07:00
Ryan Ernst cc1addeac2 Build: Find bwc version during build (#23801)
We currently have the last minor version of the previous major hardcoded
in tests like rolling upgrade. This change programatically finds this
during gradle initialization by parsing versions from Version.java.
2017-03-29 12:11:38 -07:00
Ryan Ernst d159920073 Build: Use GradleBuild task for invoking 5.x checkout build (#23770)
This commit switches from executing gradle when building the bwc testing
zip through Exec, to using GradleBuild. In addition to not depending on
gradle being in the PATH, it also has the added benefit of much better
logging while the bwc build is going on (the actual tasks show up as
tasks of a subproject within the current build).
2017-03-27 15:02:18 -07:00
Ryan Ernst 8822332355 Fetch all upstream refs for bwc checkout 2017-03-24 02:31:10 -07:00
Ryan Ernst a26321e03b Fix bwc checkout upstream remote check 2017-03-24 02:22:59 -07:00
Ryan Ernst 8c53555b28 Tests: Use local clone build of 5.x with bwc tests (#22946)
The current rest backcompat tests, which run against a mixed cluster of
5.x and 6.0 nodes, depend on snapshot builds of 5.x. However, this has
the potential for inconsistency that results in CI failures, and happens
quite often, whenever some backcompat logic is added to 5.x, but the bwc
test on master fails because the 5.x code has not yet been published as
a snapshot.

This change creates a git clone of the 5.x branch,
builds the zip distribution, and ties that into gradle substitutions for
the 5.x version.
2017-03-23 22:32:13 -07:00
Ryan Ernst f8453aca57 Packaging: Remove classpath ordering hack (#23596)
After the removal of the joda time hack we used to have, we can cleanup
the codebase handling in security, jarhell and plugins to be more picky
about uniqueness. This was originally in #18959 which was never merged.

closes #18959
2017-03-21 12:12:16 -07:00
Ryan Ernst 175bda64a0 Build: Rework integ test setup and shutdown to ensure stop runs when desired (#23304)
Gradle's finalizedBy on tasks only ensures one task runs after another,
but not immediately after. This is problematic for our integration tests
since it allows multiple project's integ test clusters to be
simultaneously. While this has not been a problem thus far (gradle 2.13
happened to keep the finalizedBy tasks close enough that no clusters
were running in parallel), with gradle 3.3 the task graph generation has
changed, and numerous clusters may be running simultaneously, causing
memory pressure, and thus generally slower tests, or even failure if the
system has a limited amount of memory (eg in a vagrant host).

This commit reworks how integ tests are configured. It adds an
`integTestCluster` extension to gradle which is equivalent to the current
`integTest.cluster` and moves the rest test runner task to
`integTestRunner`.  The `integTest` task is then just a dummy task,
which depends on the cluster runner task, as well as the cluster stop
task. This means running `integTest` in one project will both run the
rest tests, and shut down the cluster, before running `integTest` in
another project.
2017-02-22 12:43:15 -08:00
Jay Modi b234644035 Enforce Content-Type requirement on the rest layer and remove deprecated methods (#23146)
This commit enforces the requirement of Content-Type for the REST layer and removes the deprecated methods in transport
requests and their usages.

While doing this, it turns out that there are many places where *Entity classes are used from the apache http client
libraries and many of these usages did not specify the content type. The methods that do not specify a content type
explicitly have been added to forbidden apis to prevent more of these from entering our code base.

Relates #19388
2017-02-17 14:45:41 -05:00
Ryan Ernst 164079bb93 Disable javadoc for java vesion checker 2017-02-16 09:56:11 -08:00
Jason Tedor c9cde11a5e Introduce Java version checker
Today when users start Elasticsearch with their Java configuration
pointing to a pre-Java 8 install, they encounter a cryptic message:

    Exception in thread "main" java.lang.UnsupportedClassVersionError:
    org/elasticsearch/bootstrap/Elasticsearch : Unsupported major.minor
    version 52.0

They often think that they have Java 8 installed but if their JAVA_HOME
or other configuration is causing them to start with a pre-Java 8
install, this error message does not help them.

We introduce a Java version checker that runs on Java 6 as part of the
startup scripts. If the Java version is pre-Java 8, we can display a
helpful error message to the user informing them of the Java version
that the runtime was started with. Otherwise, Elasticsearch starts as it
does today.
2017-02-16 09:06:49 -05:00
Ryan Ernst 60b823c756 Add version checker tool to distributions 2017-02-16 09:06:49 -05:00
Ryan Ernst 8453cf0622 Build: Add notice file generation (#23170)
This change improves the notice file present in our distributions to
include notice and license files from each included dependency.

closes #22546
2017-02-15 09:40:16 -08:00
Jay Modi 7520a107be Optionally require a valid content type for all rest requests with content (#22691)
This change adds a strict mode for xcontent parsing on the rest layer. The strict mode will be off by default for 5.x and in a separate commit will be enabled by default for 6.0. The strict mode, which can be enabled by setting `http.content_type.required: true` in 5.x, will require that all incoming rest requests have a valid and supported content type header before the request is dispatched. In the non-strict mode, the Content-Type header will be inspected and if it is not present or not valid, we will continue with auto detection of content like we have done previously.

The content type header is parsed to the matching XContentType value with the only exception being for plain text requests. This value is then passed on with the content bytes so that we can reduce the number of places where we need to auto-detect the content type.

As part of this, many transport requests and builders were updated to provide methods that
accepted the XContentType along with the bytes and the methods that would rely on auto-detection have been deprecated.

In the non-strict mode, deprecation warnings are issued whenever a request with body doesn't provide the Content-Type header.

See #19388
2017-02-02 14:07:13 -05:00
Daniel Mitterdorfer 4073349267 Add example configuration for rotating GC log (#22715)
By default, the JVM GC log file grows without
limitation. This is inconvenient for a long running
process like Elasticsearch.

With this commit we add an example configuration
for a rotating GC log in `conig/jvm.options`.
2017-01-20 13:23:04 +01:00
Jason Tedor fc3280b3cf Expose logs base path
For certain situations, end-users need the base path for Elasticsearch
logs. Exposing this as a property is better than hard-coding the path
into the logging configuration file as otherwise the logging
configuration file could easily diverge from the Elasticsearch
configuration file. Additionally, Elasticsearch will only have
permissions to write to the log directory configured in the
Elasticsearch configuration file. This commit adds a property that
exposes this base path.

One use-case for this is configuring a rollover strategy to retain logs
for a certain period of time. As such, we add an example of this to the
documentation.

Additionally, we expose the property es.logs.cluster_name as this is
used as the name of the log files in the default configuration.

Finally, we expose es.logs.node_name in cases where node.name is
explicitly set in case users want to include the node name as part of
the name of the log files.

Relates #22625
2017-01-16 07:39:37 -05:00
Jason Tedor a6fb10826b Remove doc links from config template
The config template that ships with Elasticsearch distributions contains
links to various pieces of documentation. Links go out of date and get
broken. This commit removes such links from the config template.

Relates #22553
2017-01-11 11:23:07 -05:00
Jason Tedor f1ee2243b7 Remove disabling Netty pooled allocator
This commit reverts switching to the unpooled allocator (for now) to let
some benchmarks run to see if this is the source of an increase in GC
times.

Relates #22452
2017-01-10 14:30:15 -05:00
Nik Everett 12923ef896 Close and flush refresh listeners on shard close
Right now closing a shard looks like it strands refresh listeners,
causing tests like
`delete/50_refresh/refresh=wait_for waits until changes are visible in search`
to fail. Here is a build that fails:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+multi_cluster_search+multijob-darwin-compatibility/4/console

This attempts to fix the problem by implements `Closeable` on
`RefreshListeners` and rejecting listeners when closed. More importantly
the act of closing the instance flushes all pending listeners
so we shouldn't have any stranded listeners on close.

Because it was needed for testing, this also adds the number of
pending listeners to the `CommonStats` object and all API to which
that flows: `_cat/nodes`, `_cat/indices`, `_cat/shards`, and
`_nodes/stats`.
2017-01-06 20:03:32 -05:00
Ryan Ernst cd6e3f4cea Merge branch 'master' into keystore 2017-01-06 09:32:08 -08:00
Jason Tedor 9219d667e8 Disable the Netty recycler and pooled allocator
Netty plays a lot of games with recycling byte buffers in thread local
caches, and using a pooled byte buffer allocator to reduce pressure on
the garbage collector.

The recycler in particular appears to be fraught with peril. It appears
that there are circumstances where the recycler does not recycle quickly
enough and can exceed its capacity leading to heap exhaustion and out of
memory errors. If you spend a few minutes reading the history of the
recycler on the Netty GitHub issues, it appears it has been nothing but
a source of trouble, and the project itself has an open issue that
proposes disabling by default and possibly even removing the recycler.

The pooled byte buffer allocator has problems itself. It sizes the pool
based on the number of runtime processors and can indeed grab a very
large percentage of the heap (in some cases 50% or more). Additionally,
the Netty project continues to struggle with leaks here.

We are seeing users struggle with issues in 5.x that I think are largely
driven by some of the problems here with Netty.

This change proposes to disable the recycler, and to disable the pooled
byte buffer allocator. I think that disabling these features will return
some of the stablity that these features appear to be losing us.

I have done performance testing on my workstation with disabling these
and I do not see a difference in performance. I propose that we make
this change in master and let some nightly benchmarks run to confirm
that there is not a difference in performance. If we are comfortable
with the performance changes, I propose backporting this to all active
branches.

Relates #22452
2017-01-06 06:37:49 -05:00
javanna ccc4e414ff remove double slash from license header 2017-01-05 17:15:34 +01:00
Nik Everett 232af512f4 Switch from standalone-test to standalone-rest-test
standalone-rest-test doesn't configure unit tests and for these
integ test only tests, that is what we want.
2017-01-05 10:55:47 +01:00
Nik Everett 812f63e5ef Require either BuildPlugin or StandaloneTestBasePlugin to use RestTestPlugin
It used to be that RestTestPlugin "came with" StandaloneTestBasePlugin
but we'd like to use it with BuildPlugin for the high level rest client.
2017-01-05 10:55:47 +01:00
Ryan Ernst fb690ef748 Settings: Add infrastructure for elasticsearch keystore
This change is the first towards providing the ability to store
sensitive settings in elasticsearch. It adds the
`elasticsearch-keystore` tool, which allows managing a java keystore.
The keystore is loaded upon node startup in Elasticsearch, and used by
the Setting infrastructure when a setting is configured as secure.

There are a lot of caveats to this PR. The most important is it only
provides the tool and setting infrastructure for secure strings. It does
not yet provide for keystore passwords, keypairs, certificates, or even
convert any existing string settings to secure string settings. Those
will all come in follow up PRs. But this PR was already too big, so this
at least gets a basic version of the infrastructure in.

The two main things to look at.  The first is the `SecureSetting` class,
which extends `Setting`, but removes the assumption for the raw value of the
setting to be a string. SecureSetting provides, for now, a single
helper, `stringSetting()` to create a SecureSetting which will return a
SecureString (which is like String, but is closeable, so that the
underlying character array can be cleared). The second is the
`KeyStoreWrapper` class, which wraps the java `KeyStore` to provide a
simpler api (we do not need the entire keystore api) and also extend
the serialized format to add metadata needed for loading the keystore
with no assumptions about keystore type (so that we can change this in
the future) as well as whether the keystore has a password (so that we
can know whether prompting is necessary when we add support for keystore
passwords).
2016-12-22 16:28:34 -08:00
Nik Everett f5f2149ff2 Remove much ceremony from parsing client yaml test suites (#22311)
* Remove a checked exception, replacing it with `ParsingException`.
* Remove all Parser classes for the yaml sections, replacing them with static methods.
* Remove `ClientYamlTestFragmentParser`. Isn't used any more.
* Remove `ClientYamlTestSuiteParseContext`, replacing it with some static utility methods.

I did not rewrite the parsers using `ObjectParser` because I don't think it is worth it right now.
2016-12-22 11:00:34 -05:00
David Pilato 2ec5deef08 Link for setting page in elasticsearch.yml is outdated
Closes #22299.
2016-12-21 11:28:07 +01:00
Jason Tedor 5e68b632f8 Another fix for handling of paths on Windows
A previous fix for the handling of paths on Windows related to paths
containing multiple spaces introduced a issue where if JAVA_HOME ends
with a backslash, then Elasticsearch will refuse to start. This is not a
critical bug as a workaround exists (remove the trailing backslash), but
should be fixed nevertheless. This commit addresses this situation while
not regressing the previous fix.

Relates #22132
2016-12-20 21:08:36 -05:00
Jason Tedor 53b9ff820d Fix handling of spaces in Windows paths
This commit fixes the handling of spaces in Windows paths. The current
mechanism works fine in a path that contains a single space, but fails
on a path that contains multiple spaces. With this commit, that is no
longer the case.

Relates #21921
2016-12-01 19:36:41 -05:00
Jason Tedor b9df2e2287 Improve the out-of-the-box experience
Elasticsearch can be run in a few different ways:
 - from the command line on Linux and Windows
 - as a service on Linux and Windows

on both 32-bit client and 64-bit server VMs. We strive for a great
out-of-the-box experience any of these combinations but today it is
lacking on 32-bit client JVMs and on the Windows service. There are two
deficiencies that arise:
 - on any 32-bit client JVM we fail to start out of the box because we
   force the server JVM in jvm.options
 - when installing the Windows service, the thread stack size must be
   specified in jvm.options

This commit attempts to address these deficiencies.

We should continue to force the server JVM because there are systems
where the server JVM is not active by default (e.g., the 32-bit JDK on
Windows). This does mean that if a user tries to run with a client JVM
they will see a failure message at startup but this is the best that we
can do if we want to continue to force the server JVM. Thus, this commit
at least documents this situation.

To improve the situation with installing the Windows service, this
commit adds a default setting for the thread stack size. This default is
chosen based on the default thread stack size across all 64-bit server
JVMs. This means that if a user tries to run with a 32-bit JVM they
could otherwise see significantly higher memory usage (this situation is
complicated, it's really only on Windows where the extra memory usage is
egregious, but cutting into the 32-bit address space on any system is
bad). So this commit makes it so that the out-of-the-box experience is
improved for the Windows service on 64-bit server JVMs and we document
the need to adjust this setting on 32-bit JVMs.

Again, we are focusing on the out-of-the-box experience here and this
means optimizing for the best experience on any 64-bit server JVM as
this covers the vast majority of the user base. The users that are on
32-bit JVMs will suffer a little bit but at least now any user on any
64-bit server JVM can start Elasticsearch out of the box.

Finally, we fix some references to the jvm.options documentation.

Relates #21920
2016-12-01 17:26:29 -05:00
Jason Tedor 32df032c59 Add option to skip kernel parameters on install
During package install on systemd-based systems, we try to set
vm.max_map_count. On some systems (e.g., containers), users do not have
the ability to tune these parameters from within the container. This
commit provides an option for these users to skip setting such kernel
parameters.

Relates #21899
2016-12-01 17:23:51 -05:00
Jason Tedor b6ba4ae34b Add version 5.0.3
This commit adds version 5.0.3 and the BWC indices for version 5.0.2.

Relates #21867
2016-11-29 18:34:55 -05:00
Jason Tedor d3b444ad8a Truncate log messages from the end
Our default pattern layout truncates log messages. This is to avoid
blowing disk space from excessively log messages, which can happen if a
message contains a mapping or an large query. Yet, we trunacte from the
beginning which is probably where the most germane information is. This
commit modifies the default pattern layout to trunacte from the end.

Relates #21609
2016-11-16 21:45:14 -05:00
Nik Everett e66261eee9 Handle release of 5.0.1
Adds a version constant for it, bwc indices, and a vagrant upgrade-from
version. Also bumps the "upgrade from" version for the backwards-5.0
test and adds `skip`s for tests that don't fail against 5.0 so we skip
them during the backwards testing.

Finally, this skips the "Shrink index via API" test because it fails
consistently for me. Inconsistently for CI, but consistently for me.
I'll work on making it consistent tomorrow.
2016-11-15 19:31:28 -05:00
Yannick Welsch 6f69f54216 Use pre-JDK9 style FilePermissions on JDK9 when running from start scripts 2016-11-14 19:52:35 +01:00
Jason Tedor 404b9afeca Account for different paths for sysctl utilities
On some systems these utilities are in /usr/lib/systemd/systemd-sysctl
and /usr/sbin/sysctl, and on others the /usr is dropped. This commit
accounts for that fact.
2016-11-12 07:54:17 -05:00
Jason Tedor 1fde26347a Set vm.max_map_count on systemd package install
Our docs claim that we set vm.max_map_count automatically. This is not
quite the case. The story is that on SysV init we set vm.max_map_count
each time the service starts, which is good. On systemd, we create a
sysctl.d conf file that sets vm.map_max_count, but this is only
meaningful if the system is rebooted after package install. This commit
modifies the post-install script so that we run systemd-sysctl so that
the vm.max_map_count change occurs after package install without a
reboot.

Relates #21507
2016-11-11 16:41:54 -05:00
Luca Cavanna 06aabd9ecd Remove max_local_storage_nodes from elasticsearch.yml (#21467)
Given that the default is now 1, the comment in the config file was outdated. Also considering that the default value is production ready, we shouldn't list it among the values that need attention when going to production.

Relates to #19964
2016-11-10 16:52:38 +01:00
Jason Tedor b43ed8821f Export ES_JVM_OPTIONS for SysV init
The environment variable ES_JVM_OPTIONS allows end-users to specify a
custom location for the jvm.options file. Unfortunately, this
environment variable is not exported from the SysV init scripts. This
commit addresses this issue, and includes a test that ES_JVM_OPTIONS and
ES_JAVA_OPTS work for the SysV init packages.

Relates #21445
2016-11-09 17:52:37 -05:00
Ryan Ernst 562a30d3c6 Move licenses for core jar to core directory (#21383)
All plugins currently have their own licenses dir for the
dependencyLicenses task, but core disables this and has the check inside
distribution. This may have been better for maven, but for
gradle it makes more sense to just use the dependencyLicenses task that
automatically exists inside :core, and remove the hacked up version that
is inside distribution.
2016-11-07 15:29:35 -08:00
Ryan Ernst 7a2c984bcc Test: Remove multi process support from rest test runner (#21391)
At one point in the past when moving out the rest tests from core to
their own subproject, we had multiple test classes which evenly split up
the tests to run. However, we simplified this and went back to a single
test runner to have better reproduceability in tests. This change
removes the remnants of that multiplexing support.
2016-11-07 15:07:34 -08:00
Alexander Reelsen 5413efc570 Packaging Deb: configure start-stop-daemon to not go into background (#21343)
On ubuntu 14.04, which uses upstart, where as our debian package uses
sysvinit, there is no stdout/stderr message printed when starting up,
because the start-stop-daemon swallows it.

As Elasticsearch is started to daemonize, we can remove the background
flag from the start-stop-daemon and thus see, if the system does not have
enough memory for starting up - something that happens often on VMs, since
Elasticsearch 5.0 uses 2gb by default instead of one.

Relates #21300
Relates #12716
2016-11-07 13:59:11 +01:00
Adrien Grand 2a70f6e7b1 Upgrade to lucene-6.3.0-snapshot-a66a445. (#21309)
This addresses a bug that was introduced with https://issues.apache.org/jira/browse/LUCENE-7501.
2016-11-04 10:34:04 +01:00
Jason Tedor 799a12ad63 Fix distribution build ordering issue
Today when running gradle clean
:distribution:(integ-test-zip|tar|zip):assemble, the created archive
distribution will be missing the empty plugins directory. This is
because the empty plugins folder created in the build folder for the
copy spec task is created during configuration and then is later wiped
away by the clean task. This commit addresses this issue, by pushing
creation of the directory out of the configuration phase.

Relates #21271
2016-11-02 12:14:47 -04:00
Jason Tedor 8dd91eb2d6 Add empty plugins dir for archive distributions
Today when installing Elasticsearch from an archive distribution (tar.gz
or zip), an empty plugins folder is not included. This means that if you
install Elasticsearch and immediately run elasticsearch-plugin list, you
will receive an error message about the plugins directory missing. While
the plugins directory would be created when starting Elasticsearch for
the first time, it would be better to just include an empty plugins
directory in the archive distributions. This commit makes this the
case. Note that the package distributions already include an empty
plugins folder.

Relates #21204
2016-11-01 15:14:24 -04:00
Jason Tedor 44af41ed55 Make explicit missing settings for Windows service
When installing the Windows service, certain settings like the minimum
heap, maximum heap and thread stack size setting must be set. While
there is an error message making mention of this fact, the error message
is not explicit exactly what setting needs to be set. This commit makes
these settings explicit.

Relates #21200
2016-10-31 10:10:37 -04:00
Adrien Grand b3cc54cf0d Upgrade to lucene-6.3.0-snapshot-ed102d6 (#21150)
Lucene 6.3 is expected to be released in the next weeks so it'd be good to give
it some integration testing. I had to upgrade randomized-testing too so that
both Lucene and Elasticsearch are on the same version.
2016-10-28 14:47:15 +02:00
Jason Tedor 9c3e4d6e22 Add correct Content-Length on HEAD requests
This commit fixes responses to HEAD requests so that the value of the
Content-Length is correct per the HTTP spec. Namely, the value of this
header should be equal to the Content-Length if the request were not a
HEAD request.

This commit also fixes a memory leak on HEAD requests to the main action
that arose from the bytes on a builder not being released due to them
being dropped on the floor to ensure that the response to the main
action did not have a body.

Relates #21123
2016-10-25 23:08:19 -04:00
Tanguy Leroux 127b4a8efc Change permissions on config files (#20966)
This commit changes some default file permissions on configuration files.
2016-10-24 09:42:03 +02:00
Nik Everett 8cc22eb960 Make sure HEAD / has 0 Content-Length (#21077)
Before this commit `curl -XHEAD localhost:9200?pretty` would return
`Content-Length: 1` and a body which is fairly upsetting to standards
compliant tools. Now it'll return `Content-Length: 0` with an empty
body like every other `HEAD` request.

Relates to #21075
2016-10-21 16:44:50 -04:00
Jason Tedor 535bf92280 Upgrade Log4j 2 to version 2.7
This commit upgrades the Log4j 2 dependency to version 2.7 and removes
some hacks that we had in place to work around bugs in Log4j 2 version
2.6.2.

Relates #20805
2016-10-07 13:05:30 -05:00
Nik Everett 6705c6aa2f Further document the pom hack
We have a "HUGE HACK" that allows us to publish zip artifacts to
Sonatype's OSS repository without javadoc and source jars. We don't
include those jars because the zip is just a repackaging of the
core and module jars for which we already publish the javadoc and
source jars. So we have a hack to publish the zip artifact when the
pom says the project is of type 'pom'.
2016-09-21 11:25:19 -04:00
Nik Everett 3771d80830 Fix up nebulaRealPom
I'm not sure why we need this pom instead of the pom generated by
nebula, but if we are going to have it then we need to populate it
with appropriate stuff like project name, description, and url.
2016-09-20 11:05:23 -04:00
Ryan Ernst 85b8f29415 Build: Remove old maven deploy support (#20403)
* Build: Remove old maven deploy support

This change removes the old maven deploy that we have in parallel to
maven-publish, and makes maven-publish fully work with publishing to
maven local. Using `gradle publishToMavenLocal` should be used to
publish to .m2.

Note that there is an unfortunate hack that means for
zip artifacts we must first create/publish a dummy pom file, and then
follow that with the real pom file. It would be nice to have the pom
file contains packaging=zip, but maven central then requires sources and
javadocs. But our zips are really just attached artifacts, so we already
set the packaging type to pom for our zip files. This change just works
around a limitation of the underlying maven publishing library which
silently skips attached artifacts when the packaging type is set to pom.

relates #20164
closes #20375

* Remove unnecessary extra spacing
2016-09-19 15:10:41 -07:00
Jason Tedor b456823434 Fix leftover node attributes usage
Previously node attributes could be set via node.* but this now requires
using node.attr.*. This commit fixes some leftover usages of the old
way.
2016-09-19 07:45:31 -04:00
gfyoung b03c807368 Rename service.bat to elasticsearch-service.bat (#20496)
Closes gh-17528.
2016-09-15 15:53:44 +02:00
Ryan Ernst f885218063 Plugins: Use sysprop like with es.path.home to pass conf dir (#18870)
Currently we always pass -E to the the plugin cli with the conf dir, but
this causes a very confusing error message when not giving a specific
command to the plugin cli. This change makes path.conf pass just like
path.home. These are special settings, so passing via sysprops is the
right thing to do (it is all about how we pass between shell and java
cli).

closes #18689
2016-09-14 14:09:06 -07:00
Tanguy Leroux 3caaafa4bd [Package] Remove bin/lib/modules directories on RPM uninstall/upgrade (#20448)
When uninstalling or upgrading elasticsearch using the RPM package some empty directories remain on the filesystem:

    /usr/share/elasticsearch/bin
    /usr/share/elasticsearch/lib
    /usr/share/elasticsearch/modules
    /usr/share/elasticsearch/modules/foo

Having empty directories in modules can prevent elasticsearch to start after an upgrade: the plugins service expects to find a plugin-descriptor.properties file in every sub directory of modules.

This PR cleans things a bit so that these empty directories are removed on upgrade/removal like it was in 2.x.
2016-09-14 18:18:37 +02:00
Tanguy Leroux 26dc6f1306 [Packaging] Do not remove scripts directory on upgrade (#20452)
When upgrading elasticsearch using the RPM package, the scripts directory is removed if it's empty but it won't be recreated by the upgraded package. But after that the service won't start because the scripts dir is missing.
2016-09-14 10:11:27 +02:00
Jason Tedor 7560101ec7 Complete Elasticsearch logger names
This commit modifies the logger names within Elasticsearch to be the
fully-qualified class name as opposed removing the org.elasticsearch
prefix and dropping the class name. This change separates the root
logger from the Elasticsearch loggers (they were equated from the
removal of the org.elasticsearch prefix) and enables log levels to be
set at the class level (instead of the package level).

Relates #20457
2016-09-13 22:46:54 -04:00
Jason Tedor fbe27664a6 Fix prefix logging
Today we add a prefix when logging within Elasticsearch. This prefix
contains the node name, and index and shard-level components if
appropriate.

Due to some implementation details with Log4j 2 , this does not work for
integration tests; instead what we see is the node name for the last
node to startup. The implementation detail here is that Log4j 2 there is
only one logger for a name, message factory pair, and the key derived
from the message factory is the class name of the message factory. So,
when the last node starts up and starts setting prefixes on its message
factories, it will impact the loggers for the other nodes.

Additionally, the prefixes are lost when logging an exception. This is
due to another implementation detail in Log4j 2. Namely, since we log
exceptions using a parameterized message, Log4j 2 decides that that
means that we do not want to use the message factory that we have
provided (the prefix message factory) and so logs the exception without
the prefix.

This commit fixes both of these issues.

Relates #20429
2016-09-13 14:46:34 -04:00
Tanguy Leroux 6090c51fc5 Add quiet option to disable console logging (#20422)
This commit adds a -q/--quiet option to Elasticsearch so that it does not log anything in the console and closes stdout & stderr streams. This is useful for SystemD to avoid duplicate logs in both journalctl and /var/log/elasticsearch/elasticsearch.log while still allows the JVM to print error messages in stdout/stderr if needed.

closes #17220
2016-09-13 14:08:24 +02:00
Jason Tedor f58a33eb90 Remove unsupported environment variables check
The Elasticsearch startup scripts contain checks for the presence of
support for environment variables that were removed in the 5.x
series. These checks warn the user and fail the script if any of the
unsupported environment variables are present. This was provided as
migration step from 2.x to 5.x so that we were not just silently
ignoring environment variables that were previously set. This commit
removes these checks, as upgrades from 2.x to 6.x are not supported.

Relates #20404
2016-09-09 14:55:32 -04:00
Jason Tedor 27ff4f327c Remove allow unquoted JSON
Previous versions of Elasticsearch permitted unquoted JSON field names even though this is against the JSON spec. This leniency was disabled by default in the 5.x series of Elasticsearch but a backwards compatibility layer was added via a system property with the intention of removing this layer in 6.0.0. This commit removes this backwards compatibility layer.

Relates #20388
2016-09-08 13:36:31 -04:00
Ryan Ernst 12141d5f9d Remove -D handling in args for windows (#20378)
-D parameters used to be allowed when starting elasticsearch scripts.
However, this was removed in #18207, but the elasticsearch-plugin.bat script
was forgotten. This change removes the -D handling.
2016-09-07 15:52:34 -07:00
Simon Willnauer 11f2da5f14 Skip loading of jansi from log4j2 (#20334)
Jython shades `jansi` into it's classpath without changing it's package or
anything like that. This causes attempts to load native code on windows which
blows up tests. This change adds `log4j.skipJansi=true` system property to our
tests as well as to the JVM properties we set.
2016-09-06 05:53:00 -04:00
Jason Tedor 5fe4cb6adc Size limit deprecation logs
This commit configures the deprecation logs to be size-limited to 1 GB,
and compress these logs when they roll. The default configuration will
preserve up to four rolled logs.

Relates #20287
2016-09-01 14:25:04 -04:00
Jason Tedor c98e790772 Merge branch 'master' into log4j2
* master:
  Correct path.conf for integ-test-zip distribution
2016-08-31 22:34:47 -04:00
Jason Tedor 7c47325dc7 Correct path.conf for integ-test-zip distribution
The integ-test-zip distribution did not specify a value for
path.conf. As such, it picked up the default value of
/etc/elasticsearch. This means that on machines that have this
directory, integration tests could fail because they would try to pick
up configuration from that directory rather than from the home directory
of the exploded distribution. This commit fixes this issue by specifying
a value of path.conf for the integ-test-zip distribution.

Relates #20271
2016-08-31 21:28:40 -04:00
Jason Tedor e166459bbe Merge branch 'master' into log4j2
* master:
  Increase visibility of deprecation logger
  Skip transport client plugin installed on JDK 9
  Explicitly disable Netty key set replacement
  percolator: Fail indexing percolator queries containing either a has_child or has_parent query.
  Make it possible for Ingest Processors to access AnalysisRegistry
  Allow RestClient to send array-based headers
  Silence rest util tests until the bogusness can be simplified
  Remove unknown HttpContext-based test as it fails unpredictably on different JVMs
  Tests: Improve rest suite names and generated test names for docs tests
  Add support for a RestClient base path
2016-08-31 10:59:27 -04:00
Jason Tedor 1a805bb675 Increase visibility of deprecation logger
The deprecation logger is an important way to make visible features of
Elasticsearch that are deprecated. Yet, the default logging makes the
log messages for the deprecation logger invisible. We want these log
messages to be visible, so the default logging for the deprecation
logger should enable these log messages. This commit changes the log
level of deprecation log message to warn, and configures the deprecation
logger so that these log messages are visible out of the box.

Relates #20254
2016-08-31 10:51:17 -04:00
Jason Tedor 0c027acbf9 Explicitly disable Netty key set replacement
Netty replaces the backing set for the selector implementation. The
value of doing this is questionable, and doing this requires permissions
that we are not going to grant. This commit explicitly disables this
optimization rather than relying on it failing due to lack of
permissions.

Relates #20249
2016-08-31 08:11:03 -04:00
Jason Tedor 7da0cdec42 Introduce Log4j 2
This commit introduces Log4j 2 to the stack.
2016-08-30 13:31:24 -04:00
Mike McCandless 0ccfe69789 Upgrade to Lucene 6.2.0 2016-08-24 17:26:28 -04:00
Ali Beyad 3d7daa1a5a Removes the whitespace between the # (to comment out) and (#20094)
the setting in elasticsearch.yml, so that when a user uncomments
out a setting by just removing the #, the setting actually
takes effect. Before, it was very easy to uncomment out a
setting by just removing the #, leaving a single whitespace
character before the setting name, which would cause the
setting to not get picked up by Elasticsearch.

Closes #20090
2016-08-22 09:03:34 -04:00
Jason Tedor 743b2ed1b2 Set default min heap equal to default max heap
This commit sets the default min heap equal to the default max
heap. This is to align the default out-of-box settings with the heap
size bootstrap check.
2016-08-19 12:29:48 -04:00
Clinton Gormley 8e5ee2c6d5 RESTART_ON_UPGRADE incorrectly named ES_RESTART_ON_UPGRADE in sysconfig (#19976)
Closes #19950
2016-08-12 19:19:30 +02:00
Jason Tedor d7dbb2b595 Fix handling of spaces for jvm.options on Windows
This commit fixes the handling of spaces in the path to the jvm.options
file on Windows. The issue is that the extraneous set of quotes were
included as part of the value of ES_JVM_OPTIONS thus confusing further
downstream commands.

Relates #19951
2016-08-11 11:23:22 -04:00
Tanguy Leroux 841d5a210e Update to Jackson 2.8.1
This commit updates Jackson to the 2.8.1 version, which is more strict when it comes to build objects. It also adds the snakeyaml dependency that was previously shaded in jackson libs.

It also closes #18076
2016-08-05 12:26:06 +02:00
Jason Tedor eb6da69e9f Explicitly tell Netty to not use unsafe
With the security permissions that we grant to Netty, Netty can not
access unsafe (because it relies on having the runtime permission
accessDeclaredMembers and the reflect permission
suppressAccessChecks). Instead, we should just explicitly tell Netty to
not use unsafe. This commit adds a flag to the default jvm.options to
tell Netty to not look for unsafe.

Relates #19786
2016-08-03 15:23:34 -04:00
Nik Everett 9270e8b22b Rename client yaml test infrastructure
This makes it obvious that these tests are for running the client yaml
suites. Now that there are other ways of running tests using the REST
client against a running cluster we can't go on calling the shared
client yaml tests "REST tests". They are rest tests, but they aren't
**the** rest tests.
2016-07-26 13:53:44 -04:00
Lee Hinman e538c1c6d6 Merge remote-tracking branch 'dakrone/translog-cli' 2016-07-26 09:39:11 -06:00
Lee Hinman ac53c90ff4 Add 'elasticsearch-translog' CLI tool with 'translog' command
This adds the `bin/elasticsearch-translate` bin file that will be used
for CLI tasks pertaining to Elasticsearch. Currently it implements only
a single sub-command, `truncate-translog`, that creates a truncated
translog for a given folder.

Here's what running the tool looks like:

```
λ bin/elasticsearch-translog truncate -d data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/
Checking existing translog files
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!   WARNING: Elasticsearch MUST be stopped before running this tool   !
!                                                                     !
!   WARNING:    Documents inside of translog files will be lost       !
!                                                                     !
!   WARNING:          The following files will be DELETED!            !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
--> data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-10.tlog
--> data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-18.tlog
--> data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-21.tlog
--> data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-12.ckp
--> data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-25.ckp
--> data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-29.tlog
--> data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-2.tlog
--> data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-5.tlog
--> data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-41.ckp
--> data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-6.ckp
--> data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-37.ckp
--> data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-24.ckp
--> data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-11.ckp

Continue and DELETE files? [y/N] y
Reading translog UUID information from Lucene commit from shard at [data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/index]
Translog Generation: 3
Translog UUID      : AxqC4rocTC6e0fwsljAh-Q
Removing existing translog files
Creating new empty checkpoint at [data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog.ckp]
Creating new empty translog at [data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-3.tlog]
Done.
```

It also includes a `-b` batch operation that can be used to skip the
confirmation diaglog.

Resolves #19123
2016-07-26 08:34:07 -06:00
Nik Everett a95d4f4ee7 Add Location header and improve REST testing
This adds a header that looks like `Location: /test/test/1` to the
response for the index/create/update API. The requirement for the header
comes from https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

https://tools.ietf.org/html/rfc7231#section-7.1.2 claims that relative
URIs are OK. So we use an absolute path which should resolve to the
appropriate location.

Closes #19079

This makes large changes to our rest test infrastructure, allowing us
to write junit tests that test a running cluster via the rest client.
It does this by splitting ESRestTestCase into two classes:
* ESRestTestCase is the superclass of all tests that use the rest client
to interact with a running cluster.
* ESClientYamlSuiteTestCase is the superclass of all tests that use the
rest client to run the yaml tests. These tests are shared across all
official clients, thus the `ClientYamlSuite` part of the name.
2016-07-25 17:02:40 -04:00
Jason Tedor 2d1b0587dd Introduce Netty 4
This commit adds transport-netty4, a transport and HTTP implementation
based on Netty 4.

Relates #19526
2016-07-22 22:26:35 -04:00
Ryan Ernst e7b3bea8d3 Tweak and document gradle setup for adding transport modules to integ-test-zip 2016-07-12 12:13:52 -07:00
Simon Willnauer 4fb79707bd Fix remaining tests that either need access to the netty module or require explict configuration
Some tests still start http implicitly or miss configuring the transport clients correctly.
This commit fixes all remaining tests and adds a depdenceny to `transport-netty` from
`qa/smoke-test-http` and `modules/reindex` since they need an http server running on the nodes.

This also moves all required permissions for netty into it's module and out of core.
2016-07-12 16:29:57 +02:00
Simon Willnauer 048e4416e7 Move netty transport and http into a module
This moves all netty code and it's dependency into a module.
2016-07-11 22:21:29 +02:00
Jason Tedor d29d2f8793 Disable service in pre-uninstall
Today in the packaging removal scripts, we disable the service in
post-uninstall. Yet, this happens after service files have been
erased. On some systems, this can cause the service disable to fail
leaving behind state causing the service to be enabled on subsequent
installs. This commit moves the service disabling to the pre-uninstall
script to prevent this issue.

Relates #19328
2016-07-08 07:58:52 -04:00
Jason Tedor e86aa29f67 Die with dignity
Today when a thread encounters a fatal unrecoverable error that
threatens the stability of the JVM, Elasticsearch marches on. This
includes out of memory errors, stack overflow errors and other errors
that leave the JVM in a questionable state. Instead, the Elasticsearch
JVM should die when these errors are encountered. This commit causes
this to be the case.

Relates #19272
2016-07-07 14:44:03 -04:00
Alexander Reelsen 71b48fb16c Dependencies: Update to jopt-5.0 (#19278)
The new version of jopt allows us to remove a couple of TODOs in the code.

Closes #12368
2016-07-07 08:50:10 +02:00
Simon Willnauer a4ec0ac22f Upgrade to netty 3.10.6.Final (#19235) 2016-07-05 11:11:55 +02:00
Jason Tedor d6ddce2be8 Update SHA for JNA dependency
This commit updates the SHA for the JNA dependency. The JNA dependency
was upgraded from version 4.1.0 to version 4.2.2 in commit
7f10174362 but the SHA was not updated.
2016-06-23 11:43:18 -04:00
Adrien Grand 600cbb6ab0 Upgrade to Lucene 6.1.0. #18926 2016-06-17 09:03:00 +02:00
Adrien Grand 44c653f5a8 Upgrade to lucene-6.1.0-snapshot-3a57bea. 2016-06-10 16:18:12 +02:00
Jason Tedor 370b198f9f Remove explicit parallel new GC flag
Unless explicitly disabled, the parallel new collector is enabled
automatically as soon as the CMS collector is enabled:

  void Arguments::set_cms_and_parnew_gc_flags() {
    assert(
      !UseSerialGC && !UseParallelOldGC && !UseParallelGC,
      "Error");
    assert(UseConcMarkSweepGC, "CMS is expected to be on here");

    // If we are using CMS, we prefer to UseParNewGC,
    // unless explicitly forbidden.
    if (FLAG_IS_DEFAULT(UseParNewGC)) {
      FLAG_SET_ERGO(bool, UseParNewGC, true);
    }

While it's fine to be explicit, the UseParNewGC flag is deprecatd in JDK
8 and produces warning messages in JDK 9:

  Java HotSpot(TM) 64-Bit Server VM warning: Option UseParNewGC was
  deprecated in version 9.0 and will likely be removed in a future
  release.

Thus, we can and should just remove this flag from the default JVM
options.

Relates #18767
2016-06-07 12:05:38 -04:00
Stefan Scherer 92c6d78d4a Use java from path if JAVA_HOME is not set
This commit adds support to the Windows scripts for finding java.exe via
the path if JAVA_HOME is not set.

Relates #18685
2016-06-03 15:30:18 -04:00
Jason Tedor 8e2a7d0fe1 Rename boostrap.mlockall to bootstrap.memory_lock
The setting bootstrap.mlockall is useful on both POSIX-like systems
(POSIX mlockall) and Windows (Win32 VirtualLock). But mlockall is really
a POSIX only thing so the name should not be tied POSIX. This commit
renames the setting to "bootstrap.memory_lock".

Relates #18669
2016-06-01 16:25:51 -04:00
Adrien Grand d182e171a4 Upgrade to Lucene 6.0.1. 2016-06-01 10:31:10 +02:00
Tanguy Leroux 16307eee6f Remove extra bin/ directory in bin folder
Recent changes adds an extra bin/ directory that contains Windows bat files in the normal Bin folder of elasticsearch. For exemple the script elasticsearch.bat is located in bin/bin/elasticsearch.bat instead of bin/elasticsearch.bat.
2016-05-30 09:19:35 +02:00
Jason Tedor 41710f1028 Upgrade joda-time to 2.9.4
This commit upgrades joda-time to version 2.9.4 to integrate a bug fix
there into Elasticsearch.

Relates #18609
2016-05-27 08:51:19 -04:00
Ryan Ernst 4735b5a5fa Build: Use license, notice and readme files from root for distributions
The distributions had their own copies of these extra files, which was a
carry over from maven. This change removes the duplicate files and
copies them from the root of the project.

closes #18597
2016-05-26 09:19:38 -07:00
Ryan Ernst d505c3d5ba Build: Use dash for deb package version and underscore for rpm
This change tweaks the metadata version associated with the deb and rpm
packages. For rpm, dashes are allowed in versions, so we can use the
version as it already exists. For rpm, dashes are not allowed, so this
uses underscores instead. This only affects prerelease versions (eg
alpha/beta/rc), and usually someone doesn't specify the version, so the
inconsistency doesn't matter that much.
2016-05-24 12:11:13 -07:00
Lee Hinman 688254de41 Don't mkdir directly in deb init script
Elasticsearch will create the log and data path directories when it
starts, there is no need to do this in the init script

Resolves #18307
2016-05-20 15:52:16 -06:00
Jason Tedor 452faa220c Filter client/server VM options from jvm.options
The default jvm.options file ships with the -server flag to force the
server VM on systems where the server VM is not the default. However,
the method of starting the JVM via the Windows service does not support
the command-line flags for selecting the VM because it starts from a
DLL that is specific to the server or client VM. Thus, we need to
filter these options from the jvm.options configuration file when
installing the Windows service.

Relates #18473
2016-05-20 15:25:45 -04:00
Jason Tedor c257e2c51f Remove settings and system properties entanglement
Today when parsing settings during bootstrap, we add a system property
for every Elasticsearch setting. Additionally, settings can be set via
system properties. This commit simplifies this situation.
 - settings are no longer propogated to system properties
 - system properties can not be used to set settings
 - the "es." prefix on settings is no longer required (nor permitted)
 - test logging has a dedicated system property (tests.logger.level)

Relates #18198
2016-05-19 14:08:08 -04:00
Jason Tedor db4809d906 Remove last vestigates of /bin/sh shebangs
This commit removes the remaining /bin/sh shebangs in favor of
/bin/bash.

Relates #18448
2016-05-18 11:03:00 -04:00
Lee Hinman 4003d3f077 Merge remote-tracking branch 'dakrone/dont-mkdir-data-with-comma' 2016-05-17 13:56:59 -06:00
Lee Hinman ff5c7965ed Don't run `mkdir` when $DATA_DIR contains a comma-separated list
Resolves #16992
Resolves https://github.com/elastic/cookbook-elasticsearch/issues/441
2016-05-17 13:42:24 -06:00
Jason Tedor 10a881d860 Increase default heap size to 2g
Increase default heap size to 2g
2016-05-14 07:52:27 -04:00
Ryan Ernst 17de294460 Merge pull request #18344 from rjernst/run_distro
Build: Make run task you full zip distribution
2016-05-13 17:00:41 -07:00
Ryan Ernst d6f3d85925 Build: Make run task you full zip distribution
This change makes `gradle run` use the full zip distribution. Arguably
we could make it do the same when run inside plugins, but I started out
with this simple change. I am open to moving it into the RunTask itself
so that plugins will do this as well.
2016-05-13 16:36:26 -07:00
Ryan Ernst 5aea24137c Build: More pom generation improvements
This adds a non empty description to all generated poms, as well as
fixing distributions so they actually have a generated pom.
2016-05-13 16:32:35 -07:00
Jason Tedor bc46517f7e Restore unequal min and max default heap
After considerable discussion, we have elected to set the default min
heap to 256m and the default max heap to 2g. This is to balance the
desire for a good out-of-the-box performance experience (default max
heap of 2g) with a good out-of-the-box experience running on machines
with limited resources or running multiple instances on a single modern
developer laptop (default min heap of 1g).
2016-05-13 07:07:56 -04:00
Jason Tedor 52b09bc803 Increase default heap size to 2g
Benchmarking with the geonames data set shows that Elasticsearch truly
needs 2g of heap or the heap is a bottleneck for indexing rates. If our
goal is to satisfy the out-of-the-box-experience, we should prioritize
the out-of-the-box performance experience. Thus, the default heap should
be 2g until smaller heaps are not the bottleneck for indexing small data
sets.
2016-05-12 21:04:31 -04:00
Jason Tedor f9667dbe33 Switch init.d scripts to use bash
This commit modifies the init.d scripts to use bash now that bash is a
required dependency.

Relates #18308
2016-05-12 18:13:04 -04:00
Jason Tedor 8adfd60079 Fix Debian package bash requires clause
This commit fixes the Debian package requires clause for bash. For the
RPM it is okay to specify the binary, but for the Debian package the
package name must be specified.
2016-05-11 20:05:09 -04:00
Jason Tedor e4edee5d9a Equal default min and max heap settings
Today we encourage users to set their minimum and maximum heap settings
equal to each other to prevent the heap from resizing. Yet, the default
heap settings do not start Elasticsearch in this fashion. This commit
addresses this discrepancy by setting the default min heap to '512m' and
the default max heap to the default min heap.

Relates #16334
2016-05-11 10:48:08 -04:00
Jason Tedor 2bf585e642 Require /bin/bash in packaging
This commit adds a hard requirement to the RPM and Debian packages for
/bin/bash to be present, and adds a note regarding this to the migration
docs.

Relates #18259
2016-05-10 21:17:09 -04:00
Jason Tedor 1dfbe8b0f1 Fix handling of default path conf in plugin script
In the plugin script we set the setting default.path.conf to control the
path to the configuration file. But the default directory might not
exist in which case we can run into exceptions. This commit restores a
guard against this directory not existing.
2016-05-10 15:32:23 -04:00
Jason Tedor ef6a7631cf Switch scripts to use bash
This commit switches the command-line scripts to use bash instead of sh
so that we can take advantage of features that bash provides like
arrays.

Relates #18251
2016-05-10 15:06:51 -04:00
Jason Tedor 557fc8b437 Further simplifications of plugin script
In 7d1fd17172 the parsing of command-line
properties in the plugin script was removed. That commit missed
additional parsing of the properties for es.default.path.conf. This
commit removes that parsing and also replaces the use of eval with exec.

Relates #18239
2016-05-10 10:08:58 -04:00
Jason Tedor 7d1fd17172 Remove plugin script parsing of system properties
The plugin script parses command-line options looking for Java system
properties and extracts these arguments to pass to the java command when
starting the JVM. Since elasticsearch-plugin allows arbitrary user
arguments to the JVM via ES_JAVA_OPTS, this parsing is unnecessary. This
commit removes this unnecessary 

Relates #18207
2016-05-09 13:06:18 -04:00
Ryan Ernst 3d1be071c9 Merge branch 'master' into pom_gen 2016-05-06 12:56:51 -07:00
Jason Tedor 0eaa831f48 Preserve config files from RPM install
This commit modifies the packaging for the RPM package so that edits to
config files will not get lost during removal and upgrade.

Relates #18188
2016-05-06 13:24:54 -04:00
Ryan Ernst e16af604bf Build: Add pom generation to assemble task
In preparation for a unified release process, we need to be able to
generate the pom files independently of trying to actually publish. This
change adds back the maven-publish plugin just for that purpose. The
nexus plugin still exists for now, so that we do not break snapshots,
but that can be removed at a later time once snapshots are happenign
through the unified tools. Note I also changed the dir jars are written
into so that all our artifacts are under build/distributions.
2016-05-05 17:57:44 -07:00
Radovan Ondas 443976d03c Fix typo in message for variable setup ES_MAX_MEM
Small typo fix in startup script.

Relates #18168
2016-05-05 19:17:31 -04:00
Jason Tedor f1fb6a37c0 Fix error message on ES_MAX_MEM set
This commit fixes a typo in the error message when the now unsupported
environment variable ES_MAX_MEM is set on Windows.
2016-05-05 19:16:57 -04:00
Ryan Ernst 8ff699e41a Packaging: Make rpm not include parent dirs
closes #18162
2016-05-05 14:22:14 -07:00
Jason Tedor e11b96ca9c Default to server VM and add client VM check
Today we softly warn about running with the client VM. However, we
should really refuse to start in production mode if running with the
client VM as the performance of the client VM is too devastating for a
server application. This commit adds an option to jvm.options to ensure
that we are starting with the server VM (on all 32-bit non-Windows
platforms on server-class machines (2+ CPUs, 2+ GB physical RAM) this is
the default and on all 64-bit platforms this is the only option) and
adds a bootstrap check for the client VM.

Relates #18155
2016-05-05 10:36:21 -04:00
Jason Tedor 124e8e5a6d Remove client option for JVM for plugins script
Today we specify the client option for the JVM when executing plugin
commands. Yet, this option does nothing on a 64-bit capable JDK as such
JDKs always select the Hotspot server VM. And for 32-bit JDKs, running
plugin commands with the server VM is okay. Thus, we should just remove
this unnecessary flag and just let the default VM be selected.

Relates #18142
2016-05-04 14:11:29 -04:00
Jason Tedor 4c6cf7ee88 Pass ES_JAVA_OPTS to JVM for plugins script
This commit adds support for ES_JAVA_OPTS to the elasticsearch-plugin
script.

Relates #18140
2016-05-04 12:48:39 -04:00
Alexander Reelsen 9cadc1f40b Packaging: Fix vagrant tests on centos 6 (#17979)
The RPM init script did not include the check for `bin/elasticsearch` being
executable. This fix adds this checks and makes the tests pass.

This fixes the init script, so the tests pass on centos-6 again.
2016-04-26 11:55:50 +02:00
Jason Tedor 8a5032fae5 Remove unnecessary sleep from init script restart
Today when restarting Elasticsearch using the start-stop-daemon on
Debian-based systems using System V init, we sleep for one second
between the process successfully stopping and starting the process
again. This sleep is unnecessary as the stop function retries forever
until the previous instance successfully terminates. This commit removes
that unncessary sleep.

Relates #17966
2016-04-25 22:50:18 -04:00
Lee Hinman 9da88a99da Fix exit code
Exit with proper exit code (1) and an error message if elasticsearch
executable binary does not exists or has insufficient permissions to
execute.

Squashed commit of the following:

commit 9768d316303418ba4f9c96d3f87c376048a1b1bc
Author: Puru <tuladharpuru@gmail.com>
Date:   Fri Apr 22 23:26:47 2016 +0545

    Fixed ES_HOME typo

commit 79a2b0394297f8b02b6f71b71ba35ff79f1a684e
Author: Puru <tuladharpuru@gmail.com>
Date:   Sun Apr 10 11:00:24 2016 +0545

    Improve elasticsearch startup script test

    Added improvement as per conversation in https://github.com/elastic/elasticsearch/pull/17082#issuecomment-206459613

commit 7be38e1fefd4baa6ccdbdc14745c00f6dc052e0c
Author: Puru <tuladharpuru@gmail.com>
Date:   Wed Mar 23 13:23:52 2016 +0545

    Add elasticsearch startup script test

    The test ensures that elasticsearch startup script exists and is executable.

commit d10eed5c08260fa9c158a4487bbb3103a8d867ed
Author: Puru <tuladharpuru@gmail.com>
Date:   Wed Mar 23 12:30:25 2016 +0545

    Fixed IF syntax and failure message

commit 6dc66f616545572485b4d43bee05a4cbbf1bed72
Author: Puru <tuladharpuru@gmail.com>
Date:   Sat Mar 12 11:08:11 2016 +0545

    Fix exit code

    Exit with proper exit code (1) and an error message if elasticsearch executable binary does not exists or has insufficient permissions to execute.
2016-04-22 11:48:05 -06:00
Lee Hinman 4fca5f734a Explicitly set packaging permissions
This changes our packaging to be explicit about the permissions of files
and directories in the tar.gz, rpm, and deb packages. This is to protect
against a user having an incorrectly set umask when installing.

Additionally, plugins that are installed now have their permissions set
by the plugin installation so that plugins that may have been packaged
with incorrect permissions are secured.

Resolves #17634
2016-04-21 12:30:56 -06:00
Lee Hinman b8899cdb78 Merge remote-tracking branch 'dakrone/allow-bad-json' 2016-04-19 10:02:53 -06:00
Lee Hinman a1e8fb794c Allow JSON with unquoted field names by enabling system property
In Elasticsearch 5.0.0, by default unquoted field names in JSON will be
rejected. This can cause issues, however, for documents that were
already indexed with unquoted field names. To alleviate this, a system
property has been added that can be enabled so migration can occur.

This system property will be removed in Elasticsearch 6.0.0

Resolves #17674
2016-04-19 09:14:13 -06:00
Danilo Vaz 2e2d8c1442 Updated copyright years to include 2016 (#17808) 2016-04-18 12:39:23 +02:00
Alexander Reelsen 95579ca95a Build: Allow for file based deploy, sign packages
This allows for a local file based deploy without needed nexus
auth information.

Also signing of packages has been added, either via gradle.properties
or using system properties as a fallback.

The property build.repository allows to configure another endpoint if no
snapshot build is done.

Fix creation of .asc file for tar.gz distribution

Closes #17405
2016-04-15 17:14:05 +02:00
Jason Tedor 7297580170 Simplify JVM options
This commit simplifies the default JVM options that ship with
Elasticsearch. In particular, expert settings that were previously
configurable via environment variables have been removed from the
default configuration file. Further, the heap size settings have been
moved to the top of the file with a clearer message that is in
concordance with their importance.

Closes #17714
2016-04-13 17:59:59 -04:00
Jason Tedor f4ccf9bd4f Set variable in Windows scripts to default value
This commit sets the bad_env_var to a default value of 0 so that in case
no bad environment variables are encountered the variable is still
defined and does not later cause a parse error in the execution of these
batch files.

Relates #17675
2016-04-13 12:17:36 -04:00
Nik Everett bf2483e4b3 Setup jvm opts correctly for deb tests
Also stops warning about JAVA_OPTS when it is an empty string.
2016-04-13 11:18:19 -04:00
Jason Tedor a581d7cca4 Merge pull request #17675 from jasontedor/java-opts
Add JVM options configuration file
2016-04-12 23:07:40 -04:00
Jason Tedor 5662a14d22 Rewrite if-statement for shell compatibility
This commit rewrites an if-statement in bin/elasticsearch for
compatibility with some shells (e.g., dash on Ubuntu).
2016-04-12 22:27:08 -04:00
Jason Tedor 27867c729d Additional handling of ES_GC_LOG_FILE
This commit adds handling of the old ES_GC_LOG_FILE environment variable
to treat it like the other removed environment variables.
2016-04-12 22:26:43 -04:00
Jason Tedor bdfa063946 Add migration messages when rejecting env. vars
This commit adds migration messages to the startup scripts when
rejecting the old unsupported environment variables.
2016-04-12 15:32:31 -04:00
Jason Tedor d4cf0f021a Fail if unsupported environment variables are set
This commit adds a hard failure on startup if any unsupported
environment variables from older versions of Elasticsearch are still
set.
2016-04-12 13:29:09 -04:00
Jason Tedor 2b1c191b12 Add comments to JVM options parsing for Windows
This commit adds some comments to the portion of bin/elasticsearch.bat
that is responsible for parsing the JVM options.
2016-04-12 12:37:15 -04:00
Jason Tedor 7743c422db Warn in bin/elasticsearch if JAVA_OPTS is set
JAVA_OPTS is not a built-in mechanism for passing options to the JVM but
many people think that it is. We do not respect JAVA_OPTS, but this
commit adds a warning if it is set in case the end-user thinks that it
will affect Elasticsearch.
2016-04-12 11:51:30 -04:00
Jason Tedor 9460289bee Rename function that parses JVM options
This commit renames the function that parses JVM options in
bin/elasticsearch from jvm_options to parse_jvm_options. The reason for
the rename is because a for-loop variable was shadowing the name of this
function and changing the function name further clarifies the purpose of
the function.
2016-04-12 11:42:06 -04:00
Jason Tedor 00033b3da3 Fix placement of comment in bin/elasticsearch
This commit fixes the placement of a comment in bin/elasticsearch. The
comment was made out of place by the addition of a function definition
but order is restored.
2016-04-12 11:39:33 -04:00
Jason Tedor 3879aa2a98 Add JVM options configuration file
This commit adds a new configuration file jvm.options to centralize and
simplify management of JVM options. This separates the configuration of
the JVM from the packaging scripts (bin/elasticsearch*, bin/service.bat,
and init.d/elasticsearch) simplifying end-user operational management of
custom JVM options.
2016-04-12 11:19:16 -04:00
Adrien Grand 496c7fbd84 Upgrade Lucene 6 Release
* upgrades numerics to new Point format
* updates geo api changes
  * adds GeoPointDistanceRangeQuery as XGeoPointDistanceRangeQuery
  * cuts over to ES GeoHashUtils
2016-04-11 16:50:04 -05:00
Daniil Penkin 696fa18c61 Support for configuring Elasticsearch service display name, description and user on Windows 2016-04-07 10:39:57 +10:00
Lee Hinman 6e44ddaf0e Merge branch 'pr/11338' 2016-04-06 15:06:30 -06:00
Lee Hinman 42745c8c53 Merge pull request #14596 from girirajsharma/ES-RPM
rpm uses non-portable `--system` flag to `useradd`
2016-04-06 14:46:08 -06:00
Jason Tedor cea3c3c92f Quote path to java binary
This commit quotes the variable that contains the path to the java
binary. Without these quotes, when the arguments to eval are evaluated
the existing quotes will be removed leading to unquoted use of the path
to the java binary. If this path contains spaces, evaluation will fail.
2016-04-03 23:23:25 -04:00
Clinton Gormley 410147c754 Added RPM metadata 2016-04-02 10:38:34 +02:00
Joe Hillenbrand 81b6326891 Set MAX_OPEN_FILES to 65536
Relates to #17430
2016-03-30 16:44:37 -07:00
Mpdreamz e8776799cc Service command still had positional start command
Which caused the service to fail to start
2016-03-29 23:25:03 +02:00
Jason Tedor a0634b2442 Merge pull request #17197 from jasontedor/chown
Elasticsearch ownership for data, logs, and configs
2016-03-24 21:21:11 -04:00
Jason Tedor d22b5977de Ownership for data, logs, and configs for packages
This commit ensures that the data, logs, and config directories have the
proper ownership after the packages are installed. Additionally, this
commit ensures that the configs in /etc/elasticsearch are preserved
after removal of the RPM package.
2016-03-24 20:56:02 -04:00
Simon Willnauer 63b754aa22 Remove positional parameter from bat file 2016-03-24 11:19:25 +01:00
Alexander Reelsen b2573858b6 Version: Set version to 5.0.0-alpha1
Change version, required a minor fix in the RPM building.
In case of a alpha/beta version, the release will contain alpha/beta
as the RPM version cannot contains dashes/tildes.
2016-03-24 08:36:08 +01:00
Alexander Reelsen 4ac4f3c8bc Build: Update ospackage gradle plugin
The older version did not support signing of the packages.
2016-03-21 15:38:01 +01:00
Ryan Ernst f71f0d6010 Revert "Build: Switch to maven-publish plugin"
This reverts commit a90a2b34fc.
2016-03-18 17:22:25 -07:00
Jason Tedor 96c6ac2d3f Merge pull request #17176 from jasontedor/its-not-easy-being-green
Vagrant tests should be green
2016-03-18 15:28:37 -04:00
Jason Tedor 9f73152940 Fix plugins permissions 2016-03-17 15:35:54 -04:00
Ryan Ernst 6af4c43c4f Merge pull request #17128 from rjernst/maven_publish
Build: Switch to maven-publish plugin
2016-03-17 11:53:50 -07:00
Yannick Welsch e32da555aa Fix Windows start script to pass parameters in last position
Relates to #15320
2016-03-16 13:43:32 +01:00
Jason Tedor d2db9cf95f Fix es.path.home on Windows 2016-03-16 04:46:25 -04:00
Ryan Ernst a90a2b34fc Build: Switch to maven-publish plugin
The build currently uses the old maven support in gradle. This commit
switches to use the newer maven-publish plugin. This will allow future
changes, for example, easily publishing to artifactory.

An additional part of this change makes publishing of build-tools part
of the normal publishing, instead of requiring a separate upload step
from within buildSrc. That also sets us up for a follow up to enable
precomit checks on the buildSrc code itself.
2016-03-15 19:16:37 -07:00
Jason Tedor 618441aea3 Merge pull request #17088 from jasontedor/simplify-bootstrap-settings
Bootstrap does not set system properties
2016-03-15 19:25:16 -04:00
Jason Tedor 4ee90db13d Remove path.home command-line setting 2016-03-15 16:29:29 -04:00
Jason Tedor 5994e91b08 Fix systemd pidfile setting
This commit fixes the pidfile setting on systems that used systemd. The
issue is that the pidfile can only be set via the command line arguments
-p or --pidfile, and is no longer settable via a setting.
2016-03-15 15:57:37 -04:00
Adrien Grand 5596e31068 Upgrade to lucene-6.0.0-f0aa4fc. #17075 2016-03-14 07:58:52 +01:00
Jason Tedor 8a05c2a2be Bootstrap does not set system properties
Today, certain bootstrap properties are set and read via system
properties. This action-at-distance way of managing these properties is
rather confusing, and completely unnecessary. But another problem exists
with setting these as system properties. Namely, these system properties
are interpreted as Elasticsearch settings, not all of which are
registered. This leads to Elasticsearch failing to startup if any of
these special properties are set. Instead, these properties should be
kept as local as possible, and passed around as method parameters where
needed. This eliminates the action-at-distance way of handling these
properties, and eliminates the need to register these non-setting
properties. This commit does exactly that.

Additionally, today we use the "-D" command line flag to set the
properties, but this is confusing because "-D" is a special flag to the
JVM for setting system properties. This creates confusion because some
"-D" properties should be passed via arguments to the JVM (so via
ES_JAVA_OPTS), and some should be passed as arguments to
Elasticsearch. This commit changes the "-D" flag for Elasticsearch
settings to "-E".
2016-03-13 20:09:15 -04:00
Jason Tedor 4353b2e024 Do not pass double-dash arguments on startup
This commit addresses an issue in the init scripts which are passing
invalid command line arguments to the startup script.

Closes #17087
2016-03-13 18:58:08 -04:00
Ryan Ernst 591fb8f028 Merge branch 'master' into cli-parsing 2016-03-11 10:45:05 -08:00
Nik Everett 48191a44ab Merge pull request #15173 from rhoml/enable_es_include_on_init
Enable es_include at init
2016-03-10 14:31:50 -05:00
Lee Hinman 22e716551b Add -XX+AlwaysPreTouch JVM flag
Enables the touching of all memory pages used by the JVM heap spaces
during initialization of the HotSpot VM, which commits all memory pages
at initialization time. By default, pages are committed only as they are
needed.
2016-03-10 10:11:32 -07:00
Ryan Ernst 80ae2b0002 Fix more licenses 2016-03-09 00:10:59 -08:00
Ryan Ernst cb607a8fae Remove commons-cli sha and add jopt-simple sha 2016-03-08 19:18:14 -08:00
Robert Muir 54018a5d37 upgrade to lucene 6.0.0-snapshot-bea235f
Closes #16964

Squashed commit of the following:

commit a23f9d2d29220991aa498214530753d7a5a148c6
Merge: eec9c4e 0b0a251
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Mar 7 04:12:02 2016 -0500

    Merge branch 'master' into lucene6

commit eec9c4e5cd11e9c3e0b426f04894bb2a6dae4f21
Merge: bc67205 675d940
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 13:45:00 2016 -0500

    Merge branch 'master' into lucene6

commit bc67205bdfe1526eae277ab7856fc050ecbdb7b2
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 09:56:31 2016 -0500

    fix test bug

commit a60723b007ff12d97b1810cef473bd7b553a0327
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Mar 4 15:35:35 2016 +0100

    Fix SimpleValidateQueryIT to put braces around boosted terms

commit ae3a49d7ba7ced448d2a5262e5d8ec98671a9090
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Mar 4 15:27:25 2016 +0100

    fix multimatchquery

commit ae23fdb88a8f6d3fb7ba60fd1aaf3fd72d899aa5
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Mar 4 15:20:49 2016 +0100

    Rewrite DecayFunctionScoreIT to be independent of the similarity used

    This test relied a lot on the term scoring and compared scores
    that are dependent on the similarity. This commit changes the base query
    to be a predictable constant score query.

commit 366c2d518c35d31251033f1b6f6a93f6e2ae327d
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Mar 4 14:06:14 2016 +0100

    Fix scoring in tests due to changes to idf calculation.

    Lucene 6 uses a different default similarity as well as a different
    way to calculate IDF. In contrast to older version lucene 6 uses docCount per field
    to calculate the IDF not the # of docs in the index to overcome the sparse field
    cases.

commit dac99fd64ac2fa71b8d8d106fe68825e574c49f8
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 08:21:57 2016 -0500

    don't hardcoded expected termquery score

commit 6e9f340ba49ab10eed512df86d52a121aa775b0f
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 08:04:45 2016 -0500

    suppress deprecation warning until migrated to points

commit 3ac8908424b3fdad44a90a4f7bdb3eff7efd077d
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 07:21:43 2016 -0500

    Remove invalid test: all commits have IDs, and its illegal to do this.

commit c12976288124ad1a26467e7e848fb810548e7eab
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 07:06:14 2016 -0500

    don't test with unsupported back compat

commit 18bbfe76128570bc70883bf91ff4c44c82d27817
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 07:02:18 2016 -0500

    remove now invalid lucene 4 backcompat test

commit 7e730e572886f0ef2d3faba712e4256216ff01ec
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 06:58:52 2016 -0500

    remove now invalid lucene 4 backwards test

commit 244d2ab6868ba5ac9e0bcde3c2833743751a25ec
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 06:47:23 2016 -0500

    use 6.0 codec

commit 5f64d4a431a6fdaa1234adca23f154c2a1de8284
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 06:43:08 2016 -0500

    compile, javadocs, forbidden-apis, etc

commit 1f273cd62a7fe9ca8f8944acbbfc5cbdd3d81ccb
Merge: cd33921 29e3443
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Mar 4 10:45:29 2016 +0100

    Merge branch 'master' into lucene6

commit cd33921ac742ef9fb351012eff35f3c7dbda7264
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:58:37 2016 -0500

    fix hunspell dictionary loading

commit c7fdbd837b01f7defe9cb1c24e2ec65604b0dc96
Merge: 4d4190f d8948ba
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:41:53 2016 -0500

    Merge branch 'master' into lucene6

commit 4d4190fd82601aaafac6b8254ccb3edf218faa34
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:39:14 2016 -0500

    remove nocommit

commit 77ca69e288b1a41aa9595c921ed166c272a00ea8
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:38:24 2016 -0500

    clean up numericutils vs legacynumericutils

commit a466d696fbaad04b647ffbc0857a9439b583d0bf
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:32:43 2016 -0500

    upgrade spatial4j

commit 5412c747a8cfe638bacedbc8233163cb75cc3dc5
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:19:28 2016 -0500

    move to 6.0.0-snapshot-8eada27

commit b32bfe924626b87e540692375ece09e7c2edb189
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 11:30:09 2016 +0100

    Fix some test compile errors.

commit 6ccde35e9840b03c68d1a2cd47c7923a06edf64a
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 11:25:51 2016 +0100

    Current Lucene version is 6.0.0.

commit f62e1015d931b4cc04c778298a8fa1ba65e97ad9
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 11:20:48 2016 +0100

    Fix compile errors in NGramTokenFilterFactory.

commit 6837c6eabf96075f743649da9b9b52dd39611c58
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 10:50:59 2016 +0100

    Fix the edge ngram tokenizer/filter.

commit ccd7f070de5efcdfbeb34b9555c65c4990bf1ba6
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 10:42:44 2016 +0100

    The missing value is now accessible through a getter.

commit bd3b77f9b28e5b05daa3d49683a9922a6baf2963
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 10:41:51 2016 +0100

    Remove IndexCacheableQuery.

commit 05f3091c347aeae80eeb16349ac51d2b53cf86f7
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 10:39:43 2016 +0100

    Fix compilation of function_score queries.

commit 81cda79a2431ac78f56b0cc5a5765387f662d801
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 10:35:02 2016 +0100

    Fix compile errors in BlendedTermQuery.

commit 70994ce8dd1eca0b995870974a38e20f26f96a7b
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 23:33:03 2016 -0500

    add bug ID

commit 29d4f1a71f36f646b5a6060bed3db019564a279d
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 21:02:32 2016 -0500

    easy .store changes

commit 5e1a1e6fd665fa455e88d3a8987362fad5f44bb1
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 20:47:24 2016 -0500

    cleanups mostly around boosting

commit 333a669ec6c305ada5645d13ed1da0e19ec1d053
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 20:27:56 2016 -0500

    more simple fixes

commit bd5cd98a1e089c866b6b4a5e159400b110140ce6
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 19:49:38 2016 -0500

    more easy fixes and removal of ancient cruft

commit a68f419ee47da5f9c9ce5b372f01d707e902474c
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 19:35:02 2016 -0500

    cutover numerics

commit 4ca5dc1fa47dd5892db00899032133318fff3116
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 18:34:18 2016 -0500

    fix some constants

commit 88710a17817086e477c6c021ec346d0534b7fb88
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 18:14:25 2016 -0500

    Add spatial-extras jar as a core dependency

commit c8cd6726583e5ce3f546ed355d4eca037164a30d
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 18:03:33 2016 -0500

    update to lucene 6 jars
2016-03-07 04:12:23 -05:00
Nik Everett 97c4dd3b92 Merge pull request #16718 from biolds/master
Fix waiting for pidfile
2016-02-29 15:02:29 -05:00
Jason Tedor 79d8e225c4 Remove implicit support for Cygwin
This commit makes explicit that we do not support Cygwin.

Closes #16871
2016-02-29 14:59:42 -05:00
Nik Everett ba5be0332d Remove optional logger wrappers
Removes all our logger wrappers except the wrapper for log4j1.2. If you
depend on Elasticsearch's jar in your application you'll need to declare
log4j 1.2 and/or some bridge to your favorite logger.

We did this to simplify our builds and code. No more commons-logging like
log implementation sniffing. No more optional dependency hacks in gradle.

We might one day want to use j.u.l instead of log4j. If we do want that
we can recover its wrapper by studying this commit. We didn't go directly
to j.u.l in this commit because that is a bigger change. Our logging
configuration is based on log4j1.2 and people are used to it. So it'd
be a much more fraught breaking change to do that conversion.
2016-02-26 16:41:07 -05:00
Lee Hinman 5d359f737f Merge branch 'pr/16801' 2016-02-26 13:22:14 -07:00
James Bertouch f7ef8390db Upgrade to Jackson 2.7.1 2016-02-26 12:09:41 -05:00
Nik Everett 74754952db Remove mention of mvn from RestIT and Security 2016-02-24 19:43:56 -08:00
Mike McCandless 5fffede2b0 Upgrade to Lucene 5.5.0 official release 2016-02-20 17:34:16 -05:00
Laurent Defert 6661c4c37f Fix waiting for pidfile
Closes #16717
2016-02-18 02:00:10 +01:00
Nicholas Knize 52ee4c7027 upgrade to lucene 5.5.0-snapshot-850c6c2 2016-02-11 14:28:50 -06:00
gmarz 7a969c0bbe Windows service: Use JAVA_HOME environment variable in registry
This allows for updating Java without having to re-install the service.

Closes #13521
2016-02-11 12:31:56 -05:00
Rhommel f759604783
Adds ES_INCLUDE on rpm 2016-02-11 13:29:10 +11:00
Jim Ferenczi 7d0181b5d4 Rename bin/plugin in bin/elasticsearch-plugin 2016-02-05 10:09:14 +01:00
Robert Muir d5dc05f69e Upgrade to lucene 5.5.0-snapshot-1725675 2016-02-02 22:53:39 -05:00
Ryan Ernst a052dfeb38 Plugins: Reduce complexity of plugin cli
The plugin cli currently is extremely lenient, allowing most errors to
simply be logged. This can lead to either corrupt installations (eg
partially installed plugins), or confused users.

This change rewrites the plugin cli to have almost no leniency.
Unfortunately it was not possible to remove all leniency, due in
particular to how config files are handled.

The following functionality was simplified:
* The format of the name argument to install a plugin is now an official
  plugin name, maven coordinates, or a URL.
* Checksum files are required, and only checked, for official plugins
  and maven plugins. Checksums are also only SHA1.
* Downloading no longer uses a separate thread, and no longer has a timeout.
* Installation, and removal, attempts to be atomic. This only truly works
  when no config or bin files exist.
* config and bin directories are verified before copying is attempted.
* Permissions and user/group are no longer set on config and bin files.
  We rely on the users umask.
* config and bin directories must only contain files, no subdirectories.
* The code is reorganized so each command is a separate class. These
  classes already existed, but were embedded in the plugin cli class, as
  an extra layer between the cli code and the code running for each command.
2016-01-31 20:30:45 -08:00
Jason Tedor 26e5cd651e Default standard output to the journal in systemd
This commit modifies the default setting for standard output in the
systemd configuration to the journal instead of /dev/null. This is to
address a user pain point where Elasticsearch would fail to start but
the error message would be sent to standard output and therefore
/dev/null leading to difficult-to-debug situations.
2016-01-21 15:09:57 -05:00
Greg Marzouka 8fffc474dd Merge pull request #15549 from gmarz/master
Fix Windows service installation failure
2016-01-21 09:33:47 -05:00
Robert Muir 6e7e3a2274 Update lucene to r1725675
Adds DFI (divergence from independence) provider.
Fixes test bugs passing invalid values for BM25 parameters.
2016-01-20 03:32:51 -05:00
David Roberts febc2f08d6 Use egrep instead of grep -E for Solaris
Closes #15628
2016-01-04 17:07:43 +00:00
Adrien Grand cf52e96c42 Upgrade to lucene-5.5.0-snapshot-1721183.
Some files that implement or use the Scorer API had to be changed because of
https://issues.apache.org/jira/browse/LUCENE-6919.
2015-12-21 17:02:08 +01:00
Ryan Ernst 4ea19995cf Remove wildcard imports 2015-12-18 12:43:47 -08:00
gmarz 8004827522 Fix Windows service installation failure when ES_HOME contains parantheses
Closes #15349
2015-12-18 15:04:54 -05:00
Ryan Ernst 634c39ccfd Fix breakage of distribution integ tests 2015-12-10 19:05:47 -08:00
Ryan Ernst e29c9b9dfd Change task for building modules directory in distributions to sync.
This handles when dependencies have changed, so old ones will be
removed.
2015-12-10 17:30:58 -08:00
Ryan Ernst 2758099996 Merge pull request #15382 from rjernst/integ_test_deps
Fix module/integ-test cluster shutdown task ordering
2015-12-10 17:20:11 -08:00
Ryan Ernst f78fc6901f Build: Fix module/integ-test cluster shutdown task ordering
Both modules and integ-test-zip have integration tests (the latter being
the base rest tests). We can currently get odd behavior where
integ-test-zip's integ test does not shutdown its cluster before running
mdoule integ tests (and it then tries to shutdown all those clusters at
once after modules integ tests have run).

The underlying issue can be attributed to a bug in gradle with how cross project
mustRunAfter work with finalizers. This change works around this bug by
setting up mustRunAfter on the shutdown task itself.
2015-12-10 16:22:16 -08:00
Robert Muir 2741888498 Remove RuntimePermission("accessDeclaredMembers")
Upgrades lucene to 5.5.0-1719088, randomizedtesting to 2.3.2, and securemock to 1.2
2015-12-10 14:26:55 -05:00
Robert Muir e454fadc22 Merge branch 'master' into shave_mustache 2015-12-10 07:58:24 -05:00
Robert Muir 7595c4a3c8 Improve network docs
This makes some minor improvements (does not fix all problems!)

It reorders unicast disco in elasticsearch.yml to be right after the network host,
for better locality.

It removes the warning (unreleased) about publish addresses, lets try to really discourage setting
that unless you need to (behind a proxy server). Most people should be fine with `network.host`

Finally it reorganizes the network docs page a bit:

We add a table of 4 "basic" settings at the very beginning:

* network.host
* discovery.zen.ping.unicast.hosts
* http.port
* transport.tcp.port

The first two being the most important, which addresses to bind and talk to, and the other two
being the port numbers.

The rest of the stuff I tried to simplify and reorder under "advanced" headers.

This is just a quick stab, I still think we need more effort into this thing, but we gotta start somewhere.
2015-12-10 12:01:18 +01:00
Robert Muir 17436d4332 get gradle check passing for lang-mustache 2015-12-09 00:10:37 -05:00
Ryan Ernst 3d9d8bd45a Build: Remove duplicate runs of packaged rest tests
We currently use the full suite of packaged rest tests for each
distribution. We also used to run rest tests within core integ tests,
but this stopped working when we split out the test-framework, since the
test files are in there.

This change simplifies the code to run packaged rest tests just once,
for the integ-test-zip, and removes the unused rest tests from
test-framework. Distributions rest tests now check that all modules
were loaded.
2015-12-04 13:43:40 -08:00
Ryan Ernst 2dbd93d3b0 Tweaked module error message against non integ-test-zip and made checks
for limiting configuration to certain distributions prettier
2015-12-04 11:40:08 -08:00
Ryan Ernst 01d48e2062 Merge branch 'master' into jigsaw 2015-12-04 11:29:49 -08:00
Ryan Ernst f5b6b40a92 Fix distros to put modules dir int he right place and qa rest tests to
use the real zip
2015-12-03 23:03:14 -08:00
Ryan Ernst afedd24877 force integ tests for modules to run before distribution integ tests 2015-12-03 22:07:22 -08:00
Ryan Ernst caf77f7eea Add integ-test-zip distribution 2015-12-03 14:52:51 -08:00
Ryan Ernst 0a4a81afaf Added modules, distributions now include them (just plugins installed in
a diff dir)
2015-12-03 14:18:26 -08:00
Ryan Ernst b00c59398b Build: Cleanup test classes in buildSrc to be less groovyish 2015-12-03 01:49:54 -08:00
Robert Muir b7f497627f Merge pull request #15172 from rmuir/nuke_unused_deps
remove unused core dependencies
2015-12-02 07:20:11 -05:00
Ryan Ernst d68c6673a2 Build: Cleanup precommit task gradle code
This change attempts to simplify the gradle tasks for precommit. One
major part of that is using a "less groovy style", as well as being more
consistent about how tasks are created and where they are configured. It
also allows the things creating the tasks to set up inter task
dependencies, instead of assuming them (ie decoupling from tasks
eleswhere in the build).
2015-12-01 22:36:54 -08:00
Rhommel Lamas f623c88d99 Enable ES_INCLUDE 2015-12-02 16:37:30 +11:00
Robert Muir 6e5c301739 remove unused dependencies 2015-12-01 22:22:11 -05:00
Nik Everett 45e71329ee [build] Fix deb and rpm tests
Gradle is causing so much trouble here! Its too cute for its own good.
2015-11-28 19:07:06 -05:00
Ryan Ernst 7788c4cd69 Build: Fix rpm and deb generated poms to have correct artifactId
For all our distributions, we use "elasticsearch" as the artifactId.
This fixes the id for the generated poms of rpm and deb distributions.
2015-11-28 11:30:49 -08:00
Nik Everett efe27df096 Disable rpm and deb integ tests
They are broken and we didn't know it.
2015-11-25 12:35:30 -05:00
Michael McCandless e13b0d4bde upgrade lucene to 5.4.0-snapshot-1715952 2015-11-23 17:13:49 -05:00
Ryan Ernst b8e462b934 Build: Change project attachment into special extra-plugins dir
Currently we use the "gradle project attachment plugin" to support
building elasticsearch as part of another project. However, this plugin
has a number of issues, a large part of which is requiring consistent
use of the projectsPrefix.

This change removes projectsPrefix, and adds support for a special
extra-plugins directory in the root of elasticsearch. Any projects
checked out within this directory will be automatically added to
elasticsearch.
2015-11-22 08:44:33 -08:00
Nik Everett 452d85e582 [build] Skip package tests on windows
They aren't going to work because the packages don't include the windows
files.
2015-11-20 14:54:30 -05:00
Nik Everett 9b0a47d8e3 Fix rpm and deb distributions
and test them with vagrant
2015-11-18 14:16:42 -05:00
Ryan Ernst 50b48e7d8c Build: Make run command work within plugins
We recently got a run command with gradle, but it is sometimes useful to
run ES with a specific plugin. This is a start, by making each esplugin
have a run command which installs the plugin and runs elasticsearch in
the foreground.
2015-11-16 17:29:01 -08:00
Michael McCandless a0bf253d16 upgrade lucene 5.4 snapshot 2015-11-16 14:38:05 -05:00
Michael McCandless 9d7ca53022 upgrade lucene 5.4 snapshot 2015-11-16 14:35:17 -05:00
Adrien Grand d0853e9253 Merge pull request #14619 from jpountz/upgrade/lucene-5.4.0-snapshot-1712973
Upgrade to lucene-5.4.0-snapshot-1712973.
2015-11-09 22:38:44 +01:00
Adrien Grand d6d7af0a6c Upgrade to lucene-5.4.0-snapshot-1712973. 2015-11-09 15:53:27 +01:00
andrejserafim 2bd353d72d replacing run.bat and run.sh with gradle run
run.sh and run.bat were calling out to the old maven build system.
This is no longer in place, so we've created new gradle tasks to
start an elasticsearch node from the current codebase.

fixed #14423
2015-11-08 17:07:19 +00:00
girirajsharma 7ac3fb2c0c rpm uses non-portable `--system` flag to `useradd` 2015-11-07 11:35:37 +05:30
Ryan Ernst 95f885e11f Build: Change tar extension back to tar.gz instead of tgz
Gradle defaults to tgz extension when tar is compressed. This changes
the tar distribution back to tar.gz. Note that this also means the maven
packaging type is now tar.gz.
2015-11-03 11:24:27 -08:00
Ryan Ernst 3e4f87d4d6 Fix tar issues and build distro dependency for integ tests 2015-11-02 22:19:29 -08:00
Ryan Ernst f5556224f5 Build: Add back integ tests to distributions
rpm and deb are still skipped, but this configures rest tests to run for
zip and tgz distributions

closes #14361
2015-11-02 17:51:08 -08:00
Ryan Ernst 0dca49fae4 fix eclipse config for qa projects 2015-10-30 21:32:16 -07:00
Ryan Ernst b6dee6bd43 Merge pull request #14375 from rjernst/sweep_up_maven
Remove maven pom files and supporting ant files
2015-10-30 18:59:11 -07:00
Areek Zillur 13b60e1b92 update to lucene-5.4.x-snapshot-1711508 2015-10-30 15:42:02 -04:00
Clinton Gormley 551d255295 Merge pull request #14351 from jmferrer/master
typo error: adress -> address
2015-10-30 10:44:26 +01:00
Ryan Ernst 542522531a Build: Remove maven pom files and supporting ant files
This change removes the leftover pom files. A couple files were left for
reference, namely in qa tests that have not yet been migrated (vagrant
and multinode). The deb and rpm assemblies also still exist for
reference when finishing their setup in gradle.

See #13930
2015-10-29 23:53:49 -07:00
Ryan Ernst c86100f636 Switch build system to Gradle
See #13930
2015-10-29 11:40:19 -07:00
Robert Muir 1194cd0bdc Add threadgroup isolation.
Closes #14353

Squashed commit of the following:

commit edae0729f71ea3d3f9fa9c0d27c9effc042eb5a9
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Oct 29 14:13:42 2015 -0400

    update sha1 and simplify test

commit 635c4f245d66ad353a16267c810e02b725553fad
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Oct 29 07:01:26 2015 -0400

    Add threadgroup isolation.

    Code with `modifyThread` and `modifyThreadGroup` may only modify
    its own threadgroup (or an ancestor of that). This enforces
    what is intended by the ThreadGroup class.

    This has two immediate implications:
    1. Code without these permissions (scripts) may not create or mess with threads
    2. ES application threads cannot mess with Java system threads

    ES puts all application threads in one single group today, but in the future
    this can be organized better, and we will have more isolation in the system.
2015-10-29 14:14:27 -04:00
jmferrer 8880474f11 typo error: adress -> address 2015-10-29 11:00:21 +01:00
Adrien Grand 43958db10b Upgrade to lucene-5.4-snapshot-1710880. 2015-10-28 09:34:54 +01:00
Nicholas Knize 3d970f17f9 Fix Multi-geometry bbox
A long time coming this Upgrades to Spatial4J 0.5 which includes the fix for calculating a Multi-geometry bounding box.
2015-10-27 21:34:07 -05:00
Jason Tedor f1694e6663 Startup script exit status should catch daemonized startup failures
This commit fixes an issue where when starting Elasticsearch in
daemonized mode, a failed startup would not cause a non-zero exit code
to be returned. This can prevent the SysV init system from detecting
startup failures.

Closes #14163
2015-10-16 17:04:37 -04:00
Adrien Grand 5ae810991c Upgrade to lucene-5.4-snapshot-1708254. 2015-10-16 09:41:36 +02:00
Simon Willnauer f4eab500aa Upgrade Netty to 3.10.5.final
This fixes a critical bug in SSL handling.

See http://netty.io/news/2015/10/13/3-10-5-Final.html
2015-10-14 10:56:38 +02:00
Yannick Welsch 63366add7d Parse Java system properties in plugin.bat
Closes #13616
2015-10-13 13:33:34 +02:00
Jason Tedor 9a9a6a4b3b Remove Guava as a dependency
This commit removes Guava as a dependency. Note that Guava will remain
as a test-only dependency (transitively through Jimfs).

Closes #13224
2015-10-09 14:19:22 -04:00
Jason Tedor 81b9932ee8 Update mustache.java to version 0.9.1
This commit upgrades mustache.java to version 0.9.1. The primary motive
for this is because version 0.8.13 depends on Guava, but version 0.9.1
does not.

Relates #13224
2015-10-09 13:56:44 -04:00
javanna 648cc6defa Packaging: change permissions/ownership of config dir
When generating the rpm and dep package we now set proper group (elasticsearch) and permissions (750) to the conf dir (default /etc/elasticsearch). Same for the scripts subdirectory.

Expanded the assert_file bash function to also optionally check the group of files, so we can actually test that the group was set correctly.

Relates to #11016
Closes #14017
2015-10-08 15:40:12 +02:00
Britta Weber 71aefd5a06 remove option to configure custom config file via CONF_FILE or -Des.default.conf
It is rarely used and was not consistently handled by different distributions anyway.
This commit also adds a test for specifying CONF_DIR when installing plugins and
starting elasticsearch.

relates to #12712 and #12954
closes #5329
closes #13715
2015-10-06 19:02:43 +02:00
Robert Muir 8c4bc7d10b Nuke ES_CLASSPATH appending, JarHell fail on empty classpath elements
Closes #13880

Squashed commit of the following:

commit 316a328e5032e580ba840db993d907631334aac0
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Sep 30 16:57:47 2015 -0400

    windows is terrible

commit 0406b560c58bf833f8d77af9c7cf3386771dd9c5
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Sep 30 16:43:09 2015 -0400

    Nuke ES_CLASSPATH appending

    Out of box, ES expects its stuff to be in particular places. We should not be appending to ES_CLASSPATH, allowing users to specify stuff there, like we do in elasticsearch.bin.sh

    If the user sets it, its not going to work out of box.

    Closes #13812

commit 415d8972df28eddec322bb6d70100a1993fa95f6
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Sep 30 16:26:35 2015 -0400

    Fail hard on empty classpath elements.

    This can happen easily, if somehow old 1.x shellscripts survive and try to launch 2.x code.
    I have the feeling this happens maybe because of packaging upgrades or something.
    Either way: we can just fail hard and clear in this situation, rather than the current situation
    where CWD might be /, and we might traverse the entire filesystem until we hit an error...

    Relates to #13864
2015-09-30 18:59:27 -04:00
Robert Muir 20a203b1a2 Merge pull request #13834 from rmuir/groovy_factor_out
Factor groovy out of core into lang-groovy
2015-09-29 05:58:21 -04:00
Tanguy Leroux 9cb27ad068 Update to Jackson 2.6.2 2015-09-29 10:20:54 +02:00
Robert Muir e0d42739dd Factor groovy out of core into lang-groovy 2015-09-28 20:17:45 -04:00
Robert Muir c0d098b233 Don't let ubuntu try to install its crazy jayatana agent.
By default, our security stuff will reject this (as do other apps).
See https://bugs.launchpad.net/ubuntu/+source/jayatana/+bug/1441487

However its not really the user's fault, ubuntu screws up here by
installing this agent by default. We don't want any agents.

So instead, we drop it like this:

```
$ bin/elasticsearch
Warning: Ignoring JAVA_TOOL_OPTIONS=-Bogus1 -Bogus2
Please pass JVM parameters via JAVA_OPTS instead
[2015-09-25 23:34:39,777][INFO ][node                     ] [Doctor Bong] version[3.0.0-SNAPSHOT], pid[19044], build[2f5b6ea/2015-09-26T03:18:16Z]
...
```

Closes #13785
2015-09-25 23:32:52 -04:00
Robert Muir 689af1a6d6 Factor expressions scripts out to lang-expression plugin 2015-09-22 20:33:47 -04:00
Nik Everett 20d8e79801 Merge pull request #13514 from nik9000/dont_detect_java_7
Remove JAVA_HOME detection from the debian init script
2015-09-22 08:59:10 -04:00
Nik Everett 2a5bb4e648 [packaging] remove JAVA_HOME detection
Only the debian init script did JAVA_HOME detection. Everything else just
relied on `bin/elasticsearch`'s `which java` style detection. This strips
the detection from the debian init script so its like the rpm init script.

Closes #13403
2015-09-22 08:56:45 -04:00
André Carvalho 9ab6e01f4d Adds a validation for plugins script to check if java is set
This does the same validation as in elasticearch bin script
2015-09-17 10:34:36 -03:00
Robert Muir 01e6d8e3dc Remove java.lang.reflect.ReflectPermission "suppressAccessChecks"
Closes #13603

Squashed commit of the following:

commit 8799fb42d80297a79285beaf407b1bbecdb5854d
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Sep 16 03:32:29 2015 -0400

    Add randomizedtesting snapshot note

commit 0d874d9f0f5fddaeab8f48f9816a052dcaa691be
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Sep 16 03:11:01 2015 -0400

    Add a mechanism for insecure plugins and get all tests passing

commit 80540aeb9a264f6f299aaa3bc89df7f9b7923a60
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Sep 15 22:59:29 2015 -0400

    Really remove, we are killing this

commit 884818c1ad44ca2e7572a6998c086580be919657
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Sep 15 22:57:22 2015 -0400

    fill in TODOs

commit 34f4cb81f249edfec4d8d211da892f8c987e5948
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Sep 15 22:31:43 2015 -0400

    Publish snapshots of RR and lucene and cutover

commit d68eb9d66ce059761805c64d67e41a29098c9afa
Merge: f27e208 f62da59
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Sep 15 12:32:41 2015 -0400

    Merge branch 'master' into kill-setaccessible

commit f27e20855216dab6a6ad035d41018d8c67f3144c
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Sep 15 12:32:21 2015 -0400

    make a real lucene snapshot
2015-09-16 04:08:31 -04:00
Alexander Reelsen 1e209e3802 Plugins: Remove java opts/es java opts from plugin manager
... and run as client VM.

Reasoning: When calling the plugin manager on java 7 with additional JAVA_OPTS
that change heap configuration compared to what is set at the plugin
manager shell script. This resulted in errors.

This commit removes the JAVA_OPTS and ES_JAVA_OPTS from the plugin
manager call to prevent those settings.

Closes #12479
2015-09-15 17:04:42 +02:00
Robert Muir c1f2fc76c2 Upgrade lucene to r1702090
The semantics of the `boost` parameter for `function_score` changed. This is
due to the fact that Lucene now requires that query boosts and top-level boosts
are applied the same way.
2015-09-10 23:36:43 +02:00
gmarz 9e6115b066 Packaging: Fix Windows service start/stop issues
This commit addresses several bugs that prevented the Windows
service from being started or stopped:

- Extra white space in the concatenation of java options in
  elasticsearch.in.bat which tripped up Apache Commons Daemon
  and caused ES to startup without any params, eventually leading
  to the "path.home is not configured" exception.

- service.bat was not passing the start argument to ES

- The service could not be stopped gracefully via the stop command
  because there wasn't a method for procrun to call.

Closes #13247
Closes #13401
2015-09-09 13:31:46 -04:00
Robert Muir f216d92d19 Upgrade to lucene 5.4-snapshot r1701068 2015-09-03 15:13:33 -04:00
Robert Muir 11314336b6 bump master (3.0-snapshot) to java 8
Closes #13314

Squashed commit of the following:

commit 0e2d6c8e55e07957d0bea08f3c80f51f481ba6d4
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Sep 3 12:49:51 2015 -0400

    re-enable the rest of this test, as its one method that hangs

commit 04dc6b62d4800f574babcdbbf0e44f5c3400e0f0
Merge: 2b12805 52945b2
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Sep 3 12:48:55 2015 -0400

    Merge branch 'master' into java8

commit 2b128056ded741cddaf1d92c55920d7baf80dc95
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Sep 3 12:28:07 2015 -0400

    more jsr166e removal

commit 5826feba484389a87dbcad9a53ee5bf9a306eb61
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Sep 3 12:15:13 2015 -0400

    disable broken test

commit aabd88b85f2dd7b9b2b9308ffd52bc71aef8ab82
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Sep 3 12:11:12 2015 -0400

    Disable broken test

commit 863463303a12415dfecd6a46a6767eb07a2c7415
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Sep 3 11:38:32 2015 -0400

    bump master (3.0-snapshot) to java 8
2015-09-03 12:55:21 -04:00
Simon Willnauer 796701d52e Move version to 3.0.0-SNAPSHOT 2015-09-03 10:43:28 +02:00
Lee Hinman f25befda12 Merge remote-tracking branch 'dakrone/fix-Xloggc-args' 2015-09-02 22:04:51 -06:00
Lee Hinman 3b0b05f6cd Don't surround -Xloggc log filename with quotes
The `-Xloggc:filename.log` parameter has very strict filename semantics:

```
[A-Z][a-z][0-9]-_.%[p|t]
```

Our script specifies \" and \" to surround it, which makes Java think we
are sending: -Xloggc:"foo.log" and it fails with:

```
Invalid file name for use with -Xloggc: Filename can only contain the characters [A-Z][a-z][0-9]-_.%[p|t] but it has been "foo.log"
Note %p or %t can only be used once
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
```

We can't quote this, and we should not need to since the valid
characters don't include a space character, so we don't need to worry
about quoting.

Resolves #13277
2015-09-02 16:45:46 -06:00
David Pilato 34ee4c2d66 [build] remove shaded elasticsearch version
The shaded version of elasticsearch was built at the very beginning to avoid dependency conflicts in a specific case where:

* People use elasticsearch from Java
* People needs to embed elasticsearch jar within their own application (as it's today the only way to get a `TransportClient`)
* People also embed in their application another (most of the time older) version of dependency we are using for elasticsearch, such as: Guava, Joda, Jackson...

This conflict issue can be solved within the projects themselves by either upgrade the dependency version and use the one provided by elasticsearch or by shading elasticsearch project and relocating some conflicting packages.

Example
-------

As an example, let's say you want to use within your project `Joda 2.1` but elasticsearch `2.0.0-beta1` provides `Joda 2.8`.
Let's say you also want to run all that with shield plugin.

Create a new maven project or module with:

```xml
<groupId>fr.pilato.elasticsearch.test</groupId>
<artifactId>es-shaded</artifactId>
<version>1.0-SNAPSHOT</version>

<properties>
    <elasticsearch.version>2.0.0-beta1</elasticsearch.version>
</properties>

<dependencies>
    <dependency>
        <groupId>org.elasticsearch</groupId>
        <artifactId>elasticsearch</artifactId>
        <version>${elasticsearch.version}</version>
    </dependency>
    <dependency>
        <groupId>org.elasticsearch.plugin</groupId>
        <artifactId>shield</artifactId>
        <version>${elasticsearch.version}</version>
    </dependency>
</dependencies>
```

And now shade and relocate all packages which conflicts with your own application:

```xml
<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>2.4.1</version>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                    <configuration>
                        <relocations>
                            <relocation>
                                <pattern>org.joda</pattern>
                                <shadedPattern>fr.pilato.thirdparty.joda</shadedPattern>
                            </relocation>
                        </relocations>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>
```

You can create now a shaded version of elasticsearch + shield by running `mvn clean install`.

In your project, you can now depend on:

```xml
<dependency>
    <groupId>fr.pilato.elasticsearch.test</groupId>
    <artifactId>es-shaded</artifactId>
    <version>1.0-SNAPSHOT</version>
</dependency>
<dependency>
    <groupId>joda-time</groupId>
    <artifactId>joda-time</artifactId>
    <version>2.1</version>
</dependency>
```

Build then your TransportClient as usual:

```java
TransportClient client = TransportClient.builder()
        .settings(Settings.builder()
                        .put("path.home", ".")
                        .put("shield.user", "username:password")
                        .put("plugin.types", "org.elasticsearch.shield.ShieldPlugin")
        )
        .build();
client.addTransportAddress(new InetSocketTransportAddress(new InetSocketAddress("localhost", 9300)));

// Index some data
client.prepareIndex("test", "doc", "1").setSource("foo", "bar").setRefresh(true).get();
SearchResponse searchResponse = client.prepareSearch("test").get();
```

If you want to use your own version of Joda, then import for example `org.joda.time.DateTime`. If you want to access to the shaded version (not recommended though), import `fr.pilato.thirdparty.joda.time.DateTime`.

You can run a simple test to make sure that both classes can live together within the same JVM:

```java
CodeSource codeSource = new org.joda.time.DateTime().getClass().getProtectionDomain().getCodeSource();
System.out.println("unshaded = " + codeSource);

codeSource = new fr.pilato.thirdparty.joda.time.DateTime().getClass().getProtectionDomain().getCodeSource();
System.out.println("shaded = " + codeSource);
```

It will print:

```
unshaded = (file:/path/to/joda-time-2.1.jar <no signer certificates>)
shaded = (file:/path/to/es-shaded-1.0-SNAPSHOT.jar <no signer certificates>)
```

This PR also removes fully-loaded module.

By the way, the project can now build with Maven 3.3.3 so we can relax a bit our maven policy.
2015-09-02 11:57:10 +02:00
Adrien Grand 5d9fb2e8a6 Upgrade to lucene-5.3.0.
From a user perspective, the main benefit from this upgrade is that the new
Lucene53Codec has disk-based norms. The elasticsearch directory has been fixed
to load these norms through mmap instead of nio.

Other changes include the removal of `max_thread_states`, the fact that
PhraseQuery and BooleanQuery are now immutable, and that deleted docs are now
applied on top of the Scorer API.

This change introduces a couple of `AwaitsFix`s but I don't think it should
hold us from merging.
2015-09-01 11:58:45 +02:00
Jason Tedor e9b6949407 More portable extraction of short hostname
This commit increases the portability of extracting the short hostname
on a Unix-like system.

Closes #13107
2015-08-25 14:29:16 -04:00
David Pilato 8b85a8ff73 Update joda-time to 2.8.2
- DateTimeZone data updated to version 2015f

- Fixed to handle JDK 8u60 [#288, #291]
  Without this fix, formatting a time-zone will print "+00:00" instead of "GMT" for the GMT time-zone

- DateTimeZone data updated to version 2015e
2015-08-25 10:51:24 +02:00
Nik Everett d94630d938 Fix importing into eclipse
Renaming the distribution's artifactIds to "elasticsearch" caused the eclipse
import process to balk. Fix those modules by moving their eclipse package to
"[groupId].[artifactId]".
2015-08-21 13:36:26 -07:00
Ryan Ernst 164efaecbe Networking: Move multicast discovery to a plugin
Multicast has known issues (see #12999 and #12993). This change moves
multicast into a plugin, and deprecates it in the docs.  It also allows
for plugging in multiple zen ping implementations.

closes #13019
2015-08-20 16:43:25 -07:00
Robert Muir e2ab62596f Default to unicast discovery, with default host list of 127.0.0.1, [::1]
Fix unicast discovery to work when a host has multiple addresses.
Ban dangerous methods in java.net with forbidden APIs.
Fix ipv6 bugs and formatting of network addresses everywhere.

Closes #12999
Closes #12993

Squashed commit of the following:

commit 6c1aa001d091c5cf25212a53dc701fb704337f1e
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 14:25:43 2015 -0400

    Fix these to be correct with addresses just in case

commit 648215627e84abf58a71400e7dc9ae775efb71d6
Merge: d00561b 41d8fbe
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 13:23:09 2015 -0400

    Merge branch 'master' into unicast_all_the_way_down

commit d00561b76fd1aa5850699f7901f3dae3d4d402b7
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 16:38:50 2015 +0200

    limit local ports to 5 in UnicastZenPing

commit e2e15c594006746cbe24432694294a71cc99deb8
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 10:32:47 2015 -0400

    fix port limiting

commit 10153cb7adadda81a1f482445e703836b65cf5e2
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 10:18:37 2015 -0400

    don't serialize scopeids: that's broken

commit 2aa63d43db2baec68a2e9bc227cfeb85dfeb4f83
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 16:06:51 2015 +0200

    restore @Network

commit c840f1d1ef438826ae1ecfd5e45942a0e30dc9c0
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 16:02:30 2015 +0200

     Use NetworkAddress.formatAddress where applicable in plugins

commit 374ce878852b35d626b7a29c8c4773545b0e9ddd
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 15:34:06 2015 +0200

    Use NetworkAddress.formatAddress where applicable

commit e7a606d63f1bc43c1b62b6e17adf707c76d43a15
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 10:17:57 2015 +0200

    Add @Multicast annotation to disable multicast tests by default.

    We only run multicast tests now when we explicitly state it. A working
    multicast env is required which is not always the case.

commit 2d7d2d0347179696ab41f71f048b13305014c85b
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 09:51:28 2015 +0200

    Remove extra check for local mode in InternalTestCluster

commit dda59ac39aa136d4687b9274c2692cd77f8b8f66
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 09:37:03 2015 +0200

    Handle node mode across entire test cluster

    We used static methods reading sys properties to define the node mode
    per cluster. this had lots of problems when tests couldn't cope with
    mixed or only local mode. Now we are passing it down to the cluster from the test
    which allows to @SuppressNetworkMode / @SupressLocalMode on the test to force
    consistent node configurations.

commit 058197b7a408318995c88ce7f6762e32348de0de
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 03:19:14 2015 -0400

    really ban InetSocketAddress's trappy method and break build and go to sleep, sorry

commit ac8779185aee1e17e6f5a81766290fdfc9c603ba
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 03:16:52 2015 -0400

    Ban methods that might surprisingly cause DNS lookups

commit e64fe3dff2b11503e5f2831eb9863d64f56c5538
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 02:59:05 2015 -0400

    Add unit test

commit f15434f20fb1a3691b1cc16028597d8fae937e05
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 02:39:02 2015 -0400

    fix ipv6 formatting bugs

commit 05c2c74098052c75fbb79ea1818a295ef2e03e30
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 02:12:05 2015 -0400

    format addresses correctly so I can actually read what comes out of our logs and stats apis

commit 4f9389dcf1e8925f23153c5eb271b4ce2294dbaf
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Aug 19 21:26:52 2015 -0400

    ban dangerous methods in java.net

commit 6aacd4d9925f324903d1d099a6cf5f862aeaf677
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Aug 19 20:59:24 2015 -0400

    ban lenient method

commit f466a842c60163d1f4554bdce8a4163edb534c2c
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 00:29:00 2015 +0200

    fix tests to not mix local transport and zen unicast disco

commit 0de007a33b33fb68cf85cd86db4ca4f8ce10bbc9
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 00:10:07 2015 +0200

    fix tests to not mix local transport and zen unicast disco

commit 539f6ca6e5137e0d496239adc8684688dedcc824
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 00:02:01 2015 +0200

    fix tests to not mix local transport and zen unicast disco

commit 004c2881b25467f332acc8c9f9e92b1f0f9d314e
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Aug 19 17:51:45 2015 -0400

    Fix multinode

commit 54113af325ce31571811c49fdaae89d5687be4ba
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Aug 19 17:36:45 2015 -0400

    fix integration tests

commit 0156a77a56319d6b9737ec6a531992052e50bd59
Author: Simon Willnauer <simonw@apache.org>
Date:   Wed Aug 19 23:32:18 2015 +0200

    enable multicast in MulticastZenPingIT.java

commit 1791caa35da853ce0122485fa3fd4674c671ec6e
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Aug 19 17:23:16 2015 -0400

    Fix constant

commit 22820b53e0b2dc9fd47145c2bc29ce912a8fd484
Author: Simon Willnauer <simonw@apache.org>
Date:   Wed Aug 19 22:59:09 2015 +0200

    give it some extra ids for local transport crazyness

commit b2138fafa94a8a085813fd48356df63e57ade5b3
Author: Simon Willnauer <simonw@apache.org>
Date:   Wed Aug 19 22:51:42 2015 +0200

    pass on local addresses from configured transport rather than hard code IP addresses

commit 1bf5de1f457b081e0ce262b57d2b55d39c434156
Author: Simon Willnauer <simonw@apache.org>
Date:   Wed Aug 19 22:04:31 2015 +0200

    fix PluggableTransportModuleIT.java to use local disco and detach port limit for node local disco

commit b6706eddfa04c43947c16551359ae98a463d34aa
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Aug 19 14:16:03 2015 -0400

    Default to unicast discovery, with default host list of 127.0.0.1, [::1]
2015-08-20 14:26:40 -04:00
David Pilato dba1b52e81 [build] rpm module should be build on machine with /usr/bin/rpmbuild
We build the rpm module automatically if you have either:

* `/usr/bin/rpmbuild`
* `/usr/local/bin/rpmbuild`

available.

If your `rpmbuild` is in another location and available in your path, then run maven with `rpm` profile:

```sh
mvn deploy -Prpm
```

Closes #12984.
2015-08-19 15:37:28 +02:00
Simon Willnauer 0ffd99cca3 Drop commons-lang dependency
commons-lang really is only used by some core classes to join strings or modiy arrays.
It's not worth carrying the dependency. This commit removes the dependency on commons-lang
entirely.
2015-08-18 22:59:31 +02:00
David Pilato 692cc80523 [maven] also rename parent project artifactId
Also fixed bad scm links
2015-08-18 13:41:12 +02:00
David Pilato b5eb78875f [maven] rename maven names / ids for distribution modules 2015-08-18 13:38:49 +02:00
Robert Muir 68307aa9f3 Fix network binding for ipv4/ipv6
When elasticsearch is configured by interface (or default: loopback interfaces),
bind to all addresses on the interface rather than an arbitrary one.

If the publish address is not specified, default it from the bound addresses
based on the following sort ordering:

* ipv4/ipv6 (java.net.preferIPv4Stack, defaults to true)
* ordinary addresses
* site-local addresses
* link local addresses
* loopback addresses

One one address is published, and multicast is still always over ipv4: these
need to be future improvements.

Closes #12906
Closes #12915

Squashed commit of the following:

commit 7e60833312f329a5749f9a256b9c1331a956d98f
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 14:45:33 2015 -0400

    fix java 7 compilation oops

commit c7b9f3a42058beb061b05c6dd67fd91477fd258a
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 14:24:16 2015 -0400

    Cleanup/fix logic around custom resolvers

commit bd7065f1936e14a29c9eb8fe4ecab0ce512ac08e
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 13:29:42 2015 -0400

    Add some unit tests for utility methods

commit 0faf71cb0ee9a45462d58af3d1bf214e8a79347c
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 12:11:48 2015 -0400

    localhost all the way down

commit e198bb2bc0d1673288b96e07e6e6ad842179978c
Merge: b55d092 b93a75f
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 12:05:02 2015 -0400

    Merge branch 'master' into network_cleanup

commit b55d092811d7832bae579c5586e171e9cc1ebe9d
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 12:03:03 2015 -0400

    fix docs, fix another bug in multicast (publish host = bad here!)

commit 88c462eb302b30a82585f95413927a5cbb7d54c4
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 11:50:49 2015 -0400

    remove nocommit

commit 89547d7b10d68b23d7f24362e1f4782f5e1ca03c
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 11:49:35 2015 -0400

    fix http too

commit 9b9413aca8a3f6397b5031831f910791b685e5be
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 11:06:02 2015 -0400

    Fix transport / interface code

    Next up: multicast and then http
2015-08-17 15:43:07 -04:00
Nik Everett a9b2b033e3 [Packaging] Fix variable substitution
Systemd looks to be a bit less tolerant about $VAR than bash is. Replace
$VAR with ${VAR} in places in the systemd configuration file to get the
substitutions working.
2015-08-15 08:53:43 -07:00
Simon Willnauer b447e2ae99 Move master to [2.1.0-SNAPSHOT] 2015-08-14 23:44:06 +02:00
Lee Hinman 6f5a25d98e [DOC] Use 127.0.0.1 instead of localhost in READMEs
Users with IPv6 preferred over IPv4 may have `localhost` resolve to
`::1` instead of `127.0.0.1`, so we should be explicit so they don't run
into issues.
2015-08-14 14:47:58 -06:00
David Pilato 4e23fe2657 [build] revert maven 3.3.x fix
We need to revert #12803 as it creates some troubles in IntelliJ and also might fail when using older elasticsearch versions.
2015-08-13 15:21:06 +02:00
David Pilato fbd8f69273 Build of QA: Smoke Test Shaded Jar fails under maven 3.3.x
Build fails with maven 3.3.1 and 3.3.3. To reproduce, install one of the 3.3.x versions of maven and run `mvn clean verify` in the root directory of the project. The build will fail in the QA: Smoke Test Shaded Jar module with the following error:

```
Started J0 PID(99979@flea.local).
Suite: org.elasticsearch.shaded.test.ShadedIT
  2> NOTE: reproduce with: ant test  -Dtestcase=ShadedIT -Dtests.method=testJodaIsNotOnTheCP -Dtests.seed=2F4D23A7462CF921 -Dtests.locale= -Dtests.timezone=Asia/Baku -Dtests.asserts=true -Dtests.file.encoding=UTF-8
FAILURE 0.06s | ShadedIT.testJodaIsNotOnTheCP <<<
  > Throwable #1: junit.framework.AssertionFailedError: Expected an exception but the test passed: java.lang.ClassNotFoundException
  > at __randomizedtesting.SeedInfo.seed([2F4D23A7462CF921:3A9404F1F69FD80]:0)
  > at junit.framework.Assert.fail(Assert.java:57)
  > at java.lang.Thread.run(Thread.java:745)
  2> NOTE: reproduce with: ant test  -Dtestcase=ShadedIT -Dtests.method=testGuavaIsNotOnTheCP -Dtests.seed=2F4D23A7462CF921 -Dtests.locale= -Dtests.timezone=Asia/Baku -Dtests.asserts=true -Dtests.file.encoding=UTF-8
FAILURE 0.01s | ShadedIT.testGuavaIsNotOnTheCP <<<
  > Throwable #1: junit.framework.AssertionFailedError: Expected an exception but the test passed: java.lang.ClassNotFoundException
  > at __randomizedtesting.SeedInfo.seed([2F4D23A7462CF921:C2502FD54D83433D]:0)
  > at junit.framework.Assert.fail(Assert.java:57)
  > at java.lang.Thread.run(Thread.java:745)
  2> NOTE: reproduce with: ant test  -Dtestcase=ShadedIT -Dtests.method=testjsr166eIsNotOnTheCP -Dtests.seed=2F4D23A7462CF921 -Dtests.locale= -Dtests.timezone=Asia/Baku -Dtests.asserts=true -Dtests.file.encoding=UTF-8
FAILURE 0.01s | ShadedIT.testjsr166eIsNotOnTheCP <<<
  > Throwable #1: junit.framework.AssertionFailedError: Expected an exception but the test passed: java.lang.ClassNotFoundException
  > at __randomizedtesting.SeedInfo.seed([2F4D23A7462CF921:35593286F4269392]:0)
  > at junit.framework.Assert.fail(Assert.java:57)
  > at java.lang.Thread.run(Thread.java:745)
  2> NOTE: leaving temporary files on disk at: /Users/Shared/Jenkins/Home/workspace/elasticsearch-master/qa/smoke-test-shaded/target/J0/temp/org.elasticsearch.shaded.test.ShadedIT_2F4D23A7462CF921-001
  2> NOTE: test params are: codec=CheapBastard, sim=DefaultSimilarity, locale=, timezone=Asia/Baku
  2> NOTE: Mac OS X 10.10.4 x86_64/Oracle Corporation 1.8.0_25 (64-bit)/cpus=8,threads=1,free=482137936,total=514850816
  2> NOTE: All tests run in this JVM: [ShadedIT]
Completed [1/1] in 6.61s, 5 tests, 3 failures <<< FAILURES!

Tests with failures:
  - org.elasticsearch.shaded.test.ShadedIT.testJodaIsNotOnTheCP
  - org.elasticsearch.shaded.test.ShadedIT.testGuavaIsNotOnTheCP
  - org.elasticsearch.shaded.test.ShadedIT.testjsr166eIsNotOnTheCP
```
Please note that build doesn't fail with maven 3.2.x and it doesn't fail if mvn command is executed inside the qa/smoke-test-shaded directory. Only when the build is started from the root directory the error above can be observed.

The reason is because of the shaded version which depends on elasticsearch core.
When Maven build the module only, then elasticsearch core is not added to the dependency tree.

```sh
mvn dependency:tree -pl :smoke-test-shaded
```

```
[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ smoke-test-shaded ---
[INFO] org.elasticsearch.qa:smoke-test-shaded:jar:2.0.0-beta1-SNAPSHOT
[INFO] +- org.elasticsearch.distribution.shaded:elasticsearch:jar:2.0.0-beta1-SNAPSHOT:compile
[INFO] |  +- org.apache.lucene:lucene-core:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-backward-codecs:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-analyzers-common:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-queries:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-memory:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-highlighter:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-queryparser:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-sandbox:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-suggest:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-misc:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-join:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-grouping:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-spatial:jar:5.2.1:compile
[INFO] |  \- com.spatial4j:spatial4j:jar:0.4.1:compile
[INFO] +- org.hamcrest:hamcrest-all:jar:1.3:test
[INFO] \- org.apache.lucene:lucene-test-framework:jar:5.2.1:test
[INFO]    +- org.apache.lucene:lucene-codecs:jar:5.2.1:test
[INFO]    +- com.carrotsearch.randomizedtesting:randomizedtesting-runner:jar:2.1.16:test
[INFO]    +- junit:junit:jar:4.11:test
[INFO]    \- org.apache.ant🐜jar:1.8.2:test
```

But if shaded plugin is involved during the build, it modifies the `projectArtifactMap`:

```sh
mvn dependency:tree -pl org.elasticsearch.distribution.shaded:elasticsearch,:smoke-test-shaded
```

```
[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ smoke-test-shaded ---
[INFO] org.elasticsearch.qa:smoke-test-shaded:jar:2.0.0-beta1-SNAPSHOT
[INFO] +- org.elasticsearch.distribution.shaded:elasticsearch:jar:2.0.0-beta1-SNAPSHOT:compile
[INFO] |  \- org.elasticsearch:elasticsearch:jar:2.0.0-beta1-SNAPSHOT:compile
[INFO] |     +- org.apache.lucene:lucene-backward-codecs:jar:5.2.1:compile
[INFO] |     +- org.apache.lucene:lucene-analyzers-common:jar:5.2.1:compile
[INFO] |     +- org.apache.lucene:lucene-queries:jar:5.2.1:compile
[INFO] |     +- org.apache.lucene:lucene-memory:jar:5.2.1:compile
[INFO] |     +- org.apache.lucene:lucene-highlighter:jar:5.2.1:compile
[INFO] |     +- org.apache.lucene:lucene-queryparser:jar:5.2.1:compile
[INFO] |     |  \- org.apache.lucene:lucene-sandbox:jar:5.2.1:compile
[INFO] |     +- org.apache.lucene:lucene-suggest:jar:5.2.1:compile
[INFO] |     |  \- org.apache.lucene:lucene-misc:jar:5.2.1:compile
[INFO] |     +- org.apache.lucene:lucene-join:jar:5.2.1:compile
[INFO] |     |  \- org.apache.lucene:lucene-grouping:jar:5.2.1:compile
[INFO] |     +- org.apache.lucene:lucene-spatial:jar:5.2.1:compile
[INFO] |     |  \- com.spatial4j:spatial4j:jar:0.4.1:compile
[INFO] |     +- com.google.guava:guava:jar:18.0:compile
[INFO] |     +- com.carrotsearch:hppc:jar:0.7.1:compile
[INFO] |     +- joda-time:joda-time:jar:2.8:compile
[INFO] |     +- org.joda:joda-convert:jar:1.2:compile
[INFO] |     +- com.fasterxml.jackson.core:jackson-core:jar:2.5.3:compile
[INFO] |     +- com.fasterxml.jackson.dataformat:jackson-dataformat-smile:jar:2.5.3:compile
[INFO] |     +- com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:jar:2.5.3:compile
[INFO] |     |  \- org.yaml:snakeyaml:jar:1.12:compile
[INFO] |     +- com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:jar:2.5.3:compile
[INFO] |     +- io.netty:netty:jar:3.10.3.Final:compile
[INFO] |     +- com.ning:compress-lzf:jar:1.0.2:compile
[INFO] |     +- com.tdunning:t-digest:jar:3.0:compile
[INFO] |     +- org.hdrhistogram:HdrHistogram:jar:2.1.6:compile
[INFO] |     +- org.apache.commons:commons-lang3:jar:3.3.2:compile
[INFO] |     +- commons-cli:commons-cli:jar:1.3.1:compile
[INFO] |     \- com.twitter:jsr166e:jar:1.1.0:compile
[INFO] +- org.hamcrest:hamcrest-all:jar:1.3:test
[INFO] \- org.apache.lucene:lucene-test-framework:jar:5.2.1:test
[INFO]    +- org.apache.lucene:lucene-codecs:jar:5.2.1:test
[INFO]    +- org.apache.lucene:lucene-core:jar:5.2.1:compile
[INFO]    +- com.carrotsearch.randomizedtesting:randomizedtesting-runner:jar:2.1.16:test
[INFO]    +- junit:junit:jar:4.11:test
[INFO]    \- org.apache.ant🐜jar:1.8.2:test
```

A fix could consist of fixing something on Maven side. Probably something changed in a recent version and introduced this "issue" but it might be not really an issue. More a fix.

There are two workarounds:

1) exclude manually elasticsearch core from shaded version in smoke-test-shaded module and add manually each lucene lib needed by elasticsearch

2) add a new `elasticsearch-lucene` (lucene) POM module which simply declares all needed lucene libs in subprojects (such as the smoke tester one).

I choose the later.

Closes #12791.
2015-08-13 12:15:19 +02:00
Simon Willnauer 605253a39f Cut over master to 2.0.0-SNAPSHOT 2015-08-12 21:16:08 +02:00
Robert Muir 0f61f56b60 use spaces liberally in integration tests and fix space handling
Closes #12710
2015-08-11 18:03:41 +02:00
Adrien Grand abe3880dd9 Merge pull request #12274 from smflorentino/smflorentino-fix-winjvm-detect
Improve JVM Arch Detection
2015-08-11 16:17:26 +02:00
Nik Everett 3c76e15aab Merge pull request #12785 from nik9000/remove_conf
Fix rpm -e removing /etc/elasticsearch
2015-08-11 05:08:07 -07:00
Nik Everett 7223f7c75a Packaging: fix rpm -e removing /etc/elasticsearch
When we fixed rpm creating the /etc/elasticsearch/scripts directory we
broke removing the rpm - it lef the /etc/elasticsearch directory behind.
This fixes that.
2015-08-10 14:32:41 -07:00
Ryan Ernst 40f119d85a This method on settings loaded a class, based on a setting value, using
the default classloader. It had all kinds of leniency in how the
classname was found, and simply cannot work with plugins having isolated
classloaders.

This change removes that method. Some of the uses of it were for custom
extension points, like custom repository or discovery types. A lot were
just there to plugin mock implementations for tests. For the settings
that were legitimate, all now support plugins adding the given setting
via onModule. For those that were specific to tests for mocks, they now
use Classes.loadClass (a helper around Class.forName). This is a
temporary measure until (in a future PR) tests can change the
implementation via package private statics.

I also removed a number of unnecessary intermediate modules, added a
"jvm-example" plugin that can be filled in in the future as a smoke test
for breaking plugins, and gave some documentation to "spawn" modules
interface.

closes #12643
closes #12656
2015-08-10 14:04:45 -07:00
Scott Florentino e7aa2a1fb2 Closing #12256 Update service.bat with a safer RAM value 2015-08-10 13:22:29 -07:00
Colin Goodheart-Smithe 6abc69c488 Packaging: Makes sure all POMs contain a description
Adds an explicit description the RPM package so it doesn't inherit the description from the POM.

Closes #12550

Also, modified descriptions for deb and rpm packages to be the same and to reference the documentation rather than listing features that are out of date.
2015-08-10 15:21:24 +01:00
Nik Everett cde2502732 Merge pull request #12706 from nik9000/osx_can_build_rpm
Let OSX build rpms for linux
2015-08-10 05:48:36 -04:00
Nik Everett f84552dc98 [Packaging] Run tests in vagrant
This creates a module in qa called vagrant that can be run if you have
vagrant and virtualbox installed and will run the packaging tests in trusty
and centos-7.0. You can ask it to run tests in other linuxes. This is the full
list:
* precise aka Ubuntu 12.04
* trusty aka Ubuntu 14.04
* vivid aka Ubuntun 15.04
* wheezy aka Debian 7, the current debian oldstable distribution
* jessie aka Debian 8, the current debina stable distribution
* centos-6
* centos-7
* fedora-22
* oel-7

There is lots of documentation on how to do this in the TESTING.asciidoc.

Closes #12611
2015-08-10 05:31:42 -04:00
Nik Everett d6087667bf Merge pull request #12704 from nik9000/create_scripts
rpm and deb create scripts directory
2015-08-10 05:27:45 -04:00
Tanguy Leroux b1fd0a61da Do not kill process on service shutdown
When installed as a service with a DEB or RPM package, we should gently wait for elasticsearch to stop (flushing indices on closing can take some time) and never kill the process.

Closes #11248
2015-08-10 10:04:56 +02:00
Clinton Gormley 205262c737 Enable the license checker over distribution/* and plugins/*
Moved the license checker config into the parent pom, and overrede
the license dir/target-to-check in distributions/pom.

Disabled the license checker explicitly for projects which run integration
tests but have no licenses dir:

* core
* distribution
* qa
* plugins/delete-by-query
* plugins/mapper-size
* plugins/site-example

Closes #12752
Closes #12754
2015-08-09 18:18:59 +02:00
Clinton Gormley 7efc419041 Revert "Revert "disable license checker on windows for now, jenkins boxes have perl problems""
This reverts commit e2cbec185b.
2015-08-07 16:47:52 +02:00
Alexander Reelsen b6016b6e1e Bats testing: Remove useless systemctl check
Bats testing uncovered a useless systemctl check, that resulted in an
error, because the systemctl file was uninstalled, but we hoped to
check for an explicetely configured SystemExitCode.

In addition we did not reload the systemctl configuration when uninstalling
elasticsearch, which now is fixed as well.

Closes #12682
2015-08-07 14:42:38 +02:00
Clinton Gormley e2cbec185b Revert "disable license checker on windows for now, jenkins boxes have perl problems"
This reverts commit 6e9d8671bc.
2015-08-07 14:27:24 +02:00
Nik Everett 8e6e8ff234 Let OSX build rpms for linux
Closes #12701
2015-08-07 08:11:34 -04:00
Nik Everett f1fc2efcc8 Packaging: rpm and deb create scripts directory
Elasticsearch will create this if it doesn't exist if it cant but because
it doesn't own /etc/elasticsearch when installed by rpm and deb it can't
create /etc/elasticsearch/scripts.

Closes #12702
2015-08-07 08:10:53 -04:00
Robert Muir f50531c403 Revert "Revert "disable license checker on windows for now, jenkins boxes have perl problems""
This reverts commit 633b4868d9.
2015-08-06 23:38:24 -04:00
Robert Muir 633b4868d9 Revert "disable license checker on windows for now, jenkins boxes have perl problems"
This reverts commit 6e9d8671bc.

Lets see if they are up for the task now.
2015-08-06 22:17:45 -04:00
Robert Muir 6e7d1a828d fix totally broken batch file. thank you jenkins. windows you still suck. 2015-08-06 18:27:06 -04:00
Robert Muir 6e9d8671bc disable license checker on windows for now, jenkins boxes have perl problems 2015-08-06 18:01:04 -04:00
Robert Muir 7414d19d28 Support jenkins randomization in integration tests 2015-08-06 14:48:27 -04:00
Robert Muir 354a32b05d remove annoying maven warning 2015-08-05 22:08:12 -04:00
Robert Muir 5fe49e4420 cleanup skipping everywhere 2015-08-05 21:45:09 -04:00
Alexander Reelsen 9028fb8fbe Release: Update build release script to reflect latest changes
As the script now deploys to S3 and several things in master have
changed, this script needs to reflect the latest changes

* An unsigned RPM is built by default, so that users of older
  RPM based distros can download and use that RPM by default
* In addition a signed RPM is built, that is used for the repositories
* Paths for the new distributions have been fixed
* The check for the number of jars has been removed, as this is done
  as part of the license checking in `mvn verify`
* Checksum generation has been removed, as this is done as part of the
  mvn build
* Publishing artifacts of S3 has been removed
* Repostitory creation script has been updated
2015-08-05 09:38:42 +02:00
Robert Muir 6f9a067197 Change master branch back to 2.0-beta1 2015-08-04 15:38:21 -04:00
Robert Muir 29d73accdd Merge pull request #12639 from rmuir/improve_integ
Don't use port 9200/9300 for integration tests
2015-08-04 13:09:47 -04:00
David Pilato c61dccd189 Packaging: mvn install renames artifacts when copying
This PR:

* renames all distribution artifacts to `elasticsearch` so maven plugins will pick up the correct finalName without needing any hack.
* changes the groupId for every single distribution module as we can't have more than one module using the same groupId:artifactId
* does not attach anymore empty jar files for tar/zip/... modules as they don't contain any `src/main/java` stuff.

When you build it, you end up with:

```
$ tree ~/.m2/repository/org/elasticsearch/distribution
distribution
├── deb
│   └── elasticsearch
│       ├── 2.0.0-beta1-SNAPSHOT
│       │   ├── elasticsearch-2.0.0-beta1-SNAPSHOT.deb
│       │   ├── elasticsearch-2.0.0-beta1-SNAPSHOT.deb.md5
│       │   ├── elasticsearch-2.0.0-beta1-SNAPSHOT.deb.sha1
│       │   ├── elasticsearch-2.0.0-beta1-SNAPSHOT.pom
│       │   ├── elasticsearch-2.0.0-beta1-SNAPSHOT.pom.md5
│       │   └── elasticsearch-2.0.0-beta1-SNAPSHOT.pom.sha1
├── elasticsearch-distribution
│   ├── 2.0.0-beta1-SNAPSHOT
│   │   ├── elasticsearch-distribution-2.0.0-beta1-SNAPSHOT.pom
│   │   ├── elasticsearch-distribution-2.0.0-beta1-SNAPSHOT.pom.md5
│   │   └── elasticsearch-distribution-2.0.0-beta1-SNAPSHOT.pom.sha1
├── fully-loaded
│   └── elasticsearch
│       ├── 2.0.0-beta1-SNAPSHOT
│       │   ├── elasticsearch-2.0.0-beta1-SNAPSHOT.pom
│       │   ├── elasticsearch-2.0.0-beta1-SNAPSHOT.pom.md5
│       │   └── elasticsearch-2.0.0-beta1-SNAPSHOT.pom.sha1
├── rpm
│   └── elasticsearch
│       ├── 2.0.0-beta1-SNAPSHOT
│       │   ├── elasticsearch-2.0.0-beta1-SNAPSHOT.pom
│       │   ├── elasticsearch-2.0.0-beta1-SNAPSHOT.pom.md5
│       │   ├── elasticsearch-2.0.0-beta1-SNAPSHOT.pom.sha1
│       │   ├── elasticsearch-2.0.0-beta1-SNAPSHOT.rpm
│       │   ├── elasticsearch-2.0.0-beta1-SNAPSHOT.rpm.md5
│       │   └── elasticsearch-2.0.0-beta1-SNAPSHOT.rpm.sha1
├── shaded
│   └── elasticsearch
│       ├── 2.0.0-beta1-SNAPSHOT
│       │   ├── elasticsearch-2.0.0-beta1-SNAPSHOT.jar
│       │   ├── elasticsearch-2.0.0-beta1-SNAPSHOT.jar.md5
│       │   ├── elasticsearch-2.0.0-beta1-SNAPSHOT.jar.sha1
│       │   ├── elasticsearch-2.0.0-beta1-SNAPSHOT.pom
│       │   ├── elasticsearch-2.0.0-beta1-SNAPSHOT.pom.md5
│       │   └── elasticsearch-2.0.0-beta1-SNAPSHOT.pom.sha1
├── tar
│   └── elasticsearch
│       ├── 2.0.0-beta1-SNAPSHOT
│       │   ├── elasticsearch-2.0.0-beta1-SNAPSHOT.pom
│       │   ├── elasticsearch-2.0.0-beta1-SNAPSHOT.pom.md5
│       │   ├── elasticsearch-2.0.0-beta1-SNAPSHOT.pom.sha1
│       │   ├── elasticsearch-2.0.0-beta1-SNAPSHOT.tar.gz
│       │   ├── elasticsearch-2.0.0-beta1-SNAPSHOT.tar.gz.md5
│       │   └── elasticsearch-2.0.0-beta1-SNAPSHOT.tar.gz.sha1
└── zip
    └── elasticsearch
        └── 2.0.0-beta1-SNAPSHOT
            ├── elasticsearch-2.0.0-beta1-SNAPSHOT.pom
            ├── elasticsearch-2.0.0-beta1-SNAPSHOT.pom.md5
            ├── elasticsearch-2.0.0-beta1-SNAPSHOT.pom.sha1
            ├── elasticsearch-2.0.0-beta1-SNAPSHOT.zip
            ├── elasticsearch-2.0.0-beta1-SNAPSHOT.zip.md5
            ├── elasticsearch-2.0.0-beta1-SNAPSHOT.zip.sha1
            └──
```

Closes #12536
2015-08-04 15:34:11 +02:00
Robert Muir cf6fe6cdcd Don't use port 9200/9300 for integration tests 2015-08-04 09:25:44 -04:00
Nik Everett e8ccdf91fc Merge pull request #12610 from nik9000/bug/12504_2.0
Plugin script: Fix spaces
2015-08-04 09:05:38 -04:00
Robert Muir 5059c978e3 Fix distribution checks to pass unzipped directory to license checker 2015-08-04 07:54:55 -04:00
Simon Willnauer 6753f7f03e Cut over master to 2.0.0-SNAPSHOT 2015-08-04 10:54:12 +02:00
Robert Muir c059101fbb Fix missed test class 2015-08-03 21:56:43 -04:00
Ryan Ernst 1e12d03252 Tests: Rename base tests cases to use "TestCase" suffix
Most of the abstract base test classes we have were previously @Ignored.
However, there were also some other tests ignored. Having two ways to
quiet tests is confusing, and clearly it has caused some tests
to get lost in the fold.

This change moves all base test classes to use the "TestCase" suffix,
which is not picked up by the test class name pattern. It also removes
@Ignore from (almost) all tests, and adds it to forbidden apis.
And since we were renaming, I shorted base test class names to use
"ES" instead of "Elasticsearch". I type this a lot of types a day,
and I have heard others express a similar desire for a shorter name.

closes #10659
2015-08-03 17:43:00 -07:00
Scott Florentino 21a7518271 Merge branch 'master' into smflorentino-fix-winjvm-detect 2015-08-03 17:26:25 -07:00
Nik Everett 1b5bf17ae0 Plugin script: Fix spaces
Fixes ES_HOME with spaces and installing plugins from a local directory
with spaces.

Closes #12504
2015-08-03 15:40:18 -04:00
Robert Muir 0e027d82c9 Merge pull request #12602 from rmuir/core_integ_tests
Add integration test phase for elasticsearch core/
2015-08-03 10:15:37 -04:00
Robert Muir 379ffdb8ee Add integration test phase for elasticsearch core/ 2015-08-03 09:43:11 -04:00
Robert Muir e1886a9b12 Test that shaded jar contains only shaded classes 2015-08-03 08:56:45 -04:00
Robert Muir a5a27a5cd9 Add jar hell check test for the shaded jar 2015-08-03 05:54:55 -04:00
Robert Muir aaf0976cb7 clear up dependency hell a bit more 2015-08-01 14:02:14 -04:00
Robert Muir eaaa06b6b3 fix shading to create correct POM and not drag in wacky dependencies 2015-08-01 12:44:00 -04:00
Robert Muir 6f33ea4f62 move shading to distribution/shaded 2015-08-01 11:37:43 -04:00
Robert Muir fef55348f9 clean up .zip handling so its not uselessly downloaded in the build 2015-08-01 09:10:03 -04:00
Robert Muir 8d5b5ad862 restore my lost code changes and add measures to detect/prevent this in the future 2015-07-31 13:00:24 -04:00
Robert Muir 3d7da5a826 remove unused properties sections from distribution modules 2015-07-31 10:20:50 -04:00
Robert Muir f804ec3cd9 cleanup 2015-07-31 10:05:11 -04:00
Robert Muir 2068fbfd1e Merge pull request #12549 from rmuir/distribution_tests_1
add integration tests for tar/deb/rpm packaging
2015-07-30 09:18:09 -04:00
Alexander Reelsen 69d7f1a78a Startup: Remove getopt parsing in shell script, use java CLITool
In order to ensure, we have the same experience across operating systems
and shells, this commit uses the java CLI parser instead of the shell
getopt parsing to parse arguments.

This also allows for support for paths, which contain spaces.

Also commons-cli depdency was upgraded to 1.3.1 and tests have been added.

Changes

* new exit code, OK_AND_EXIT, allowing to tell the caller to exit, as everything
  went as expected (e.g. when running a version output)

BWC breaking:

* execute() returns an ExitStatus instead of an integer, otherwise there is no
  possibility to signal by a command, if the JVM should be exited after a run.
  This affects plugins, that have command line tools
* -v used to be version, but is a verbose flag by default in the current CLI infra,
  must be -V or --version now
* -X has been removed - the current implementation was useless anyway, as
  it prefixed those properties with "es.". You should use
  ES_JAVA_OPTS/JAVA_OPTS for JVM configuration
2015-07-30 13:20:29 +02:00
Adrien Grand 6893643bef Build: Set the correct sha1 for HDRHistogram. 2015-07-30 10:24:55 +02:00
Robert Muir ddfea366de Add integration tests for distribution/rpm 2015-07-29 22:26:45 -04:00
Robert Muir 2ed8e697d5 Add integration tests for distribution/deb 2015-07-29 21:34:46 -04:00
Robert Muir 1c5a558164 Add integration tests for distribution/tar 2015-07-29 15:33:29 -04:00
Robert Muir c6d6eaef4d Add a run.sh to run from current source code with debugger
The maven magic needed for this is now heavier
2015-07-29 14:21:43 -04:00
Alexander Reelsen c423319dda Build: Remove profile to create attached RPM
This is no longer necessary, as the RPM is now built as
a primary artifact, so no need to deploy it as a secondary
artifact anymore.

Closes #12529
2015-07-29 15:54:26 +02:00
Robert Muir c6abf28273 Disable license checker temporarily to get regression builds working again
x
2015-07-29 08:39:25 -04:00
Chris Earle e1c99bc8b4 Add -XX:+PrintGCDateStamps when using GC Logs
Now that Elasticsearch requires Java 7 or later, it's safe to add `-XX:+PrintGCDateStamps` to get human readable times alongside JVM times.
2015-07-27 15:17:54 -04:00
Alexander Reelsen 9628d2632f Build: Split packages into submodules
This change creates a proper `distribution` modules in which we have today packaging for
all of our four current packages:

* zip
* tar.gz
* rpm
* deb

Licenes have moved into the distribution project as well. So have the config/ and the bin/ directory
from the core/ project.

The RPM package is now built, if rpmbuild exists.

The bats tests have been moved as well.

Also the zip distribution now executes the REST integration tests.
2015-07-27 17:50:54 +02:00