Commit Graph

527 Commits

Author SHA1 Message Date
Lee Hinman feb244c14a Remove cluster name from data path
Previously Elasticsearch used $DATA_DIR/$CLUSTER_NAME/nodes for the path
where data is stored, this commit changes that to be $DATA_DIR/nodes.

On startup, if the old folder structure is detected it will be used.
This behavior will be removed in Elasticsearch 6.0

Resolves #17810
2016-06-07 10:13:48 -06:00
Martijn van Groningen f611f1c99e ingest: Move processors from core to ingest-common module.
Folded grok processor into ingest-common module.

The rest tests have been moved to ingest-common module as well, because these tests don't run in the rest-api-spec module but in the distribution:integ-test-zip module
and adding a test plugin there felt just wrong to me. I think this is ok. I left a tiny ingest rest test behind in that tests with an empty pipeline.

Removed messy tests, these tests were already covered in the rest tests

Added ingest test plugin in test infra so that each module testing integration with ingest doesn't need write its own plugin

Moved reindex ingest tests to qa module

Closes #18490
2016-06-07 17:32:52 +02:00
Tanguy Leroux a1172d816c Implement ctx.op = "delete" on _update_by_query and _reindex
closes #18043
2016-06-06 11:11:29 +02:00
Nik Everett 08f7f79b2e Wrap lines at 140 characters (:qa projects) 2016-06-05 15:55:44 -04:00
GUILLAUME GROSSETIE 9411f18f27 Display plugins versions
This is useful to determine if a plugin needs to be updated when using deployment automation solution (like Ansible).
2016-06-01 17:14:13 +02:00
Simon Willnauer 93c5a9dacd [TEST] Set BWC version to 5.0.0-SNAP since this is it's min compat version 2016-06-01 09:11:08 +02:00
Ryan Ernst 454de6a8f2 Tests: Remove unnecessary evil jarhell tests
We have 3 evil tests for jarhell. They have been failing in java 9
because of how evil they are. The first checks the leniency we add for
jarhell in the jdk itself. This is unecessary, since if the leniency
wasn't there, we would already be failing all jarhell checks. The second
is checking the compile version is compatible with the jdk. This is
simpler since we don't need to fake the java version: we know 1.7 should
be compatibile with both java 8 and 9, so we can use that as a constant.
Finally the last test checks if the java version system property is
broken. This is simply something we should not check, we have to trust
that java specifies it correctly, and again, if it was broken, all
jarhell checks would be broken.
2016-05-30 21:39:56 -07:00
Jason Tedor 04cae88ff4 Do not use Lucene SuppressForbidden
Lucene SuppressForbidden is marked lucene.internal and should not be
used outside of Lucene. This commit removes the uses of this class
within Elasticsearch. Instead,
org.elasticsearch.common.SuppressForbidden should be used, which was
already the case in most places.
2016-05-30 10:57:33 -04:00
Robert Muir 3f06d9f3b8 Merge pull request #18600 from rmuir/new_script_exception
replace ScriptException with a better one
2016-05-26 17:51:34 -04:00
Jason Tedor f16f65741e Fix when plugins directory is symlink
This commit fixes an issue with the plugins directory being a symbolic
link. Namely, the install plugins command attempts to always create the
plugins directory just in case it does not exist. The JDK method used
here guarantees that the directory is created, and an exception is not
thrown if the directory could not be created because it already
exists. The problem is that this JDK method does not respect symlinks so
its internal existence checks fails, it proceeds to attempt to create
the directory, but the directory creation fails because the symlink
exists. This is documented as being not an issue. We work around this by
checking if there is a symlink where we expect the plugins directory to
be, and only attempt to create if not. We add a unit test that plugin
installation to a symlinked plugins directory works as expected.
2016-05-26 14:10:32 -04:00
Robert Muir f037807117 replace ScriptException with a better one 2016-05-26 11:43:29 -04:00
Jason Tedor d29844e597 Remove custom plugins path
This commit removes the ability to specify a custom plugins
path. Instead, the plugins path will always be a subdirectory called
"plugins" off of the home directory.
2016-05-26 10:16:25 -04:00
Jason Tedor 0f529e10a8 Fix plugin command name in remove plugin command
This commit fixes the name of the plugin command that is output when a
user attempts to remove a plugin that does not exist.
2016-05-26 10:14:39 -04:00
Ryan Ernst c951e03edc Remove unused line in official plugins test. 2016-05-25 14:38:10 -07:00
Ryan Ernst 45adab0cb8 Add test that x-pack is in official plugins list 2016-05-25 14:23:57 -07:00
Ryan Ernst 5e7b8d7788 Add test for official plugins list being sorted 2016-05-25 14:21:33 -07:00
Tal Levy edfbdf2748 add ability to specify multiple grok patterns (#18074)
- now you can specify a list of grok patterns to match your field with
and the first one to successfully match wins.
- only non-null captures will be inserted into your matched document.

Fixes #17903.
2016-05-25 12:20:39 -07:00
Jason Tedor 84dfa360b1 Fix list of poi dependencies for ingest-attachment
This commit fixes the list of poi dependencies for the ingest-attachment
plugin after they were upgraded in
1d40c4bbc1.
2016-05-24 17:03:25 -04:00
Clinton Gormley 9c9bea9258 Set version to 5.0.0-alpha3 (#18550)
* Set version to 5.0.0-alpha3

* Updated version in qa/backwards tests too
2016-05-24 16:46:05 +02:00
Tanguy Leroux 1f011f9dea Remove Delete-By-Query plugin
closes #18469
2016-05-24 13:28:20 +02:00
Lee Hinman fdfd2a2f18 Remove ScriptMode class in favor of boolean true/false
This removes the ScriptMode class entirely, which was an enum with two
options (ON and OFF) which essentially boiled down to true and false.
Now the boolean values are used instead.
2016-05-20 15:01:30 -06:00
Jason Tedor c257e2c51f Remove settings and system properties entanglement
Today when parsing settings during bootstrap, we add a system property
for every Elasticsearch setting. Additionally, settings can be set via
system properties. This commit simplifies this situation.
 - settings are no longer propogated to system properties
 - system properties can not be used to set settings
 - the "es." prefix on settings is no longer required (nor permitted)
 - test logging has a dedicated system property (tests.logger.level)

Relates #18198
2016-05-19 14:08:08 -04:00
Tanguy Leroux a01ecb20ea Port Delete By Query to Reindex infrastructure
closes #16883
2016-05-19 16:07:50 +02:00
Tanguy Leroux 35d3bdab84 Add Google Cloud Storage repository plugin
Closes #12880
2016-05-19 13:26:23 +02:00
Jason Tedor db4809d906 Remove last vestigates of /bin/sh shebangs
This commit removes the remaining /bin/sh shebangs in favor of
/bin/bash.

Relates #18448
2016-05-18 11:03:00 -04:00
Lee Hinman ff5c7965ed Don't run `mkdir` when $DATA_DIR contains a comma-separated list
Resolves #16992
Resolves https://github.com/elastic/cookbook-elasticsearch/issues/441
2016-05-17 13:42:24 -06:00
Lee Hinman efff3918d8 Remove support for mulitple languages per scripting engine 2016-05-13 09:24:31 -06:00
Jason Tedor 9ce96f5792 Add tests that packages depend on bash
This commit adds bats tests that the RPM and Debian packages depend on
bash.

Relates #18292
2016-05-12 09:04:24 -04:00
Gabriel Moskovicz 0660386976 Add plugin information for Verbose mode
Relates #18051
2016-05-10 11:23:17 -04:00
Jason Tedor 7d1fd17172 Remove plugin script parsing of system properties
The plugin script parses command-line options looking for Java system
properties and extracts these arguments to pass to the java command when
starting the JVM. Since elasticsearch-plugin allows arbitrary user
arguments to the JVM via ES_JAVA_OPTS, this parsing is unnecessary. This
commit removes this unnecessary 

Relates #18207
2016-05-09 13:06:18 -04:00
Jason Tedor 3f3fa59406 Account for rpm behavior difference on directories
This commit modifies the packaging tests to account for the fact that
rpm behaves differently with respect to preserving directories marked as
"CONFIG | NOREPLACE" on older versions versus newer versions. Older
versions will leave the directory as-is while newer versions will append
the suffix ".rpmsave" to the directory name.

Relates #18216
2016-05-09 12:25:52 -04:00
Jason Tedor 0eaa831f48 Preserve config files from RPM install
This commit modifies the packaging for the RPM package so that edits to
config files will not get lost during removal and upgrade.

Relates #18188
2016-05-06 13:24:54 -04:00
Adrien Grand e88ac11633 Add back Version.V_5_0_0. #18176
This was lost whene releasing alpha2 since the version constant got renamed.
2016-05-06 12:30:22 +02:00
Ryan Ernst 8fc51380de Tests: improve logging for vagrant to emit entire output on failure
This change makes the vagrant tasks extend LoggedExec, so that the
entire vagrant output can be dumped on failure (and completely logged
when using --info). It should help for debugging issues like #18122.
2016-05-04 17:29:23 -07:00
Jason Tedor 4c6cf7ee88 Pass ES_JAVA_OPTS to JVM for plugins script
This commit adds support for ES_JAVA_OPTS to the elasticsearch-plugin
script.

Relates #18140
2016-05-04 12:48:39 -04:00
Alexander Reelsen f71eb0b888 Version: Set version to 5.0.0-alpha2 2016-04-26 09:30:26 +02:00
Lee Hinman 9da88a99da Fix exit code
Exit with proper exit code (1) and an error message if elasticsearch
executable binary does not exists or has insufficient permissions to
execute.

Squashed commit of the following:

commit 9768d316303418ba4f9c96d3f87c376048a1b1bc
Author: Puru <tuladharpuru@gmail.com>
Date:   Fri Apr 22 23:26:47 2016 +0545

    Fixed ES_HOME typo

commit 79a2b0394297f8b02b6f71b71ba35ff79f1a684e
Author: Puru <tuladharpuru@gmail.com>
Date:   Sun Apr 10 11:00:24 2016 +0545

    Improve elasticsearch startup script test

    Added improvement as per conversation in https://github.com/elastic/elasticsearch/pull/17082#issuecomment-206459613

commit 7be38e1fefd4baa6ccdbdc14745c00f6dc052e0c
Author: Puru <tuladharpuru@gmail.com>
Date:   Wed Mar 23 13:23:52 2016 +0545

    Add elasticsearch startup script test

    The test ensures that elasticsearch startup script exists and is executable.

commit d10eed5c08260fa9c158a4487bbb3103a8d867ed
Author: Puru <tuladharpuru@gmail.com>
Date:   Wed Mar 23 12:30:25 2016 +0545

    Fixed IF syntax and failure message

commit 6dc66f616545572485b4d43bee05a4cbbf1bed72
Author: Puru <tuladharpuru@gmail.com>
Date:   Sat Mar 12 11:08:11 2016 +0545

    Fix exit code

    Exit with proper exit code (1) and an error message if elasticsearch executable binary does not exists or has insufficient permissions to execute.
2016-04-22 11:48:05 -06:00
Lee Hinman 2539d94bc9 Merge remote-tracking branch 'dakrone/vagrant-umask' 2016-04-21 12:31:07 -06:00
Lee Hinman 4fca5f734a Explicitly set packaging permissions
This changes our packaging to be explicit about the permissions of files
and directories in the tar.gz, rpm, and deb packages. This is to protect
against a user having an incorrectly set umask when installing.

Additionally, plugins that are installed now have their permissions set
by the plugin installation so that plugins that may have been packaged
with incorrect permissions are secured.

Resolves #17634
2016-04-21 12:30:56 -06:00
Martijn van Groningen dd2184ab25 ingest: Streamline option naming for several processors:
* `rename` processor, renamed `to` to `target_field`
* `date` processor, renamed `match_field` to `field` and renamed `match_formats` to `formats`
* `geoip` processor, renamed `source_field` to `field` and renamed `fields` to `properties`
* `attachment` processor, renamed `source_field` to `field` and renamed `fields` to `properties`

Closes #17835
2016-04-21 13:40:43 +02:00
Lee Hinman a1e8fb794c Allow JSON with unquoted field names by enabling system property
In Elasticsearch 5.0.0, by default unquoted field names in JSON will be
rejected. This can cause issues, however, for documents that were
already indexed with unquoted field names. To alleviate this, a system
property has been added that can be enabled so migration can occur.

This system property will be removed in Elasticsearch 6.0.0

Resolves #17674
2016-04-19 09:14:13 -06:00
Lee Hinman 66adcb2962 [TEST] Fix extra backslash causing sed to hang
This caused sed to hang and the output never to show for the smoke test
command
2016-04-13 10:24:39 -06:00
Adrien Grand 82849a787a Add back the Version.V_5_0_0 constant. #17688 2016-04-13 10:00:37 +02:00
Jason Tedor a581d7cca4 Merge pull request #17675 from jasontedor/java-opts
Add JVM options configuration file
2016-04-12 23:07:40 -04:00
Jason Tedor 3879aa2a98 Add JVM options configuration file
This commit adds a new configuration file jvm.options to centralize and
simplify management of JVM options. This separates the configuration of
the JVM from the packaging scripts (bin/elasticsearch*, bin/service.bat,
and init.d/elasticsearch) simplifying end-user operational management of
custom JVM options.
2016-04-12 11:19:16 -04:00
Lee Hinman dd8afc9567 [TEST] Add extra logging for vagrant test output
The $user, $group, and $privileges are echoed when they are being
compared.
2016-04-11 09:36:50 -06:00
Adrien Grand 4391594e4c Use Settings.builder instead of settingsBuilder in ESSmokeClientTestCase. 2016-04-08 18:37:05 +02:00
Ryan Ernst 19567fb794 Add stop task for vagrant which wraps all halt tasks, instead of using clean 2016-04-06 14:15:23 -07:00
Ryan Ernst bfc708ee58 Merge pull request #17573 from rjernst/vagrant_boxes_param
Tests: Add vagrant.boxes gradle property
2016-04-06 13:33:45 -07:00
Ryan Ernst dc376fb21b Tweak packagingTest description formatting 2016-04-06 13:08:38 -07:00
Ryan Ernst 9794d371ef Always add halt tasks for all available boxes 2016-04-06 13:07:11 -07:00
Ryan Ernst 9f24de7ca8 Tests: Add vagrant.boxes gradle property
This change makes specifying which boxes to run vagrant tests on a
little easier. Previously there were two tasks, checkPackages and
checkPackagesAllDistros. With this change, there is a single
packagingTest task. The boxes to run on are specified using the
gradle property vagrant.boxes, which can be easily specified on the
command line, or in a gradle properties file. There are also two
alias names, 'sample' for a yum and apt box, and 'all' for all boxes.
2016-04-06 12:36:35 -07:00
Jason Tedor eb7ecb7296 Add test of plugin script if JAVA_HOME has a space 2016-04-06 13:28:56 -04:00
Clinton Gormley b87beeb05f Rename update-by-query REST tests to update_by_query 2016-03-29 13:13:49 +02:00
Nik Everett 0e6141e675 Replace is_true: took with took >= 0
This prevents tests from failing on machines that can finish the request
less than half a millisecond.
2016-03-28 13:03:48 -04:00
Jason Tedor 85b06f48cf Revert "Add debug logging for Vagrant upgrade test"
This reverts commit 7ecfa6e2ad.
2016-03-24 21:59:32 -04:00
Jason Tedor a0634b2442 Merge pull request #17197 from jasontedor/chown
Elasticsearch ownership for data, logs, and configs
2016-03-24 21:21:11 -04:00
Jason Tedor d22b5977de Ownership for data, logs, and configs for packages
This commit ensures that the data, logs, and config directories have the
proper ownership after the packages are installed. Additionally, this
commit ensures that the configs in /etc/elasticsearch are preserved
after removal of the RPM package.
2016-03-24 20:56:02 -04:00
Tal Levy cabc4b1636 add on_failure exception metadata to ingest document for verbose simulate 2016-03-24 16:11:47 -07:00
Alexander Reelsen baec7bc23c Tests: Fix vagrant test on debian
Debian asks during installation, if the configuration file should be updated.
This is asked via a prompt and thus hangs.

This adds an option to always update to the newer config file, so automated
installation keeps working.
2016-03-24 14:21:33 +01:00
Simon Willnauer e4e08b28ff [TEST] Use to wait for the node to start - it will wait for state recovery and not return 503 2016-03-24 11:06:40 +01:00
Jim Ferenczi da42f199bd Enforce isolated mode for all plugins
This commit removes the isolated option, each plugin have its own classloader.
2016-03-24 09:17:33 +01:00
Alexander Reelsen b2573858b6 Version: Set version to 5.0.0-alpha1
Change version, required a minor fix in the RPM building.
In case of a alpha/beta version, the release will contain alpha/beta
as the RPM version cannot contains dashes/tildes.
2016-03-24 08:36:08 +01:00
Jason Tedor 7ecfa6e2ad Add debug logging for Vagrant upgrade test 2016-03-23 23:12:16 -04:00
Jason Tedor 17dd60dd31 Merge pull request #17208 from jasontedor/install-plugin-permissions
Install plugin permissions
2016-03-23 18:44:47 -04:00
Jason Tedor 04e6e6c3e0 Fix cluster health API call 2016-03-23 17:06:45 -04:00
Jason Tedor 033167de5b Wait for yellow indices when running upgrade test
This commit makes the Vagrant upgrade test wait for yellow indices
before attempting to get documents from the upgraded Elasticsearch node.
2016-03-23 15:57:21 -04:00
Jason Tedor 8004c51c17 Add max size virtual memory check
This commit adds a bootstrap check on Linux and OS X for the max size of
virtual memory (address space) to the user running the Elasticsearch
process.

Closes #16935
2016-03-22 11:52:36 -04:00
Jason Tedor 5dc48e71d0 Use mock filesystem during install plugins tests
This commit sets up the default filesystem used during install plugins
tests. A hack is neeeded to handle the temporary directory because the
system property "java.io.tmpdir" will have been initialized to a value
that is sensible for the default filesystem, but not necessarily to a
value that makes sense for the mock filesystem in use during the
tests. This property is restored after each test.
2016-03-22 10:25:27 -04:00
Martijn van Groningen 8f22a01bbd ingest: Give the `foreach` processor access to the rest of the document.
Closes #17147
2016-03-22 10:32:13 +01:00
Jason Tedor 6db6c15d06 Add tests of POSIX handling for installing plugins
This commit refactors the unit tests for installing plugins to test
against mock filesystems (as well as the native filesystem) for better
test coverage. This commit also adds tests that cover the POSIX
attributes handling when installing plugins (e.g., ensuring that the
plugins directory has the right permissions, the bin directory has
execute permissions, and the config directory has the same owner and
group as its parent).
2016-03-21 09:02:25 -04:00
Jason Tedor a7793f7271 Cutover to elastic Vagrant boxes
This commit cuts the Vagrant tests over to the elastic Vagrant boxes.
2016-03-20 22:11:30 -04:00
Jason Tedor c3f6ad89ef Simplify module or plugin file checks 2016-03-18 11:50:49 -04:00
Jason Tedor d4abfb2a87 Centralize umask utilities in bats tests 2016-03-18 08:25:21 -04:00
Jason Tedor 6bd19cff67 Centralize existence check for modules and plugins 2016-03-17 22:34:33 -04:00
Jason Tedor 6ecfc7adbe Permissions tests for modules and plugins 2016-03-17 22:16:09 -04:00
Jason Tedor 9f73152940 Fix plugins permissions 2016-03-17 15:35:54 -04:00
Jason Tedor dacb96ba61 Fix plugin installation logging 2016-03-17 15:35:54 -04:00
Jason Tedor ba6820caeb Fix plugin file permissions tests 2016-03-17 15:35:54 -04:00
Jason Tedor ffc6ba9e20 Fix installed plugins check 2016-03-17 15:35:53 -04:00
Jason Tedor 9ed4646296 Plugin site-example no longer exists 2016-03-17 15:35:53 -04:00
Jason Tedor 9e739da1d9 Add new plugins 2016-03-17 15:35:53 -04:00
Jason Tedor ace45e8bda Add module tests 2016-03-17 15:35:53 -04:00
Jason Tedor b46dd3f01c es.path.conf can not be empty 2016-03-16 23:38:04 -04:00
Jason Tedor 618441aea3 Merge pull request #17088 from jasontedor/simplify-bootstrap-settings
Bootstrap does not set system properties
2016-03-15 19:25:16 -04:00
Jason Tedor 66ba044ec5 Use setting in integration test cluster config 2016-03-15 17:45:17 -04:00
Simon Willnauer cbaa480c16 [TEST] Let the windows machine be slow as hell 2016-03-15 15:35:44 +01:00
Jason Tedor 8a05c2a2be Bootstrap does not set system properties
Today, certain bootstrap properties are set and read via system
properties. This action-at-distance way of managing these properties is
rather confusing, and completely unnecessary. But another problem exists
with setting these as system properties. Namely, these system properties
are interpreted as Elasticsearch settings, not all of which are
registered. This leads to Elasticsearch failing to startup if any of
these special properties are set. Instead, these properties should be
kept as local as possible, and passed around as method parameters where
needed. This eliminates the action-at-distance way of handling these
properties, and eliminates the need to register these non-setting
properties. This commit does exactly that.

Additionally, today we use the "-D" command line flag to set the
properties, but this is confusing because "-D" is a special flag to the
JVM for setting system properties. This creates confusion because some
"-D" properties should be passed via arguments to the JVM (so via
ES_JAVA_OPTS), and some should be passed as arguments to
Elasticsearch. This commit changes the "-D" flag for Elasticsearch
settings to "-E".
2016-03-13 20:09:15 -04:00
Simon Willnauer 121e7c8ca4 Add infrastructure to run REST tests on a multi-version cluster
This change adds the infrastructure to run the rest tests on a multi-node
cluster that users 2 different minor versions of elasticsearch. It doesn't implement
any dedicated BWC tests but rather leverages the existing REST tests.

Since we don't have a real version to test against, the tests uses the current version
until the first minor / RC is released to ensure the infrastructure works.

Relates to #14406
Closes #17072
2016-03-13 10:52:39 +01:00
Ryan Ernst 8b26c260d1 Plugins: Enforce plugin zip does not contain zip entries outside of the unzip dir
When unzipping a plugin zip, the zip entries are resolved relative to
the directory being unzipped into. However, there are currently no
checks that the entry name was not absolute, or relatively points
outside of the plugin dir. This change adds a check for those two cases.
2016-03-11 14:53:14 -08:00
Ryan Ernst 5bd7da5659 Addressed PR feedback
* Fix tests still referring to -E
* add comment about missing classes
* rename writer constant
2016-03-11 11:46:23 -08:00
Ryan Ernst 591fb8f028 Merge branch 'master' into cli-parsing 2016-03-11 10:45:05 -08:00
Nik Everett b2eec96045 [reindex] Make search failure cause rest failure
Indexing failures have caused the reindex http request to fail for a while
now. Both search and indexing failures cause it to abort. But search
failures didn't cause a non-200 response code from the http api. This
fixes that.

Also slips in a fix to some infrequently failing rest tests.

Closes #16037
2016-03-10 13:47:49 -05:00
Nik Everett 94798a3672 [reindex][test] Fix timeout value 2016-03-10 13:12:29 -05:00
Nik Everett 378e0d2c0c [reindex][test] Script changes destination index 2016-03-10 13:06:54 -05:00
Nik Everett b8d931d23c [reindex] Timeout if sub-requests timeout
Sadly, it isn't easy to simulate a timeout during an integration test, you
just have to cause one. Groovy's sleep should do the job.
2016-03-10 13:05:23 -05:00
Boaz Leskes cd12241e9f Decouple the TransportService and ClusterService #16872
Currently, the cluster service is tightly coupled to the transport service by both managing node connections and requiring the bound address in order to create the local disco node. This commit introduces a new NodeConnectionsService which is in charge of node connection management and makes it possible to remove all network related calls from the cluster service. The local DiscoNode is now created by DiscoveryNodeService and is set both the cluster service and the transport service during node start up.

Closes #16788
Closes #16872
2016-03-10 11:45:15 +01:00
Ryan Ernst 80198accc1 Removed old cli stuff, and add tests for new Command behavior 2016-03-08 14:13:55 -08:00
Ryan Ernst e5c852f767 Convert bootstrapcli parser to jopt-simple 2016-03-08 13:39:37 -08:00
Ryan Ernst 45b5ab24fe Moved MockTerminal and created a base test case for cli commands. 2016-03-07 12:42:15 -08:00
Ryan Ernst 7a49cd1287 Merge branch 'master' into cli-parsing 2016-03-06 13:29:56 -08:00
Ryan Ernst 5a94f8b3e6 Remove outdated nocommit and tweak assert to output bad exception message on failure. 2016-03-06 11:34:10 -08:00
Ryan Ernst ed7934ee79 Cli: Simplify test terminals
This commit simplifies and consolidates the two different
implementations of terminals used in tests. There is now a single
MockTerminal which captures output, and allows accessing as one large
string (with unix style \n as newlines), as well as configuring
input.
2016-03-06 11:30:30 -08:00
Ryan Ernst 687aadef00 Merge branch 'master' into cli-parsing 2016-03-04 12:13:17 -08:00
Martijn van Groningen 8ee17d0a86 test: replace test with rest test 2016-03-04 14:28:36 +01:00
Ryan Ernst 209da28bb2 Removed check file command tests, check file command is going away 2016-03-03 09:37:33 -08:00
Jason Tedor e75a0da4d5 Add max number of processes check
This commit adds a bootstrap check on Linux for the max number of
processes available to the user running the Elasticsearch process.

Closes #16919
2016-03-03 11:40:10 -05:00
Ryan Ernst 354ede717b Removed old help files and improved plugin cli tests 2016-03-01 11:48:52 -08:00
Nik Everett c7c8bb357a Merge pull request #16861 from nik9000/reindex_is_ready
Reindex required some parsing changes for search requests to support
differing defaults from the regular search api.
2016-03-01 10:02:48 -05:00
Ryan Ernst 8cd919c687 Added jopt simple option parser and switched plugin cli to use it 2016-02-29 19:52:42 -08:00
Nik Everett 95cc3e38fc Check test naming conventions on all modules
The big win here is catching tests that are incorrectly named and will
be skipped by gradle, providing a false sense of security.

The whole thing takes about 10 seconds on my Macbook Air, not counting
compiling the test classes, which seems worth it. Because this runs as
a gradle task with propery UP-TO-DATE handling it can be skipped if the
tests haven't been changed which should save some time.

I chose to keep this in test:framework rather than a new subproject of
buildSrc because ESIntegTestCase and doesn't inroduce any additional
dependencies.
2016-02-29 16:31:49 -05:00
Boaz Leskes 195b43d66e Remove DiscoveryService and reduce guice to just Discovery #16821
DiscoveryService was a bridge into the discovery universe. This is unneeded and we can just access discovery directly or do things in a different way.

One of those different ways, is not having a dedicated discovery implementation for each our dicovery plugins but rather reuse ZenDiscovery.

UnicastHostProviders are now classified by discovery type, removing unneeded checks on plugins.

Closes #16821
2016-02-29 20:23:38 +01:00
Nik Everett d4113d77cf Don't try to install reindex plugin in bats tests 2016-02-29 10:28:30 -05:00
Nik Everett c38119bae9 Merge branch 'master' into feature/reindex 2016-02-26 16:59:54 -05:00
Nik Everett 7c4420bf3d Merge pull request #16807 from nik9000/not_mvn
Remove mention of mvn from java source files
2016-02-25 11:05:46 -05:00
Nik Everett 49f704fd6a Remove mention of mvn from ramaining java 2016-02-24 19:54:47 -08:00
Simon Willnauer d3b2ccb49d fix line length 2016-02-22 10:51:30 -08:00
Simon Willnauer 387f0473dc Assert that we can write in all data-path on startup
Today we might start a node and some of the paths might not have the
required permissions. This commit goes through all data directories as
well as index, shard and state directories and ensures we have write access.
To make this work across all OS etc. we are trying to write a real file
and remove it again in each of those directories
2016-02-22 10:32:29 -08:00
Nik Everett 821a20f582 Merge branch 'master' into feature/reindex 2016-02-11 17:41:05 -05:00
Nik Everett 18808b7576 Move reindex from a plugin to a module 2016-02-11 17:39:49 -05:00
Nik Everett 30107b4a74 Teach reindex to stop when cancelled
All we do is check the cancelled flag and stop the request at a few key
points.

Adds the cancellation cause to the status so any request that is cancelled
but doesn't die can be seen in the task list.
2016-02-11 15:21:22 -05:00
Jim Ferenczi cb50e73f7c Add missing try with resources in InstallPluginCommandTest, this should fix the build on windows. 2016-02-11 15:13:25 +01:00
Nik Everett 0da30d5eae Merge branch 'master' into feature/reindex 2016-02-10 14:07:41 -05:00
Jim Ferenczi a520e7c7ad Fix InstallPluginCommandTests failures on windows build 2016-02-10 13:02:29 +01:00
Jim Ferenczi b146f3ecb3 Pack all the plugin files into a single folder named elasticsearch at the root of the plugin zip. 2016-02-10 10:13:05 +01:00
Simon Willnauer 9b2f40f627 Merge pull request #16471 from s1monw/no_sys_props_leaked
Apply system properties after all arguemnts are parsed in BootstrapCLIParser
2016-02-05 14:23:00 +01:00
Simon Willnauer 1e18c9a7b6 Apply system properties after all arguemnts are parsed in BootstrapCLIParser
One of our tests leaked a system property here since we failed after appling some
system properties in BootstrapCLIParser. This is not a huge deal in production since
we exit the JVM if we fail on that. Yet for correctnes we should only apply them if
we manage to parse them all.
This also caused a test failure lately on CI but on an unrelated test:
  https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+periodic/314/console
2016-02-05 11:21:00 +01:00
Jim Ferenczi 7d0181b5d4 Rename bin/plugin in bin/elasticsearch-plugin 2016-02-05 10:09:14 +01:00
Martijn van Groningen 7a6adfd93a ingest: Added foreach processor.
This processor is useful when all elements of a json array need to be processed in the same way.
This avoids that a processor needs to be defined for each element in an array.
Also it is very likely that it is unknown how many elements are inside an json array.
2016-02-04 23:44:01 +01:00
Nik Everett bb4d8b79fe Merge branch 'master' into feature/reindex 2016-02-04 09:23:49 -05:00
Ryan Ernst e97345984d Remove format args from readText and readSecret 2016-02-04 02:03:17 -08:00
Ryan Ernst 2265c5d9e9 Tests: Fix bootstrap cli tests to not clear all properties previous tests set 2016-02-03 15:09:17 -08:00
Ryan Ernst 14f03a9750 Tests: Force tribe node to use the same "node.mode" as the tribe client nodes. 2016-02-03 14:37:53 -08:00
Simon Willnauer 4a4e523357 Merge branch 'master' into make_settings_strict 2016-02-03 11:34:12 +01:00
Martijn van Groningen 64ac037a82 Added an ingest qa that tests processor real world like configurations.
Renamed `ingest-with-mustache` to `smoke-test-ingest-with-all-dependencies`
Also renamed `ingest-disabled` to `smoke-test-ingest-disabled` so that the name is more inline with other qa smoke test modules.
2016-02-02 22:37:24 +01:00
Ryan Ernst 9f47b376da Plugin cli: Improve maven coordinates detection
Identifying when a plugin id is maven coordinates is currently done by
checking if the plugin id contains 2 colons. However, a valid url could
have 2 colons, for example when a port is specified. This change adds
another check, ensuring the plugin id with maven coordinates does not
contain a slash, which only a url would have.

closes #16376
2016-02-02 12:17:56 -08:00
Simon Willnauer 818a9eefb2 Make settings validation strict
This commit enableds strict settings validation on node startup. All settings
passed to elasticsearch either through system properties, yaml files or any other
way to pass settings must be registered and valid. Settings that are unknown ie. due to
typos or due to deprecation or removal will cause the node to NOT start up. Plugins
have to declare all their settings on the `SettingsModule#registerSetting` and settings for
plugins that are not installed must be removed.

This commit also removes the ability to specify the nodes name via `-Des.name` or just `name` in the
configuration files. The node name must be prefixed with the node prexif like `node.name: Boom`. Left over
usage of `name` will also cause startup to fail.
2016-02-02 11:32:44 +01:00
Ryan Ernst a2c37c0989 CliTool: Allow unexpected exceptions to propagate
Cli tools currently catch all exceptions, and only print the exception
message, except when a special system property is set. Even with this
flag set, certain exceptions, like IOException, are captured and their
stack trace is always lost.

This change adds a UserError class, which can be used a cli tools to
specify a message to the user, as well as an exit status. All other
exceptions are propagated out of main, so java will exit with non-zero
and print the stack trace.
2016-02-01 16:35:22 -08:00
Nik Everett c1d4478067 Merge branch 'master' into feature/reindex 2016-02-01 18:33:09 -05:00
Tal Levy dcb88909d5 Merge pull request #16324 from talevy/ingest_on_failure_tag
Add processor tags to on_failure metadata in ingest pipeline
2016-02-01 10:13:26 -08:00
Ryan Ernst 3787f437ec Merge branch 'master' into remove_multicast 2016-02-01 07:25:45 -08:00
Ryan Ernst 661e3c99f7 Merge pull request #16336 from rjernst/demangle
Reduce complexity of plugin cli
2016-02-01 07:20:04 -08:00
Ryan Ernst a052dfeb38 Plugins: Reduce complexity of plugin cli
The plugin cli currently is extremely lenient, allowing most errors to
simply be logged. This can lead to either corrupt installations (eg
partially installed plugins), or confused users.

This change rewrites the plugin cli to have almost no leniency.
Unfortunately it was not possible to remove all leniency, due in
particular to how config files are handled.

The following functionality was simplified:
* The format of the name argument to install a plugin is now an official
  plugin name, maven coordinates, or a URL.
* Checksum files are required, and only checked, for official plugins
  and maven plugins. Checksums are also only SHA1.
* Downloading no longer uses a separate thread, and no longer has a timeout.
* Installation, and removal, attempts to be atomic. This only truly works
  when no config or bin files exist.
* config and bin directories are verified before copying is attempted.
* Permissions and user/group are no longer set on config and bin files.
  We rely on the users umask.
* config and bin directories must only contain files, no subdirectories.
* The code is reorganized so each command is a separate class. These
  classes already existed, but were embedded in the plugin cli class, as
  an extra layer between the cli code and the code running for each command.
2016-01-31 20:30:45 -08:00
Ryan Ernst b8f08c35ec Plugin: Remove multicast plugin
closes #16310.
2016-01-29 18:41:31 -08:00
Tal Levy 61e4283a16 Add processor tags to on_failure metadata in ingest pipeline
closes #16202
2016-01-29 14:36:11 -08:00
Simon Willnauer e24fac644a Fix AzureRepositoryF to handle exceptions on close
Fix TribeUnitTests to handle exceptions on close
2016-01-29 17:34:02 +01:00
Jack Conradson 5b836dbb11 Renamed the scripting language Plan A to Painless.
Closes #16245
2016-01-27 10:37:34 -08:00
Simon Willnauer bbba1e5d7f Convert `config.ignore_system_properties` to new settings infra 2016-01-27 16:10:06 +01:00
javanna 533af17068 Tribe node: pass path.conf to inner tribe clients
If we don't do this, and some path.conf is set when starting the tribe node, that path.conf will be ignored and the inner tribe clients will try to read elsewhere, where they most likely don't have permissions to read from.

Closes #16253
Closes #16258
2016-01-27 13:19:56 +01:00
Jason Tedor 087e55cc51 Script mode settings
This commit converts the script mode settings to the new settings
infrastructure. This is a major refactoring of the handling of script
mode settings. This refactoring is necessary because these settings are
determined at runtime based on the registered script engines and the
registered script contexts.
2016-01-27 06:26:04 -05:00
javanna 61630c2b27 migrate node.local and node.mode to new Setting infra 2016-01-26 14:40:46 +01:00
javanna a306836e1a adapt to upstream changes, mustache supports now lists and arrays 2016-01-25 18:01:25 +01:00
javanna 36d98478bf Merge branch 'master' into feature/ingest 2016-01-25 18:01:09 +01:00
Yannick Welsch 296b48b9d1 Move discovery.* settings to new setting infrastructure
Closes #16182
2016-01-22 15:35:00 +01:00
Daniel Mitterdorfer e9bb3d31a3 Convert "path.*" and "pidfile" to new settings infra 2016-01-22 15:14:13 +01:00
Ryan Ernst df24019261 Merge pull request #16038 from rjernst/remove_site_plugin
Plugins: Remove site plugins
2016-01-21 12:32:22 -08:00
Martijn van Groningen 44465c94f8 Merge remote-tracking branch 'es/master' into feature/ingest 2016-01-21 11:46:27 +01:00
Nik Everett 3178d24bea Fix tests I just broke 2016-01-19 18:27:52 -05:00
Nik Everett 2cb7e8ce76 Use our standard xlint with standalone-test
We were not changing the xlint settings there at all. Also cleans up some
generic array warnings that this found by switching them to an ArrayList.
2016-01-19 17:54:47 -05:00
Martijn van Groningen 602a0f183e Merge remote-tracking branch 'es/master' into feature/ingest 2016-01-19 22:01:38 +01:00
Nik Everett 0d6d77328d Merge branch 'master' into feature/reindex 2016-01-19 13:27:24 -05:00
Ryan Ernst ef4f0a8699 Test: Make rest test framework accept http directly for the test cluster
The rest test framework, because it used to be tightly integrated with
ESIntegTestCase, currently expects the addresses for the test cluster to
be passed using the transport protocol port. However, it only uses this
to then find the http address.

This change makes ESRestTestCase extend from ESTestCase instead of
ESIntegTestCase, and changes the sysprop used to tests.rest.cluster,
which now takes the http address.

closes #15459
2016-01-18 16:44:14 -08:00
Ryan Ernst 3b78267c71 Plugins: Remove site plugins
Site plugins used to be used for things like kibana and marvel, but
there is no longer a need since kibana (and marvel as a kibana plugin)
uses node.js. This change removes site plugins, as well as the flag for
jvm plugins. Now all plugins are jvm plugins.
2016-01-16 22:45:37 -08:00
Nik Everett 645b053061 Merge branch 'master' into feature/reindex 2016-01-15 14:04:37 -05:00
Martijn van Groningen f3883343cb Move the pipeline configuration from the dedicated index to the cluster state.
Closes #15842
2016-01-13 22:59:36 +01:00
Nik Everett 85797aeb50 Basic reindex and update_by_query
This creates an reindex plugin with a very basic implementation that is
very like delete-by-query. New we'll integrate it with the task managament
work but for now this works.
2016-01-13 09:45:49 -05:00
javanna b111c7bbd5 redirect ingest requests to an ingest node
Rather than failing the request, when a node with node.ingest set to false receives an index or bulk request with a pipeline id, it should try to redirect the request to another node with node.ingest set to true. If there are no node with ingest set to true based on the current cluster state, an exception will be returned and the request will fail. Note that in case there are no ingest nodes and bulk has a pipeline id  specified only for a subset of index requests, the whole bulk will fail.
2016-01-12 12:31:11 +01:00
javanna bac1206161 remove use of request headers/context for pipeline id in favour of instance members added to IndexRequest and BulkRequest
Now that the ingest infra is part of es core we can remove some code that was required by the plugin and have a better integration with es core. We allow to specify the pipeline id in bulk and index as a request parameter, we have a REST filter that parses it and adds it to the relevant action request. That is not required anymore, as we can add this logic to RestIndexAction and RestBulkAction directly, no need for a filter. Also, we can allow to specify a pipeline id for each index requests in a bulk request. The small downside of this is that the ingest filter has to go over each item of a bulk request, all the time, to figure out whether they have a pipeline id.
2016-01-11 18:11:15 +01:00
Martijn van Groningen cd2155311f renamed ingest plugin to ingest-geoip plugin, since it only contains the geoip processor 2016-01-08 22:50:54 +01:00
javanna 8675784e3f adapt error message when node.ingest is set to false and we fail executing a pipeline 2016-01-08 10:39:39 +01:00
javanna ae69d46f92 move processors that have no deps to core, also move to core rest spec and tests and set node.inget to true by default 2016-01-08 10:39:39 +01:00
javanna 03fe38681e renamed qa package o.e.plugin.ingest to o.e.ingest
This way InternalTemplateService constructor can be set back to package private visibility
2016-01-07 15:51:52 +01:00
javanna 18aabd67c8 adapt qa tests for when ingest.node is set to false
CRUD and simulate apis work now fine, every node has the pipelines in memory, but node.ingest disables ingestion, meaning that any index or bulk request with a pipeline id is going to fail
2016-01-07 13:21:06 +01:00
javanna 9079a7e891 wip: move all the ingest infra to core 2016-01-06 19:10:44 +01:00
javanna 94469d75f9 revert rename InternalTemplateService -> MustacheTemplateService 2016-01-06 17:31:39 +01:00
javanna 635b9b5a46 clarified TemplateService comments
We will keep this abstractions as it's convenient, otherwise IngestDocument would depend on ScriptService directly, and would explicitly rely on mustache which is not even part of core. better to have the interface in core, and the impl as part of the ingest plugin, which relies on mustache, shipped with core by default.
2016-01-06 17:31:39 +01:00
javanna 2478aafa46 move ingest api to core 2016-01-06 17:31:39 +01:00
Martijn van Groningen e275af8a58 Merge remote-tracking branch 'es/master' into feature/ingest 2016-01-06 17:28:08 +01:00
javanna 1e8995d984 move all the processors under the same package org.elasticsearch.ingest.processor 2016-01-06 12:36:28 +01:00
Tal Levy f34ce9ddf4 add on_failure context to ingest metadata during executeOnFailure 2016-01-04 11:24:27 -08:00
Lee Hinman 55e6cd50e8 [TEST] Use a longer timeout for plugin manager in test 2015-12-29 10:07:31 -07:00
Robert Muir 180ab2493e Improve thirdPartyAudit check, round 3 2015-12-28 22:38:55 -05:00
Martijn van Groningen dbbb296322 added a `node.ingest` setting that controls whether ingest is active or not. Defaults to `false`.
If `node.ingest` isn't active then ingest related API calls fail and if the `pipeline_id` parameter is set then index and bulk requests fail.
2015-12-22 22:38:49 +01:00
Tal Levy 44d64c8a45 rename pipeline_id param to pipeline 2015-12-22 12:30:04 -08:00
javanna 46f99a11a0 Add append processor
The append processor allows to append one or more values to an existing list; add a new list with the provided values if the field doesn't exist yet, or convert an existing scalar into a list and add the provided values to the newly created  list.

This required adapting of IngestDocument#appendFieldValue behaviour, also added support for templating to it.

Closes #14324
2015-12-22 16:11:43 +01:00
javanna 1b7dc45c28 adapt to upstream changes: remove wildcard imports from qa/ingest-with-mustache 2015-12-22 13:42:15 +01:00
javanna f214271d89 Merge branch 'master' into feature/ingest 2015-12-22 11:14:55 +01:00
Ryan Ernst 9cb4c82c58 Build: Add fixture capabilities to integ tests
This change adds a Fixture class for use by gradle. A Fixture is an
external process that integration tests will use. It can be added as a
dependsOn for integTest, and will automatically be shutdown upon success
or failure, as well as relevant information dumped on failure. There is
also an example fixture in this change.
2015-12-19 15:46:21 -08:00
Ryan Ernst 4ea19995cf Remove wildcard imports 2015-12-18 12:43:47 -08:00
Martijn van Groningen e8a8e22e09 Add template infrastructure, removed meta processor and added template support to set and remove processor.
Added ingest wide template infrastructure to IngestDocument
Added a TemplateService interface that the ingest framework uses
Added a TemplateService implementation that the ingest plugin provides that delegates to the ES' script service
Cut SetProcessor over to use the template infrastructure for the `field` and `value` settings.
Removed the MetaDataProcessor
Removed dependency on mustache library
Added qa  ingest mustache rest test so that the ingest and mustache integration can be tested.
2015-12-18 17:35:53 +01:00
Robert Muir 6692e42d9a thirdPartyAudit round 2
This fixes the `lenient` parameter to be `missingClasses`. I will remove this boolean and we can handle them via the normal whitelist.
It also adds a check for sheisty classes (jar hell with the jdk).
This is inspired by the lucene "sheisty" classes check, but it has false positives. This check is more evil, it validates every class file against the extension classloader as a resource, to see if it exists there. If so: jar hell.

This jar hell is a problem for several reasons:

1. causes insanely-hard-to-debug problems (like bugs in forbidden-apis)
2. hides problems (like internal api access)
3. the code you think is executing, is not really executing
4. security permissions are not what you think they are
5. brings in unnecessary dependencies
6. its jar hell

The more difficult problems are stuff like jython, where these classes are simply 'uberjared' directly in, so you cant just fix them by removing a bogus dependency. And there is a legit reason for them to do that, they want to support java 1.4.
2015-12-17 02:35:00 -05:00
Robert Muir 42138007db add some more comments about internal api usage 2015-12-16 18:56:02 -05:00
Robert Muir ee79d46583 Add gradle thirdPartyAudit to precommit tasks 2015-12-16 16:38:16 -05:00
Robert Muir 2d42e99c7a smoke test plugins does not test any plugins
Currently the build has a bug and it loads 0 plugins.
2015-12-15 00:05:14 -05:00
Ryan Ernst a0c69fe7f9 Remove forbidden suppressions for InetSocketAddress 2015-12-11 18:20:09 -08:00
Ryan Ernst 5c8a0da1fd Build: Change rest integ tests to not have hardcoded ports
This change removes hardcoded ports from cluster formation. It passes
port 0 for http and transport, and then uses a special property to have
the node log the ports used for http and transport (just for tests).
This does not yet work for multi node tests. This brings us one step
closer to working with --parallel.
2015-12-11 17:36:08 -08:00
Adrien Grand 0f51d81410 Fix settings for TribeUnitTests. 2015-12-11 20:00:38 +01:00
Jason Tedor 3383c24be0 Remove and forbid use of Collections#shuffle(List) and Random#<init>()
This commit removes and now forbids all uses of
Collections#shuffle(List) and Random#<init>() across the codebase. The
rationale for removing and forbidding these methods is to increase test
reproducibility. As these methods use non-reproducible seeds, production
code and tests that rely on these methods contribute to
non-reproducbility of tests.

Instead of Collections#shuffle(List) the method
Collections#shuffle(List, Random) can be used. All that is required then
is a reproducible source of randomness. Consequently, the utility class
Randomness has been added to assist in creating reproducible sources of
randomness.

Instead of Random#<init>(), Random#<init>(long) with a reproducible seed
or the aforementioned Randomess class can be used.

Closes #15287
2015-12-11 11:16:38 -05:00
Ryan Ernst 6f166dd0ff Remove NodeBuilder
The NodeBuilder is currently used to construct a Node. However, this is
really just yet-another-builder that wraps around a Settings.Builder
witha couple convenience methods. But there are very few uses of these
convenience methods.  This change removes NodeBuilder, in favor of just
using the Node constructor.
2015-12-10 00:01:44 -08:00
Jack Conradson da5b07ae13 Added a new scripting language (PlanA).
Closes #15136
2015-12-09 16:32:37 -08:00
Ryan Ernst 25d60e152f Tribe: Fix tribe node to load config file for internal client nodes
The tribe node creates one local client node for each cluster it
connects to. Refactorings in #13383 broke this so that each local client
node now tries to load the full elasticsearch.yml that the real tribe
node uses.

This change fixes the problem by adding a TribeClientNode which is a
subclass of Node. The Environment the node uses is now passed in (in
place of Settings), and the TribeClientNode simply does not use
InternalSettingsPreparer.prepareEnvironment.

The tests around tribe nodes are not great. The existing tests pass, but
I also manually tested by creating 2 local clusters, and configuring and
starting a tribe node. With this I was able to see in the logs the tribe
node connecting to each cluster.

closes #13383
2015-12-07 20:18:06 -08:00
Ryan Ernst 3d9d8bd45a Build: Remove duplicate runs of packaged rest tests
We currently use the full suite of packaged rest tests for each
distribution. We also used to run rest tests within core integ tests,
but this stopped working when we split out the test-framework, since the
test files are in there.

This change simplifies the code to run packaged rest tests just once,
for the integ-test-zip, and removes the unused rest tests from
test-framework. Distributions rest tests now check that all modules
were loaded.
2015-12-04 13:43:40 -08:00
Robert Muir 784ebb1e1b fix bats tests 2015-12-04 03:25:43 -05:00
Ryan Ernst f5b6b40a92 Fix distros to put modules dir int he right place and qa rest tests to
use the real zip
2015-12-03 23:03:14 -08:00
Ryan Ernst 7245d34f51 add modules check to plugins smoke test 2015-12-03 22:48:51 -08:00
Robert Muir 699b140d69 update lists, rename stuff to be clearer, remove public method that is unused 2015-12-04 01:43:07 -05:00
Robert Muir 88c538879b add test for modules/ 2015-12-03 22:37:27 -05:00
Robert Muir ee827198ff set ActiveProcessLimit=1 on windows
This blocks process creation, similar to the approach for BSD.
2015-11-27 01:50:09 -05:00
Ryan Ernst 1e0f929281 Build: Simplify adding plugins that are another project in the build
The current mechanism for adding plugins to the integTest cluster is to
have a FileCollection. This works well for the integTests for a single
plugin, which automatically adds itself to be installed. However, for qa
tests where many plugins may be installed, and from other projects, it
is cumbersome to add configurations, dependencies and dependsOn
statements over and over. This simplifies installing a plugin from
another project by moving this common setup into the cluster
configuration code.
2015-11-24 12:53:11 -08:00
Ryan Ernst 7d3da7e6ee Build: Get multi node smoke tests working
This change adds back the multi node smoke test, as well as making the
cluster formation for any test allow multiple nodes. The main changes in
cluster formation are abstracting out the node specific configuration to
a helper struct, as well as making a single wait task that waits for all
nodes after their start tasks have run. The output on failure was also
improved to log which node's info is being printed.
2015-11-22 14:49:05 -08:00
Ryan Ernst b8e462b934 Build: Change project attachment into special extra-plugins dir
Currently we use the "gradle project attachment plugin" to support
building elasticsearch as part of another project. However, this plugin
has a number of issues, a large part of which is requiring consistent
use of the projectsPrefix.

This change removes projectsPrefix, and adds support for a special
extra-plugins directory in the root of elasticsearch. Any projects
checked out within this directory will be automatically added to
elasticsearch.
2015-11-22 08:44:33 -08:00
Robert Muir 30529c008d Ban write access to system properties
* Forbid System.setProperties & co in forbidden APIs.
* Ban property write access at runtime with security manager.

Plugins that need to modify system properties will need to request permission in their plugin-security.policy
2015-11-21 22:33:06 -05:00
Nik Everett 9b0a47d8e3 Fix rpm and deb distributions
and test them with vagrant
2015-11-18 14:16:42 -05:00
Robert Muir 776bb288b5 fix gradle check under jigsaw
closes #14726

Squashed commit of the following:

commit 5b591e98570e3fa481b2816a44063b98bff36ddf
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Nov 13 00:54:08 2015 -0500

    add assumption for self-signing in PluginManagerTests

commit ed11e5371b6f71591dc41c6f60d033502cfcf029
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Nov 13 00:20:59 2015 -0500

    show error output from integ test startup

commit d8b187a10e95d89a0e775333dcbe1aaa903fb376
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Nov 12 22:14:11 2015 -0500

    fix gradle check under jigsaw
2015-11-13 00:57:12 -05:00
Robert Muir 720ebe347d Handle system policy correctly.
Just suck in the system policy, so its compatible with any version of java.
It means it also respects configuration (e.g. for monitoring agents)

Closes #14704
2015-11-12 18:08:13 -05:00
Robert Muir e88896e40e apply a few post-merge cleanups 2015-11-08 02:31:26 -05:00
Ryan Ernst 9ee315a9c8 Build: Improve integ test to match ant behavior
Many other improvements:
* Use spaces in ES path
* Use space in path for plugin file installation
* Use a different cwd than ES home
* Use jps to ensure process being stopped is actually elasticsearch
* Stop ES if pid file already exists
* Delete pid file when successfully killed

Also, refactored the cluster formation code to be a little more organized.

closes #14464
2015-11-07 00:45:19 -08:00
Ryan Ernst 634c495e3d Build: Use projectsPrefix in project references for vagrant tests
It is important we use projectsPrefix whenever addinga direct project
dependency, so that attachment still works.
2015-11-06 10:52:19 -08:00
Nik Everett 047e3b9b69 Merge pull request #14531 from nik9000/gradle_vagrant
Gradle-ify vagrant tests
2015-11-05 14:29:06 -05:00
Nik Everett 1dd00dddd5 [test] Gradle-ify vagrant tests
This gets the tar and tar_plugins tests working in gradle. It does so by
adding a subproject, qa/vagrant, which adds the following tasks:

Verification
------------
checkPackages - Check the packages against a representative sample of the
                linux distributions we have in our Vagrantfile
checkPackagesAllDistros - Check the packages against all the linux
                          distributions we have in our Vagrantfile

Package Verification
--------------------
checkCentos6 - Run packaging tests against centos-6
checkCentos7 - Run packaging tests against centos-7
checkDebian8 - Run packaging tests against debian-8
checkFedora22 - Run packaging tests against fedora-22
checkOel7 - Run packaging tests against oel-7
checkOpensuse13 - Run packaging tests against opensuse-13
checkSles12 - Run packaging tests against sles-12
checkUbuntu1204 - Run packaging tests against ubuntu-1204
checkUbuntu1404 - Run packaging tests against ubuntu-1404
checkUbuntu1504 - Run packaging tests against ubuntu-1504

Vagrant
-------
smokeTestCentos6 - Smoke test the centos-6 VM
smokeTestCentos7 - Smoke test the centos-7 VM
smokeTestDebian8 - Smoke test the debian-8 VM
smokeTestFedora22 - Smoke test the fedora-22 VM
smokeTestOel7 - Smoke test the oel-7 VM
smokeTestOpensuse13 - Smoke test the opensuse-13 VM
smokeTestSles12 - Smoke test the sles-12 VM
smokeTestUbuntu1204 - Smoke test the ubuntu-1204 VM
smokeTestUbuntu1404 - Smoke test the ubuntu-1404 VM
smokeTestUbuntu1504 - Smoke test the ubuntu-1504 VM
vagrantHaltCentos6 - Shutdown the vagrant VM running centos-6
vagrantHaltCentos7 - Shutdown the vagrant VM running centos-7
vagrantHaltDebian8 - Shutdown the vagrant VM running debian-8
vagrantHaltFedora22 - Shutdown the vagrant VM running fedora-22
vagrantHaltOel7 - Shutdown the vagrant VM running oel-7
vagrantHaltOpensuse13 - Shutdown the vagrant VM running opensuse-13
vagrantHaltSles12 - Shutdown the vagrant VM running sles-12
vagrantHaltUbuntu1204 - Shutdown the vagrant VM running ubuntu-1204
vagrantHaltUbuntu1404 - Shutdown the vagrant VM running ubuntu-1404
vagrantHaltUbuntu1504 - Shutdown the vagrant VM running ubuntu-1504
vagrantSmokeTest - Smoke test some representative distros from the Vagrantfile
vagrantSmokeTestAllDistros - Smoke test all distros from the Vagrantfile
vagrantUpCentos6 - Startup a vagrant VM running centos-6
vagrantUpCentos7 - Startup a vagrant VM running centos-7
vagrantUpDebian8 - Startup a vagrant VM running debian-8
vagrantUpFedora22 - Startup a vagrant VM running fedora-22
vagrantUpOel7 - Startup a vagrant VM running oel-7
vagrantUpOpensuse13 - Startup a vagrant VM running opensuse-13
vagrantUpSles12 - Startup a vagrant VM running sles-12
vagrantUpUbuntu1204 - Startup a vagrant VM running ubuntu-1204
vagrantUpUbuntu1404 - Startup a vagrant VM running ubuntu-1404
vagrantUpUbuntu1504 - Startup a vagrant VM running ubuntu-1504

It does not make the "check" task depend on "checkPackages" so running the
vagrant tests is still optional. They are slow and depend on vagrant and
virtualbox.

The Package Verification tasks are useful for testing individual distros.

The Vagrant tasks are listed in `gradle tasks` primarily for discoverability.
2015-11-05 14:28:30 -05:00
Robert Muir 3d8b4dae33 remove the ability for code to change file permissions, this was
only needed for pluginmanager's test, and pluginmanager doesn't
even run with securitymanager yet.
2015-11-04 00:31:15 -05:00
Robert Muir 6be9954d28 remove jimfs dep in tests 2015-11-03 23:29:14 -05:00
Robert Muir 602feac915 move tests never running in jenkins to new evil tests module 2015-11-03 21:42:22 -05:00
Ryan Ernst 542522531a Build: Remove maven pom files and supporting ant files
This change removes the leftover pom files. A couple files were left for
reference, namely in qa tests that have not yet been migrated (vagrant
and multinode). The deb and rpm assemblies also still exist for
reference when finishing their setup in gradle.

See #13930
2015-10-29 23:53:49 -07:00
Ryan Ernst c86100f636 Switch build system to Gradle
See #13930
2015-10-29 11:40:19 -07:00
Nik Everett 2cc97a0d3e Remove and ban @Test
There are three ways `@Test` was used. Way one:

```java
@Test
public void flubTheBlort() {
```

This way was always replaced with:

```java
public void testFlubTheBlort() {
```

Or, maybe with a better method name if I was feeling generous.

Way two:

```java
@Test(throws=IllegalArgumentException.class)
public void testFoo() {
    methodThatThrows();
}
```

This way of using `@Test` is actually pretty OK, but to get the tools to ban
`@Test` entirely it can't be used. Instead:

```java
public void testFoo() {
    try {
        methodThatThrows();
        fail("Expected IllegalArgumentException");
    } catch (IllegalArgumentException e ) {
        assertThat(e.getMessage(), containsString("something"));
    }
}
```

This is longer but tests more than the old ways and is much more precise.
Compare:

```java
@Test(throws=IllegalArgumentException.class)
public void testFoo() {
    some();
    copy();
    and();
    pasted();
    methodThatThrows();
    code();  // <---- This was left here by mistake and is never called
}
```

to:

```java
@Test(throws=IllegalArgumentException.class)
public void testFoo() {
    some();
    copy();
    and();
    pasted();
    try {
        methodThatThrows();
        fail("Expected IllegalArgumentException");
    } catch (IllegalArgumentException e ) {
        assertThat(e.getMessage(), containsString("something"));
    }
}
```

The final use of test is:

```java
@Test(timeout=1000)
public void testFoo() {
    methodThatWasSlow();
}
```

This is the most insidious use of `@Test` because its tempting but tragically
flawed. Its flaws are:
1. Hard and fast timeouts can look like they are asserting that something is
faster and even do an ok job of it when you compare the timings on the same
machine but as soon as you take them to another machine they start to be
invalid. On a slow VM both the new and old methods fail. On a super-fast
machine the slower and faster ways succeed.
2. Tests often contain slow `assert` calls so the performance of tests isn't
sure to predict the performance of non-test code.
3. These timeouts are rude to debuggers because the test just drops out from
under it after the timeout.

Confusingly, timeouts are useful in tests because it'd be rude for a broken
test to cause CI to abort the whole build after it hits a global timeout. But
those timeouts should be very very long "backstop" timeouts and aren't useful
assertions about speed.

For all its flaws `@Test(timeout=1000)` doesn't have a good replacement __in__
__tests__. Nightly benchmarks like http://benchmarks.elasticsearch.org/ are
useful here because they run on the same machine but they aren't quick to check
and it takes lots of time to figure out the regressions. Sometimes its useful
to compare dueling implementations but that requires keeping both
implementations around. All and all we don't have a satisfactory answer to the
question "what do you replace `@Test(timeout=1000)`" with. So we handle each
occurrence on a case by case basis.

For files with `@Test` this also:
1. Removes excess blank lines. They don't help anything.
2. Removes underscores from method names. Those would fail any code style
checks we ever care to run and don't add to readability. Since I did this manually
I didn't do it consistently.
3. Make sure all test method names start with `test`. Some used to end in `Test` or start
with `verify` or `check` and they were picked up using the annotation. Without the
annotation they always need to start with `test`.
4. Organizes imports using the rules we generate for Eclipse. For the most part
this just removes `*` imports which is a win all on its own. It was "required"
to quickly remove `@Test`.
5. Removes unneeded casts. This is just a setting I have enabled in Eclipse and
forgot to turn off before I did this work. It probably isn't hurting anything.
6. Removes trailing whitespace. Again, another Eclipse setting I forgot to turn
off that doesn't hurt anything. Hopefully.
7. Swaps some tests override superclass tests to make them empty with
`assumeTrue` so that the reasoning for the skips is logged in the test run and
it doesn't "look like" that thing is being tested when it isn't.
8. Adds an oxford comma to an error message.

The total test count doesn't change. I know. I counted.
```bash
git checkout master && mvn clean && mvn install | tee with_test
git no_test_annotation master && mvn clean && mvn install | tee not_test
grep 'Tests summary' with_test > with_test_summary
grep 'Tests summary' not_test > not_test_summary
diff with_test_summary not_test_summary
```

These differ somewhat because some tests are skipped based on the random seed.
The total shouldn't differ. But it does!
```
1c1
< [INFO] Tests summary: 564 suites (1 ignored), 3171 tests, 31 ignored (31 assumptions)
---
> [INFO] Tests summary: 564 suites (1 ignored), 3167 tests, 17 ignored (17 assumptions)
```

These are the core unit tests. So we dig further:
```bash
cat with_test | perl -pe 's/\n// if /^Suite/;s/.*\n// if /IGNOR/;s/.*\n// if /Assumption #/;s/.*\n// if /HEARTBEAT/;s/Completed .+?,//' | grep Suite > with_test_suites
cat not_test | perl -pe 's/\n// if /^Suite/;s/.*\n// if /IGNOR/;s/.*\n// if /Assumption #/;s/.*\n// if /HEARTBEAT/;s/Completed .+?,//' | grep Suite > not_test_suites
diff <(sort with_test_suites) <(sort not_test_suites)
```

The four tests with lower test numbers are all extend `AbstractQueryTestCase`
and all have a method that looks like this:

```java
@Override
public void testToQuery() throws IOException {
    assumeTrue("test runs only when at least a type is registered", getCurrentTypes().length > 0);
    super.testToQuery();
}
```

It looks like this method was being double counted on master and isn't anymore.

Closes #14028
2015-10-20 17:37:36 -04:00
javanna 736823163f Plugins: plugin script to set proper plugin bin dir attributes
This commit makes sure that the plugin script looks at user, group and permissions of the elasticsearch bin dir and copies them over to the plugin bin subdirectory, whatever they are, so that they get properly setup depending on how elasticsearch was installed. We also make sure that execute permissions are added for files (we already did this before).

Relates to #11016
Closes #14088
2015-10-14 14:59:38 +02:00
javanna 4d7d29f65b Plugins: plugin script to set proper plugin config dir attributes
Depending on how elasticsearch is installed, we have two scenarios to take into account that relate to user, group and permissions assigned to the config directory:

1) deb/rpm package: /etc/elasticsearch is root:elasticsearch 750 and the plugin script is run from root user
2) tar/zip archive: es config dir is most likely elasticsearch:elasticsearch and the plugin script is most likely run from elasticsearch user

When the plugin script copies over the plugin config dir within the es config dir, it should take care of setting the proper user, group and permissions, which vary depending on how elasticsearch was installed in the first place. Should be root:elasticsearch 750 if installed from a package, or elasticsearch:elasticsearch if installed from an archive.

This commit makes sure that the plugin script looks at user, group and permissions of the config dir and copies them over to the plugin config subdirectory, whatever they are, so that they get properly setup depending on how elasticsearch was installed in the first place. We also make sure that execute permissions are left untouched for files.

Relates to #11016
Closes #14048
2015-10-13 15:56:26 +02:00
Jason Tedor 9a9a6a4b3b Remove Guava as a dependency
This commit removes Guava as a dependency. Note that Guava will remain
as a test-only dependency (transitively through Jimfs).

Closes #13224
2015-10-09 14:19:22 -04:00
javanna 648cc6defa Packaging: change permissions/ownership of config dir
When generating the rpm and dep package we now set proper group (elasticsearch) and permissions (750) to the conf dir (default /etc/elasticsearch). Same for the scripts subdirectory.

Expanded the assert_file bash function to also optionally check the group of files, so we can actually test that the group was set correctly.

Relates to #11016
Closes #14017
2015-10-08 15:40:12 +02:00
David Pilato c73ab50df1 Rename cloud-gce plugin to discovery-gce plugin
Follow up azure and aws splits, we need to be consistent and rename `cloud-gce` to `discovery-gce`.
2015-10-08 06:53:37 +02:00
Britta Weber 2473e7499b name variables and don't always use the timeout 2015-10-06 20:37:44 +02:00
Britta Weber 71aefd5a06 remove option to configure custom config file via CONF_FILE or -Des.default.conf
It is rarely used and was not consistently handled by different distributions anyway.
This commit also adds a test for specifying CONF_DIR when installing plugins and
starting elasticsearch.

relates to #12712 and #12954
closes #5329
closes #13715
2015-10-06 19:02:43 +02:00
Britta Weber aa19a4134d vagrant tests: fix plugin outupt test
package installation creates the plugin directory already so when a plugin
is installed it prints the additional line
Plugins directory [/tmp/elasticsearch/plugins] does not exist. Creating...
2015-10-06 17:37:25 +02:00
Nik Everett e71bc472cc Merge pull request #13821 from nik9000/jayatana_test
Test that Jayatana is ignored
2015-10-06 09:14:28 -04:00
Britta Weber ed7c051586 Merge pull request #13861 from brwe/vagrant-install-groovy
install groovy plugin before running script test
2015-10-06 14:18:36 +02:00
Britta Weber 9492be65d4 plugin cli tool should not create empty log files
Plugin cli tools configures logging with whatever is in the logging.yml.
If a file appender is configured for any of the logs this will cause creation
of an empty log file. If a plugin was for example installed as root it will
create empty logs at es.home/logs.
This is problematic when for example plugins are installed as root and es is run
as service. Logs will then be created in /usr/share/elasticsearch/logs
and can later not be removed by for example dpkg -r or -purge.

To avoid this, configure the logger to use an appender that writes to the same
output that plugin cli tool does. This allows other components that are called
from Plugin cli tool to write to the same terminal that plugin cli tool writes to
by using the logging mechanism already in place.
The logging conf is not read at all pb plugin cli tool.

As a side effect, the loging level for components that are called
from the plugin command such as the jar hell check can now be configured
with -Des.logger.level which makes it easier to debug the jar hell check.
2015-10-06 14:13:24 +02:00
Britta Weber b4350e31be install groovy plugin before running script test
groovy moved to a plugin but the tests rely on it
see #13834
2015-10-05 00:23:48 +02:00
Britta Weber 1a67440989 add lang-groovy to plugin vagrant test 2015-09-29 17:56:29 +02:00
Robert Muir e0d42739dd Factor groovy out of core into lang-groovy 2015-09-28 20:17:45 -04:00
Nik Everett e45b2f7f0c [test] Jayatana is ignored
Installs javatana in vivid, emulates its on-login actions when starting
elasticsearch and verifies that elasticsearch turns off javatana.

Relates to #13813
2015-09-28 03:33:05 +02:00
Britta Weber d8ff91dbad [test] add lang-expression plugin to vagrant tests
we need this now that #13726 has been merged
2015-09-23 17:55:21 +02:00
Robert Muir 689af1a6d6 Factor expressions scripts out to lang-expression plugin 2015-09-22 20:33:47 -04:00
Nik Everett 4d47015a0c Merge pull request #13579 from nik9000/run_plugin_as_elasticsearch_in_tar
Run bin/plugin as elasticsearch in tar distro
2015-09-22 11:19:19 -04:00
Nik Everett b521c606ff [test] Run bin/plugin as right user
Before this commit he tests always run bin/plugin as root which is somewhat
unrealistic and causes trouble (log files owned by root instead of
elasticsearch). After this commit `bin/plugin` runs as root when elasticsearch
is installed via the repository and as elasticsearch otherwise which is much
more realistic.

This also adds extra timeout to starting elasticsearch which is required
when all the plugins are installed. And it fixes up a problem with logging
elasticsearch's log if elasticsearch doesn't start which came up multiple
time while debugging this problem.

Also adds docs recommending running `bin/plugin` as the user that owns the
Elasticsearch files or root if installed with the packages.

Closes #13557
2015-09-22 11:17:57 -04:00
Alexander Reelsen 2b46a72a6c BATS testing: Add SLES-12 to list of tested virtual machines
This adds SuSe Linux Enterprise Server 12 to the list of tested VMs.
SLES 12 is using systemd, so that the current RPM works
out of the box.

SLES12 however is already quite old and does not ship with java8, so this
required adding an opensuse repo.
2015-09-22 15:02:22 +02:00
Nik Everett a6c8ac536f Merge pull request #13696 from nik9000/fix_azure_vagrant
Fix plugin tests
2015-09-22 09:01:45 -04:00
Ryan Ernst de2566a023 Merge pull request #13704 from rjernst/more_rest_spec_cleanup
Move a couple more rest-api-spec resource dirs into resources
2015-09-21 22:10:58 -07:00
Ryan Ernst 093791e82d Test: Move a couple more rest-api-spec resource dirs into resources
A couple were left behind. This fixes them and cleans up the test
resources for qa tests a little bit.
2015-09-21 21:57:51 -07:00
Robert Muir f401350a97 Fix straggler in qa tests 2015-09-21 23:43:24 -04:00