Commit Graph

483 Commits

Author SHA1 Message Date
Jason Tedor 83338686b9 Remove stale install plugins assertion
When installing a plugin when the plugins directory does not exist, the
install plugin command outputs a line saying that it is creating this
directory. The packaging tests for the archive distributions accounted
for this including an assertion that this line was output. The packages
have since been updated to include an empty plugins folder, so this line
will no longer be output. This commit removes this stale assertion from
the packaging tests.

Relates #21275
2016-11-02 13:07:26 -04:00
Jason Tedor 8dd91eb2d6 Add empty plugins dir for archive distributions
Today when installing Elasticsearch from an archive distribution (tar.gz
or zip), an empty plugins folder is not included. This means that if you
install Elasticsearch and immediately run elasticsearch-plugin list, you
will receive an error message about the plugins directory missing. While
the plugins directory would be created when starting Elasticsearch for
the first time, it would be better to just include an empty plugins
directory in the archive distributions. This commit makes this the
case. Note that the package distributions already include an empty
plugins folder.

Relates #21204
2016-11-01 15:14:24 -04:00
Jason Tedor 344a8028f8 Add analysis-ukrainian plugin to packaging tests
This commit adds the analysis-ukrainian plugin to the packaging tests.

Relates #21219
2016-10-31 16:38:33 -04:00
Nik Everett f11a217546 Rewrite vagrant upgrade test's dependency resolution
This is cleaner and it doesn't fail in Jenkins. I'm not sure
why the old way failed in jenkins but not locally, but computers
are basically magic so I take what I can get. Here is a jenkins
failure:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+packaging-tests/238/consoleFull
2016-10-28 17:02:24 -04:00
Nik Everett 267a7b427b Fix the package upgrade tests for 5.0.0 (#21178)
5.0+'s deb and rpm packages aren't pushed to maven cnertral so
instead we have to download them from artifacts.elastic.org.
2016-10-28 14:00:43 -04:00
Nik Everett 9e3eacec35 Fix generation of versions to upgrade from
Vagrant tests use a static list of dependencies to upgrade from
and we weren't including 5.0.0 deps in that list. Also when the
list was incorrect we weren't sorting the "current" list so it
was difficult to read.

Also adds 2.4.1 to the list but *doesn't* add 5.0.0 because we
still can't resolve it yet. We still only print an error when
the list is wrong but don't abort the build. We'll abort the build
once we've fixed resolution for 5.0.0 and we can re-add it.
2016-10-28 10:45:36 -04:00
Nik Everett bce34e4f1e Disable out of date versions exception
We are upgrading from out of date versions in our tests right now and we
can't fix that because the current versions to upgrade from aren't in
maven central. We'll resolve the resolution issue soon, but for now
let's get the build green.
2016-10-28 09:47:47 -04:00
Ali Beyad 733d74229b Revert "Updates vagrant tests upgrade version number" (#21171) 2016-10-28 09:40:04 -04:00
Ali Beyad 557506ba6e Updates the versions the vagrant tests can upgrade from to 5.0.0 (#21151) 2016-10-27 16:44:04 -04:00
Tanguy Leroux 127b4a8efc Change permissions on config files (#20966)
This commit changes some default file permissions on configuration files.
2016-10-24 09:42:03 +02:00
Tanguy Leroux 268d5ba97a Add JVM option "-XX:-AssumeMP" in SystemD tests (#21063)
Since with j`ava-1.8.0-openjdk-1.8.0.111-1.b15.el7_2.x86_64`, the OpenJDK packaged for CentOS and OEL override the default value (`false`) for the JVM option `AssumeMP` and force it to `true` (see [this patch](https://git.centos.org/blob/rpms!!java-1.8.0-openjdk.git/ab03fcc7a277355a837dd4c8500f8f90201ea353/SOURCES!always_assumemp.patch))

Because it is forced to true by default for these packages, the following warning message is printed to the standard output when the Vagrant box has only 1 CPU:
> OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N

This message will then fail the test introduced in #20422  where we check if no entries have been added to the journal after the service has been started.

This commit restore the default value for the `AssumeMP` option for CentOS and OracleServer.
2016-10-24 09:30:55 +02:00
Tanguy Leroux 5495b66797 Mute journalctl output check in packaging tests
This commit mutes a check on the output of journalctl after the Elasticsearch's systemd service has been started. It expected no entries in the journal but since OpenJDK build 1.8.0_111-b15 the following warning message is printed:
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
2016-10-21 10:52:11 +02:00
Tal Levy 38c650f376 make painless the default scripting language for ScriptProcessor (#20981)
- fixes a bug in the docs that mentions `lang` as optional
- now `lang` defaults to "painless"
2016-10-18 16:22:01 -07:00
Boaz Leskes c3987156ab Remove local discovery in favor of a simpler `MockZenPings` (#20960)
`LocalDiscovery` is a discovery implementation that uses static in memory maps to keep track of current live nodes. This is used extensively in our tests in order to speed up cluster formation (i.e., shortcut the 3 second ping period used by `ZenDiscovery` by default). This is sad as that mean that most of the test run using a different discovery semantics than what is used in production. Instead of replacing the entire discovery logic, we can use a similar approach to only shortcut the pinging components.
2016-10-18 21:12:15 +02:00
Dimitrios Liappis b0bb72ccd7 Deprecate EOL'ed Ubuntu 15.04 from Vagrantfile
* Deprecate EOL'ed Ubuntu 15.04 from Vagrantfile

Relates pr #20921
2016-10-17 17:27:01 +03:00
Jason Tedor 05adc14231 Revert "Display plugins versions"
This reverts commit 9411f18f27.

Relates #20807
2016-10-11 11:59:06 -04:00
Alexander Reelsen 2e447479c0 Tests: Fix ingest-attachment vagrant tests
The vagrant tests are checking for a certain jar version.

Relates #20710
2016-10-10 14:54:35 +02:00
Simon Willnauer 194a6b1df0 Remove LocalTransport in favor of MockTcpTransport (#20695)
This change proposes the removal of all non-tcp transport implementations. The
mock transport can be used by default to run tests instead of local transport that has
roughly the same performance compared to TCP or at least not noticeably slower.

This is a master only change, deprecation notice in 5.x will be committed as a
separate change.
2016-10-07 11:27:47 +02:00
Jason Tedor 51d53791fe Remove lenient URL parameter parsing
Today when parsing a request, Elasticsearch silently ignores incorrect
(including parameters with typos) or unused parameters. This is bad as
it leads to requests having unintended behavior (e.g., if a user hits
the _analyze API and misspell the "tokenizer" then Elasticsearch will
just use the standard analyzer, completely against intentions).

This commit removes lenient URL parameter parsing. The strategy is
simple: when a request is handled and a parameter is touched, we mark it
as such. Before the request is actually executed, we check to ensure
that all parameters have been consumed. If there are remaining
parameters yet to be consumed, we fail the request with a list of the
unconsumed parameters. An exception has to be made for parameters that
format the response (as opposed to controlling the request); for this
case, handlers are able to provide a list of parameters that should be
excluded from tripping the unconsumed parameters check because those
parameters will be used in formatting the response.

Additionally, some inconsistencies between the parameters in the code
and in the docs are corrected.

Relates #20722
2016-10-04 12:45:29 -04:00
Simon Willnauer 7e3863d2d8 [TEST] Fix EvilSystemPropertyTests to be test order independent 2016-09-29 13:26:14 +02:00
Simon Willnauer f2e6862803 Add a hard limit for `index.number_of_shard` (#20682)
this change adds a hard limit to `index.number_of_shard` that prevents
indices from being created that have more than 1024 shards. This is still
a huge limit and can only be changed via settings a system property.
2016-09-29 11:03:30 +02:00
Jason Tedor a6e33494ab Provide error message when plugin id is missing
Today when executing the install plugin command without a plugin id, we
end up throwing an NPE because the plugin id is null yet we just keep
going (ultimatley we try to lookup the null plugin id in a set, the
direct cause of the NPE). This commit modifies the install command so
that a missing plugin id is detected and help is provided to the user.

Relates #20660
2016-09-26 08:09:15 -04:00
Jay Modi 0573e03aa1 Pass classpath plugins to tribe nodes
When testing tribe nodes in an integration test, we should pass the classpath
plugins of the node down to the tribe client nodes. Without this the tribe client
nodes could be prevented from communicating with the tribes.
2016-09-22 07:29:07 -04:00
Jason Tedor 12234c067a Ensure logging is initialized in CLI tools
Today when CLI tools are executed, logging statements can intentionally
or unintentionally be executed when logging is not configured. This
leads to log messages that the status logger is not configured. This
commit reworks logging configuration for CLI tools so that logging is
always configured.

Relates #20575
2016-09-20 08:28:27 -04:00
Ali Beyad 50584c4103 Merge pull request #20532 from rjernst/rolling_upgrades
This PR introduces backward compatibility index tests to test the rolling upgrade process amongst Elasticsearch instances within the same major version. The test executes in three phases. In the first phase, we form a cluster of 2 ES instances on an old version. In the second phase, we keep one of the nodes from the old cluster, kill the other node, but preserve its data directory and start an instance of the current version of ES using the same data directory as the killed instance. In the third phase, we kill the other old version ES instance from the first phase and launch a new instance, using the same data directory as the killed instance. Therefore, during phase 3, we have fully migrated and have all current versions of ES running. In each phase, we run REST tests that index documents and search them, ensuring at each stage that the documents from the previous phase are still there.

Note that because we haven't released a GA yet of 5.0, the tests currently don't start an old version cluster in the first phase. Once GA is released, this will be changed to make the backward compatibility version 5.0, while the current version in the cluster will be 5.x.
2016-09-19 16:14:38 -04:00
Ali Beyad 8afc83047f Change the timeout of the rolling upgrades test from 40 mins to 5 mins
to still allow accounting for slow VMs
2016-09-19 15:45:41 -04:00
David Pilato dfd1eebdd0 Remove mapper attachments plugin
We now have in 5.0.0 `ingest-attachment` plugin.
We can remove `mapper-attachments` plugin for 6.0.

Closes #18837.
2016-09-19 09:01:16 +02:00
Ali Beyad 98230d035a Adds a preserveIndicesUponCompletion method to ESRestTestCase
that can be overridden by subclasses if the test must not
delete indices it created after exiting.
2016-09-16 19:21:26 -04:00
Ali Beyad 83adc87015 Removes stopNodeUponCompletion in favor of moving the stop
nodes task to the final part of the cluster test task execution
graph.
2016-09-16 16:32:43 -04:00
Ali Beyad 22e6bc8359 Disables unit tests for rolling upgrades, as there are only
rest integration tests.
2016-09-16 11:32:28 -04:00
Ali Beyad ce86ed1fdd Merge remote-tracking branch 'upstream/master' into rolling_upgrades 2016-09-16 10:43:38 -04:00
Ali Beyad 56f97500c6 In the rolling upgrades tests, we do not want to stop nodes
automatically between tasks, as we want some of the nodes from
the previous task to continue running in the next task. This
commit enables a cluster configuration setting to not stop
nodes automatically after a task runs, but instead the creator
of the test task must stop the running nodes explicitly in a
cleanup phase.
2016-09-16 10:36:55 -04:00
Ali Beyad ba072ec18e When Elasticsearch nodes are started in gradle to form a
cluster, we wait for the cluster health to indicate the
necessary nodes have formed a cluster.  This check was an
exact value (equality) check.  However, if we are trying to
connect the nodes in the cluster to nodes from a previously
formed cluster (of the same name), then we will have more
nodes returned by the cluster health check than the current
task's configured number of nodes.  Hence, this check needs
to be a >= check.  This commit fixes it.
2016-09-16 09:41:26 -04:00
Ali Beyad ec7e383783 Better checks for the cluster being up in the rolling upgrades tests. 2016-09-15 15:08:11 -04:00
Jason Tedor 7132fcd7ac Give useful error message if log config is missing
Today when starting Elasticsearch without a Log4j 2 configuration file,
we end up throwing an array index out of bounds exception. This is
because we are passing no configuration files to Log4j. Instead, we
should throw a useful error message to the user. This commit modifies
the Log4j configuration setup to throw a user exception if no Log4j
configuration files are present in the config directory.

Relates #20493
2016-09-15 07:44:05 -04:00
Ali Beyad 3f79874042 Prevent the rolling upgrades rest tests from cleaning up indices
after finishing if a the tests.rest.preserve_indices system property
is set
2016-09-14 23:34:19 -04:00
Tanguy Leroux 5d373e7f94 [TEST] Fix SystemD test on Fedora 24
The journalctl command on Fedora24 returns an exit code 0 if no log entry exist in the journal, whereas other distributions return 1.
2016-09-14 21:14:30 +02:00
Tanguy Leroux f3f9b499ad [Packaging] Run BATS upgrade tests against the current version (#20453)
BATS upgrade tests fails on master branch because it tries to install 2.x versions to upgrade from instead of 5.x versions. And since #18554 we should only test upgrades from 5.0.0-alpha4 versions.

This commit changes the vagrant tests so that it tries to list all the previous releases from version N-1. If nothing is found, it will fetch the current version and will run the upgrade tests with it. It works nicely with the current master 6.0.0-alpha1-SNAPSHOT. Once 5.0.0 is released it should run the test with it.
2016-09-14 18:26:03 +02:00
Tanguy Leroux 3caaafa4bd [Package] Remove bin/lib/modules directories on RPM uninstall/upgrade (#20448)
When uninstalling or upgrading elasticsearch using the RPM package some empty directories remain on the filesystem:

    /usr/share/elasticsearch/bin
    /usr/share/elasticsearch/lib
    /usr/share/elasticsearch/modules
    /usr/share/elasticsearch/modules/foo

Having empty directories in modules can prevent elasticsearch to start after an upgrade: the plugins service expects to find a plugin-descriptor.properties file in every sub directory of modules.

This PR cleans things a bit so that these empty directories are removed on upgrade/removal like it was in 2.x.
2016-09-14 18:18:37 +02:00
Jason Tedor 875387936b Simplify Log4j shutdown hack test
The Log4j shutdown hack test tests that a hack we have in place to
workaround a bug in Log4j during shutdown is effective. Log4j can use
JMX to control logging levels, but we disable this through the use of a
system property log4j2.disable.jmx (mainly because there is no need for
this feature, but it also means granting additional security
permissions). The bug in Log4j is that during shutdown, it neglects to
check whether or not its usage of JMX is disable and so it attempts to
unregister management beans, leading to a permissions violation. The
test works by attempting to shutdown Log4j and thus triggering the bad
code path. With the Log4j hack in place, we have introduced jar hell so
that its our code running instead of code from the Log4j jar. Our code
correctly checks that the usage of JMX is disabled and thus does not
trip on a permissions violation. The test was a little complicated in
that it attempted to just grant the minimal permissions needed for Log4j
to do its thing, but this can sometimes lead to other unwanted
permissions violations because the permissions put in place are more
restrictive necessary. This commit simplifies this situation by
rewriting the test to only deny Log4j the sole permission needed to
trigger the bug.

Relates #20476
2016-09-14 11:25:28 -04:00
Tanguy Leroux 26dc6f1306 [Packaging] Do not remove scripts directory on upgrade (#20452)
When upgrading elasticsearch using the RPM package, the scripts directory is removed if it's empty but it won't be recreated by the upgraded package. But after that the service won't start because the scripts dir is missing.
2016-09-14 10:11:27 +02:00
Jason Tedor 0eff7daf5b Fix logging hierarchy configs
Today when setting the logging level via the command-line or an API
call, the expectation is that the logging level should trickle down the
hiearchy to descendant loggers. However, this is not necessarily the
case. For example, if loggers x and x.y are already configured then
setting the logging level on x will not descend to x.y. This is because
the logging config for x.y has already been forked from the logging
config for x. Therefore, we must explicitly descend the hierarchy when
setting the logging level and that is what this commit does.

Relates #20463
2016-09-13 22:46:14 -04:00
Ali Beyad 4431720c3d File-based discovery plugin (#20394)
This commit introduces a new plugin for file-based unicast hosts
discovery. This allows specifying the unicast hosts participating
in discovery through a `unicast_hosts.txt` file located in the
`config/discovery-file` directory. The plugin will use the hosts 
specified in this file as the set of hosts to ping during discovery.

The format of the `unicast_hosts.txt` file is to have one host/port
entry per line. The hosts file is read and parsed every time
discovery makes ping requests, thus a new version of the file that
is published to the config directory will automatically be picked
up.

Closes #20323
2016-09-13 20:52:39 -04:00
Jason Tedor 482f00f93c Fix logger defaults test
This commit fixes the test EvilLoggerConfigurationTests#testDefaults.
2016-09-13 17:27:44 -04:00
Jason Tedor fbe27664a6 Fix prefix logging
Today we add a prefix when logging within Elasticsearch. This prefix
contains the node name, and index and shard-level components if
appropriate.

Due to some implementation details with Log4j 2 , this does not work for
integration tests; instead what we see is the node name for the last
node to startup. The implementation detail here is that Log4j 2 there is
only one logger for a name, message factory pair, and the key derived
from the message factory is the class name of the message factory. So,
when the last node starts up and starts setting prefixes on its message
factories, it will impact the loggers for the other nodes.

Additionally, the prefixes are lost when logging an exception. This is
due to another implementation detail in Log4j 2. Namely, since we log
exceptions using a parameterized message, Log4j 2 decides that that
means that we do not want to use the message factory that we have
provided (the prefix message factory) and so logs the exception without
the prefix.

This commit fixes both of these issues.

Relates #20429
2016-09-13 14:46:34 -04:00
Ali Beyad 513ed58d17 Added a indices flush to the rolling upgrades rest tests to ensure
they are on disk for the upgraded nodes to pick up.
2016-09-13 12:36:20 -04:00
Tanguy Leroux 6090c51fc5 Add quiet option to disable console logging (#20422)
This commit adds a -q/--quiet option to Elasticsearch so that it does not log anything in the console and closes stdout & stderr streams. This is useful for SystemD to avoid duplicate logs in both journalctl and /var/log/elasticsearch/elasticsearch.log while still allows the JVM to print error messages in stdout/stderr if needed.

closes #17220
2016-09-13 14:08:24 +02:00
Tanguy Leroux 567093cf78 Add Ubuntu-16.04 to Vagrant VMs (#20425) 2016-09-13 09:26:56 +02:00
Jason Tedor 262a5ee311 Use HTTPS for downloading released versions
This commit changes the protocol used for downloading the Maven metadata
for released versions of Elasticsearch from HTTP to HTTPS.
2016-09-12 19:48:12 -04:00
Lee Hinman 3c10f90b47 [TEST] Fix plugin list comparison
The plugin command now displays the version of the plugin, which is
compared to a string without the version. This removes the version from
the string.
2016-09-12 17:02:44 -06:00