Commit Graph

611 Commits

Author SHA1 Message Date
Jason Tedor 221caa1c5e Refactor handling for bad default permissions
This commit refactors the handling of bad default permissions that come
from the system security policy.

Relates #21735
2016-11-22 10:26:36 -05:00
Nik Everett c79371fd5b Remove lang-python and lang-javascript (#20734)
They were deprecated in 5.0. We are concentrating on making
Painless awesome rather than supporting every language possible.

Closes #20698
2016-11-21 22:13:25 -05:00
David Roberts 6daeb56969 Set execute permissions for native plugin programs (#21657) 2016-11-21 09:20:09 +00:00
David Roberts 2321782703 Fix Windows test failure (#21647) 2016-11-18 09:51:42 +00:00
Simon Willnauer de04aad994 Remove `modules/transport_netty_3` in favor of `netty_4` (#21590)
We kept `netty_3` as a fallback in the 5.x series but now that master
is 6.0 we don't need this or in other words all issues coming up with
netty 4 will be blockers for 6.0.
2016-11-17 12:44:42 +01:00
David Roberts 116593e5f5 Adjust bootstrap sequence (#21543)
Added the ability for plugins to spawn a controller process at startup
2016-11-17 09:58:09 +00:00
Jason Tedor d06a8903fd Merge branch 'master' into feature/seq_no
* master: (22 commits)
  Add proper toString() method to UpdateTask (#21582)
  Fix `InternalEngine#isThrottled` to not always return `false`. (#21592)
  add `ignore_missing` option to SplitProcessor (#20982)
  fix trace_match behavior for when there is only one grok pattern (#21413)
  Remove dead code from GetResponse.java
  Fixes date range query using epoch with timezone (#21542)
  Do not cache term queries. (#21566)
  Updated dynamic mapper section
  Docs: Clarify date_histogram bucket sizes for DST time zones
  Handle release of 5.0.1
  Fix skip reason for stats API parameters test
  Reduce skip version for stats API parameter tests
  Strict level parsing for indices stats
  Remove cluster update task when task times out (#21578)
  [DOCS] Mention "all-fields" mode doesn't search across nested documents
  InternalTestCluster: when restarting a node we should validate the cluster is formed via the node we just restarted
  Fixed bad asciidoc in boolean mapping docs
  Fixed bad asciidoc ID in node stats
  Be strict when parsing values searching for booleans (#21555)
  Fix time zone rounding edge case for DST overlaps
  ...
2016-11-16 09:10:35 -05:00
Jason Tedor b2b7595fa7 Temporarily set BWC version to 6.0.0 for seq. no
There is not yet a BWC layer in sequence numbers. This commit sets the
BWC version to 6.0.0 for the BWC and rolling upgrade tests until this
BWC layer is built.
2016-11-16 09:09:38 -05:00
Jason Tedor db5f51b839 Revert "disable backwards-5.0 tests as there is no BWC layer in the seq# related components"
This reverts commit db87837c72.
2016-11-16 09:09:35 -05:00
Boaz Leskes db87837c72 disable backwards-5.0 tests as there is no BWC layer in the seq# related components 2016-11-16 08:04:08 +00:00
Nik Everett e66261eee9 Handle release of 5.0.1
Adds a version constant for it, bwc indices, and a vagrant upgrade-from
version. Also bumps the "upgrade from" version for the backwards-5.0
test and adds `skip`s for tests that don't fail against 5.0 so we skip
them during the backwards testing.

Finally, this skips the "Shrink index via API" test because it fails
consistently for me. Inconsistently for CI, but consistently for me.
I'll work on making it consistent tomorrow.
2016-11-15 19:31:28 -05:00
Tanguy Leroux aa871f839a [Tests] Do not pass all env vars to sudo (#21562)
In #21348 the command executed to run the packaging tests has been changed to "sudo -E bats ...", forcing all environment variables from the vagrant user to be passed to the `sudo` command. This breaks a test on opensuse-13 (the one where it checks that elasticsearch cannot be started when `java` is not found) because all the PATH from the user is passed to the sudo command.

This commit restores the previous behavior while allowing only necessary testing environment variables to be passed using a /etc/sudoers.d file.
2016-11-15 15:19:34 +01:00
Ryan Ernst c7bd4f3454 Tests: Add TestZenDiscovery and replace uses of MockZenPing with it (#21488)
This changes adds a test discovery (which internally uses the existing
mock zenping by default). Having the mock the test framework selects be a discovery
greatly simplifies discovery setup (no more weird callback to a Node
method).
2016-11-14 21:46:10 -08:00
Jason Tedor 795c121852 Remove explicit task paths in tribe node test
This commit removes some explicit paths in the tribe node smoke test so
that these are easier to maintain (in the case of reorganization).
2016-11-14 13:52:57 -05:00
Jason Tedor 491a945ac8 Add socket permissions for tribe nodes
Today when a node starts, we create dynamic socket permissions based on
the configured HTTP ports and transport ports. If no ports are
configured, we use the default port ranges. When a tribe node starts, a
tribe node creates an internal node client for connecting to each remote
cluster. If neither an explicit HTTP port nor transport ports were
specified, the default port ranges are large enough for the tribe node
and its internal node clients. If an explicit HTTP port or transport
port was specified for the tribe node, then socket permissions for those
ports will be created, but not for the internal node clients. Whether
the internal node clients have explicit ports specified, or attempt to
bind within the default range, socket permissions for these will not
have been created and the internal node clients will hit a permissions
issue when attempting to bind. This commit addresses this issue by also
accounting for tribe nodes when creating the dynamic socket
permissions. Additionally, we add our first real integration test for
tribe nodes.
2016-11-14 11:58:44 -05:00
Simon Willnauer 26375256ff Enable 5.x to 6.x BWC tests (#21537)
This commit enables real BWC testing against a 5.1 snapshot. All
REST tests plus rolling upgrade test now run against a mixed version
cross major version cluster.
2016-11-14 17:03:57 +01:00
Simon Willnauer bdc942fa72 Enable 5.x to 6.x BWC tests
This commit enables real BWC testing against a 5.1 snapshot. All
REST tests plus rolling upgrade test now run against a mixed version
cross major version cluster.
2016-11-14 14:26:49 +01:00
Tanguy Leroux 9cade1cc18 Add Vagrant Gradle plugin
This commit changes the current :elactisearch:qa:vagrant build file and transforms it into a Gradle plugin in order to reuse it in other projects.

Most of the code from the build.gradle file has been moved into the VagrantTestPlugin class. To avoid duplicated VMs when running vagrant tests, the Gradle plugin sets the following environment variables before running vagrant commands:
VAGRANT_CWD: absolute path to the folder that contains the Vagrantfile
VAGRANT_PROJECT_DIR: absolute path to the Gradle project that use the VagrantTestPlugin

The VAGRANT_PROJECT_DIR is used to share project folders and files with the vagrant VM. These folders and files are exported when running the task `gradle vagrantSetUp` which:
- collects all project archives dependencies and copies them into `${project.buildDir}/bats/archives`
- copy all project bats testing files from 'src/test/resources/packaging/tests' into `${project.buildDir}/bats/tests`
- copy all project bats utils files from 'src/test/resources/packaging/utils' into `${project.buildDir}/bats/utils`

It is also possible to inherit and grab the archives/tests/utils files from project dependencies using the plugin configuration:

apply plugin: 'elasticsearch.vagrant'
esvagrant {
    inheritTestUtils true|false
    inheritTestArchives true|false
    inheritTests true|false
}
dependencies {
    // Inherit Bats test utils from :qa:vagrant project
    bats project(path: ':qa:vagrant', configuration: 'bats')
}

The folders `${project.buildDir}/bats/archives`, `${project.buildDir}/bats/tests` and `${project.buildDir}/bats/utils` are then exported to the vagrant VMs and mapped to the BATS_ARCHIVES, BATS_TESTS and BATS_UTILS environnement variables.

The following Gradle tasks have also be renamed:

* gradle vagrantSetUp
This task copies all the necessary files to the project build directory (was `prepareTestRoot`)

* gradle vagrantSmokeTest
This task starts the VMs and echoes a "Hello world" within each VM (was: `smokeTest`)
2016-11-14 10:33:05 +01:00
Jason Tedor 404b9afeca Account for different paths for sysctl utilities
On some systems these utilities are in /usr/lib/systemd/systemd-sysctl
and /usr/sbin/sysctl, and on others the /usr is dropped. This commit
accounts for that fact.
2016-11-12 07:54:17 -05:00
Jason Tedor 1fde26347a Set vm.max_map_count on systemd package install
Our docs claim that we set vm.max_map_count automatically. This is not
quite the case. The story is that on SysV init we set vm.max_map_count
each time the service starts, which is good. On systemd, we create a
sysctl.d conf file that sets vm.map_max_count, but this is only
meaningful if the system is rebooted after package install. This commit
modifies the post-install script so that we run systemd-sysctl so that
the vm.max_map_count change occurs after package install without a
reboot.

Relates #21507
2016-11-11 16:41:54 -05:00
Tanguy Leroux 3f7f8e4b97 Adapt ES_JVM_OPTIONS packaging test to ubuntu-1204
This commit adapts the "[INIT.D] start Elasticsearch with custom JVM options" packaging test so that it works on ubuntu-1204.

Related to #21445
2016-11-10 11:11:47 +01:00
Tanguy Leroux 4b94eb5cb8 Add VirtualBox version check (#21370)
This commit ensure that VirtualBox is available in version 5.1+ in the system before running packaging tests. It also check for Vagrant version is now greater than 1.8.6.
2016-11-10 09:00:13 +01:00
Jason Tedor b43ed8821f Export ES_JVM_OPTIONS for SysV init
The environment variable ES_JVM_OPTIONS allows end-users to specify a
custom location for the jvm.options file. Unfortunately, this
environment variable is not exported from the SysV init scripts. This
commit addresses this issue, and includes a test that ES_JVM_OPTIONS and
ES_JAVA_OPTS work for the SysV init packages.

Relates #21445
2016-11-09 17:52:37 -05:00
Ryan Ernst 7a2c984bcc Test: Remove multi process support from rest test runner (#21391)
At one point in the past when moving out the rest tests from core to
their own subproject, we had multiple test classes which evenly split up
the tests to run. However, we simplified this and went back to a single
test runner to have better reproduceability in tests. This change
removes the remnants of that multiplexing support.
2016-11-07 15:07:34 -08:00
Boaz Leskes 68a0d20ccd increase logging level to DEBUG in backwards-5.0 tests 2016-11-07 22:21:47 +01:00
Jason Tedor f16c308efd Assert status logger does not warn on Log4j usage
Today if you start Elasticsearch with the status logger configured to
the warn level, or use a transport client with the default status logger
level, you will see warn messages about deprecation loggers being
created with different message factories and that formatting might be
broken. This happens because the deprecation logger is constructed using
the message factory from its parent, an artifact leftover from the first
Log4j 2 implementation that used a custom message factory. When that
custom message factory was removed, this constructor invocation should
have been changed to not explicitly use the message factory from the
parent. This commit fixes this invocation. However, we also had some
status checking to all tests to ensure that there are no warn status log
messages that might indicate a configuration problem with Log4j 2. These
assertions blow up badly without the fix for the deprecation logger
construction, and also caught a misconfiguration in one of the logging
tests.

Relates #21339
2016-11-04 14:19:59 -04:00
Christoph Büscher f4594d4302 Removing plugin that isn't installed shouldn't trigger usage information
The usage information for `elasticsearch-plugin` is quiet verbose and makes the
actual error message that is shown when trying to remove an non-existing plugin
hard to spot. This changes the error code to not trigger printing the usage
information.

Closes #21250
2016-11-04 11:46:20 +01:00
Ryan Ernst dc6ed7b8d4 Remove pluggability of ZenPing (#21049)
Plugins: Remove pluggability of ZenPing

ZenPing is the part of zen discovery which knows how to ping nodes.
There is only one alternative implementation, which is just for testing.
This change removes the ability to add custom zen pings, and instead
hooks in the MockZenPing for tests through an overridden method in
MockNode. This also folds in the ZenPingService (which was really just a
single method) into ZenDiscovery, and removes the idea of having
multiple ZenPing instances. Finally, this was the last usage of the
ExtensionPoint classes, so that is also removed here.
2016-11-03 08:20:20 -07:00
Jason Tedor 83338686b9 Remove stale install plugins assertion
When installing a plugin when the plugins directory does not exist, the
install plugin command outputs a line saying that it is creating this
directory. The packaging tests for the archive distributions accounted
for this including an assertion that this line was output. The packages
have since been updated to include an empty plugins folder, so this line
will no longer be output. This commit removes this stale assertion from
the packaging tests.

Relates #21275
2016-11-02 13:07:26 -04:00
Jason Tedor 8dd91eb2d6 Add empty plugins dir for archive distributions
Today when installing Elasticsearch from an archive distribution (tar.gz
or zip), an empty plugins folder is not included. This means that if you
install Elasticsearch and immediately run elasticsearch-plugin list, you
will receive an error message about the plugins directory missing. While
the plugins directory would be created when starting Elasticsearch for
the first time, it would be better to just include an empty plugins
directory in the archive distributions. This commit makes this the
case. Note that the package distributions already include an empty
plugins folder.

Relates #21204
2016-11-01 15:14:24 -04:00
Jason Tedor 344a8028f8 Add analysis-ukrainian plugin to packaging tests
This commit adds the analysis-ukrainian plugin to the packaging tests.

Relates #21219
2016-10-31 16:38:33 -04:00
Nik Everett f11a217546 Rewrite vagrant upgrade test's dependency resolution
This is cleaner and it doesn't fail in Jenkins. I'm not sure
why the old way failed in jenkins but not locally, but computers
are basically magic so I take what I can get. Here is a jenkins
failure:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+packaging-tests/238/consoleFull
2016-10-28 17:02:24 -04:00
Nik Everett 267a7b427b Fix the package upgrade tests for 5.0.0 (#21178)
5.0+'s deb and rpm packages aren't pushed to maven cnertral so
instead we have to download them from artifacts.elastic.org.
2016-10-28 14:00:43 -04:00
Nik Everett 9e3eacec35 Fix generation of versions to upgrade from
Vagrant tests use a static list of dependencies to upgrade from
and we weren't including 5.0.0 deps in that list. Also when the
list was incorrect we weren't sorting the "current" list so it
was difficult to read.

Also adds 2.4.1 to the list but *doesn't* add 5.0.0 because we
still can't resolve it yet. We still only print an error when
the list is wrong but don't abort the build. We'll abort the build
once we've fixed resolution for 5.0.0 and we can re-add it.
2016-10-28 10:45:36 -04:00
Nik Everett bce34e4f1e Disable out of date versions exception
We are upgrading from out of date versions in our tests right now and we
can't fix that because the current versions to upgrade from aren't in
maven central. We'll resolve the resolution issue soon, but for now
let's get the build green.
2016-10-28 09:47:47 -04:00
Ali Beyad 733d74229b Revert "Updates vagrant tests upgrade version number" (#21171) 2016-10-28 09:40:04 -04:00
Ali Beyad 557506ba6e Updates the versions the vagrant tests can upgrade from to 5.0.0 (#21151) 2016-10-27 16:44:04 -04:00
Tanguy Leroux 127b4a8efc Change permissions on config files (#20966)
This commit changes some default file permissions on configuration files.
2016-10-24 09:42:03 +02:00
Tanguy Leroux 268d5ba97a Add JVM option "-XX:-AssumeMP" in SystemD tests (#21063)
Since with j`ava-1.8.0-openjdk-1.8.0.111-1.b15.el7_2.x86_64`, the OpenJDK packaged for CentOS and OEL override the default value (`false`) for the JVM option `AssumeMP` and force it to `true` (see [this patch](https://git.centos.org/blob/rpms!!java-1.8.0-openjdk.git/ab03fcc7a277355a837dd4c8500f8f90201ea353/SOURCES!always_assumemp.patch))

Because it is forced to true by default for these packages, the following warning message is printed to the standard output when the Vagrant box has only 1 CPU:
> OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N

This message will then fail the test introduced in #20422  where we check if no entries have been added to the journal after the service has been started.

This commit restore the default value for the `AssumeMP` option for CentOS and OracleServer.
2016-10-24 09:30:55 +02:00
Tanguy Leroux 5495b66797 Mute journalctl output check in packaging tests
This commit mutes a check on the output of journalctl after the Elasticsearch's systemd service has been started. It expected no entries in the journal but since OpenJDK build 1.8.0_111-b15 the following warning message is printed:
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
2016-10-21 10:52:11 +02:00
Tal Levy 38c650f376 make painless the default scripting language for ScriptProcessor (#20981)
- fixes a bug in the docs that mentions `lang` as optional
- now `lang` defaults to "painless"
2016-10-18 16:22:01 -07:00
Boaz Leskes c3987156ab Remove local discovery in favor of a simpler `MockZenPings` (#20960)
`LocalDiscovery` is a discovery implementation that uses static in memory maps to keep track of current live nodes. This is used extensively in our tests in order to speed up cluster formation (i.e., shortcut the 3 second ping period used by `ZenDiscovery` by default). This is sad as that mean that most of the test run using a different discovery semantics than what is used in production. Instead of replacing the entire discovery logic, we can use a similar approach to only shortcut the pinging components.
2016-10-18 21:12:15 +02:00
Dimitrios Liappis b0bb72ccd7 Deprecate EOL'ed Ubuntu 15.04 from Vagrantfile
* Deprecate EOL'ed Ubuntu 15.04 from Vagrantfile

Relates pr #20921
2016-10-17 17:27:01 +03:00
Jason Tedor 05adc14231 Revert "Display plugins versions"
This reverts commit 9411f18f27.

Relates #20807
2016-10-11 11:59:06 -04:00
Alexander Reelsen 2e447479c0 Tests: Fix ingest-attachment vagrant tests
The vagrant tests are checking for a certain jar version.

Relates #20710
2016-10-10 14:54:35 +02:00
Simon Willnauer 194a6b1df0 Remove LocalTransport in favor of MockTcpTransport (#20695)
This change proposes the removal of all non-tcp transport implementations. The
mock transport can be used by default to run tests instead of local transport that has
roughly the same performance compared to TCP or at least not noticeably slower.

This is a master only change, deprecation notice in 5.x will be committed as a
separate change.
2016-10-07 11:27:47 +02:00
Jason Tedor 51d53791fe Remove lenient URL parameter parsing
Today when parsing a request, Elasticsearch silently ignores incorrect
(including parameters with typos) or unused parameters. This is bad as
it leads to requests having unintended behavior (e.g., if a user hits
the _analyze API and misspell the "tokenizer" then Elasticsearch will
just use the standard analyzer, completely against intentions).

This commit removes lenient URL parameter parsing. The strategy is
simple: when a request is handled and a parameter is touched, we mark it
as such. Before the request is actually executed, we check to ensure
that all parameters have been consumed. If there are remaining
parameters yet to be consumed, we fail the request with a list of the
unconsumed parameters. An exception has to be made for parameters that
format the response (as opposed to controlling the request); for this
case, handlers are able to provide a list of parameters that should be
excluded from tripping the unconsumed parameters check because those
parameters will be used in formatting the response.

Additionally, some inconsistencies between the parameters in the code
and in the docs are corrected.

Relates #20722
2016-10-04 12:45:29 -04:00
Simon Willnauer 7e3863d2d8 [TEST] Fix EvilSystemPropertyTests to be test order independent 2016-09-29 13:26:14 +02:00
Simon Willnauer f2e6862803 Add a hard limit for `index.number_of_shard` (#20682)
this change adds a hard limit to `index.number_of_shard` that prevents
indices from being created that have more than 1024 shards. This is still
a huge limit and can only be changed via settings a system property.
2016-09-29 11:03:30 +02:00
Jason Tedor a6e33494ab Provide error message when plugin id is missing
Today when executing the install plugin command without a plugin id, we
end up throwing an NPE because the plugin id is null yet we just keep
going (ultimatley we try to lookup the null plugin id in a set, the
direct cause of the NPE). This commit modifies the install command so
that a missing plugin id is detected and help is provided to the user.

Relates #20660
2016-09-26 08:09:15 -04:00
Jay Modi 0573e03aa1 Pass classpath plugins to tribe nodes
When testing tribe nodes in an integration test, we should pass the classpath
plugins of the node down to the tribe client nodes. Without this the tribe client
nodes could be prevented from communicating with the tribes.
2016-09-22 07:29:07 -04:00
Jason Tedor 12234c067a Ensure logging is initialized in CLI tools
Today when CLI tools are executed, logging statements can intentionally
or unintentionally be executed when logging is not configured. This
leads to log messages that the status logger is not configured. This
commit reworks logging configuration for CLI tools so that logging is
always configured.

Relates #20575
2016-09-20 08:28:27 -04:00
Ali Beyad 50584c4103 Merge pull request #20532 from rjernst/rolling_upgrades
This PR introduces backward compatibility index tests to test the rolling upgrade process amongst Elasticsearch instances within the same major version. The test executes in three phases. In the first phase, we form a cluster of 2 ES instances on an old version. In the second phase, we keep one of the nodes from the old cluster, kill the other node, but preserve its data directory and start an instance of the current version of ES using the same data directory as the killed instance. In the third phase, we kill the other old version ES instance from the first phase and launch a new instance, using the same data directory as the killed instance. Therefore, during phase 3, we have fully migrated and have all current versions of ES running. In each phase, we run REST tests that index documents and search them, ensuring at each stage that the documents from the previous phase are still there.

Note that because we haven't released a GA yet of 5.0, the tests currently don't start an old version cluster in the first phase. Once GA is released, this will be changed to make the backward compatibility version 5.0, while the current version in the cluster will be 5.x.
2016-09-19 16:14:38 -04:00
Ali Beyad 8afc83047f Change the timeout of the rolling upgrades test from 40 mins to 5 mins
to still allow accounting for slow VMs
2016-09-19 15:45:41 -04:00
David Pilato dfd1eebdd0 Remove mapper attachments plugin
We now have in 5.0.0 `ingest-attachment` plugin.
We can remove `mapper-attachments` plugin for 6.0.

Closes #18837.
2016-09-19 09:01:16 +02:00
Ali Beyad 98230d035a Adds a preserveIndicesUponCompletion method to ESRestTestCase
that can be overridden by subclasses if the test must not
delete indices it created after exiting.
2016-09-16 19:21:26 -04:00
Ali Beyad 83adc87015 Removes stopNodeUponCompletion in favor of moving the stop
nodes task to the final part of the cluster test task execution
graph.
2016-09-16 16:32:43 -04:00
Ali Beyad 22e6bc8359 Disables unit tests for rolling upgrades, as there are only
rest integration tests.
2016-09-16 11:32:28 -04:00
Ali Beyad ce86ed1fdd Merge remote-tracking branch 'upstream/master' into rolling_upgrades 2016-09-16 10:43:38 -04:00
Ali Beyad 56f97500c6 In the rolling upgrades tests, we do not want to stop nodes
automatically between tasks, as we want some of the nodes from
the previous task to continue running in the next task. This
commit enables a cluster configuration setting to not stop
nodes automatically after a task runs, but instead the creator
of the test task must stop the running nodes explicitly in a
cleanup phase.
2016-09-16 10:36:55 -04:00
Ali Beyad ba072ec18e When Elasticsearch nodes are started in gradle to form a
cluster, we wait for the cluster health to indicate the
necessary nodes have formed a cluster.  This check was an
exact value (equality) check.  However, if we are trying to
connect the nodes in the cluster to nodes from a previously
formed cluster (of the same name), then we will have more
nodes returned by the cluster health check than the current
task's configured number of nodes.  Hence, this check needs
to be a >= check.  This commit fixes it.
2016-09-16 09:41:26 -04:00
Ali Beyad ec7e383783 Better checks for the cluster being up in the rolling upgrades tests. 2016-09-15 15:08:11 -04:00
Jason Tedor 7132fcd7ac Give useful error message if log config is missing
Today when starting Elasticsearch without a Log4j 2 configuration file,
we end up throwing an array index out of bounds exception. This is
because we are passing no configuration files to Log4j. Instead, we
should throw a useful error message to the user. This commit modifies
the Log4j configuration setup to throw a user exception if no Log4j
configuration files are present in the config directory.

Relates #20493
2016-09-15 07:44:05 -04:00
Ali Beyad 3f79874042 Prevent the rolling upgrades rest tests from cleaning up indices
after finishing if a the tests.rest.preserve_indices system property
is set
2016-09-14 23:34:19 -04:00
Tanguy Leroux 5d373e7f94 [TEST] Fix SystemD test on Fedora 24
The journalctl command on Fedora24 returns an exit code 0 if no log entry exist in the journal, whereas other distributions return 1.
2016-09-14 21:14:30 +02:00
Tanguy Leroux f3f9b499ad [Packaging] Run BATS upgrade tests against the current version (#20453)
BATS upgrade tests fails on master branch because it tries to install 2.x versions to upgrade from instead of 5.x versions. And since #18554 we should only test upgrades from 5.0.0-alpha4 versions.

This commit changes the vagrant tests so that it tries to list all the previous releases from version N-1. If nothing is found, it will fetch the current version and will run the upgrade tests with it. It works nicely with the current master 6.0.0-alpha1-SNAPSHOT. Once 5.0.0 is released it should run the test with it.
2016-09-14 18:26:03 +02:00
Tanguy Leroux 3caaafa4bd [Package] Remove bin/lib/modules directories on RPM uninstall/upgrade (#20448)
When uninstalling or upgrading elasticsearch using the RPM package some empty directories remain on the filesystem:

    /usr/share/elasticsearch/bin
    /usr/share/elasticsearch/lib
    /usr/share/elasticsearch/modules
    /usr/share/elasticsearch/modules/foo

Having empty directories in modules can prevent elasticsearch to start after an upgrade: the plugins service expects to find a plugin-descriptor.properties file in every sub directory of modules.

This PR cleans things a bit so that these empty directories are removed on upgrade/removal like it was in 2.x.
2016-09-14 18:18:37 +02:00
Jason Tedor 875387936b Simplify Log4j shutdown hack test
The Log4j shutdown hack test tests that a hack we have in place to
workaround a bug in Log4j during shutdown is effective. Log4j can use
JMX to control logging levels, but we disable this through the use of a
system property log4j2.disable.jmx (mainly because there is no need for
this feature, but it also means granting additional security
permissions). The bug in Log4j is that during shutdown, it neglects to
check whether or not its usage of JMX is disable and so it attempts to
unregister management beans, leading to a permissions violation. The
test works by attempting to shutdown Log4j and thus triggering the bad
code path. With the Log4j hack in place, we have introduced jar hell so
that its our code running instead of code from the Log4j jar. Our code
correctly checks that the usage of JMX is disabled and thus does not
trip on a permissions violation. The test was a little complicated in
that it attempted to just grant the minimal permissions needed for Log4j
to do its thing, but this can sometimes lead to other unwanted
permissions violations because the permissions put in place are more
restrictive necessary. This commit simplifies this situation by
rewriting the test to only deny Log4j the sole permission needed to
trigger the bug.

Relates #20476
2016-09-14 11:25:28 -04:00
Tanguy Leroux 26dc6f1306 [Packaging] Do not remove scripts directory on upgrade (#20452)
When upgrading elasticsearch using the RPM package, the scripts directory is removed if it's empty but it won't be recreated by the upgraded package. But after that the service won't start because the scripts dir is missing.
2016-09-14 10:11:27 +02:00
Jason Tedor 0eff7daf5b Fix logging hierarchy configs
Today when setting the logging level via the command-line or an API
call, the expectation is that the logging level should trickle down the
hiearchy to descendant loggers. However, this is not necessarily the
case. For example, if loggers x and x.y are already configured then
setting the logging level on x will not descend to x.y. This is because
the logging config for x.y has already been forked from the logging
config for x. Therefore, we must explicitly descend the hierarchy when
setting the logging level and that is what this commit does.

Relates #20463
2016-09-13 22:46:14 -04:00
Ali Beyad 4431720c3d File-based discovery plugin (#20394)
This commit introduces a new plugin for file-based unicast hosts
discovery. This allows specifying the unicast hosts participating
in discovery through a `unicast_hosts.txt` file located in the
`config/discovery-file` directory. The plugin will use the hosts 
specified in this file as the set of hosts to ping during discovery.

The format of the `unicast_hosts.txt` file is to have one host/port
entry per line. The hosts file is read and parsed every time
discovery makes ping requests, thus a new version of the file that
is published to the config directory will automatically be picked
up.

Closes #20323
2016-09-13 20:52:39 -04:00
Jason Tedor 482f00f93c Fix logger defaults test
This commit fixes the test EvilLoggerConfigurationTests#testDefaults.
2016-09-13 17:27:44 -04:00
Jason Tedor fbe27664a6 Fix prefix logging
Today we add a prefix when logging within Elasticsearch. This prefix
contains the node name, and index and shard-level components if
appropriate.

Due to some implementation details with Log4j 2 , this does not work for
integration tests; instead what we see is the node name for the last
node to startup. The implementation detail here is that Log4j 2 there is
only one logger for a name, message factory pair, and the key derived
from the message factory is the class name of the message factory. So,
when the last node starts up and starts setting prefixes on its message
factories, it will impact the loggers for the other nodes.

Additionally, the prefixes are lost when logging an exception. This is
due to another implementation detail in Log4j 2. Namely, since we log
exceptions using a parameterized message, Log4j 2 decides that that
means that we do not want to use the message factory that we have
provided (the prefix message factory) and so logs the exception without
the prefix.

This commit fixes both of these issues.

Relates #20429
2016-09-13 14:46:34 -04:00
Ali Beyad 513ed58d17 Added a indices flush to the rolling upgrades rest tests to ensure
they are on disk for the upgraded nodes to pick up.
2016-09-13 12:36:20 -04:00
Tanguy Leroux 6090c51fc5 Add quiet option to disable console logging (#20422)
This commit adds a -q/--quiet option to Elasticsearch so that it does not log anything in the console and closes stdout & stderr streams. This is useful for SystemD to avoid duplicate logs in both journalctl and /var/log/elasticsearch/elasticsearch.log while still allows the JVM to print error messages in stdout/stderr if needed.

closes #17220
2016-09-13 14:08:24 +02:00
Tanguy Leroux 567093cf78 Add Ubuntu-16.04 to Vagrant VMs (#20425) 2016-09-13 09:26:56 +02:00
Jason Tedor 262a5ee311 Use HTTPS for downloading released versions
This commit changes the protocol used for downloading the Maven metadata
for released versions of Elasticsearch from HTTP to HTTPS.
2016-09-12 19:48:12 -04:00
Lee Hinman 3c10f90b47 [TEST] Fix plugin list comparison
The plugin command now displays the version of the plugin, which is
compared to a string without the version. This removes the version from
the string.
2016-09-12 17:02:44 -06:00
Ali Beyad 0c7dd1865c Reworking yaml tests for rolling upgrades 2016-09-12 18:46:06 -04:00
Lee Hinman 3439796df3 Merge branch 'pr/18683' 2016-09-12 16:24:09 -06:00
Ali Beyad 0b3eb11712 Changed rest-api-spec tests folder structure 2016-09-12 16:05:36 -04:00
Jason Tedor d547b79224 Separate configs for logging tests
The evil logger tests rely on external configuration. This configuration
is shared between these tests which means that changing the
configuration for one test can cause an unrelated test to fail. In
particular, removing the appenders on the root logger so that inherited
loggers in one test do not have a console and file appender by default
breaks tests that were expecting the root logger to have these
appenders. This commit separates these configs so that these tests are
not subject to this problem.
2016-09-09 17:43:21 -04:00
Jason Tedor 55a2f26b21 Logging shutdown hack
Log4j has a bug where on shutdown it ignores that JMX might be disabled;
since it does not respect this on shutdown, it proceeds to attempt to
access JMX leading to a security exception that should have otherwise
not occurred had it respected that JMX is disabled. This commit
intentionally introduces jar hell with the Server class to work around
this bug until a fix is released.

Relates #20389
2016-09-09 11:59:15 -04:00
Jason Tedor d8475488b8 Disable console logging
Previously we would disable console logging in certain circumstances
(for example, if Elasticsearch is not in the foreground, or if
Elasticsearch is in the foreground but an exception was thrown during
bootstrap). This commit makes this handling work with Log4j 2. This will
prevent users from seeing double bootstrap check failure messages.

Relates #20387
2016-09-09 09:15:35 -04:00
Jason Tedor 27ff4f327c Remove allow unquoted JSON
Previous versions of Elasticsearch permitted unquoted JSON field names even though this is against the JSON spec. This leniency was disabled by default in the 5.x series of Elasticsearch but a backwards compatibility layer was added via a system property with the intention of removing this layer in 6.0.0. This commit removes this backwards compatibility layer.

Relates #20388
2016-09-08 13:36:31 -04:00
Jason Tedor 8fc38f623b Remove logging configuration warning
The 5.x series of Elasticsearch emits a warning if any of the old
logging configuration formats are present. This commit removes that
warning.

Relates #20386
2016-09-08 10:57:48 -04:00
Jason Tedor de43565abc Do not log full bootstrap checks exception
By default, when an exception causes the JVM to terminate, the stack
trace is printed. In the case of failing bootstrap checks, this stack
trace is useless to the user, and might even distract them from seeing
that the bootstrap checks failed for reasons under their control. With
this commit, we cause the stack trace for a failing bootstrap check to
be truncated.

We also modify some methods to not declare that they throw the top level
checked exception type Exception, but instead explicitly declare the
exceptions that they throw. These exceptions are caught and wrapped in a
BootstrapException so that we can percolate only two exception types out
of Bootstrap#init as checked exception, BootstrapException and
NodeValidationException.

Relates #19989
2016-09-08 10:56:11 -04:00
Simon Willnauer f319545814 Prepare master branch to be 6.0.0-alpha1 2016-09-08 12:55:30 +02:00
Jason Tedor b8396cd2d6 Shutdown logging in logging configuration tests
The logging configuration tests write to log files which are deleted at
the end of the test. If these files are not closed, some operating
systems will complain when these deletes are performed. This commit
ensures that the logging system is properly shutdown so that these files
can be properly deleted.
2016-09-07 21:27:39 -04:00
Jason Tedor 98a278ed95 Shutdown logging in logging evil tests
The evil logging tests write to log files which are deleted at the end
of the test. If these files are not closed, some operating systems will
complain when these deletes are performed. This commit ensures that the
logging system is properly shutdown so that these files can be properly
deleted.
2016-09-07 13:43:53 -04:00
Ryan Ernst bd2de367cc Use 5.0 alpha5 for bwc version, so we have transport.ports
Also fixed bug to ensure unicast host is writtent in yaml quotes
2016-09-06 16:08:27 -07:00
Ryan Ernst e6ac27fcfa Merge branch 'master' into rolling_upgrades 2016-09-06 15:52:03 -07:00
Ryan Ernst a844b085f1 Made node config always have a unicast transport uri closure 2016-09-06 15:51:14 -07:00
Jason Tedor f427d7fe74 More verbose message on preserving plugin config
This commit expands on the message printed when config files are
preserved when removing a plugin to give the user an indication of the
reason the config files are preserved.
2016-09-06 08:51:12 -04:00
Jason Tedor 75956604eb Print message when removing plugin with config
When removing a plugin with a config directory, we preserve the config
directory. This is because the workflow for upgrading a plugin involves
removing and then installing the plugin again and losing the plugin
config in this case would be terrible. This commit causes a message
regarding this to be printed in case the user wants to manually delete
these files.
2016-09-06 08:01:43 -04:00
Ali Beyad 5d8aa6b4fe Adds tests for rolling upgrades to execute 2016-09-03 01:34:39 -04:00
Jason Tedor 40f889b825 Warn if unsupported logging configuration present
This commit adds a warning that an unsupported logging configuration is
present and points users to the new logging configuration file.

Relates #20309
2016-09-02 18:36:57 -04:00
Ryan Ernst ecaf6ef001 Add rolling upgrade test project 2016-08-31 16:45:03 -07:00
Jason Tedor 76ab02e002 Merge branch 'master' into log4j2
* master:
  Avoid NPE in LoggingListener
  Randomly use Netty 3 plugin in some tests
  Skip smoke test client on JDK 9
  Revert "Don't allow XContentBuilder#writeValue(TimeValue)"
  [docs] Remove coming in 2.0.0
  Don't allow XContentBuilder#writeValue(TimeValue)
  [doc] Remove leftover from CONSOLE conversion
  Parameter improvements to Cluster Health API wait for shards (#20223)
  Add 2.4.0 to packaging tests list
  Docs: clarify scale is applied at origin+offest (#20242)
2016-08-31 16:37:55 -04:00
Jason Tedor 07d1a72395 Update packaging tests for Log4j 2
This commit updates the packaging tests for Log4j 2. Namely, these tests
make assertions about logging.yml that should now be about
log4j2.properties.
2016-08-31 15:52:28 -04:00
Jason Tedor 54083f7d6e Randomly use Netty 3 plugin in some tests
When Netty 4 was introduced, it was not the default network
implementation. Some tests were constructed to randomly use Netty 4
instead of the default network implementation. When Netty 4 was made the
default implementation, these tests were not updated. Thus, these tests
are randomly choosing between the default network implementation (Netty
4) and Netty 4. This commit updates these tests to reverse the role of
Netty 3 and Netty 4 so that the randomization is choosing between Netty
3 and the default (again, now Netty 4).

Relates #20265
2016-08-31 15:41:39 -04:00
Jason Tedor 5877cb2329 Skip smoke test client on JDK 9
This commit adds an assumption to SmokeTestClientIT tests on JDK 9. The
underlying issue is that Netty attempts to access sun.nio.ch but this
package is not exported from java.base on JDK 9. This throws an uncaught
InaccessibleObjectException causing the test to fail. This assumption
can be removed when Netty 4.1.6 is released as it will include a fix for
this scenario.

Relates #20260
2016-08-31 14:57:12 -04:00
Igor Motov 6cac3e9a8d Add 2.4.0 to packaging tests list 2016-08-31 11:23:20 -04:00
Jason Tedor ac8c2e98ab Enable console logging for CLI tools
This commit enables CLI tools to have console logging. For the CLI
tools, we skip configuring the logging infrastructure via the config
file, and instead set the level only via a system property.
2016-08-31 09:05:26 -04:00
Jason Tedor 1f6a4be544 Fix failing evil logging configuration tests
This commit fixes failing evil logging configuration tests. The test for
resolving multiple configuration files was failing after
9a58fc2348 removed some of the
configuration needed for this test. The solution is revert the removal
of that configuration, but remove additivity from the test logger to
prevent the evil logger tests from failing.
2016-08-30 21:00:41 -04:00
Jason Tedor 9a58fc2348 Fix failing evil logger tests
This commit fixes failing evil logger tests. The tests were failing
after inadvertently configuring appenders on the parent and child
logger.
2016-08-30 18:35:08 -04:00
Jason Tedor 7da0cdec42 Introduce Log4j 2
This commit introduces Log4j 2 to the stack.
2016-08-30 13:31:24 -04:00
Igor Motov b36fbc4452 Add support for parameters to the script ingest processor
The script processor should support `params` to be consistent with all other script consumers.
2016-08-24 16:49:48 -04:00
Nik Everett 3b8701199f Fix packaging tests to test upgrading on disk groovy script 2016-08-23 11:55:11 -04:00
Nik Everett e22074c97f Add description to task in vagrant build 2016-08-23 09:58:32 -04:00
Tal Levy 84bf24b1e9 remove ability to set field value in script-processor configuration (#19981) 2016-08-15 10:57:39 -07:00
Jason Tedor 1f0673c9bd Default max local storage nodes to one
This commit defaults the max local storage nodes to one. The motivation
for this change is that a default value greather than one is dangerous
as users sometimes end up unknowingly starting a second node and start
thinking that they have encountered data loss.

Relates #19964
2016-08-12 09:26:20 -04:00
Nik Everett 9f8f2ea54b Remove ESIntegTestCase#pluginList
It was a useful method in 1.7 when javac's type inference wasn't as
good, but now we can just replace it with `Arrays.asList`.
2016-08-11 15:44:02 -04:00
Lee Hinman 5849c488b5 Merge remote-tracking branch 'dakrone/compliation-breaker' 2016-08-09 11:57:26 -06:00
Clinton Gormley 0dd6f63c49 Fixed missing changes in the version bump to alpha6 2016-08-09 18:52:03 +02:00
Lee Hinman 2be52eff09 Circuit break the number of inline scripts compiled per minute
When compiling many dynamically changing scripts, parameterized
scripts (<https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-scripting-using.html#prefer-params>)
should be preferred. This enforces a limit to the number of scripts that
can be compiled within a minute. A new dynamic setting is added -
`script.max_compilations_per_minute`, which defaults to 15.

If more dynamic scripts are sent, a user will get the following
exception:

```json
{
  "error" : {
    "root_cause" : [
      {
        "type" : "circuit_breaking_exception",
        "reason" : "[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead",
        "bytes_wanted" : 0,
        "bytes_limit" : 0
      }
    ],
    "type" : "search_phase_execution_exception",
    "reason" : "all shards failed",
    "phase" : "query",
    "grouped" : true,
    "failed_shards" : [
      {
        "shard" : 0,
        "index" : "i",
        "node" : "a5V1eXcZRYiIk8lecjZ4Jw",
        "reason" : {
          "type" : "general_script_exception",
          "reason" : "Failed to compile inline script [\"aaaaaaaaaaaaaaaa\"] using lang [painless]",
          "caused_by" : {
            "type" : "circuit_breaking_exception",
            "reason" : "[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead",
            "bytes_wanted" : 0,
            "bytes_limit" : 0
          }
        }
      }
    ],
    "caused_by" : {
      "type" : "general_script_exception",
      "reason" : "Failed to compile inline script [\"aaaaaaaaaaaaaaaa\"] using lang [painless]",
      "caused_by" : {
        "type" : "circuit_breaking_exception",
        "reason" : "[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead",
        "bytes_wanted" : 0,
        "bytes_limit" : 0
      }
    }
  },
  "status" : 500
}
```

This also fixes a bug in `ScriptService` where requests being executed
concurrently on a single node could cause a script to be compiled
multiple times (many in the case of a powerful node with many shards)
due to no synchronization between checking the cache and compiling the
script. There is now synchronization so that a script being compiled
will only be compiled once regardless of the number of concurrent
searches on a node.

Relates to #19396
2016-08-09 10:26:27 -06:00
Jason Tedor 920a21e55c Fix parsing in test set max number of threads
This commit fixes a test bug in
EvilJNANativesTests#testSetMaximumNumberOfThreads. Namely, the test was
not checking whether or not the value from /proc/self/limits was equal
to "unlimited" before attempting to parse as a long. This commit fixes
that error.
2016-08-07 13:05:07 -04:00
Jason Tedor a62740bbd2 Avoid early initializing Netty
Today when we load the Netty plugins, we indirectly cause several Netty
classes to initialize. This is because we attempt to load some classes
by name, and loading these classes is done in a way that triggers a long
chain of class initializers within Netty. We should not do this, this
can lead to log messages before the logger is loader, and it leads to
initialization in cases when the classes would never be needed (for
example, Netty 3 class initialization is never needed if Netty 4 is
used, and vice versa). This commit avoids this early initialization of
these classes by removing the need for the early loading.

Relates #19819
2016-08-05 14:58:33 -04:00
Nik Everett f97b1a94b8 Add 2.3.5 to packaging tests list
This should make the packaging tests happy again.
2016-08-03 10:49:49 -04:00
Martijn van Groningen a91bb29585 ingest: Made the response format of the get pipeline api match with the response format of the index template api
Closes #19585
2016-07-29 17:58:30 +02:00
Martijn van Groningen 24d7fa6d54 ingest: Change the `foreach` processor to use the `_ingest._value` ingest metadata attribute to store the current array element being processed.
Closes #19592
2016-07-27 09:35:09 +02:00
Nik Everett 9270e8b22b Rename client yaml test infrastructure
This makes it obvious that these tests are for running the client yaml
suites. Now that there are other ways of running tests using the REST
client against a running cluster we can't go on calling the shared
client yaml tests "REST tests". They are rest tests, but they aren't
**the** rest tests.
2016-07-26 13:53:44 -04:00
Nik Everett a95d4f4ee7 Add Location header and improve REST testing
This adds a header that looks like `Location: /test/test/1` to the
response for the index/create/update API. The requirement for the header
comes from https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

https://tools.ietf.org/html/rfc7231#section-7.1.2 claims that relative
URIs are OK. So we use an absolute path which should resolve to the
appropriate location.

Closes #19079

This makes large changes to our rest test infrastructure, allowing us
to write junit tests that test a running cluster via the rest client.
It does this by splitting ESRestTestCase into two classes:
* ESRestTestCase is the superclass of all tests that use the rest client
to interact with a running cluster.
* ESClientYamlSuiteTestCase is the superclass of all tests that use the
rest client to run the yaml tests. These tests are shared across all
official clients, thus the `ClientYamlSuite` part of the name.
2016-07-25 17:02:40 -04:00
Jason Tedor 2d1b0587dd Introduce Netty 4
This commit adds transport-netty4, a transport and HTTP implementation
based on Netty 4.

Relates #19526
2016-07-22 22:26:35 -04:00
javanna db8beeba3b Merge branch 'master' into feature/async_rest_client 2016-07-22 15:51:03 +02:00
Nik Everett 4d89aa97e9 Vagrant tests should use plugin zips
Fixes failing packaging tests:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+packaging-tests/1049/console
2016-07-21 10:02:51 -04:00
Tal Levy f7cd86ef6d rethrow script compilation exceptions into ingest configuration exceptions (#19318)
* rethrow script compilation exceptions into ingest configuration exceptions
* update readProcessor to rethrow any exception as an ElasticsearchException
2016-07-20 10:37:56 -07:00
Nik Everett 3a82c613e4 Migrate query registration from push to pull
Remove `ParseField` constants used for names where there are no deprecated
names and just use the `String` version of the registration method instead.

This is step 2 in cleaning up the plugin interface for extending
search time actions. Aggregations are next.

This is breaking for plugins because those that register a new query should
now implement `SearchPlugin` rather than `onModule(SearchModule)`.
2016-07-20 12:33:51 -04:00
javanna a9b5c5adbe restore throws IOException clause on all performRequest sync methods
We throw IOException, which is the exception that is going to be thrown in 99% of the cases. A more generic exception can happen, and if it is a runtime one we just let it bubble up as is, otherwise we wrap it into runtime one so that we don't require to catch Exception everywhere, which seems odd.

Also adjusted javadocs for all performRequest methods
2016-07-19 15:18:05 +02:00
javanna 1bb33cf572 Remove RestClient#JSON_CONTENT_TYPE constant, already available in ContentType class 2016-07-19 15:17:12 +02:00
javanna 1fbec71243 Rest client: introduce async performRequest method and use async client under the hood for sync requests too
The new method accepts the usual parameters (method, endpoint, params, entity and headers) plus a response listener and an async response consumer. Shortcut methods are also added that don't require params, entity and the async response consumer optional.

There are a few relevant api changes as a consequence of the move to async client that affect sync methods:
- Response doesn't implement Closeable anymore, responses don't need to be closed
- performRequest throws Exception rather than just IOException, as that is the the exception that we get from the FutureCallback#failed method in the async http client
- ssl configuration is a bit simpler, one only needs to call setSSLStrategy from a custom HttpClientConfigCallback, that doesn't end up overridng any other default around connection pooling (it used to happen with the sync client and make ssl configuration more complex)

Relates to #19055
2016-07-19 15:15:58 +02:00
Ryan Ernst 3f6c0feee3 Merge pull request #19461 from rjernst/plugin_default_config
Simplify plugin configuration for rest tests
2016-07-18 14:07:15 -07:00
Simon Willnauer 8394544548 Add a dedicated client/transport project for transport-client (#19435)
The `client/transport` project adds a new jar build project that
pulls in all dependencies and configures all required modules.

Preinstalled modules are:
 * transport-netty
 * lang-mustache
 * reindex
 * percolator

The `TransportClient` classes are still in core
while `TransportClient.Builder` has only a protected construcutor
such that users are redirected to use the new `TransportClientBuilder`
from the new jar.

Closes #19412
2016-07-18 15:42:24 +02:00
Ryan Ernst 2fb3cdceff Build: Simplify plugin configuration for rest tests
This change removes the multiple ways that plugins can be added to the
integ test cluster. It also removes the use of the default
configuration, and instead adds a zip configuration to all plugins. This
will enable using project substitutions with plugins, which must be done
with the default configuration.
2016-07-15 14:34:21 -07:00
Ali Beyad 19d0dbcd17 Removes waiting for yellow cluster health upon index (#19460)
creation in the REST tests, as we no longer need it due
to index creation now waiting for active shard copies
before returning (by default, it waits for the primary of
each shard, which is the same as ensuring yellow health).

Relates #19450
2016-07-15 17:18:34 -04:00
Ryan Ernst 4b9932d4a8 Merge branch 'master' into rest_headers 2016-07-14 19:03:53 -07:00
Jason Tedor 31c648eee8 Rename transport-netty to transport-netty3
This commit renames the Netty 3 transport module from transport-netty to
transport-netty3. This is to make room for a Netty 4 transport module,
transport-netty4.

Relates #19439
2016-07-14 22:03:14 -04:00
Ryan Ernst 0b514f82a0 Plugins: Make rest headers registration pull based
Currently custom headers that should be passed through rest requests are
registered by depending on the RestController in guice and calling a
registration method. This change moves that registration to a getter for
plugins, and makes the RestController take the set of headers on
construction.
2016-07-14 18:45:53 -07:00
Simon Willnauer 5616251f22 Remove `node.mode` and `node.local` settings (#19428)
Today `node.mode` and `node.local` serve almost the same purpose, they
are a shortcut for `discovery.type` and `transport.type`. If `node.local: true`
or `node.mode: local` is set elasticsearch will start in _local_ mode which means
only nodes within the same JVM are discovered and a non-network based transport
is used. The _local_ mode it only really used in tests or if nodes are embedded.
For both, embedding and tests explicit configuration via `discovery.type` and `transport.type`
should be preferred.

This change removes all the usage of these settings and by-default doesn't
configure a default transport implemenation since netty is now a module. Yet, to make
the user expericence flawless, plugins or modules can set a `http.type.default` and
`transport.type.default`. Plugins set this via `PluginService#additionalSettings()`
which enforces _set-once_ which prevents node startup if set multiple times. This means
that our distributions will just startup with netty transport since it's packaged as a
module unless `transport.type` or `http.transport.type` is explicitly set.

This change also found a bunch of bugs since several NamedWriteables were not registered if a
transport client is used. Now that we don't rely on the `node.mode` leniency which is inherited
instead of using explicit settings, `TransportClient` uses `AssertingLocalTransport` which detects these problems since it serializes all messages.

Closes #16234
2016-07-14 13:21:10 +02:00
Tal Levy 8fd01554bc update foreach processor to only support one applied processor. (#19402)
Closes #19345.
2016-07-13 13:13:00 -07:00
Nik Everett f084469c27 Fix syntax for template tests 2016-07-13 11:34:06 -04:00
Nik Everett e9d292b450 Fix names of tar tests 2016-07-13 11:33:55 -04:00
Martijn van Groningen 2c3165d080 Removed deprecated 1.x script and template syntax
Closes #13729
2016-07-13 15:07:36 +02:00
Simon Willnauer 814c7224f9 Merge pull request #19392 from elastic/modularize_netty
This moves all netty related code into modules/transport-netty the module is build as a zip file as well as a JAR to serve as a dependency for transport client. For the time being this is required otherwise we have no network based impl. for transport client users. This might be subject to change given that we move forward http client.
2016-07-13 09:52:03 +02:00
Simon Willnauer ce28aa9160 [TEST] HttpCompressionIT is a real IT 2016-07-12 23:49:17 +02:00
Simon Willnauer c463083537 minor cleanups and an additional BogusPlugin for HttpSmokeTestCase 2016-07-12 17:55:05 +02:00
Simon Willnauer 2d80a53b09 fix test plugin visibility 2016-07-12 17:42:29 +02:00
Simon Willnauer 9cb247287f consolidate security code in on place an allow test based on the jar dependency to opt out of netty internal property setting assertion 2016-07-12 17:41:21 +02:00
Simon Willnauer 4fb79707bd Fix remaining tests that either need access to the netty module or require explict configuration
Some tests still start http implicitly or miss configuring the transport clients correctly.
This commit fixes all remaining tests and adds a depdenceny to `transport-netty` from
`qa/smoke-test-http` and `modules/reindex` since they need an http server running on the nodes.

This also moves all required permissions for netty into it's module and out of core.
2016-07-12 16:29:57 +02:00
javanna 512b8be791 RestClient: simplify ssl configuration and make http config callback functional friendly 2016-07-12 13:25:55 +02:00