Commit Graph

1341 Commits

Author SHA1 Message Date
Britta Weber 9492be65d4 plugin cli tool should not create empty log files
Plugin cli tools configures logging with whatever is in the logging.yml.
If a file appender is configured for any of the logs this will cause creation
of an empty log file. If a plugin was for example installed as root it will
create empty logs at es.home/logs.
This is problematic when for example plugins are installed as root and es is run
as service. Logs will then be created in /usr/share/elasticsearch/logs
and can later not be removed by for example dpkg -r or -purge.

To avoid this, configure the logger to use an appender that writes to the same
output that plugin cli tool does. This allows other components that are called
from Plugin cli tool to write to the same terminal that plugin cli tool writes to
by using the logging mechanism already in place.
The logging conf is not read at all pb plugin cli tool.

As a side effect, the loging level for components that are called
from the plugin command such as the jar hell check can now be configured
with -Des.logger.level which makes it easier to debug the jar hell check.
2015-10-06 14:13:24 +02:00
Britta Weber b4350e31be install groovy plugin before running script test
groovy moved to a plugin but the tests rely on it
see #13834
2015-10-05 00:23:48 +02:00
Britta Weber 1a67440989 add lang-groovy to plugin vagrant test 2015-09-29 17:56:29 +02:00
Robert Muir e0d42739dd Factor groovy out of core into lang-groovy 2015-09-28 20:17:45 -04:00
Nik Everett e45b2f7f0c [test] Jayatana is ignored
Installs javatana in vivid, emulates its on-login actions when starting
elasticsearch and verifies that elasticsearch turns off javatana.

Relates to #13813
2015-09-28 03:33:05 +02:00
Britta Weber d8ff91dbad [test] add lang-expression plugin to vagrant tests
we need this now that #13726 has been merged
2015-09-23 17:55:21 +02:00
Robert Muir 689af1a6d6 Factor expressions scripts out to lang-expression plugin 2015-09-22 20:33:47 -04:00
Nik Everett 4d47015a0c Merge pull request #13579 from nik9000/run_plugin_as_elasticsearch_in_tar
Run bin/plugin as elasticsearch in tar distro
2015-09-22 11:19:19 -04:00
Nik Everett b521c606ff [test] Run bin/plugin as right user
Before this commit he tests always run bin/plugin as root which is somewhat
unrealistic and causes trouble (log files owned by root instead of
elasticsearch). After this commit `bin/plugin` runs as root when elasticsearch
is installed via the repository and as elasticsearch otherwise which is much
more realistic.

This also adds extra timeout to starting elasticsearch which is required
when all the plugins are installed. And it fixes up a problem with logging
elasticsearch's log if elasticsearch doesn't start which came up multiple
time while debugging this problem.

Also adds docs recommending running `bin/plugin` as the user that owns the
Elasticsearch files or root if installed with the packages.

Closes #13557
2015-09-22 11:17:57 -04:00
Alexander Reelsen 2b46a72a6c BATS testing: Add SLES-12 to list of tested virtual machines
This adds SuSe Linux Enterprise Server 12 to the list of tested VMs.
SLES 12 is using systemd, so that the current RPM works
out of the box.

SLES12 however is already quite old and does not ship with java8, so this
required adding an opensuse repo.
2015-09-22 15:02:22 +02:00
Nik Everett a6c8ac536f Merge pull request #13696 from nik9000/fix_azure_vagrant
Fix plugin tests
2015-09-22 09:01:45 -04:00
Ryan Ernst de2566a023 Merge pull request #13704 from rjernst/more_rest_spec_cleanup
Move a couple more rest-api-spec resource dirs into resources
2015-09-21 22:10:58 -07:00
Ryan Ernst 093791e82d Test: Move a couple more rest-api-spec resource dirs into resources
A couple were left behind. This fixes them and cleans up the test
resources for qa tests a little bit.
2015-09-21 21:57:51 -07:00
Robert Muir f401350a97 Fix straggler in qa tests 2015-09-21 23:43:24 -04:00
Nik Everett fe24bcaa39 [test] Fix plugin tests
Fix the vagrant tests after azure was split into 3 plugins. The tests
need to list all the plugins and some dependency so we can make sure the
plugin can be installed and uninstalled.
2015-09-21 15:47:37 -04:00
Nik Everett 1998c2ca4d Merge pull request #13633 from andrestc/plugins-script-check-java
Adds a validation for plugins script to check if java is set
2015-09-21 14:40:26 -04:00
David Pilato f230eabc15 [cloud-azure] Split azure plugin in 3 plugins
Until now we had a cloud-azure plugin which is providing 3 distinct features:

* discovery on Azure
* snapshot/restore on Aure
* SMB store

This commit splits the plugin by feature so people can use either one or the other or both features.

Doc is updated accordingly.
2015-09-21 17:55:23 +02:00
André Carvalho bb0b661d9b Fix script testing for centos-7 2015-09-20 00:28:25 -03:00
André Carvalho 0213aafaad Limits variable scope 2015-09-18 13:49:05 -03:00
André Carvalho ab1bd5db53 Adds test case for plugins script 2015-09-17 22:22:20 -03:00
André Carvalho 139a65770a Adds test case for elasticsearch script 2015-09-17 22:22:09 -03:00
Britta Weber c9cd70d419 [test] start elasticsearch and verify it does what it should after installing in custom dir 2015-09-17 17:24:13 +02:00
Britta Weber d4161ff938 [test] add test for 'plugin list'
Tests that the plugins that are reported with 'plugin list' are the
same as in the plugins pom.
2015-09-17 10:25:35 +02:00
Nik Everett 8d02efd088 [test] Packaging test for filesystem scripts
Adds a tests for loading scripts from the filesystem for search templates
and for search filters.

Closes #13184
2015-09-16 10:44:21 -04:00
Nik Everett 9257478b47 Merge pull request #13593 from nik9000/sles_vagrant
Add opensuse-13 to packaging tests
2015-09-16 10:12:31 -04:00
Nik Everett 058d385942 [test] Add opensuse-13 to packaging tests
This gets opensuse-13 working with vagrant and the packaging tests. They pass
with some minor tweaks.

Closes #13507
2015-09-16 10:09:39 -04:00
Nik Everett bfc5f47fe2 [test] Fix centos-6 tests
Right now we execute some debian-isms in the init.d tests. This switches to
trying both the debian and centos ways to stop services from starting
automatically.
2015-09-15 17:55:12 -04:00
Nik Everett 9c91af9b7f Merge pull request #13422 from nik9000/java_8_for_package_tests
Packaging tests use Java 8
2015-09-14 08:26:04 -04:00
Robert Muir 3e517794e9 make logger final so its not detected as a static leak 2015-09-11 00:35:45 -04:00
Nik Everett 606896e4b6 [test] Packaging test for aws plugin's parts
The AWS plugin was broken into discovery-ec2 and repository-s3 so we can't
test the old plugin and must test the new ones.

Fixed some wording issues in test names.
2015-09-10 14:33:33 -04:00
Nik Everett afabdd1569 [test] Start with plugins in package tests
This changes the packaging tests to start Elasticsearch with all plugins
installed and checks `_cat/plugins?h=c` against the list of plugins in
the plugins directory. If the list differs, error! So it proves that the
plugins can be installed using bin/plugin as shipped in the rpm and deb
packages.

Closes #13254
2015-09-10 13:43:50 -04:00
Nik Everett 800fb5f7f5 [test] Document implementation choices
There are two other obvious ways to implement the "packages don't start
elasticsearch" checks but when you work through them they aren't as nice
as the implementation of the checks that we use now. This just adds
documentation to that effect.
2015-09-10 11:58:26 -04:00
Nik Everett 54e78b61b9 [test] Test that packages don't start service
We don't want either the deb or rpm package to start elasticsearch as soon
as they install nor do we want the package to register elasticsearch to
start on restart. That action is reserved for the administrator. This adds
tests for that.

Closes #13122
2015-09-10 11:58:26 -04:00
Nik Everett 4cabe39096 [test] Packaging tests use Java 8
To do this we:
1. All the rpm based distros we test support Java 8. We just ask to install
it.
2. There is a ppa that works for the Ubuntus. We just add that for them.
3. Debian Jessie has Java 8 in its backports. We just add that repository.
4. Debian Wheezy doesn't have Java 8 easily accessible so we drop it. We
could add it back with Orache Java 8 at a later date but that will take a
few more backflips and won't support things like vagrant-cachier.

This required a ton of rebuilding of vagrant boxes so it also fixes:
1. apt-get update is run too frequently
2. Lots of weird warning messages are spit out of apt-get
3. Switch from the chef provided based images to those provided by boxcutter.
The chef images has left vagrant atlas!

Closes #13366
2015-09-09 08:04:40 -04:00
Nik Everett 8d3df330ff [test] Add test for package reinstall after remove
Closes #13286
2015-09-07 09:07:07 -04:00
Nik Everett 72ea5327aa Merge pull request #13287 from nik9000/package_upgrade_test
Test upgrading from an older version
2015-09-04 15:35:36 -04:00
Nik Everett 04d254e571 [test] Documentation for packaging tests 2015-09-04 15:34:10 -04:00
David Pilato 30aa231f8e [plugin] split cloud-aws in repository-s3 and discovery-ec2
Until now we had a cloud-aws plugin which is providing 2 disctinct features:

* discovery on EC2
* snapshot/restore on S3

This commit splits the plugin by feature so people can use either one or the other or both features.

Doc is updated accordingly.
2015-09-03 11:12:20 +02:00
Simon Willnauer 796701d52e Move version to 3.0.0-SNAPSHOT 2015-09-03 10:43:28 +02:00
Nik Everett ab5c981ed2 [test] Test upgrading from an older version
Adds a test for upgrading from 2.0.0-beta1 to the version of that is built.

Closes #13183
2015-09-02 15:16:53 -04:00
David Pilato 4d05832a0b [qa] Add smoke test client module
As we log a lot, we hit a default limit:

```
The test or suite printed 9450 bytes to stdout and stderr, even though the limit was set to 8192 bytes. Increase the limit with @Limit, ignore it completely with @SuppressSysoutChecks or run with -Dtests.verbose=true
```

(cherry picked from commit 0cb325d)
2015-09-02 20:40:07 +02:00
David Pilato 1ffc6cd6a7 [qa] Add smoke test client module
Fix previous commit. A `pom` project does not run any test...
And the cluster name is set externally so we can't assert that it's `elasticsearch`.
2015-09-02 19:28:04 +02:00
David Pilato 1a8a2c9bc2 [qa] Add smoke test client module
This commit adds a new smoke test for testing client as a end Java user.

It starts a cluster in `pre-integration-test` phase, then execute the client operations defined as JUnit tests within `integration-test` phase and then stop the external cluster in `post-integration-test` phase.

You can also run test classes from your IDE.

* Start an external node on your machine with `bin/elasticsearch` (note that you can test Java API regressions if you run an older or newer node version)
* Run the JUnit test. By default, it will run tests on `localhost:9300` but you can change this setting using system property `tests.cluster`. It also expects the default `cluster.name` (`elasticsearch`).

This commit also starts adding [snippets as defined by Maven](https://maven.apache.org/guides/mini/guide-snippet-macro.html) to help keeping automatically synchronized the Java reference guide with the current code.

Our documentation builder tool does not support snippets though but we will most likely support it at some point.
2015-09-02 17:34:54 +02:00
Nik Everett 467a459f95 Merge pull request #13083 from nik9000/bats_less_skip
Clean up more bats tests
2015-09-02 08:45:15 -04:00
David Pilato 34ee4c2d66 [build] remove shaded elasticsearch version
The shaded version of elasticsearch was built at the very beginning to avoid dependency conflicts in a specific case where:

* People use elasticsearch from Java
* People needs to embed elasticsearch jar within their own application (as it's today the only way to get a `TransportClient`)
* People also embed in their application another (most of the time older) version of dependency we are using for elasticsearch, such as: Guava, Joda, Jackson...

This conflict issue can be solved within the projects themselves by either upgrade the dependency version and use the one provided by elasticsearch or by shading elasticsearch project and relocating some conflicting packages.

Example
-------

As an example, let's say you want to use within your project `Joda 2.1` but elasticsearch `2.0.0-beta1` provides `Joda 2.8`.
Let's say you also want to run all that with shield plugin.

Create a new maven project or module with:

```xml
<groupId>fr.pilato.elasticsearch.test</groupId>
<artifactId>es-shaded</artifactId>
<version>1.0-SNAPSHOT</version>

<properties>
    <elasticsearch.version>2.0.0-beta1</elasticsearch.version>
</properties>

<dependencies>
    <dependency>
        <groupId>org.elasticsearch</groupId>
        <artifactId>elasticsearch</artifactId>
        <version>${elasticsearch.version}</version>
    </dependency>
    <dependency>
        <groupId>org.elasticsearch.plugin</groupId>
        <artifactId>shield</artifactId>
        <version>${elasticsearch.version}</version>
    </dependency>
</dependencies>
```

And now shade and relocate all packages which conflicts with your own application:

```xml
<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>2.4.1</version>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                    <configuration>
                        <relocations>
                            <relocation>
                                <pattern>org.joda</pattern>
                                <shadedPattern>fr.pilato.thirdparty.joda</shadedPattern>
                            </relocation>
                        </relocations>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>
```

You can create now a shaded version of elasticsearch + shield by running `mvn clean install`.

In your project, you can now depend on:

```xml
<dependency>
    <groupId>fr.pilato.elasticsearch.test</groupId>
    <artifactId>es-shaded</artifactId>
    <version>1.0-SNAPSHOT</version>
</dependency>
<dependency>
    <groupId>joda-time</groupId>
    <artifactId>joda-time</artifactId>
    <version>2.1</version>
</dependency>
```

Build then your TransportClient as usual:

```java
TransportClient client = TransportClient.builder()
        .settings(Settings.builder()
                        .put("path.home", ".")
                        .put("shield.user", "username:password")
                        .put("plugin.types", "org.elasticsearch.shield.ShieldPlugin")
        )
        .build();
client.addTransportAddress(new InetSocketTransportAddress(new InetSocketAddress("localhost", 9300)));

// Index some data
client.prepareIndex("test", "doc", "1").setSource("foo", "bar").setRefresh(true).get();
SearchResponse searchResponse = client.prepareSearch("test").get();
```

If you want to use your own version of Joda, then import for example `org.joda.time.DateTime`. If you want to access to the shaded version (not recommended though), import `fr.pilato.thirdparty.joda.time.DateTime`.

You can run a simple test to make sure that both classes can live together within the same JVM:

```java
CodeSource codeSource = new org.joda.time.DateTime().getClass().getProtectionDomain().getCodeSource();
System.out.println("unshaded = " + codeSource);

codeSource = new fr.pilato.thirdparty.joda.time.DateTime().getClass().getProtectionDomain().getCodeSource();
System.out.println("shaded = " + codeSource);
```

It will print:

```
unshaded = (file:/path/to/joda-time-2.1.jar <no signer certificates>)
shaded = (file:/path/to/es-shaded-1.0-SNAPSHOT.jar <no signer certificates>)
```

This PR also removes fully-loaded module.

By the way, the project can now build with Maven 3.3.3 so we can relax a bit our maven policy.
2015-09-02 11:57:10 +02:00
Nik Everett 38147da43e [tests] Fix exit code check for systemctl 2015-09-01 15:24:04 -04:00
Nik Everett 99d2f0463e [packaging] clean up more bats tests
This cleans up deb, rpm, systemd, and sysvinit tests:
1. Move skip_not_rpm, skip_not_dpkg, etc to the setup() methods for faster
runtime and cleaner code.
2. Removed lots of needless invocations of `run`
3. Created install_package for use in the systemd and sysvinit tests.
4. Removed lots of needless stderr to stdout redirects.

Closes #13075
Related to #13074
2015-09-01 14:57:47 -04:00
Nik Everett e4410482fe Packaging: Install all plugins during bats tests
Related to #12717
2015-09-01 13:43:56 -04:00
Jason Tedor a8bace9f97 Remove and forbid final uses of ImmutableList 2015-08-31 14:35:23 -04:00
Nik Everett 23c1766cdc [packaging] Lock vagrant to virtualbox
Virtualbox is the default virtualization provier for vagrant but folks
override that from time to time. If they do then the build will fail because
the boxes used by the build don't usually support non-virtualbox providers.

Closes #13217
2015-08-31 10:45:46 -04:00
Nik Everett 577dcd7c26 Merge pull request #12903 from nik9000/cleanup_tar
Clean up the tar tests
2015-08-24 10:00:21 -04:00
Ryan Ernst d0f5ce58d8 Add multicast plugin to help and qa 2015-08-20 22:04:17 -07:00
Simon Willnauer 3eee8b387e only verify both nodes join the cluster 2015-08-20 21:51:13 +02:00
Simon Willnauer 0ffd99cca3 Drop commons-lang dependency
commons-lang really is only used by some core classes to join strings or modiy arrays.
It's not worth carrying the dependency. This commit removes the dependency on commons-lang
entirely.
2015-08-18 22:59:31 +02:00
David Pilato 0599f85d2d [build] simplify ant script for plugins
Now we are using short names for artifactId (see #12879) so we don't need anymore to transform long names `elasticsearch-pluginname` to short names `pluginname` in ant script when we install a plugin.
Modify also convert-plugin-name
Clean up remaining plugins with old format
And fix vagrant tests
2015-08-18 19:41:37 +02:00
David Pilato 20851a4e4a Merge branch 'pr/shorten-artifactids' 2015-08-18 15:25:38 +02:00
David Pilato 4a3ea799ec [qa] multinode tests fails when you run low on disk space (85%)
In #12853 we actually introduced a test regression. Now as we wait for yellow instead of green, we might have some pending tasks.
This commit simplify all that and only checks the number of nodes within the cluster.
2015-08-18 14:36:12 +02:00
David Pilato 807d35e96f [maven] change murmur3 plugin groupId and name 2015-08-18 13:45:59 +02:00
David Pilato 692cc80523 [maven] also rename parent project artifactId
Also fixed bad scm links
2015-08-18 13:41:12 +02:00
David Pilato d21afc8090 [maven] rename artifactIds from `elasticsearch-something` to `something`
In plugins, we are using non consistent naming. We use `elasticsearch-cloud-aws` as the artifactId, which generates a jar file called `elasticsearch-cloud-aws-VERSION.jar`.

But when you want to install the plugin, you will end up with a shorter name for the plugin `cloud-aws`.

```
bin/plugin install cloud-aws
```

This commit changes that and use consistent names for `artifactId`, so `finalName`.

Also changed maven names.
2015-08-18 13:38:48 +02:00
David Pilato da65493965 [qa] multinode tests fails when you run low on disk space (85%)
Indeed, we check within the test suite that we have not unassigned shards.

But when the test starts on my machine I get:

```
[elasticsearch] [2015-08-13 12:03:18,801][INFO ][org.elasticsearch.cluster.routing.allocation.decider] [Kehl of Tauran] low disk watermark [85%] exceeded on [eLujVjWAQ8OHdhscmaf0AQ][Jackhammer] free: 59.8gb[12.8%], replicas will not be assigned to this node
```

```
  2> REPRODUCE WITH: mvn verify -Pdev -Dskip.unit.tests -Dtests.seed=2AE3A3B7B13CE3D6 -Dtests.class=org.elasticsearch.smoketest.SmokeTestMultiIT -Dtests.method="test {yaml=smoke_test_multinode/10_basic/cluster health basic test, one index}" -Des.logger.level=ERROR -Dtests.assertion.disabled=false -Dtests.security.manager=true -Dtests.heap.size=512m -Dtests.locale=ar_YE -Dtests.timezone=Asia/Hong_Kong -Dtests.rest.suite=smoke_test_multinode
FAILURE 38.5s | SmokeTestMultiIT.test {yaml=smoke_test_multinode/10_basic/cluster health basic test, one index} <<<
   > Throwable #1: java.lang.AssertionError: expected [2xx] status code but api [cluster.health] returned [408 Request Timeout] [{"cluster_name":"prepare_release","status":"yellow","timed_out":true,"number_of_nodes":2,"number_of_data_nodes":2,"active_primary_shards":3,"active_shards":3,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":3,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":50.0}]
```

We don't check anymore if we have unassigned shards and we wait for `yellow` status instead of `green`.

Closes #12852.
2015-08-18 13:21:19 +02:00
Adrien Grand c169386dd4 Merge pull request #12931 from jpountz/fix/murmur3_defaults
Move the `murmur3` field to a plugin and fix defaults.
2015-08-18 12:09:32 +02:00
Adrien Grand a91b3fcbb9 Move the `murmur3` field to a plugin and fix defaults.
This move the `murmur3` field to the `mapper-murmur3` plugin and fixes its
defaults so that values will not be indexed by default, as the only purpose
of this field is to speed up `cardinality` aggregations on high-cardinality
string fields, which only requires doc values.

I also removed the `rehash` option from the `cardinality` aggregation as it
doesn't bring much value (rehashing is cheap) and allowed to remove the
coupling between the `cardinality` aggregation and the `murmur3` field.

Close #12874
2015-08-18 11:41:52 +02:00
Nik Everett 0b650ed203 Add tests for plugins with bin directory
Also removes all mention of shield:
```bash
$ find $BATS -type f -exec grep -Hi shield {} \;
$
```
2015-08-17 10:53:16 -07:00
Nik Everett 391ea379e2 Test: Use jvm-example for testing bin/plugin
Related to #12651
2015-08-17 10:53:16 -07:00
Nik Everett 513ac4471a Tests: Clean up the tar tests
1. Move `clean_before_test` to the first test so its more explicit.
2. Move `skip_not_tar_gz` to setup because it was run first in every test.
3. Remove calls to `run` that only check the status. Its simpler to just
execute the command. Its better because std-out will be captured and replayed
on error.
4. Switch from `su` to `sudo` because `su` was breaking `bats`'s error
reporting.
2015-08-17 10:31:08 -07:00
Nik Everett 708198ddd0 Merge pull request #12898 from nik9000/vagrant_name
Change qa/vagrant artifactId
2015-08-16 10:11:12 -07:00
Nik Everett 42300938aa Merge pull request #12904 from nik9000/remove_ES_CLEAN_BEFORE_TEST
Remove ES_CLEAN_BEFORE_TEST
2015-08-15 07:30:19 -07:00
Nik Everett 1d54cff167 Test: Remove ES_CLEAN_BEFORE_TEST
In the bats test ES_CLEAN_BEFORE_TEST was used to clean the environment
before running the tests. Unfortunately the tests don't work unless you
specify it every time. This removes that option and always runs the clean.
2015-08-14 15:12:52 -07:00
Simon Willnauer b447e2ae99 Move master to [2.1.0-SNAPSHOT] 2015-08-14 23:44:06 +02:00
Nik Everett 11d74dc26d Testing: Change qa/vagrant artifactId
Related to #12651
2015-08-14 13:18:23 -07:00
David Pilato 4e23fe2657 [build] revert maven 3.3.x fix
We need to revert #12803 as it creates some troubles in IntelliJ and also might fail when using older elasticsearch versions.
2015-08-13 15:21:06 +02:00
David Pilato fbd8f69273 Build of QA: Smoke Test Shaded Jar fails under maven 3.3.x
Build fails with maven 3.3.1 and 3.3.3. To reproduce, install one of the 3.3.x versions of maven and run `mvn clean verify` in the root directory of the project. The build will fail in the QA: Smoke Test Shaded Jar module with the following error:

```
Started J0 PID(99979@flea.local).
Suite: org.elasticsearch.shaded.test.ShadedIT
  2> NOTE: reproduce with: ant test  -Dtestcase=ShadedIT -Dtests.method=testJodaIsNotOnTheCP -Dtests.seed=2F4D23A7462CF921 -Dtests.locale= -Dtests.timezone=Asia/Baku -Dtests.asserts=true -Dtests.file.encoding=UTF-8
FAILURE 0.06s | ShadedIT.testJodaIsNotOnTheCP <<<
  > Throwable #1: junit.framework.AssertionFailedError: Expected an exception but the test passed: java.lang.ClassNotFoundException
  > at __randomizedtesting.SeedInfo.seed([2F4D23A7462CF921:3A9404F1F69FD80]:0)
  > at junit.framework.Assert.fail(Assert.java:57)
  > at java.lang.Thread.run(Thread.java:745)
  2> NOTE: reproduce with: ant test  -Dtestcase=ShadedIT -Dtests.method=testGuavaIsNotOnTheCP -Dtests.seed=2F4D23A7462CF921 -Dtests.locale= -Dtests.timezone=Asia/Baku -Dtests.asserts=true -Dtests.file.encoding=UTF-8
FAILURE 0.01s | ShadedIT.testGuavaIsNotOnTheCP <<<
  > Throwable #1: junit.framework.AssertionFailedError: Expected an exception but the test passed: java.lang.ClassNotFoundException
  > at __randomizedtesting.SeedInfo.seed([2F4D23A7462CF921:C2502FD54D83433D]:0)
  > at junit.framework.Assert.fail(Assert.java:57)
  > at java.lang.Thread.run(Thread.java:745)
  2> NOTE: reproduce with: ant test  -Dtestcase=ShadedIT -Dtests.method=testjsr166eIsNotOnTheCP -Dtests.seed=2F4D23A7462CF921 -Dtests.locale= -Dtests.timezone=Asia/Baku -Dtests.asserts=true -Dtests.file.encoding=UTF-8
FAILURE 0.01s | ShadedIT.testjsr166eIsNotOnTheCP <<<
  > Throwable #1: junit.framework.AssertionFailedError: Expected an exception but the test passed: java.lang.ClassNotFoundException
  > at __randomizedtesting.SeedInfo.seed([2F4D23A7462CF921:35593286F4269392]:0)
  > at junit.framework.Assert.fail(Assert.java:57)
  > at java.lang.Thread.run(Thread.java:745)
  2> NOTE: leaving temporary files on disk at: /Users/Shared/Jenkins/Home/workspace/elasticsearch-master/qa/smoke-test-shaded/target/J0/temp/org.elasticsearch.shaded.test.ShadedIT_2F4D23A7462CF921-001
  2> NOTE: test params are: codec=CheapBastard, sim=DefaultSimilarity, locale=, timezone=Asia/Baku
  2> NOTE: Mac OS X 10.10.4 x86_64/Oracle Corporation 1.8.0_25 (64-bit)/cpus=8,threads=1,free=482137936,total=514850816
  2> NOTE: All tests run in this JVM: [ShadedIT]
Completed [1/1] in 6.61s, 5 tests, 3 failures <<< FAILURES!

Tests with failures:
  - org.elasticsearch.shaded.test.ShadedIT.testJodaIsNotOnTheCP
  - org.elasticsearch.shaded.test.ShadedIT.testGuavaIsNotOnTheCP
  - org.elasticsearch.shaded.test.ShadedIT.testjsr166eIsNotOnTheCP
```
Please note that build doesn't fail with maven 3.2.x and it doesn't fail if mvn command is executed inside the qa/smoke-test-shaded directory. Only when the build is started from the root directory the error above can be observed.

The reason is because of the shaded version which depends on elasticsearch core.
When Maven build the module only, then elasticsearch core is not added to the dependency tree.

```sh
mvn dependency:tree -pl :smoke-test-shaded
```

```
[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ smoke-test-shaded ---
[INFO] org.elasticsearch.qa:smoke-test-shaded:jar:2.0.0-beta1-SNAPSHOT
[INFO] +- org.elasticsearch.distribution.shaded:elasticsearch:jar:2.0.0-beta1-SNAPSHOT:compile
[INFO] |  +- org.apache.lucene:lucene-core:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-backward-codecs:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-analyzers-common:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-queries:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-memory:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-highlighter:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-queryparser:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-sandbox:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-suggest:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-misc:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-join:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-grouping:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-spatial:jar:5.2.1:compile
[INFO] |  \- com.spatial4j:spatial4j:jar:0.4.1:compile
[INFO] +- org.hamcrest:hamcrest-all:jar:1.3:test
[INFO] \- org.apache.lucene:lucene-test-framework:jar:5.2.1:test
[INFO]    +- org.apache.lucene:lucene-codecs:jar:5.2.1:test
[INFO]    +- com.carrotsearch.randomizedtesting:randomizedtesting-runner:jar:2.1.16:test
[INFO]    +- junit:junit:jar:4.11:test
[INFO]    \- org.apache.ant🐜jar:1.8.2:test
```

But if shaded plugin is involved during the build, it modifies the `projectArtifactMap`:

```sh
mvn dependency:tree -pl org.elasticsearch.distribution.shaded:elasticsearch,:smoke-test-shaded
```

```
[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ smoke-test-shaded ---
[INFO] org.elasticsearch.qa:smoke-test-shaded:jar:2.0.0-beta1-SNAPSHOT
[INFO] +- org.elasticsearch.distribution.shaded:elasticsearch:jar:2.0.0-beta1-SNAPSHOT:compile
[INFO] |  \- org.elasticsearch:elasticsearch:jar:2.0.0-beta1-SNAPSHOT:compile
[INFO] |     +- org.apache.lucene:lucene-backward-codecs:jar:5.2.1:compile
[INFO] |     +- org.apache.lucene:lucene-analyzers-common:jar:5.2.1:compile
[INFO] |     +- org.apache.lucene:lucene-queries:jar:5.2.1:compile
[INFO] |     +- org.apache.lucene:lucene-memory:jar:5.2.1:compile
[INFO] |     +- org.apache.lucene:lucene-highlighter:jar:5.2.1:compile
[INFO] |     +- org.apache.lucene:lucene-queryparser:jar:5.2.1:compile
[INFO] |     |  \- org.apache.lucene:lucene-sandbox:jar:5.2.1:compile
[INFO] |     +- org.apache.lucene:lucene-suggest:jar:5.2.1:compile
[INFO] |     |  \- org.apache.lucene:lucene-misc:jar:5.2.1:compile
[INFO] |     +- org.apache.lucene:lucene-join:jar:5.2.1:compile
[INFO] |     |  \- org.apache.lucene:lucene-grouping:jar:5.2.1:compile
[INFO] |     +- org.apache.lucene:lucene-spatial:jar:5.2.1:compile
[INFO] |     |  \- com.spatial4j:spatial4j:jar:0.4.1:compile
[INFO] |     +- com.google.guava:guava:jar:18.0:compile
[INFO] |     +- com.carrotsearch:hppc:jar:0.7.1:compile
[INFO] |     +- joda-time:joda-time:jar:2.8:compile
[INFO] |     +- org.joda:joda-convert:jar:1.2:compile
[INFO] |     +- com.fasterxml.jackson.core:jackson-core:jar:2.5.3:compile
[INFO] |     +- com.fasterxml.jackson.dataformat:jackson-dataformat-smile:jar:2.5.3:compile
[INFO] |     +- com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:jar:2.5.3:compile
[INFO] |     |  \- org.yaml:snakeyaml:jar:1.12:compile
[INFO] |     +- com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:jar:2.5.3:compile
[INFO] |     +- io.netty:netty:jar:3.10.3.Final:compile
[INFO] |     +- com.ning:compress-lzf:jar:1.0.2:compile
[INFO] |     +- com.tdunning:t-digest:jar:3.0:compile
[INFO] |     +- org.hdrhistogram:HdrHistogram:jar:2.1.6:compile
[INFO] |     +- org.apache.commons:commons-lang3:jar:3.3.2:compile
[INFO] |     +- commons-cli:commons-cli:jar:1.3.1:compile
[INFO] |     \- com.twitter:jsr166e:jar:1.1.0:compile
[INFO] +- org.hamcrest:hamcrest-all:jar:1.3:test
[INFO] \- org.apache.lucene:lucene-test-framework:jar:5.2.1:test
[INFO]    +- org.apache.lucene:lucene-codecs:jar:5.2.1:test
[INFO]    +- org.apache.lucene:lucene-core:jar:5.2.1:compile
[INFO]    +- com.carrotsearch.randomizedtesting:randomizedtesting-runner:jar:2.1.16:test
[INFO]    +- junit:junit:jar:4.11:test
[INFO]    \- org.apache.ant🐜jar:1.8.2:test
```

A fix could consist of fixing something on Maven side. Probably something changed in a recent version and introduced this "issue" but it might be not really an issue. More a fix.

There are two workarounds:

1) exclude manually elasticsearch core from shaded version in smoke-test-shaded module and add manually each lucene lib needed by elasticsearch

2) add a new `elasticsearch-lucene` (lucene) POM module which simply declares all needed lucene libs in subprojects (such as the smoke tester one).

I choose the later.

Closes #12791.
2015-08-13 12:15:19 +02:00
Simon Willnauer 605253a39f Cut over master to 2.0.0-SNAPSHOT 2015-08-12 21:16:08 +02:00
Nik Everett ce00241440 Merge pull request #12769 from nik9000/automate_bats
Move vagrant activation to a parameter
2015-08-12 07:40:50 -07:00
Nik Everett 999001746a Move vagrant activation to a parameter
Closes #12611
2015-08-12 07:40:09 -07:00
Simon Willnauer 76564bbe6a TearDown secondary node after tests 2015-08-11 13:08:32 +02:00
Simon Willnauer b3b04e05f5 add multinode to pom.xml 2015-08-11 12:15:35 +02:00
Simon Willnauer d1bc099c91 Add multi-node IT infrastructure
This adds the infrastrucutre to run integration tests with more than one node.
 * it adds relevant macros and targets to integration-tests.xml to start unicast nodes
 * there is a qa/smoke-test-multinode project that simulates such a setup

this commit is soely the infrastructure and doesn't hook up any projects to use this.
For reliability and stability reasons this should be used with care and only if it's really
needed.

Closes #12718
2015-08-11 11:36:15 +02:00
Ryan Ernst 40f119d85a This method on settings loaded a class, based on a setting value, using
the default classloader. It had all kinds of leniency in how the
classname was found, and simply cannot work with plugins having isolated
classloaders.

This change removes that method. Some of the uses of it were for custom
extension points, like custom repository or discovery types. A lot were
just there to plugin mock implementations for tests. For the settings
that were legitimate, all now support plugins adding the given setting
via onModule. For those that were specific to tests for mocks, they now
use Classes.loadClass (a helper around Class.forName). This is a
temporary measure until (in a future PR) tests can change the
implementation via package private statics.

I also removed a number of unnecessary intermediate modules, added a
"jvm-example" plugin that can be filled in in the future as a smoke test
for breaking plugins, and gave some documentation to "spawn" modules
interface.

closes #12643
closes #12656
2015-08-10 14:04:45 -07:00
Simon Willnauer 13a347239a Don't run shaded tests as unit tests
If we run the tests as a reactor build we reference the dependencies
before they are shaded. This causes problems since we verify that unshaded versions
of a transitive dependency is not present. This commit moves the verification tests
into the integration test that always runs with the shaded version of the jar.
2015-08-10 16:29:19 +02:00
Adrien Grand 1dfc022f89 Merge pull request #12765 from jpountz/enhancement/convert-plugin-name
Tests: Move qa's convert-plugin-name macrodef to dev-tools.
2015-08-10 12:03:56 +02:00
Nik Everett f84552dc98 [Packaging] Run tests in vagrant
This creates a module in qa called vagrant that can be run if you have
vagrant and virtualbox installed and will run the packaging tests in trusty
and centos-7.0. You can ask it to run tests in other linuxes. This is the full
list:
* precise aka Ubuntu 12.04
* trusty aka Ubuntu 14.04
* vivid aka Ubuntun 15.04
* wheezy aka Debian 7, the current debian oldstable distribution
* jessie aka Debian 8, the current debina stable distribution
* centos-6
* centos-7
* fedora-22
* oel-7

There is lots of documentation on how to do this in the TESTING.asciidoc.

Closes #12611
2015-08-10 05:31:42 -04:00
Adrien Grand 9bf854752b Tests: Move qa's convert-plugin-name macrodef to dev-tools. 2015-08-10 11:07:52 +02:00
Clinton Gormley 205262c737 Enable the license checker over distribution/* and plugins/*
Moved the license checker config into the parent pom, and overrede
the license dir/target-to-check in distributions/pom.

Disabled the license checker explicitly for projects which run integration
tests but have no licenses dir:

* core
* distribution
* qa
* plugins/delete-by-query
* plugins/mapper-size
* plugins/site-example

Closes #12752
Closes #12754
2015-08-09 18:18:59 +02:00
Simon Willnauer f1e80b7f9c Add basic tests for sharded JAR
this commit adds a simple integration test that starts a
node from a shaded jar, indexes a doc and retrieves it. It
also has some basic unittests that try to load shaded classes and ensure
that their counterpart is not in the classpath.

Closes #12711
2015-08-07 14:14:33 +02:00
Robert Muir d1a5068b77 Merge branch 'master' into integ_randomization 2015-08-06 14:51:00 -04:00
Robert Muir 7414d19d28 Support jenkins randomization in integration tests 2015-08-06 14:48:27 -04:00
Adrien Grand 5dd5936fad Move the `_size` mapper to a plugin.
This is one of our esoteric metadata mappers so I think we should distribute
it in a plugin rather than in elasticsearch core.

This introduces one limitation: the value of the `_size` parameter is not
retrievable for documents that are only in the transaction log.
2015-08-06 20:35:22 +02:00
Robert Muir 5fe49e4420 cleanup skipping everywhere 2015-08-05 21:45:09 -04:00
Robert Muir 15e5247e03 Get plugin smoketester running in jenkins.
We have a smoke_test_plugins.py, but its a bit slow, not integrated
into our build, etc.

I converted this into an integration test. It is definitely uglier
but more robust and fast (e.g. 20 seconds time to verify).

Also there is refactoring of existing integ tests logic, like printing
out commands we execute and stuff
2015-08-05 20:24:36 -04:00