Commit Graph

315 Commits

Author SHA1 Message Date
Lee Hinman 178dc7ebaf Revert "Add no-Charset version of Files.readAllLines to forbidden apis"
This reverts commit 88765320a9.
2015-08-14 15:05:24 -06:00
Lee Hinman 88765320a9 Add no-Charset version of Files.readAllLines to forbidden apis 2015-08-14 14:55:22 -06:00
Alexander Reelsen e869c87119 Build: Adding aws-maven extension to dev-tools and rest-api-spec 2015-08-14 17:30:00 +02:00
Alexander Reelsen 3893544f98 Release scripts: Split prepare_release into two scripts
In order to reflect our RC release process, we need to separate
the prepare_release script into two separate scripts.

One script now updates the documentation. That one can be executed
anytime and needs to be pushed after that.

The other script updates the version in Version.java and all pom.xml
files, but does not commit anymore. This allows to create a non snapshot
version locally, run mvn deploy, push the artifacts into S3 and, upon
successful tests, simply release them on sonatype. This also allows for
updates, because the S3 snapshot will include the commitId in their repo
as already pushed before.
2015-08-14 16:44:39 +02:00
Clinton Gormley 1eaa8ee7b3 Add an optional "prefix_to_ignore" parameter to the license checker
If core plugins are to be renamed to not include the "elasticsearch-"
prefix, then we need a way of telling the license checker which
JAR files to ignore.
2015-08-14 15:41:05 +02:00
Simon Willnauer c1405d26b5 fixtypo 2015-08-14 15:27:23 +02:00
Simon Willnauer dd1b0d84a6 Also deploy top level artifacts to S3
This commit adds the S3 wagon release profile also to dev-tools
and rest-api-spec and makes the actual repository path / bucket
configurable.
2015-08-14 15:21:06 +02:00
Alexander Reelsen 0f3ada159e Release: Create pre release script
Refactored a part out of the release script, so the user can
change the version locally as well as move the documentation
and change the Version.java

The background of this change is to have a very simple release
process that puts stuff into a staging environment, so the beta
release can be tested, before it is officially released.

This means the build_release script can be removed soon.
2015-08-14 11:04:32 +02:00
Lee Hinman 2eebbcf450 [TEST] Use parent path for path.shared_data
On our Jenkins instances the ${path.home} and createTempDir() locations
share a different parent, so the custom index locations are not within
the ${path.shared_data} directory. This is a hack to fix it until we can
find a way to unify the createTempDir() and `path.shared_data` settings
inside the tests
2015-08-12 15:00:43 -06:00
Simon Willnauer 605253a39f Cut over master to 2.0.0-SNAPSHOT 2015-08-12 21:16:08 +02:00
Lee Hinman d1d2c6efbb [TEST] Add `path.shared_data` that was mistakenly rebased out 2015-08-12 12:05:07 -06:00
Clinton Gormley 68cc5ddf65 More outdates Perl libs in Suse 11 2015-08-12 14:08:10 +02:00
Clinton Gormley 77b5233d6d Check license script fails with perl 5.10.0 - include updated File::Temp lib 2015-08-12 11:01:29 +02:00
jaymode 4da02073ce allow specifying additional arguments in startup-elasticsearch macrodef
Previously we had additional.args as a argument to the startup-elasticsearch macrodef and this was
being used to set some additional elasticsearch settings. This adds the ability to specify additional
arguments back using a element called additional-args.
2015-08-11 09:12:43 -04:00
Simon Willnauer a8580d11f5 Improve arg handling in ant script
This commit is a spinn-off from #12710 which improves the handling of process
arguments ot play nice with spaces etc.
2015-08-11 13:08:32 +02:00
Simon Willnauer d1bc099c91 Add multi-node IT infrastructure
This adds the infrastrucutre to run integration tests with more than one node.
 * it adds relevant macros and targets to integration-tests.xml to start unicast nodes
 * there is a qa/smoke-test-multinode project that simulates such a setup

this commit is soely the infrastructure and doesn't hook up any projects to use this.
For reliability and stability reasons this should be used with care and only if it's really
needed.

Closes #12718
2015-08-11 11:36:15 +02:00
Colin Goodheart-Smithe 6abc69c488 Packaging: Makes sure all POMs contain a description
Adds an explicit description the RPM package so it doesn't inherit the description from the POM.

Closes #12550

Also, modified descriptions for deb and rpm packages to be the same and to reference the documentation rather than listing features that are out of date.
2015-08-10 15:21:24 +01:00
Adrien Grand 1dfc022f89 Merge pull request #12765 from jpountz/enhancement/convert-plugin-name
Tests: Move qa's convert-plugin-name macrodef to dev-tools.
2015-08-10 12:03:56 +02:00
Nik Everett f84552dc98 [Packaging] Run tests in vagrant
This creates a module in qa called vagrant that can be run if you have
vagrant and virtualbox installed and will run the packaging tests in trusty
and centos-7.0. You can ask it to run tests in other linuxes. This is the full
list:
* precise aka Ubuntu 12.04
* trusty aka Ubuntu 14.04
* vivid aka Ubuntun 15.04
* wheezy aka Debian 7, the current debian oldstable distribution
* jessie aka Debian 8, the current debina stable distribution
* centos-6
* centos-7
* fedora-22
* oel-7

There is lots of documentation on how to do this in the TESTING.asciidoc.

Closes #12611
2015-08-10 05:31:42 -04:00
Adrien Grand 9bf854752b Tests: Move qa's convert-plugin-name macrodef to dev-tools. 2015-08-10 11:07:52 +02:00
Clinton Gormley d1585782d4 Removed perl 5.8 support from the license checker
No point in keeping 5.8 support because it relies on modules
that are shipped with 5.9.
2015-08-09 19:54:32 +02:00
Clinton Gormley bc7dbce4fe Enable the license checker for plugins without third party deps too 2015-08-09 19:07:45 +02:00
Clinton Gormley 205262c737 Enable the license checker over distribution/* and plugins/*
Moved the license checker config into the parent pom, and overrede
the license dir/target-to-check in distributions/pom.

Disabled the license checker explicitly for projects which run integration
tests but have no licenses dir:

* core
* distribution
* qa
* plugins/delete-by-query
* plugins/mapper-size
* plugins/site-example

Closes #12752
Closes #12754
2015-08-09 18:18:59 +02:00
Simon Willnauer f1e80b7f9c Add basic tests for sharded JAR
this commit adds a simple integration test that starts a
node from a shaded jar, indexes a doc and retrieves it. It
also has some basic unittests that try to load shaded classes and ensure
that their counterpart is not in the classpath.

Closes #12711
2015-08-07 14:14:33 +02:00
Robert Muir 7414d19d28 Support jenkins randomization in integration tests 2015-08-06 14:48:27 -04:00
Robert Muir 57b1b72af6 Factor out waiting for easier redefinition 2015-08-06 08:28:08 -04:00
Robert Muir 5fe49e4420 cleanup skipping everywhere 2015-08-05 21:45:09 -04:00
Robert Muir c62f0655d3 fix/cleanup pidfile stuff 2015-08-05 20:34:24 -04:00
Robert Muir 15e5247e03 Get plugin smoketester running in jenkins.
We have a smoke_test_plugins.py, but its a bit slow, not integrated
into our build, etc.

I converted this into an integration test. It is definitely uglier
but more robust and fast (e.g. 20 seconds time to verify).

Also there is refactoring of existing integ tests logic, like printing
out commands we execute and stuff
2015-08-05 20:24:36 -04:00
Clinton Gormley 60f8fa4451 Added pure perl fallback for Digest::SHA for old windows boxes 2015-08-05 13:58:18 +02:00
Alexander Reelsen 9028fb8fbe Release: Update build release script to reflect latest changes
As the script now deploys to S3 and several things in master have
changed, this script needs to reflect the latest changes

* An unsigned RPM is built by default, so that users of older
  RPM based distros can download and use that RPM by default
* In addition a signed RPM is built, that is used for the repositories
* Paths for the new distributions have been fixed
* The check for the number of jars has been removed, as this is done
  as part of the license checking in `mvn verify`
* Checksum generation has been removed, as this is done as part of the
  mvn build
* Publishing artifacts of S3 has been removed
* Repostitory creation script has been updated
2015-08-05 09:38:42 +02:00
Robert Muir ead4f0fe1f Merge pull request #12660 from rmuir/improve_integ_tests
improve integration tests output when ES cannot be started.
2015-08-04 22:52:10 -04:00
Robert Muir 3c9fb6f3a2 improve integration tests output when ES cannot be started.
This can happen for a number of reasons, including bugs.
Today you will get a super crappy failure, telling you a .pid file
was not found... you can go look in target/integ-tests/elasticsearch-xxx/logs
and examine the log file, but thats kinda a pain and not easy if its a jenkins
server.

Instead we can fail like this:
```
start-external-cluster-with-plugin:
     [echo] Installing plugin elasticsearch-example-jvm-plugin...
    [mkdir] Created dir: /home/rmuir/workspace/elasticsearch/plugins/jvm-example/target/integ-tests/temp
     [exec] -> Installing elasticsearch-example-jvm-plugin...
     [exec] Plugins directory [/home/rmuir/workspace/elasticsearch/plugins/jvm-example/target/integ-tests/elasticsearch-2.0.0-SNAPSHOT/plugins] does not exist. Creating...
     [exec] Trying file:/home/rmuir/workspace/elasticsearch/plugins/jvm-example/target/releases/elasticsearch-example-jvm-plugin-2.0.0-SNAPSHOT.zip ...
     [exec] Downloading ...........DONE
     [exec] PluginInfo{name='example-jvm-plugin', description='Demonstrates all the pluggable Java entry points in Elasticsearch', site=false, jvm=true, classname=org.elasticsearch.plugin.example.ExampleJvmPlugin, isolated=true, version='2.0.0-SNAPSHOT'}
     [exec] Installed example-jvm-plugin into /home/rmuir/workspace/elasticsearch/plugins/jvm-example/target/integ-tests/elasticsearch-2.0.0-SNAPSHOT/plugins/example-jvm-plugin
     [echo] Starting up external cluster...
     [echo] [2015-08-04 22:02:55,130][INFO ][org.elasticsearch.node   ] [smoke_tester] version[2.0.0-SNAPSHOT], pid[4321], build[e2a47d8/2015-08-05T00:50:08Z]
     [echo] [2015-08-04 22:02:55,130][INFO ][org.elasticsearch.node   ] [smoke_tester] initializing ...
     [echo] [2015-08-04 22:02:55,259][INFO ][org.elasticsearch.plugins] [smoke_tester] loaded [uber-plugin], sites []
     [echo] [2015-08-04 22:02:55,260][ERROR][org.elasticsearch.bootstrap] Exception
     [echo] java.lang.NullPointerException
     [echo] 	at org.elasticsearch.common.settings.Settings$Builder.put(Settings.java:1051)
     [echo] 	at org.elasticsearch.plugins.PluginsService.updatedSettings(PluginsService.java:208)
     [echo] 	at org.elasticsearch.node.Node.<init>(Node.java:148)
     [echo] 	at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:157)
     [echo] 	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:177)
     [echo] 	at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:272)
     [echo] 	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:28)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 25.178 s
[INFO] Finished at: 2015-08-04T22:03:14-05:00
[INFO] Final Memory: 32M/515M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.8:run (integ-setup) on project elasticsearch-example-jvm-plugin: An Ant BuildException has occured: The following error occurred while executing this line:
[ERROR] /home/rmuir/workspace/elasticsearch/plugins/jvm-example/target/dev-tools/ant/integration-tests.xml:176: The following error occurred while executing this line:
[ERROR] /home/rmuir/workspace/elasticsearch/plugins/jvm-example/target/dev-tools/ant/integration-tests.xml:142: ES instance did not start
```
2015-08-04 22:16:30 -04:00
Igor Motov f71c9a25a1 Check for incompatible mappings while upgrading old indices
Conflicting mappings that were allowed before v2.0 can cause runaway shard failures on upgrade. This commit adds a check that prevents a cluster from starting if it contains such indices as well as restoring such indices from a snapshot into already running cluster.

Closes #11857
2015-08-04 18:13:29 -06:00
Robert Muir 39508c4d7a Merge pull request #12653 from rmuir/look_ma_im_a_snapshot_again
Change master branch back to 2.0-beta1
2015-08-04 16:10:59 -04:00
Robert Muir 6f9a067197 Change master branch back to 2.0-beta1 2015-08-04 15:38:21 -04:00
Clinton Gormley 17922d25b8 Change license-check script to work with perl 5.8 2015-08-04 21:31:31 +02:00
Robert Muir 29d73accdd Merge pull request #12639 from rmuir/improve_integ
Don't use port 9200/9300 for integration tests
2015-08-04 13:09:47 -04:00
Robert Muir abb43c3cc6 Merge branch 'master' into maven/distribution-groupid 2015-08-04 10:00:59 -04:00
Robert Muir f1019ace3b Fix plugins integ dependencies and apply cleanups 2015-08-04 09:59:19 -04:00
Simon Willnauer 96ad1911fd Use correct classname in all-signatures.txt 2015-08-04 15:36:18 +02:00
Robert Muir cf6fe6cdcd Don't use port 9200/9300 for integration tests 2015-08-04 09:25:44 -04:00
Clinton Gormley d0bd769b85 Added missing dependency to check-license script 2015-08-04 15:00:50 +02:00
Simon Willnauer 2b2657279d Use UTC instead of default timezone for creation date in CAT endpoint
this change was added recently which uses default timezone for the creation
date on CAT endpoints. We should be consistent and use UTC across the board.
This commit adds #getDefaultTimzone() to forbidden API and fixes the REST tests.

Relates to #11688
2015-08-04 14:38:23 +02:00
Clinton Gormley 20a5fc7e30 The check-license script now accepts either a directory or a .zip file only
Called as:

    check_license_and_sha.pl --check path/to/licenses path/to/extracted/package/

or

    check_license_and_sha.pl --check path/to/licenses path/to/file.zip
2015-08-04 13:07:58 +02:00
Simon Willnauer 6753f7f03e Cut over master to 2.0.0-SNAPSHOT 2015-08-04 10:54:12 +02:00
Ryan Ernst 1e12d03252 Tests: Rename base tests cases to use "TestCase" suffix
Most of the abstract base test classes we have were previously @Ignored.
However, there were also some other tests ignored. Having two ways to
quiet tests is confusing, and clearly it has caused some tests
to get lost in the fold.

This change moves all base test classes to use the "TestCase" suffix,
which is not picked up by the test class name pattern. It also removes
@Ignore from (almost) all tests, and adds it to forbidden apis.
And since we were renaming, I shorted base test class names to use
"ES" instead of "Elasticsearch". I type this a lot of types a day,
and I have heard others express a similar desire for a shorter name.

closes #10659
2015-08-03 17:43:00 -07:00
Ryan Ernst 32a5c20b33 Tests: Forbid @Slow
Now that all uses of @Slow are gone, we can forbid this annotations use
so tests are not confusing. Follow up to #12617.
2015-08-03 16:43:09 -07:00
Robert Muir e1886a9b12 Test that shaded jar contains only shaded classes 2015-08-03 08:56:45 -04:00
Robert Muir a5a27a5cd9 Add jar hell check test for the shaded jar 2015-08-03 05:54:55 -04:00
Robert Muir fef55348f9 clean up .zip handling so its not uselessly downloaded in the build 2015-08-01 09:10:03 -04:00
Robert Muir 8d5b5ad862 restore my lost code changes and add measures to detect/prevent this in the future 2015-07-31 13:00:24 -04:00
Robert Muir ddfea366de Add integration tests for distribution/rpm 2015-07-29 22:26:45 -04:00
Robert Muir 2ed8e697d5 Add integration tests for distribution/deb 2015-07-29 21:34:46 -04:00
Robert Muir 1c5a558164 Add integration tests for distribution/tar 2015-07-29 15:33:29 -04:00
Robert Muir c6d6eaef4d Add a run.sh to run from current source code with debugger
The maven magic needed for this is now heavier
2015-07-29 14:21:43 -04:00
Adrien Grand a832987ada Forbid Files.isHidden.
As Robert pointed out on #12465, it has the undesirable property of relying on
the operating system. So it would be better to use a simple rule such as
checking whether the file name starts with a dot.
2015-07-27 16:13:04 +02:00
Robert Muir d1783ab11e improve plugin file some more 2015-07-23 11:37:55 -04:00
Robert Muir 4b2bda4195 make java.version mandatory for jvm plugins
JarHell has a low level check, but its more of a best effort one,
only checking if X-Compile-Target-JDK is set in the manifest. This
is the case for all lucene- and elasticsearch- generated jars, but
lets just be explicit for plugins.
2015-07-23 10:58:52 -04:00
Clinton Gormley 235452329a In the license checker, check for the existence of the specfied package before trying to unpack it 2015-07-23 14:31:42 +02:00
Robert Muir 4040f194f5 Refactor pluginservice
Closes #12367

Squashed commit of the following:

commit 9453c411798121aa5439c52e95301f60a022ba5f
Merge: 3511a9c 828d8c7
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Jul 22 08:22:41 2015 -0400

    Merge branch 'master' into refactor_pluginservice

commit 3511a9c616503c447de9f0df9b4e9db3e22abd58
Author: Ryan Ernst <ryan@iernst.net>
Date:   Tue Jul 21 21:50:15 2015 -0700

    Remove duplicated constant

commit 4a9b5b4621b0ef2e74c1e017d9c8cf624dd27713
Author: Ryan Ernst <ryan@iernst.net>
Date:   Tue Jul 21 21:01:57 2015 -0700

    Add check that plugin must specify at least site or jvm

commit 19aef2f0596153a549ef4b7f4483694de41e101b
Author: Ryan Ernst <ryan@iernst.net>
Date:   Tue Jul 21 20:52:58 2015 -0700

    Change plugin "plugin" property to "classname"

commit 07ae396f30ed592b7499a086adca72d3f327fe4c
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 23:36:05 2015 -0400

    remove test with no methods

commit 550e73bf3d0f94562f4dde95239409dc5a24ce25
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 23:31:58 2015 -0400

    fix loading to use classname

commit 04463aed12046da0da5cac2a24c3ace51a79f799
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 23:24:19 2015 -0400

    rename to classname

commit 9f3afadd1caf89448c2eb913757036da48758b2d
Author: Ryan Ernst <ryan@iernst.net>
Date:   Tue Jul 21 20:18:46 2015 -0700

    moved PluginInfo and refactored parsing from properties file

commit df63ccc1b8b7cc64d3e59d23f6c8e827825eba87
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 23:08:26 2015 -0400

    fix test

commit c7febd844be358707823186a8c7a2d21e37540c9
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 23:03:44 2015 -0400

    remove test

commit 017b3410cf9d2b7fca1b8653e6f1ebe2f2519257
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 22:58:31 2015 -0400

    fix test

commit c9922938df48041ad43bbb3ed6746f71bc846629
Merge: ad59af4 01ea89a
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 22:37:28 2015 -0400

    Merge branch 'master' into refactor_pluginservice

commit ad59af465e1f1ac58897e63e0c25fcce641148a7
Author: Areek Zillur <areek.zillur@elasticsearch.com>
Date:   Tue Jul 21 19:30:26 2015 -0400

    [TEST] Verify expected number of nodes in cluster before issuing shardStores request

commit f0f5a1e087255215b93656550fbc6bd89b8b3205
Author: Lee Hinman <lee@writequit.org>
Date:   Tue Jul 21 11:27:28 2015 -0600

    Ignore EngineClosedException during translog fysnc

    When performing an operation on a primary, the state is captured and the
    operation is performed on the primary shard. The original request is
    then modified to increment the version of the operation as preparation
    for it to be sent to the replicas.

    If the request first fails on the primary during the translog sync
    (because the Engine is already closed due to shadow primaries closing
    the engine on relocation), then the operation is retried on the new primary
    after being modified for the replica shards. It will then fail due to the
    version being incorrect (the document does not yet exist but the request
    expects a version of "1").

    Order of operations:

    - Request is executed against primary
    - Request is modified (version incremented) so it can be sent to replicas
    - Engine's translog is fsync'd if necessary (failing, and throwing an exception)
    - Modified request is retried against new primary

    This change ignores the exception where the engine is already closed
    when syncing the translog (similar to how we ignore exceptions when
    refreshing the shard if the ?refresh=true flag is used).

commit 4ac68bb1658688550ced0c4f479dee6d8b617777
Author: Shay Banon <kimchy@gmail.com>
Date:   Tue Jul 21 22:37:29 2015 +0200

    Replica allocator unit tests
    First batch of unit tests to verify the behavior of replica allocator

commit 94609fc5943c8d85adc751b553847ab4cebe58a3
Author: Jason Tedor <jason@tedor.me>
Date:   Tue Jul 21 14:04:46 2015 -0400

    Correctly list blobs in Azure storage to prevent snapshot corruption and do not unnecessarily duplicate Lucene segments in Azure Storage

    This commit addresses an issue that was leading to snapshot corruption for snapshots stored as blobs in Azure Storage.

    The underlying issue is that in cases when multiple snapshots of an index were taken and persisted into Azure Storage, snapshots subsequent
    to the first would repeatedly overwrite the snapshot files. This issue does render useless all snapshots except the final snapshot.

    The root cause of this is due to String concatenation involving null. In particular, to list all of the blobs in a snapshot directory in
    Azure the code would use the method listBlobsByPrefix where the prefix is null. In the listBlobsByPrefix method, the path keyPath + prefix
    is constructed. However, per 5.1.11, 5.4 and 15.18.1 of the Java Language Specification, the reference null is first converted to the string
    "null" before performing the concatenation. This leads to no blobs being returned and therefore the snapshot mechanism would operate as if
    it were writing the first snapshot of the index. The fix is simply to check if prefix is null and handle the concatenation accordingly.

    Upon fixing this issue so that subsequent snapshots would no longer overwrite earlier snapshots, it was discovered that the snapshot metadata
    returned by the listBlobsByPrefix method was not sufficient for the snapshot layer to detect whether or not the Lucene segments had already
    been copied to the Azure storage layer in an earlier snapshot. This led the snapshot layer to unnecessarily duplicate these Lucene segments
    in Azure Storage.

    The root cause of this is due to known behavior in the CloudBlobContainer.getBlockBlobReference method in the Azure API. Namely, this method
    does not fetch blob attributes from Azure. As such, the lengths of all the blobs appeared to the snapshot layer to be of length zero and
    therefore they would compare as not equal to any new blobs that the snapshot layer is going to persist. To remediate this, the method
    CloudBlockBlob.downloadAttributes must be invoked. This will fetch the attributes from Azure Storage so that a proper comparison of the
    blobs can be performed.

    Closes elastic/elasticsearch-cloud-azure#51, closes elastic/elasticsearch-cloud-azure#99

commit cf1d481ce5dda0a45805e42f3b2e0e1e5d028b9e
Author: Lee Hinman <lee@writequit.org>
Date:   Mon Jul 20 08:41:55 2015 -0600

    Unit tests for `nodesAndVersions` on shared filesystems

    With the `recover_on_any_node` setting, these unit tests check that the
    correct node list and versions are returned.

commit 3c27cc32395c3624f7c794904d9ea4faf2eccbfb
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 14:15:59 2015 -0400

    don't fail junit4 integration tests if there are no tests.

    instead fail the failsafe plugin, which means the external cluster will still get shut down

commit 95d2756c5a8c21a157fa844273fc83dfa3c00aea
Author: Alexander Reelsen <alexander@reelsen.net>
Date:   Tue Jul 21 17:16:53 2015 +0200

    Testing: Fix help displaying tests under windows

    The help files are using a unix based file separator, where as
    the test relies on the help being based on the file system separator.

    This commit fixes the test to remove all `\r` characters before
    comparing strings.

    The test has also been moved into its own CliToolTestCase, as it does
    not need to be an integration test.

commit 944f06ea36bd836f007f8eaade8f571d6140aad9
Author: Clinton Gormley <clint@traveljury.com>
Date:   Tue Jul 21 18:04:52 2015 +0200

    Refactored check_license_and_sha.pl to accept a license dir and package path

    In preparation for the move to building the core zip, tar.gz, rpm, and deb as separate modules, refactored check_license_and_sha.pl to:

    * accept a license dir and path to the package to check on the command line
    * to be able to extract zip, tar.gz, deb, and rpm
    * all packages except rpm will work on Windows

commit 2585431e8dfa5c82a2cc5b304cd03eee9bed7a4c
Author: Chris Earle <pickypg@users.noreply.github.com>
Date:   Tue Jul 21 08:35:28 2015 -0700

    Updating breaking changes

    - field names cannot be mapped with `.` in them
    - fixed asciidoc issue where the list was not recognized as a list

commit de299b9d3f4615b12e2226a1e2eff5a38ecaf15f
Author: Shay Banon <kimchy@gmail.com>
Date:   Tue Jul 21 13:27:52 2015 +0200

    Replace primaryPostAllocated flag and use UnassignedInfo
    There is no need to maintain additional state as to if a primary was allocated post api creation on the index routing table, we hold all this information already in the UnassignedInfo class.
    closes #12374

commit 43080bff40f60bedce5bdbc92df302f73aeb9cae
Author: Alexander Reelsen <alexander@reelsen.net>
Date:   Tue Jul 21 15:45:05 2015 +0200

    PluginManager: Fix bin/plugin calls in scripts/bats test

    The release and smoke test python scripts used to install
    plugins in the old fashion.

    Also the BATS testing suite installed/removed plugins in that
    way. Here the marvel tests have been removed, as marvel currently
    does not work with the master branch.

    In addition documentation has been updated as well, where it was
    still missing.

commit b81ccba48993bc13c7678e6d979fd96998499233
Author: Boaz Leskes <b.leskes@gmail.com>
Date:   Tue Jul 21 11:37:50 2015 +0200

    Discovery: make sure NodeJoinController.ElectionCallback is always called from the update cluster state thread

    This is important for correct handling of the joining thread. This causes assertions to trip in our test runs. See http://build-us-00.elastic.co/job/es_g1gc_master_metal/11653/ as an example

    Closes #12372

commit 331853790bf29e34fb248ebc4c1ba585b44f5cab
Author: Boaz Leskes <b.leskes@gmail.com>
Date:   Tue Jul 21 15:54:36 2015 +0200

    Remove left over no commit from TransportReplicationAction

    It asks to double check thread pool rejection. I did and don't see problems with it.

commit e5724931bbc1603e37faa977af4235507f4811f5
Author: Alexander Reelsen <alexander@reelsen.net>
Date:   Tue Jul 21 15:31:57 2015 +0200

    CliTool: Various PluginManager fixes

    The new plugin manager parser was not called correctly in the scripts.
    In addition the plugin manager now creates a plugins/ directory in case
    it does not exist.

    Also the integration tests called the plugin manager in the deprecated way.

commit 7a815a370f83ff12ffb12717ac2fe62571311279
Author: Alexander Reelsen <alexander@reelsen.net>
Date:   Tue Jul 21 13:54:18 2015 +0200

    CLITool: Port PluginManager to use CLITool

    In order to unify the handling and reuse the CLITool infrastructure
    the plugin manager should make use of this as well.

    This obsolets the -i and --install options but requires the user
    to use `install` as the first argument of the CLI.

    This is basically just a port of the existing functionality, which
    is also the reason why this is not a refactoring of the plugin manager,
    which will come in a separate commit.

commit 7f171eba7b71ac5682a355684b6da703ffbfccc7
Author: Martijn van Groningen <martijn.v.groningen@gmail.com>
Date:   Tue Jul 21 10:44:21 2015 +0200

    Remove custom execute local logic in TransportSingleShardAction and TransportInstanceSingleOperationAction and rely on transport service to execute locally. (forking thread etc.)

    Change TransportInstanceSingleOperationAction to have shardActionHandler to, so we can execute locally without endless spinning.

commit 0f38e3eca6b570f74b552e70b4673f47934442e1
Author: Ryan Ernst <ryan@iernst.net>
Date:   Tue Jul 21 17:36:12 2015 -0700

    More readMetadata tests and pickiness

commit 880b47281bd69bd37807e8252934321b089c9f8e
Author: Ryan Ernst <ryan@iernst.net>
Date:   Tue Jul 21 14:42:09 2015 -0700

    Started unit tests for plugin service

commit cd7c8ddd7b8c4f3457824b493bffb19c156c7899
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 07:21:07 2015 -0400

    fix tests

commit 673454f0b14f072f66ed70e32110fae4f7aad642
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 06:58:25 2015 -0400

    refactor pluginservice
2015-07-22 10:45:45 -04:00
Clinton Gormley f84757e8bb Refactored check_license_and_sha.pl to accept a license dir and package path
In preparation for the move to building the core zip, tar.gz, rpm, and deb as separate modules, refactored check_license_and_sha.pl to:

* accept a license dir and path to the package to check on the command line
* to be able to extract zip, tar.gz, deb, and rpm
* all packages except rpm will work on Windows
2015-07-21 18:04:52 +02:00
Alexander Reelsen d3e454780f PluginManager: Fix bin/plugin calls in scripts/bats test
The release and smoke test python scripts used to install
plugins in the old fashion.

Also the BATS testing suite installed/removed plugins in that
way. Here the marvel tests have been removed, as marvel currently
does not work with the master branch.

In addition documentation has been updated as well, where it was
still missing.
2015-07-21 16:17:59 +02:00
Alexander Reelsen 999d5ab801 CliTool: Various PluginManager fixes
The new plugin manager parser was not called correctly in the scripts.
In addition the plugin manager now creates a plugins/ directory in case
it does not exist.

Also the integration tests called the plugin manager in the deprecated way.
2015-07-21 15:31:57 +02:00
Robert Muir cd9d1c617f Merge pull request #12344 from rmuir/2.1.16
Update randomizedtesting to 2.1.16
2015-07-20 11:13:57 -04:00
Robert Muir a0a6299840 Update randomizedtesting to 2.1.16 2015-07-20 11:02:50 -04:00
Robert Muir 48bc49c093 Remove broken `exec` build target, replace with something better.
Currently this target is "yet another way" to run elasticsearch,
which we can't maintain. It also has the problem that it doesnt
ensure its running on the latest source code, doesn't configure
any scratch space properly, won't work with securitymanager, list
goes on.

Even if we made it work, it would break every day, since its untested.

Instead, `mvn package -Drun -DskipTests` will run packaging, and then
startup bin/elasticsearch (like integration tests, but in foreground).

It also enables debugger socket on port 8000, for people that like
IDE debuggers and not system.out.println.

Its a little slower to get started because of all the shading/RPM/DEB
building going on in `package` but that is just what it is right now
until that stuff is moved out.
2015-07-17 00:03:33 -04:00
Robert Muir 9e7882e604 protect logic under skipTests for stupid pom packaging 2015-07-16 20:49:16 -04:00
Robert Muir 6b7561ac9a Use junit4 for running integration tests, too
failsafe uses surefire, which sucks. It also mean integ tests act alien right now.
I would rather have the consistency, e.g. things formatted the same way, running integ tests under security manager, etc.
2015-07-16 19:43:33 -04:00
Robert Muir 4804bf747b skip this target too, to not confuse POM builds 2015-07-15 00:30:26 -04:00
Robert Muir 68143dedf2 Refactor integration tests
1. tests don't have a bogus test dependency on zips anymore,
   instead we handle this in pre-integration-test. This reduces
   lots of confusion for e.g. mvn clean test.
2. refactor integ logic so that core/ and plugin/ share it.
   previously they were duplicates but the above change simplifies life.
   it also makes it easier for doing more interesting stuff
2015-07-14 23:37:04 -04:00
Igor Motov 24a93840d5 Add url repository whitelist
Require urls for URL repository to be listed in repositories.url.allowed_urls setting. This change ensures that only authorized URLs can be accessed by elasticsearch
2015-07-14 18:38:26 -04:00
uboness b40186652c updated the elasticsearch versioning format
Moving to from `X.Y.Z.beta1`/`X.Y.Z.RC1` to `X.Y.Z-beta1`/`X.Y.Z-rc1`
2015-07-13 20:26:37 +02:00
Robert Muir c2c8956347 enforce plugins are installed with correct name, and install correctly in integ tests 2015-07-10 19:20:25 -04:00
Simon Willnauer e0708813a9 Make 2.0.0.beta1-SNAPSHOT the current version.
Today everything is tight to having the next version as the latest.
In order to work towards 2.0.0.beta1 we need to fix all the usage of
2.0.0-SNAPSHOT to reflect the version we will release soon.
Usually we do this on the release branch but to simplify things I wanna
keep this on master for now and move to 2.1.0-SNAPSHOT on master once
we created a 2.0 branch.

Closes #12148
2015-07-09 21:24:32 +02:00
Robert Muir 8ef73b7dba Get integration tests working on windows.
We also run our license checker in `mvn verify`, but there
are problems with checksum calculation on windows there, so I've
disabled the license checker on windows to prevent those false fails.
2015-07-08 02:40:00 -04:00
Robert Muir fcddec532a Use extract-pid when killing, too 2015-07-08 00:10:25 -04:00
Robert Muir 7a9f74de80 print PID when starting ES in integ tests too. 2015-07-08 00:04:27 -04:00
Robert Muir fb62d945aa isolate integ tests a little better.
They are mostly just ES clients, but we can set tempdir and the usual
stuff to contain them better.
2015-07-07 23:10:18 -04:00
Robert Muir 85a25bdf73 harden logic around integ test workspace and process mgmt
there is more to do here, but this is already a lot more robust.

* don't clean workspace in teardown, it might be useful for debugging if stuff fails.
* kill ES/clean workspace in setup, so things always work even in the case of ^C
* use pidfile to kill
* fail if kill errors
* refactor a bit more logic here
2015-07-07 20:11:50 -04:00
Robert Muir 5884628049 Integ setup should unzip org.elasticsearch:elasticsearch:zip
This property is set by maven, and unlike the current hack, during
a multimodule build will be set to the correct thing.

Otherwise today sometimes we run integ tests with outdated ES
artifacts, which makes for incredibly confusing failures.

Closes #12101
2015-07-07 19:04:46 -04:00
Robert Muir 023cb0b174 make integ testing a bit more picky 2015-07-07 10:21:40 -04:00
Robert Muir 23dfb5f1a5 don't try to run tests for packaging=pom 2015-07-06 15:49:57 -04:00
Robert Muir 9a146b9e75 Get delete by query rest tests running 2015-07-06 15:13:50 -04:00
Robert Muir 75285cee88 factor out this helper 2015-07-06 14:19:28 -04:00
Robert Muir 7595104ec3 Factor integration tests logic to separate build file 2015-07-06 13:59:16 -04:00
David Pilato e7a6b51bab [maven] change groupId / artifactId
When we generate our project, we can get something like:

```
├── dev-tools
├── elasticsearch
├── elasticsearch-parent
├── elasticsearch-plugin
├── plugin
│   ├── elasticsearch-analysis-icu
│   ├── elasticsearch-analysis-kuromoji
│   ├── elasticsearch-analysis-phonetic
│   ├── elasticsearch-analysis-smartcn
│   ├── elasticsearch-analysis-stempel
│   ├── elasticsearch-cloud-aws
│   ├── elasticsearch-cloud-azure
│   ├── elasticsearch-cloud-gce
│   ├── elasticsearch-delete-by-query
│   ├── elasticsearch-lang-javascript
│   └── elasticsearch-lang-python
├── rest-api-spec
└── securemock
```

I propose here to use a common naming for artifacts: start always with `elasticsearch-`.
Also, move `elasticsearch-plugin` to `org.elasticsearch.plugin` groupId.

So we could have:

```
├── elasticsearch
├── elasticsearch-dev-tools
├── elasticsearch-parent
├── elasticsearch-rest-api-spec
├── elasticsearch-securemock
├── plugin
│   ├── elasticsearch-analysis-icu
│   ├── elasticsearch-analysis-kuromoji
│   ├── elasticsearch-analysis-phonetic
│   ├── elasticsearch-analysis-smartcn
│   ├── elasticsearch-analysis-stempel
│   ├── elasticsearch-cloud-aws
│   ├── elasticsearch-cloud-azure
│   ├── elasticsearch-cloud-gce
│   ├── elasticsearch-delete-by-query
│   ├── elasticsearch-lang-javascript
│   ├── elasticsearch-lang-python
│   └── elasticsearch-plugin
```
2015-07-06 17:17:07 +02:00
Robert Muir 3f4b8df00d Merge pull request #12026 from rmuir/integ_tests
add integration test harness to maven build
2015-07-06 10:16:54 -04:00
Clinton Gormley c5d3c46943 Added build and test labels to es release notes script 2015-07-05 20:24:46 +02:00
Tanguy Leroux 9495816cb7 Remove sigar completely 2015-07-03 15:49:17 +02:00
Michael McCandless 275fdcc08d fix silly smoke_test_plugin bugs when ES actually succeeds in starting with the installed plugins 2015-07-03 05:59:16 -04:00
Robert Muir 80871bae2b Add simple integ testing infra 2015-07-03 02:12:01 -04:00
Michael McCandless 48b85421ec fix smoke_test_plugins.py: install the .zip under releases, not the .jar 2015-06-30 15:03:54 -04:00
Michael McCandless 189ef91e3c first cut 2015-06-30 14:14:45 -04:00
Simon Willnauer e7eb9cf4de Ban java serialization
We had several problems with Java Serializatin in the past. At some point
in the Java 1.7.x series JDKs where not compatible anymore when java
serialization (ObjectStream) was used to exchange objects. In elasticsearch
we used this to serialize exceptions across the wire which caused several problems
with incompatible JDKs. While causing lot of trouble this essentially prevented
users from moving forward and upgrade their JVMs. To prevent these kind of issues
this commit removes the dependency on java serialization entirely and bans the
usage of ObjectOutputStream and ObjectInputStream entirely.

Yet, we can't fully serialize all exception anymore such that this commit
is best effort and adds hand written serialization to all elasticsearch exceptions
as well to a selected set of JDK and Lucene exceptions. (see StreamOutput#writeThrowable /
StreamInput.readThrowable). Stacktraces should be preserved for all exceptions while
several names might be replaced with ElasticsearchException if there is no mapping for
the given exception.
2015-06-30 14:51:43 +02:00
Alexander Reelsen f26672c184 Release: Build two RPMS, signed and unsigned
In order to support older RPM based distributions like CentOS5,
we should have one RPM available, which is not signed.

This commit creates an unsigned RPM first, then moves it over to
target/releases during the build, then builds a signed RPM.

The unsigned one is uploaded via S3, where as the signed one is
used for the repositories.

In addition, you can now build an RPM without having to specify
any gpg credentials due to offloading this into a maven profile
that is only activated when specifying `rpm.sign` property.

Closes #11587
2015-06-30 14:22:20 +02:00
Clinton Gormley fa40680736 Build: If SHA files have changed, explain how to update them in the license check exception 2015-06-30 11:29:35 +02:00
Simon Willnauer 91ab808ce2 Revert accidential modification of release script in this files previous commit
commit 05db5dc2c8
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Jun 5 13:12:05 2015 +0200

    create parent pom project from its original location
2015-06-25 17:18:34 +02:00
Clinton Gormley a3d1a50865 Build: tar on linux needs the --wildcard option, but not supported on OSX
Removing '*.jar'  filter when untarring during the license check
2015-06-23 14:07:10 +02:00
Clinton Gormley 9fb3bf06c5 Changed the license checker to use the ZIP file as the source of JARs to check.
Also checks that the tar.gz file (if present) contains the same JARs as the
ZIP file.
2015-06-23 12:50:31 +02:00