Commit Graph

453 Commits

Author SHA1 Message Date
Martijn Laarman a80317c4b3 cmd /C needs to be quoted as a whole
To support spaces in both the command as well as its arguments cmd needs
be called like this:

cmd /C ""c:\a b\c.bat" "argument 1" "argument2""

ant was running

cmd /C "c:\a b\c.bat" "argument 1" "argument2"

which in windows causes to be preprocessed to

cmd /C c:\a b\c.bat" "argument 1" "argument2

Which would make it appear as though ant was not properly quoting (which
it did sort of).
2015-08-31 13:33:10 +02:00
Jason Tedor 3067cacb66 Remove and forbid use of com.google.common.collect.Lists
This commit removes and now forbids all uses of
com.google.common.collect.Lists across the codebase. This is the first
of many steps in the eventual removal of Guava as a dependency.
2015-08-27 22:14:33 -04:00
Simon Willnauer f64a875e03 use provided version in smoke test file paths 2015-08-27 23:20:01 +02:00
Ryan Ernst 38b8f20cc5 Make 0.x and 1.x indexes still work with get-bwc-version 2015-08-27 10:19:59 -07:00
Ryan Ernst 1029b119d9 Tests: Fix generation scripts for bwc indexes, and add 2.0 beta1 index 2015-08-26 19:36:26 -07:00
Simon Willnauer 5579b8fad9 [BUILD] Add publish instructions to the release script 2015-08-26 16:22:14 +02:00
Simon Willnauer d4a9089253 [BUILD] Allow smoke-testing other artifact but staging artifacts 2015-08-26 16:07:50 +02:00
Britta Weber 12aeafcbed Merge pull request #12961 from brwe/jps-check
check with jps that the pid file contains a pid that actually is an e…
2015-08-26 14:29:13 +02:00
Clinton Gormley 13727ba6fd Release notes: Add upgrades category 2015-08-25 17:16:14 +02:00
Simon Willnauer 3175be7a25 Add smoke-tester to mail template 2015-08-25 12:19:24 +02:00
Simon Willnauer aaca188571 Merge pull request #13078 from s1monw/add_smoke_tester
Add RC smoke tester that checks basic functionality
2015-08-25 10:53:43 +02:00
Nik Everett 0de493e35d Merge pull request #13070 from nik9000/eclipse_fix
Fix some issues developing Elasticsearch with Eclipse
2015-08-24 15:35:50 -04:00
Simon Willnauer 3a01f7b3b4 [BUILD] Python3 doesn't have dict.iter_items - it's replaced by dict.items() 2015-08-24 12:52:09 +02:00
Alexander Reelsen cb68a2196d Release: Prevent running whole lifecycle twice 2015-08-24 10:26:49 +02:00
Nik Everett bb9e77f876 Eclipse: Fix launch configuration
We had a file in dev-tools/ElasticSearch.launch which tried to launch
elasticsearch but failed somewhat epically because of the security manager
and files having moved. This recreates it with
`-Des.security.manager.enabled=false` to get it working again. Its not as nice
as testing with the security manager in place but its better than waiting
minutes for maven to package and startup elasticsearch.
2015-08-21 19:07:03 -07:00
Simon Willnauer 5458d07ea2 Refactor script for RC creation
The script now allows to run all required steps at once and alternatively
prints out manual instructions to run the steps individually. It also has
flags and options to run debug builds from a local checkout.
2015-08-21 14:36:59 +02:00
Jason Tedor 31b80e4f3f Merge pull request #13010 from jasontedor/feature/improve-java-version-comparison
Improve java version comparison and explicitly enforce a version format
2015-08-21 08:30:50 -04:00
Jason Tedor 5ff1f8a058 Enforce version format for java.version for plugins
Currently we implicitly enforce a version format on the java.version
property for plugins via JarHell.checkJavaVersion. We should explicitly
enforce this version format and specify it in the documentation.

This commit adds an explicit enforcement of the version format on the
java.version property for plugins and updates the documentation for
plugin-descriptor.properties accordingly.

Closes #13009
2015-08-21 08:28:41 -04:00
Ryan Ernst 164efaecbe Networking: Move multicast discovery to a plugin
Multicast has known issues (see #12999 and #12993). This change moves
multicast into a plugin, and deprecates it in the docs.  It also allows
for plugging in multiple zen ping implementations.

closes #13019
2015-08-20 16:43:25 -07:00
Robert Muir e2ab62596f Default to unicast discovery, with default host list of 127.0.0.1, [::1]
Fix unicast discovery to work when a host has multiple addresses.
Ban dangerous methods in java.net with forbidden APIs.
Fix ipv6 bugs and formatting of network addresses everywhere.

Closes #12999
Closes #12993

Squashed commit of the following:

commit 6c1aa001d091c5cf25212a53dc701fb704337f1e
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 14:25:43 2015 -0400

    Fix these to be correct with addresses just in case

commit 648215627e84abf58a71400e7dc9ae775efb71d6
Merge: d00561b 41d8fbe
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 13:23:09 2015 -0400

    Merge branch 'master' into unicast_all_the_way_down

commit d00561b76fd1aa5850699f7901f3dae3d4d402b7
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 16:38:50 2015 +0200

    limit local ports to 5 in UnicastZenPing

commit e2e15c594006746cbe24432694294a71cc99deb8
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 10:32:47 2015 -0400

    fix port limiting

commit 10153cb7adadda81a1f482445e703836b65cf5e2
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 10:18:37 2015 -0400

    don't serialize scopeids: that's broken

commit 2aa63d43db2baec68a2e9bc227cfeb85dfeb4f83
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 16:06:51 2015 +0200

    restore @Network

commit c840f1d1ef438826ae1ecfd5e45942a0e30dc9c0
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 16:02:30 2015 +0200

     Use NetworkAddress.formatAddress where applicable in plugins

commit 374ce878852b35d626b7a29c8c4773545b0e9ddd
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 15:34:06 2015 +0200

    Use NetworkAddress.formatAddress where applicable

commit e7a606d63f1bc43c1b62b6e17adf707c76d43a15
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 10:17:57 2015 +0200

    Add @Multicast annotation to disable multicast tests by default.

    We only run multicast tests now when we explicitly state it. A working
    multicast env is required which is not always the case.

commit 2d7d2d0347179696ab41f71f048b13305014c85b
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 09:51:28 2015 +0200

    Remove extra check for local mode in InternalTestCluster

commit dda59ac39aa136d4687b9274c2692cd77f8b8f66
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 09:37:03 2015 +0200

    Handle node mode across entire test cluster

    We used static methods reading sys properties to define the node mode
    per cluster. this had lots of problems when tests couldn't cope with
    mixed or only local mode. Now we are passing it down to the cluster from the test
    which allows to @SuppressNetworkMode / @SupressLocalMode on the test to force
    consistent node configurations.

commit 058197b7a408318995c88ce7f6762e32348de0de
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 03:19:14 2015 -0400

    really ban InetSocketAddress's trappy method and break build and go to sleep, sorry

commit ac8779185aee1e17e6f5a81766290fdfc9c603ba
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 03:16:52 2015 -0400

    Ban methods that might surprisingly cause DNS lookups

commit e64fe3dff2b11503e5f2831eb9863d64f56c5538
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 02:59:05 2015 -0400

    Add unit test

commit f15434f20fb1a3691b1cc16028597d8fae937e05
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 02:39:02 2015 -0400

    fix ipv6 formatting bugs

commit 05c2c74098052c75fbb79ea1818a295ef2e03e30
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 02:12:05 2015 -0400

    format addresses correctly so I can actually read what comes out of our logs and stats apis

commit 4f9389dcf1e8925f23153c5eb271b4ce2294dbaf
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Aug 19 21:26:52 2015 -0400

    ban dangerous methods in java.net

commit 6aacd4d9925f324903d1d099a6cf5f862aeaf677
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Aug 19 20:59:24 2015 -0400

    ban lenient method

commit f466a842c60163d1f4554bdce8a4163edb534c2c
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 00:29:00 2015 +0200

    fix tests to not mix local transport and zen unicast disco

commit 0de007a33b33fb68cf85cd86db4ca4f8ce10bbc9
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 00:10:07 2015 +0200

    fix tests to not mix local transport and zen unicast disco

commit 539f6ca6e5137e0d496239adc8684688dedcc824
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 00:02:01 2015 +0200

    fix tests to not mix local transport and zen unicast disco

commit 004c2881b25467f332acc8c9f9e92b1f0f9d314e
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Aug 19 17:51:45 2015 -0400

    Fix multinode

commit 54113af325ce31571811c49fdaae89d5687be4ba
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Aug 19 17:36:45 2015 -0400

    fix integration tests

commit 0156a77a56319d6b9737ec6a531992052e50bd59
Author: Simon Willnauer <simonw@apache.org>
Date:   Wed Aug 19 23:32:18 2015 +0200

    enable multicast in MulticastZenPingIT.java

commit 1791caa35da853ce0122485fa3fd4674c671ec6e
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Aug 19 17:23:16 2015 -0400

    Fix constant

commit 22820b53e0b2dc9fd47145c2bc29ce912a8fd484
Author: Simon Willnauer <simonw@apache.org>
Date:   Wed Aug 19 22:59:09 2015 +0200

    give it some extra ids for local transport crazyness

commit b2138fafa94a8a085813fd48356df63e57ade5b3
Author: Simon Willnauer <simonw@apache.org>
Date:   Wed Aug 19 22:51:42 2015 +0200

    pass on local addresses from configured transport rather than hard code IP addresses

commit 1bf5de1f457b081e0ce262b57d2b55d39c434156
Author: Simon Willnauer <simonw@apache.org>
Date:   Wed Aug 19 22:04:31 2015 +0200

    fix PluggableTransportModuleIT.java to use local disco and detach port limit for node local disco

commit b6706eddfa04c43947c16551359ae98a463d34aa
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Aug 19 14:16:03 2015 -0400

    Default to unicast discovery, with default host list of 127.0.0.1, [::1]
2015-08-20 14:26:40 -04:00
Britta Weber c0d31a2c9e Merge pull request #12965 from brwe/es-running-already
fail with better error message if elasticsearch was started already
2015-08-19 17:56:56 +02:00
Britta Weber 4cc6359756 fail with better error message if elasticsearch was started already
Before it failed with
.../es/qa/smoke-test-plugins/target/integ-tests/es.pid doesn't exist
which was confusing.
2015-08-19 17:56:05 +02:00
Christoph Büscher 8454d49552 Plugins: Add 'name' property to plugin descriptor file to determine plugin name
At the moment, when installing from an url, a user provides the plugin name on
the command line like:

* bin/plugin install [plugin-name] --url [url]

This can lead to problems when picking an already existing name from another
plugin, and can potentially overwrite plugins already installed with that name.

This, this PR introduces a mandatory `name` property to the plugin descriptor
file which replaces the name formerly provided by the user.

With the addition of the `name` property to the plugin descriptor file, the user
does not need to specify the plugin name any longer when installing from a file
or url. Because of this, all arguments to `plugin install` command are now
either treated as a symbolic name, a URL or a file without the need to specify
this with an explicit option.

The new syntax for `plugin install` is now:

bin/plugin install [name or url]

* downloads official plugin
bin/plugin install analysis-kuromoji

* downloads github plugin
bin/plugin install lmenezes/elasticsearch-kopf

* install from URL or file
bin/plugin install http://link.to/foo.zip
bin/plugin install file:/path/to/foo.zip

If the argument does not parse to a valid URL, it is assumed to be a name and the
download location is resolved like before. Regardless of the source location of
the plugin, it is extracted to a temporary directory and the `name` property from
the descriptor file is used to determine the final install location.

Relates to #12715
2015-08-19 12:13:01 +02:00
Britta Weber 04141b61c4 make property local and use java home to find jps 2015-08-19 01:32:55 +02:00
Robert Muir e07f039659 Workaround JDK bug 8034057
This causes a FileSystemException when trying to retrieve FileStore for a path,
and falsely returns false for Files.isWritable

Squashed commit of the following:

commit d2cc0d966f3bc94aa836b316a42b3c5724bc01ef
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Aug 18 15:49:48 2015 -0400

    fixes for the non-bogus comments

commit 6e0a272f5f8ef7358654ded8ff4ffc31831fa5c7
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Aug 18 15:30:43 2015 -0400

    Fix isWritable too

commit 2a8764ca118fc4c950bfc60d0b97de873e0e82ad
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Aug 18 14:49:50 2015 -0400

    try to workaround filestore bug
2015-08-18 16:44:57 -04:00
David Pilato 0599f85d2d [build] simplify ant script for plugins
Now we are using short names for artifactId (see #12879) so we don't need anymore to transform long names `elasticsearch-pluginname` to short names `pluginname` in ant script when we install a plugin.
Modify also convert-plugin-name
Clean up remaining plugins with old format
And fix vagrant tests
2015-08-18 19:41:37 +02:00
Simon Willnauer 1e511eda28 Remove usage or `InetAddress#getLocalHost`
this method is very confusing and if it's used it's likely the wrong thing
with respect to the actual bound / published address. This change discourages
it's use and removes all useage. It's replaced with the actual published address
most of the time.
2015-08-18 16:48:34 +02:00
Britta Weber f031c17681 check with jps that the pid file contains a pid that actually is an elasticsearch process
relates to #12063
2015-08-18 16:24:40 +02:00
David Pilato c189c1a1ec [maven] also rename artifactId for dev-tools and rest-api-spec 2015-08-18 13:41:12 +02:00
David Pilato 0014728a0d [maven] rename maven names for parent, dev-tools and rest spec
Follow up for previous commit:

```
[INFO] Build Tools and Resources .......................... SUCCESS [  0.158 s]
[INFO] Rest API Specification ............................. SUCCESS [  0.002 s]
[INFO] Elasticsearch Parent POM ........................... SUCCESS [  0.122 s]
```
2015-08-18 13:38:49 +02:00
David Pilato 2b6c5762f4 [build] display ignored artifact when checking licenses 2015-08-18 13:38:48 +02:00
Robert Muir 68307aa9f3 Fix network binding for ipv4/ipv6
When elasticsearch is configured by interface (or default: loopback interfaces),
bind to all addresses on the interface rather than an arbitrary one.

If the publish address is not specified, default it from the bound addresses
based on the following sort ordering:

* ipv4/ipv6 (java.net.preferIPv4Stack, defaults to true)
* ordinary addresses
* site-local addresses
* link local addresses
* loopback addresses

One one address is published, and multicast is still always over ipv4: these
need to be future improvements.

Closes #12906
Closes #12915

Squashed commit of the following:

commit 7e60833312f329a5749f9a256b9c1331a956d98f
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 14:45:33 2015 -0400

    fix java 7 compilation oops

commit c7b9f3a42058beb061b05c6dd67fd91477fd258a
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 14:24:16 2015 -0400

    Cleanup/fix logic around custom resolvers

commit bd7065f1936e14a29c9eb8fe4ecab0ce512ac08e
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 13:29:42 2015 -0400

    Add some unit tests for utility methods

commit 0faf71cb0ee9a45462d58af3d1bf214e8a79347c
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 12:11:48 2015 -0400

    localhost all the way down

commit e198bb2bc0d1673288b96e07e6e6ad842179978c
Merge: b55d092 b93a75f
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 12:05:02 2015 -0400

    Merge branch 'master' into network_cleanup

commit b55d092811d7832bae579c5586e171e9cc1ebe9d
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 12:03:03 2015 -0400

    fix docs, fix another bug in multicast (publish host = bad here!)

commit 88c462eb302b30a82585f95413927a5cbb7d54c4
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 11:50:49 2015 -0400

    remove nocommit

commit 89547d7b10d68b23d7f24362e1f4782f5e1ca03c
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 11:49:35 2015 -0400

    fix http too

commit 9b9413aca8a3f6397b5031831f910791b685e5be
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 11:06:02 2015 -0400

    Fix transport / interface code

    Next up: multicast and then http
2015-08-17 15:43:07 -04:00
Alexander Reelsen f95a509538 Release script: Set versions for non inherited projects
rest-api-spec and dev-tools dont have the elasticsearch-parent
set as a parent and thus need a separate mvn run to change the
plugin version.
2015-08-17 16:40:29 +02:00
Alexander Reelsen a75adaaaaa Prepare release script: fix python compilation error 2015-08-17 15:52:36 +02:00
Alexander Reelsen d1c93fb573 Release: Remove aws-maven plugin/improve release docs
In order to have consistent deploys across several repositories,
we should deploy to sonatype first, then mirror those contents,
and then upload to s3.

This means, the aws wagon is not needed anymore.
2015-08-17 15:39:22 +02:00
Alexander Reelsen c3b3b0e6f8 Release: Replace python search/replace with mvn versions:set plugin
mvn has a versions:set plugin, that can be easily invoked and does not
require the python script to parse the files and hope that there are no
other snapshot mentions.
2015-08-17 11:47:24 +02:00
Simon Willnauer e7d075f6ae Add elasticsearch version as a prefix for the staging URL
This is purely for maintainance reasons since it easier to see if we can drop
certain stageing urls if we have the version next to the hash.
I also removed the gpg passphrase from the example URL since it's better to get prompted?
2015-08-17 11:05:49 +02:00
Simon Willnauer b447e2ae99 Move master to [2.1.0-SNAPSHOT] 2015-08-14 23:44:06 +02:00
Lee Hinman 178dc7ebaf Revert "Add no-Charset version of Files.readAllLines to forbidden apis"
This reverts commit 88765320a9.
2015-08-14 15:05:24 -06:00
Lee Hinman 88765320a9 Add no-Charset version of Files.readAllLines to forbidden apis 2015-08-14 14:55:22 -06:00
Alexander Reelsen e869c87119 Build: Adding aws-maven extension to dev-tools and rest-api-spec 2015-08-14 17:30:00 +02:00
Alexander Reelsen 3893544f98 Release scripts: Split prepare_release into two scripts
In order to reflect our RC release process, we need to separate
the prepare_release script into two separate scripts.

One script now updates the documentation. That one can be executed
anytime and needs to be pushed after that.

The other script updates the version in Version.java and all pom.xml
files, but does not commit anymore. This allows to create a non snapshot
version locally, run mvn deploy, push the artifacts into S3 and, upon
successful tests, simply release them on sonatype. This also allows for
updates, because the S3 snapshot will include the commitId in their repo
as already pushed before.
2015-08-14 16:44:39 +02:00
Clinton Gormley 1eaa8ee7b3 Add an optional "prefix_to_ignore" parameter to the license checker
If core plugins are to be renamed to not include the "elasticsearch-"
prefix, then we need a way of telling the license checker which
JAR files to ignore.
2015-08-14 15:41:05 +02:00
Simon Willnauer c1405d26b5 fixtypo 2015-08-14 15:27:23 +02:00
Simon Willnauer dd1b0d84a6 Also deploy top level artifacts to S3
This commit adds the S3 wagon release profile also to dev-tools
and rest-api-spec and makes the actual repository path / bucket
configurable.
2015-08-14 15:21:06 +02:00
Alexander Reelsen 0f3ada159e Release: Create pre release script
Refactored a part out of the release script, so the user can
change the version locally as well as move the documentation
and change the Version.java

The background of this change is to have a very simple release
process that puts stuff into a staging environment, so the beta
release can be tested, before it is officially released.

This means the build_release script can be removed soon.
2015-08-14 11:04:32 +02:00
Lee Hinman 2eebbcf450 [TEST] Use parent path for path.shared_data
On our Jenkins instances the ${path.home} and createTempDir() locations
share a different parent, so the custom index locations are not within
the ${path.shared_data} directory. This is a hack to fix it until we can
find a way to unify the createTempDir() and `path.shared_data` settings
inside the tests
2015-08-12 15:00:43 -06:00
Simon Willnauer 605253a39f Cut over master to 2.0.0-SNAPSHOT 2015-08-12 21:16:08 +02:00
Lee Hinman d1d2c6efbb [TEST] Add `path.shared_data` that was mistakenly rebased out 2015-08-12 12:05:07 -06:00
Clinton Gormley 68cc5ddf65 More outdates Perl libs in Suse 11 2015-08-12 14:08:10 +02:00
Clinton Gormley 77b5233d6d Check license script fails with perl 5.10.0 - include updated File::Temp lib 2015-08-12 11:01:29 +02:00
jaymode 4da02073ce allow specifying additional arguments in startup-elasticsearch macrodef
Previously we had additional.args as a argument to the startup-elasticsearch macrodef and this was
being used to set some additional elasticsearch settings. This adds the ability to specify additional
arguments back using a element called additional-args.
2015-08-11 09:12:43 -04:00
Simon Willnauer a8580d11f5 Improve arg handling in ant script
This commit is a spinn-off from #12710 which improves the handling of process
arguments ot play nice with spaces etc.
2015-08-11 13:08:32 +02:00
Simon Willnauer d1bc099c91 Add multi-node IT infrastructure
This adds the infrastrucutre to run integration tests with more than one node.
 * it adds relevant macros and targets to integration-tests.xml to start unicast nodes
 * there is a qa/smoke-test-multinode project that simulates such a setup

this commit is soely the infrastructure and doesn't hook up any projects to use this.
For reliability and stability reasons this should be used with care and only if it's really
needed.

Closes #12718
2015-08-11 11:36:15 +02:00
Colin Goodheart-Smithe 6abc69c488 Packaging: Makes sure all POMs contain a description
Adds an explicit description the RPM package so it doesn't inherit the description from the POM.

Closes #12550

Also, modified descriptions for deb and rpm packages to be the same and to reference the documentation rather than listing features that are out of date.
2015-08-10 15:21:24 +01:00
Adrien Grand 1dfc022f89 Merge pull request #12765 from jpountz/enhancement/convert-plugin-name
Tests: Move qa's convert-plugin-name macrodef to dev-tools.
2015-08-10 12:03:56 +02:00
Nik Everett f84552dc98 [Packaging] Run tests in vagrant
This creates a module in qa called vagrant that can be run if you have
vagrant and virtualbox installed and will run the packaging tests in trusty
and centos-7.0. You can ask it to run tests in other linuxes. This is the full
list:
* precise aka Ubuntu 12.04
* trusty aka Ubuntu 14.04
* vivid aka Ubuntun 15.04
* wheezy aka Debian 7, the current debian oldstable distribution
* jessie aka Debian 8, the current debina stable distribution
* centos-6
* centos-7
* fedora-22
* oel-7

There is lots of documentation on how to do this in the TESTING.asciidoc.

Closes #12611
2015-08-10 05:31:42 -04:00
Adrien Grand 9bf854752b Tests: Move qa's convert-plugin-name macrodef to dev-tools. 2015-08-10 11:07:52 +02:00
Clinton Gormley d1585782d4 Removed perl 5.8 support from the license checker
No point in keeping 5.8 support because it relies on modules
that are shipped with 5.9.
2015-08-09 19:54:32 +02:00
Clinton Gormley bc7dbce4fe Enable the license checker for plugins without third party deps too 2015-08-09 19:07:45 +02:00
Clinton Gormley 205262c737 Enable the license checker over distribution/* and plugins/*
Moved the license checker config into the parent pom, and overrede
the license dir/target-to-check in distributions/pom.

Disabled the license checker explicitly for projects which run integration
tests but have no licenses dir:

* core
* distribution
* qa
* plugins/delete-by-query
* plugins/mapper-size
* plugins/site-example

Closes #12752
Closes #12754
2015-08-09 18:18:59 +02:00
Simon Willnauer f1e80b7f9c Add basic tests for sharded JAR
this commit adds a simple integration test that starts a
node from a shaded jar, indexes a doc and retrieves it. It
also has some basic unittests that try to load shaded classes and ensure
that their counterpart is not in the classpath.

Closes #12711
2015-08-07 14:14:33 +02:00
Robert Muir 7414d19d28 Support jenkins randomization in integration tests 2015-08-06 14:48:27 -04:00
Robert Muir 57b1b72af6 Factor out waiting for easier redefinition 2015-08-06 08:28:08 -04:00
Robert Muir 5fe49e4420 cleanup skipping everywhere 2015-08-05 21:45:09 -04:00
Robert Muir c62f0655d3 fix/cleanup pidfile stuff 2015-08-05 20:34:24 -04:00
Robert Muir 15e5247e03 Get plugin smoketester running in jenkins.
We have a smoke_test_plugins.py, but its a bit slow, not integrated
into our build, etc.

I converted this into an integration test. It is definitely uglier
but more robust and fast (e.g. 20 seconds time to verify).

Also there is refactoring of existing integ tests logic, like printing
out commands we execute and stuff
2015-08-05 20:24:36 -04:00
Clinton Gormley 60f8fa4451 Added pure perl fallback for Digest::SHA for old windows boxes 2015-08-05 13:58:18 +02:00
Alexander Reelsen 9028fb8fbe Release: Update build release script to reflect latest changes
As the script now deploys to S3 and several things in master have
changed, this script needs to reflect the latest changes

* An unsigned RPM is built by default, so that users of older
  RPM based distros can download and use that RPM by default
* In addition a signed RPM is built, that is used for the repositories
* Paths for the new distributions have been fixed
* The check for the number of jars has been removed, as this is done
  as part of the license checking in `mvn verify`
* Checksum generation has been removed, as this is done as part of the
  mvn build
* Publishing artifacts of S3 has been removed
* Repostitory creation script has been updated
2015-08-05 09:38:42 +02:00
Robert Muir ead4f0fe1f Merge pull request #12660 from rmuir/improve_integ_tests
improve integration tests output when ES cannot be started.
2015-08-04 22:52:10 -04:00
Robert Muir 3c9fb6f3a2 improve integration tests output when ES cannot be started.
This can happen for a number of reasons, including bugs.
Today you will get a super crappy failure, telling you a .pid file
was not found... you can go look in target/integ-tests/elasticsearch-xxx/logs
and examine the log file, but thats kinda a pain and not easy if its a jenkins
server.

Instead we can fail like this:
```
start-external-cluster-with-plugin:
     [echo] Installing plugin elasticsearch-example-jvm-plugin...
    [mkdir] Created dir: /home/rmuir/workspace/elasticsearch/plugins/jvm-example/target/integ-tests/temp
     [exec] -> Installing elasticsearch-example-jvm-plugin...
     [exec] Plugins directory [/home/rmuir/workspace/elasticsearch/plugins/jvm-example/target/integ-tests/elasticsearch-2.0.0-SNAPSHOT/plugins] does not exist. Creating...
     [exec] Trying file:/home/rmuir/workspace/elasticsearch/plugins/jvm-example/target/releases/elasticsearch-example-jvm-plugin-2.0.0-SNAPSHOT.zip ...
     [exec] Downloading ...........DONE
     [exec] PluginInfo{name='example-jvm-plugin', description='Demonstrates all the pluggable Java entry points in Elasticsearch', site=false, jvm=true, classname=org.elasticsearch.plugin.example.ExampleJvmPlugin, isolated=true, version='2.0.0-SNAPSHOT'}
     [exec] Installed example-jvm-plugin into /home/rmuir/workspace/elasticsearch/plugins/jvm-example/target/integ-tests/elasticsearch-2.0.0-SNAPSHOT/plugins/example-jvm-plugin
     [echo] Starting up external cluster...
     [echo] [2015-08-04 22:02:55,130][INFO ][org.elasticsearch.node   ] [smoke_tester] version[2.0.0-SNAPSHOT], pid[4321], build[e2a47d8/2015-08-05T00:50:08Z]
     [echo] [2015-08-04 22:02:55,130][INFO ][org.elasticsearch.node   ] [smoke_tester] initializing ...
     [echo] [2015-08-04 22:02:55,259][INFO ][org.elasticsearch.plugins] [smoke_tester] loaded [uber-plugin], sites []
     [echo] [2015-08-04 22:02:55,260][ERROR][org.elasticsearch.bootstrap] Exception
     [echo] java.lang.NullPointerException
     [echo] 	at org.elasticsearch.common.settings.Settings$Builder.put(Settings.java:1051)
     [echo] 	at org.elasticsearch.plugins.PluginsService.updatedSettings(PluginsService.java:208)
     [echo] 	at org.elasticsearch.node.Node.<init>(Node.java:148)
     [echo] 	at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:157)
     [echo] 	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:177)
     [echo] 	at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:272)
     [echo] 	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:28)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 25.178 s
[INFO] Finished at: 2015-08-04T22:03:14-05:00
[INFO] Final Memory: 32M/515M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.8:run (integ-setup) on project elasticsearch-example-jvm-plugin: An Ant BuildException has occured: The following error occurred while executing this line:
[ERROR] /home/rmuir/workspace/elasticsearch/plugins/jvm-example/target/dev-tools/ant/integration-tests.xml:176: The following error occurred while executing this line:
[ERROR] /home/rmuir/workspace/elasticsearch/plugins/jvm-example/target/dev-tools/ant/integration-tests.xml:142: ES instance did not start
```
2015-08-04 22:16:30 -04:00
Igor Motov f71c9a25a1 Check for incompatible mappings while upgrading old indices
Conflicting mappings that were allowed before v2.0 can cause runaway shard failures on upgrade. This commit adds a check that prevents a cluster from starting if it contains such indices as well as restoring such indices from a snapshot into already running cluster.

Closes #11857
2015-08-04 18:13:29 -06:00
Robert Muir 39508c4d7a Merge pull request #12653 from rmuir/look_ma_im_a_snapshot_again
Change master branch back to 2.0-beta1
2015-08-04 16:10:59 -04:00
Robert Muir 6f9a067197 Change master branch back to 2.0-beta1 2015-08-04 15:38:21 -04:00
Clinton Gormley 17922d25b8 Change license-check script to work with perl 5.8 2015-08-04 21:31:31 +02:00
Robert Muir 29d73accdd Merge pull request #12639 from rmuir/improve_integ
Don't use port 9200/9300 for integration tests
2015-08-04 13:09:47 -04:00
Robert Muir abb43c3cc6 Merge branch 'master' into maven/distribution-groupid 2015-08-04 10:00:59 -04:00
Robert Muir f1019ace3b Fix plugins integ dependencies and apply cleanups 2015-08-04 09:59:19 -04:00
Simon Willnauer 96ad1911fd Use correct classname in all-signatures.txt 2015-08-04 15:36:18 +02:00
Robert Muir cf6fe6cdcd Don't use port 9200/9300 for integration tests 2015-08-04 09:25:44 -04:00
Clinton Gormley d0bd769b85 Added missing dependency to check-license script 2015-08-04 15:00:50 +02:00
Simon Willnauer 2b2657279d Use UTC instead of default timezone for creation date in CAT endpoint
this change was added recently which uses default timezone for the creation
date on CAT endpoints. We should be consistent and use UTC across the board.
This commit adds #getDefaultTimzone() to forbidden API and fixes the REST tests.

Relates to #11688
2015-08-04 14:38:23 +02:00
Clinton Gormley 20a5fc7e30 The check-license script now accepts either a directory or a .zip file only
Called as:

    check_license_and_sha.pl --check path/to/licenses path/to/extracted/package/

or

    check_license_and_sha.pl --check path/to/licenses path/to/file.zip
2015-08-04 13:07:58 +02:00
Simon Willnauer 6753f7f03e Cut over master to 2.0.0-SNAPSHOT 2015-08-04 10:54:12 +02:00
Ryan Ernst 1e12d03252 Tests: Rename base tests cases to use "TestCase" suffix
Most of the abstract base test classes we have were previously @Ignored.
However, there were also some other tests ignored. Having two ways to
quiet tests is confusing, and clearly it has caused some tests
to get lost in the fold.

This change moves all base test classes to use the "TestCase" suffix,
which is not picked up by the test class name pattern. It also removes
@Ignore from (almost) all tests, and adds it to forbidden apis.
And since we were renaming, I shorted base test class names to use
"ES" instead of "Elasticsearch". I type this a lot of types a day,
and I have heard others express a similar desire for a shorter name.

closes #10659
2015-08-03 17:43:00 -07:00
Ryan Ernst 32a5c20b33 Tests: Forbid @Slow
Now that all uses of @Slow are gone, we can forbid this annotations use
so tests are not confusing. Follow up to #12617.
2015-08-03 16:43:09 -07:00
Robert Muir e1886a9b12 Test that shaded jar contains only shaded classes 2015-08-03 08:56:45 -04:00
Robert Muir a5a27a5cd9 Add jar hell check test for the shaded jar 2015-08-03 05:54:55 -04:00
Robert Muir fef55348f9 clean up .zip handling so its not uselessly downloaded in the build 2015-08-01 09:10:03 -04:00
Robert Muir 8d5b5ad862 restore my lost code changes and add measures to detect/prevent this in the future 2015-07-31 13:00:24 -04:00
Robert Muir ddfea366de Add integration tests for distribution/rpm 2015-07-29 22:26:45 -04:00
Robert Muir 2ed8e697d5 Add integration tests for distribution/deb 2015-07-29 21:34:46 -04:00
Robert Muir 1c5a558164 Add integration tests for distribution/tar 2015-07-29 15:33:29 -04:00
Robert Muir c6d6eaef4d Add a run.sh to run from current source code with debugger
The maven magic needed for this is now heavier
2015-07-29 14:21:43 -04:00
Adrien Grand a832987ada Forbid Files.isHidden.
As Robert pointed out on #12465, it has the undesirable property of relying on
the operating system. So it would be better to use a simple rule such as
checking whether the file name starts with a dot.
2015-07-27 16:13:04 +02:00
Robert Muir d1783ab11e improve plugin file some more 2015-07-23 11:37:55 -04:00
Robert Muir 4b2bda4195 make java.version mandatory for jvm plugins
JarHell has a low level check, but its more of a best effort one,
only checking if X-Compile-Target-JDK is set in the manifest. This
is the case for all lucene- and elasticsearch- generated jars, but
lets just be explicit for plugins.
2015-07-23 10:58:52 -04:00
Clinton Gormley 235452329a In the license checker, check for the existence of the specfied package before trying to unpack it 2015-07-23 14:31:42 +02:00
Robert Muir 4040f194f5 Refactor pluginservice
Closes #12367

Squashed commit of the following:

commit 9453c411798121aa5439c52e95301f60a022ba5f
Merge: 3511a9c 828d8c7
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Jul 22 08:22:41 2015 -0400

    Merge branch 'master' into refactor_pluginservice

commit 3511a9c616503c447de9f0df9b4e9db3e22abd58
Author: Ryan Ernst <ryan@iernst.net>
Date:   Tue Jul 21 21:50:15 2015 -0700

    Remove duplicated constant

commit 4a9b5b4621b0ef2e74c1e017d9c8cf624dd27713
Author: Ryan Ernst <ryan@iernst.net>
Date:   Tue Jul 21 21:01:57 2015 -0700

    Add check that plugin must specify at least site or jvm

commit 19aef2f0596153a549ef4b7f4483694de41e101b
Author: Ryan Ernst <ryan@iernst.net>
Date:   Tue Jul 21 20:52:58 2015 -0700

    Change plugin "plugin" property to "classname"

commit 07ae396f30ed592b7499a086adca72d3f327fe4c
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 23:36:05 2015 -0400

    remove test with no methods

commit 550e73bf3d0f94562f4dde95239409dc5a24ce25
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 23:31:58 2015 -0400

    fix loading to use classname

commit 04463aed12046da0da5cac2a24c3ace51a79f799
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 23:24:19 2015 -0400

    rename to classname

commit 9f3afadd1caf89448c2eb913757036da48758b2d
Author: Ryan Ernst <ryan@iernst.net>
Date:   Tue Jul 21 20:18:46 2015 -0700

    moved PluginInfo and refactored parsing from properties file

commit df63ccc1b8b7cc64d3e59d23f6c8e827825eba87
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 23:08:26 2015 -0400

    fix test

commit c7febd844be358707823186a8c7a2d21e37540c9
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 23:03:44 2015 -0400

    remove test

commit 017b3410cf9d2b7fca1b8653e6f1ebe2f2519257
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 22:58:31 2015 -0400

    fix test

commit c9922938df48041ad43bbb3ed6746f71bc846629
Merge: ad59af4 01ea89a
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 22:37:28 2015 -0400

    Merge branch 'master' into refactor_pluginservice

commit ad59af465e1f1ac58897e63e0c25fcce641148a7
Author: Areek Zillur <areek.zillur@elasticsearch.com>
Date:   Tue Jul 21 19:30:26 2015 -0400

    [TEST] Verify expected number of nodes in cluster before issuing shardStores request

commit f0f5a1e087255215b93656550fbc6bd89b8b3205
Author: Lee Hinman <lee@writequit.org>
Date:   Tue Jul 21 11:27:28 2015 -0600

    Ignore EngineClosedException during translog fysnc

    When performing an operation on a primary, the state is captured and the
    operation is performed on the primary shard. The original request is
    then modified to increment the version of the operation as preparation
    for it to be sent to the replicas.

    If the request first fails on the primary during the translog sync
    (because the Engine is already closed due to shadow primaries closing
    the engine on relocation), then the operation is retried on the new primary
    after being modified for the replica shards. It will then fail due to the
    version being incorrect (the document does not yet exist but the request
    expects a version of "1").

    Order of operations:

    - Request is executed against primary
    - Request is modified (version incremented) so it can be sent to replicas
    - Engine's translog is fsync'd if necessary (failing, and throwing an exception)
    - Modified request is retried against new primary

    This change ignores the exception where the engine is already closed
    when syncing the translog (similar to how we ignore exceptions when
    refreshing the shard if the ?refresh=true flag is used).

commit 4ac68bb1658688550ced0c4f479dee6d8b617777
Author: Shay Banon <kimchy@gmail.com>
Date:   Tue Jul 21 22:37:29 2015 +0200

    Replica allocator unit tests
    First batch of unit tests to verify the behavior of replica allocator

commit 94609fc5943c8d85adc751b553847ab4cebe58a3
Author: Jason Tedor <jason@tedor.me>
Date:   Tue Jul 21 14:04:46 2015 -0400

    Correctly list blobs in Azure storage to prevent snapshot corruption and do not unnecessarily duplicate Lucene segments in Azure Storage

    This commit addresses an issue that was leading to snapshot corruption for snapshots stored as blobs in Azure Storage.

    The underlying issue is that in cases when multiple snapshots of an index were taken and persisted into Azure Storage, snapshots subsequent
    to the first would repeatedly overwrite the snapshot files. This issue does render useless all snapshots except the final snapshot.

    The root cause of this is due to String concatenation involving null. In particular, to list all of the blobs in a snapshot directory in
    Azure the code would use the method listBlobsByPrefix where the prefix is null. In the listBlobsByPrefix method, the path keyPath + prefix
    is constructed. However, per 5.1.11, 5.4 and 15.18.1 of the Java Language Specification, the reference null is first converted to the string
    "null" before performing the concatenation. This leads to no blobs being returned and therefore the snapshot mechanism would operate as if
    it were writing the first snapshot of the index. The fix is simply to check if prefix is null and handle the concatenation accordingly.

    Upon fixing this issue so that subsequent snapshots would no longer overwrite earlier snapshots, it was discovered that the snapshot metadata
    returned by the listBlobsByPrefix method was not sufficient for the snapshot layer to detect whether or not the Lucene segments had already
    been copied to the Azure storage layer in an earlier snapshot. This led the snapshot layer to unnecessarily duplicate these Lucene segments
    in Azure Storage.

    The root cause of this is due to known behavior in the CloudBlobContainer.getBlockBlobReference method in the Azure API. Namely, this method
    does not fetch blob attributes from Azure. As such, the lengths of all the blobs appeared to the snapshot layer to be of length zero and
    therefore they would compare as not equal to any new blobs that the snapshot layer is going to persist. To remediate this, the method
    CloudBlockBlob.downloadAttributes must be invoked. This will fetch the attributes from Azure Storage so that a proper comparison of the
    blobs can be performed.

    Closes elastic/elasticsearch-cloud-azure#51, closes elastic/elasticsearch-cloud-azure#99

commit cf1d481ce5dda0a45805e42f3b2e0e1e5d028b9e
Author: Lee Hinman <lee@writequit.org>
Date:   Mon Jul 20 08:41:55 2015 -0600

    Unit tests for `nodesAndVersions` on shared filesystems

    With the `recover_on_any_node` setting, these unit tests check that the
    correct node list and versions are returned.

commit 3c27cc32395c3624f7c794904d9ea4faf2eccbfb
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 14:15:59 2015 -0400

    don't fail junit4 integration tests if there are no tests.

    instead fail the failsafe plugin, which means the external cluster will still get shut down

commit 95d2756c5a8c21a157fa844273fc83dfa3c00aea
Author: Alexander Reelsen <alexander@reelsen.net>
Date:   Tue Jul 21 17:16:53 2015 +0200

    Testing: Fix help displaying tests under windows

    The help files are using a unix based file separator, where as
    the test relies on the help being based on the file system separator.

    This commit fixes the test to remove all `\r` characters before
    comparing strings.

    The test has also been moved into its own CliToolTestCase, as it does
    not need to be an integration test.

commit 944f06ea36bd836f007f8eaade8f571d6140aad9
Author: Clinton Gormley <clint@traveljury.com>
Date:   Tue Jul 21 18:04:52 2015 +0200

    Refactored check_license_and_sha.pl to accept a license dir and package path

    In preparation for the move to building the core zip, tar.gz, rpm, and deb as separate modules, refactored check_license_and_sha.pl to:

    * accept a license dir and path to the package to check on the command line
    * to be able to extract zip, tar.gz, deb, and rpm
    * all packages except rpm will work on Windows

commit 2585431e8dfa5c82a2cc5b304cd03eee9bed7a4c
Author: Chris Earle <pickypg@users.noreply.github.com>
Date:   Tue Jul 21 08:35:28 2015 -0700

    Updating breaking changes

    - field names cannot be mapped with `.` in them
    - fixed asciidoc issue where the list was not recognized as a list

commit de299b9d3f4615b12e2226a1e2eff5a38ecaf15f
Author: Shay Banon <kimchy@gmail.com>
Date:   Tue Jul 21 13:27:52 2015 +0200

    Replace primaryPostAllocated flag and use UnassignedInfo
    There is no need to maintain additional state as to if a primary was allocated post api creation on the index routing table, we hold all this information already in the UnassignedInfo class.
    closes #12374

commit 43080bff40f60bedce5bdbc92df302f73aeb9cae
Author: Alexander Reelsen <alexander@reelsen.net>
Date:   Tue Jul 21 15:45:05 2015 +0200

    PluginManager: Fix bin/plugin calls in scripts/bats test

    The release and smoke test python scripts used to install
    plugins in the old fashion.

    Also the BATS testing suite installed/removed plugins in that
    way. Here the marvel tests have been removed, as marvel currently
    does not work with the master branch.

    In addition documentation has been updated as well, where it was
    still missing.

commit b81ccba48993bc13c7678e6d979fd96998499233
Author: Boaz Leskes <b.leskes@gmail.com>
Date:   Tue Jul 21 11:37:50 2015 +0200

    Discovery: make sure NodeJoinController.ElectionCallback is always called from the update cluster state thread

    This is important for correct handling of the joining thread. This causes assertions to trip in our test runs. See http://build-us-00.elastic.co/job/es_g1gc_master_metal/11653/ as an example

    Closes #12372

commit 331853790bf29e34fb248ebc4c1ba585b44f5cab
Author: Boaz Leskes <b.leskes@gmail.com>
Date:   Tue Jul 21 15:54:36 2015 +0200

    Remove left over no commit from TransportReplicationAction

    It asks to double check thread pool rejection. I did and don't see problems with it.

commit e5724931bbc1603e37faa977af4235507f4811f5
Author: Alexander Reelsen <alexander@reelsen.net>
Date:   Tue Jul 21 15:31:57 2015 +0200

    CliTool: Various PluginManager fixes

    The new plugin manager parser was not called correctly in the scripts.
    In addition the plugin manager now creates a plugins/ directory in case
    it does not exist.

    Also the integration tests called the plugin manager in the deprecated way.

commit 7a815a370f83ff12ffb12717ac2fe62571311279
Author: Alexander Reelsen <alexander@reelsen.net>
Date:   Tue Jul 21 13:54:18 2015 +0200

    CLITool: Port PluginManager to use CLITool

    In order to unify the handling and reuse the CLITool infrastructure
    the plugin manager should make use of this as well.

    This obsolets the -i and --install options but requires the user
    to use `install` as the first argument of the CLI.

    This is basically just a port of the existing functionality, which
    is also the reason why this is not a refactoring of the plugin manager,
    which will come in a separate commit.

commit 7f171eba7b71ac5682a355684b6da703ffbfccc7
Author: Martijn van Groningen <martijn.v.groningen@gmail.com>
Date:   Tue Jul 21 10:44:21 2015 +0200

    Remove custom execute local logic in TransportSingleShardAction and TransportInstanceSingleOperationAction and rely on transport service to execute locally. (forking thread etc.)

    Change TransportInstanceSingleOperationAction to have shardActionHandler to, so we can execute locally without endless spinning.

commit 0f38e3eca6b570f74b552e70b4673f47934442e1
Author: Ryan Ernst <ryan@iernst.net>
Date:   Tue Jul 21 17:36:12 2015 -0700

    More readMetadata tests and pickiness

commit 880b47281bd69bd37807e8252934321b089c9f8e
Author: Ryan Ernst <ryan@iernst.net>
Date:   Tue Jul 21 14:42:09 2015 -0700

    Started unit tests for plugin service

commit cd7c8ddd7b8c4f3457824b493bffb19c156c7899
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 07:21:07 2015 -0400

    fix tests

commit 673454f0b14f072f66ed70e32110fae4f7aad642
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 06:58:25 2015 -0400

    refactor pluginservice
2015-07-22 10:45:45 -04:00
Clinton Gormley f84757e8bb Refactored check_license_and_sha.pl to accept a license dir and package path
In preparation for the move to building the core zip, tar.gz, rpm, and deb as separate modules, refactored check_license_and_sha.pl to:

* accept a license dir and path to the package to check on the command line
* to be able to extract zip, tar.gz, deb, and rpm
* all packages except rpm will work on Windows
2015-07-21 18:04:52 +02:00
Alexander Reelsen d3e454780f PluginManager: Fix bin/plugin calls in scripts/bats test
The release and smoke test python scripts used to install
plugins in the old fashion.

Also the BATS testing suite installed/removed plugins in that
way. Here the marvel tests have been removed, as marvel currently
does not work with the master branch.

In addition documentation has been updated as well, where it was
still missing.
2015-07-21 16:17:59 +02:00
Alexander Reelsen 999d5ab801 CliTool: Various PluginManager fixes
The new plugin manager parser was not called correctly in the scripts.
In addition the plugin manager now creates a plugins/ directory in case
it does not exist.

Also the integration tests called the plugin manager in the deprecated way.
2015-07-21 15:31:57 +02:00
Robert Muir cd9d1c617f Merge pull request #12344 from rmuir/2.1.16
Update randomizedtesting to 2.1.16
2015-07-20 11:13:57 -04:00
Robert Muir a0a6299840 Update randomizedtesting to 2.1.16 2015-07-20 11:02:50 -04:00
Robert Muir 48bc49c093 Remove broken `exec` build target, replace with something better.
Currently this target is "yet another way" to run elasticsearch,
which we can't maintain. It also has the problem that it doesnt
ensure its running on the latest source code, doesn't configure
any scratch space properly, won't work with securitymanager, list
goes on.

Even if we made it work, it would break every day, since its untested.

Instead, `mvn package -Drun -DskipTests` will run packaging, and then
startup bin/elasticsearch (like integration tests, but in foreground).

It also enables debugger socket on port 8000, for people that like
IDE debuggers and not system.out.println.

Its a little slower to get started because of all the shading/RPM/DEB
building going on in `package` but that is just what it is right now
until that stuff is moved out.
2015-07-17 00:03:33 -04:00
Robert Muir 9e7882e604 protect logic under skipTests for stupid pom packaging 2015-07-16 20:49:16 -04:00
Robert Muir 6b7561ac9a Use junit4 for running integration tests, too
failsafe uses surefire, which sucks. It also mean integ tests act alien right now.
I would rather have the consistency, e.g. things formatted the same way, running integ tests under security manager, etc.
2015-07-16 19:43:33 -04:00
Robert Muir 4804bf747b skip this target too, to not confuse POM builds 2015-07-15 00:30:26 -04:00
Robert Muir 68143dedf2 Refactor integration tests
1. tests don't have a bogus test dependency on zips anymore,
   instead we handle this in pre-integration-test. This reduces
   lots of confusion for e.g. mvn clean test.
2. refactor integ logic so that core/ and plugin/ share it.
   previously they were duplicates but the above change simplifies life.
   it also makes it easier for doing more interesting stuff
2015-07-14 23:37:04 -04:00
Igor Motov 24a93840d5 Add url repository whitelist
Require urls for URL repository to be listed in repositories.url.allowed_urls setting. This change ensures that only authorized URLs can be accessed by elasticsearch
2015-07-14 18:38:26 -04:00
uboness b40186652c updated the elasticsearch versioning format
Moving to from `X.Y.Z.beta1`/`X.Y.Z.RC1` to `X.Y.Z-beta1`/`X.Y.Z-rc1`
2015-07-13 20:26:37 +02:00
Robert Muir c2c8956347 enforce plugins are installed with correct name, and install correctly in integ tests 2015-07-10 19:20:25 -04:00
Simon Willnauer e0708813a9 Make 2.0.0.beta1-SNAPSHOT the current version.
Today everything is tight to having the next version as the latest.
In order to work towards 2.0.0.beta1 we need to fix all the usage of
2.0.0-SNAPSHOT to reflect the version we will release soon.
Usually we do this on the release branch but to simplify things I wanna
keep this on master for now and move to 2.1.0-SNAPSHOT on master once
we created a 2.0 branch.

Closes #12148
2015-07-09 21:24:32 +02:00
Robert Muir 8ef73b7dba Get integration tests working on windows.
We also run our license checker in `mvn verify`, but there
are problems with checksum calculation on windows there, so I've
disabled the license checker on windows to prevent those false fails.
2015-07-08 02:40:00 -04:00
Robert Muir fcddec532a Use extract-pid when killing, too 2015-07-08 00:10:25 -04:00
Robert Muir 7a9f74de80 print PID when starting ES in integ tests too. 2015-07-08 00:04:27 -04:00
Robert Muir fb62d945aa isolate integ tests a little better.
They are mostly just ES clients, but we can set tempdir and the usual
stuff to contain them better.
2015-07-07 23:10:18 -04:00
Robert Muir 85a25bdf73 harden logic around integ test workspace and process mgmt
there is more to do here, but this is already a lot more robust.

* don't clean workspace in teardown, it might be useful for debugging if stuff fails.
* kill ES/clean workspace in setup, so things always work even in the case of ^C
* use pidfile to kill
* fail if kill errors
* refactor a bit more logic here
2015-07-07 20:11:50 -04:00
Robert Muir 5884628049 Integ setup should unzip org.elasticsearch:elasticsearch:zip
This property is set by maven, and unlike the current hack, during
a multimodule build will be set to the correct thing.

Otherwise today sometimes we run integ tests with outdated ES
artifacts, which makes for incredibly confusing failures.

Closes #12101
2015-07-07 19:04:46 -04:00
Robert Muir 023cb0b174 make integ testing a bit more picky 2015-07-07 10:21:40 -04:00
Robert Muir 23dfb5f1a5 don't try to run tests for packaging=pom 2015-07-06 15:49:57 -04:00
Robert Muir 9a146b9e75 Get delete by query rest tests running 2015-07-06 15:13:50 -04:00
Robert Muir 75285cee88 factor out this helper 2015-07-06 14:19:28 -04:00
Robert Muir 7595104ec3 Factor integration tests logic to separate build file 2015-07-06 13:59:16 -04:00
David Pilato e7a6b51bab [maven] change groupId / artifactId
When we generate our project, we can get something like:

```
├── dev-tools
├── elasticsearch
├── elasticsearch-parent
├── elasticsearch-plugin
├── plugin
│   ├── elasticsearch-analysis-icu
│   ├── elasticsearch-analysis-kuromoji
│   ├── elasticsearch-analysis-phonetic
│   ├── elasticsearch-analysis-smartcn
│   ├── elasticsearch-analysis-stempel
│   ├── elasticsearch-cloud-aws
│   ├── elasticsearch-cloud-azure
│   ├── elasticsearch-cloud-gce
│   ├── elasticsearch-delete-by-query
│   ├── elasticsearch-lang-javascript
│   └── elasticsearch-lang-python
├── rest-api-spec
└── securemock
```

I propose here to use a common naming for artifacts: start always with `elasticsearch-`.
Also, move `elasticsearch-plugin` to `org.elasticsearch.plugin` groupId.

So we could have:

```
├── elasticsearch
├── elasticsearch-dev-tools
├── elasticsearch-parent
├── elasticsearch-rest-api-spec
├── elasticsearch-securemock
├── plugin
│   ├── elasticsearch-analysis-icu
│   ├── elasticsearch-analysis-kuromoji
│   ├── elasticsearch-analysis-phonetic
│   ├── elasticsearch-analysis-smartcn
│   ├── elasticsearch-analysis-stempel
│   ├── elasticsearch-cloud-aws
│   ├── elasticsearch-cloud-azure
│   ├── elasticsearch-cloud-gce
│   ├── elasticsearch-delete-by-query
│   ├── elasticsearch-lang-javascript
│   ├── elasticsearch-lang-python
│   └── elasticsearch-plugin
```
2015-07-06 17:17:07 +02:00
Robert Muir 3f4b8df00d Merge pull request #12026 from rmuir/integ_tests
add integration test harness to maven build
2015-07-06 10:16:54 -04:00
Clinton Gormley c5d3c46943 Added build and test labels to es release notes script 2015-07-05 20:24:46 +02:00
Tanguy Leroux 9495816cb7 Remove sigar completely 2015-07-03 15:49:17 +02:00
Michael McCandless 275fdcc08d fix silly smoke_test_plugin bugs when ES actually succeeds in starting with the installed plugins 2015-07-03 05:59:16 -04:00
Robert Muir 80871bae2b Add simple integ testing infra 2015-07-03 02:12:01 -04:00
Michael McCandless 48b85421ec fix smoke_test_plugins.py: install the .zip under releases, not the .jar 2015-06-30 15:03:54 -04:00
Michael McCandless 189ef91e3c first cut 2015-06-30 14:14:45 -04:00
Simon Willnauer e7eb9cf4de Ban java serialization
We had several problems with Java Serializatin in the past. At some point
in the Java 1.7.x series JDKs where not compatible anymore when java
serialization (ObjectStream) was used to exchange objects. In elasticsearch
we used this to serialize exceptions across the wire which caused several problems
with incompatible JDKs. While causing lot of trouble this essentially prevented
users from moving forward and upgrade their JVMs. To prevent these kind of issues
this commit removes the dependency on java serialization entirely and bans the
usage of ObjectOutputStream and ObjectInputStream entirely.

Yet, we can't fully serialize all exception anymore such that this commit
is best effort and adds hand written serialization to all elasticsearch exceptions
as well to a selected set of JDK and Lucene exceptions. (see StreamOutput#writeThrowable /
StreamInput.readThrowable). Stacktraces should be preserved for all exceptions while
several names might be replaced with ElasticsearchException if there is no mapping for
the given exception.
2015-06-30 14:51:43 +02:00
Alexander Reelsen f26672c184 Release: Build two RPMS, signed and unsigned
In order to support older RPM based distributions like CentOS5,
we should have one RPM available, which is not signed.

This commit creates an unsigned RPM first, then moves it over to
target/releases during the build, then builds a signed RPM.

The unsigned one is uploaded via S3, where as the signed one is
used for the repositories.

In addition, you can now build an RPM without having to specify
any gpg credentials due to offloading this into a maven profile
that is only activated when specifying `rpm.sign` property.

Closes #11587
2015-06-30 14:22:20 +02:00
Clinton Gormley fa40680736 Build: If SHA files have changed, explain how to update them in the license check exception 2015-06-30 11:29:35 +02:00
Simon Willnauer 91ab808ce2 Revert accidential modification of release script in this files previous commit
commit 05db5dc2c8
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Jun 5 13:12:05 2015 +0200

    create parent pom project from its original location
2015-06-25 17:18:34 +02:00
Clinton Gormley a3d1a50865 Build: tar on linux needs the --wildcard option, but not supported on OSX
Removing '*.jar'  filter when untarring during the license check
2015-06-23 14:07:10 +02:00
Clinton Gormley 9fb3bf06c5 Changed the license checker to use the ZIP file as the source of JARs to check.
Also checks that the tar.gz file (if present) contains the same JARs as the
ZIP file.
2015-06-23 12:50:31 +02:00
Simon Willnauer 1b2a3d0af6 Add @Repeat to forbidden APIs
@Repeat should not be committed just like @Seed.
Use -Pdev to run annotated methods.
2015-06-18 20:34:02 +02:00
Ryan Ernst 9157a11047 Build: Add Iterators.emptyIterator to forbidden apis
As a follow up to #11741, this forbids Iterators.emptyIterator in
favor the of builtin Collections.emptyIterator.
2015-06-18 10:12:58 -07:00
Simon Willnauer 2a63249441 Add DateTime ctors without timezone to forbidden APIs
Using DateTime with default timezone is asking for trouble and should
be added to forbidden APIs
2015-06-18 10:43:45 +02:00
Clinton Gormley 05d512f417 Packaging: Add LICENSE, NOTICE, and sha1 files and tests for all core dependencies
Added a licenses/ directory to core which contains a sha1 file for each JAR
dependency, and one or more LICENSE files and one NOTICE file for each
project.

Also adds dev-tools/src/main/resources/license-check/check_license_and_sha.pl
which checks that the licenses/ dir is up to date during a mvn verify,
and which can be used to update the sha1 files when upgrading dependencies.

Closes #2794
Closes #10684
Closes #11705
2015-06-17 18:06:00 +02:00
Alexander Reelsen a54d4e4aa8 Versioning: Adding 1.6.1 development version & 1.6.0 bwc index 2015-06-09 16:30:02 +02:00
Simon Willnauer 05db5dc2c8 create parent pom project from its original location 2015-06-05 13:12:05 +02:00
Michael McCandless e1197dfea9 Merge branch 'master' into require_units
Conflicts:
	src/main/java/org/elasticsearch/action/bulk/BulkRequest.java
	src/main/java/org/elasticsearch/cluster/metadata/MetaDataIndexUpgradeService.java
	src/main/java/org/elasticsearch/node/internal/InternalSettingsPreparer.java
	src/test/java/org/elasticsearch/snapshots/DedicatedClusterSnapshotRestoreTests.java
2015-06-04 13:59:10 -04:00
Michael McCandless 68d6427944 add missing units to index settings if index was created before 2.0 2015-05-30 04:39:03 -04:00
Clinton Gormley bd381b86ca Release notes: Add HTML header with UTF-8 encoding 2015-05-29 21:02:38 +02:00
Alexander Reelsen b465e19e5f Release script: Always check for valid environment
In order to be sure that a release can be executed on the local machine,
the build_release script now checks for environment variables and tries
to execute a  couple of commands.

In order to easily check for a correctly setup environment, you can
run the following commands, which exits early and does not trigger a
release process.

```
python3 dev-tools/build_release.py --check-only
```
2015-05-26 14:43:29 +02:00
Robert Muir b462fd712a factor out static analysis 2015-05-23 01:33:37 -04:00
Robert Muir 5330e3423f remove build duplication 2015-05-22 23:23:59 -04:00
Igor Motov 21ed6bb90c Core: Don't allow indices containing too-old segments to be opened
When index is introduced into the cluster via cluster upgrade, restore or as a dangled index the MetaDataIndexUpgradeService checks if this index can be upgraded to the current version. If upgrade is not possible, the newly upgraded cluster startup and restore process are aborted, the dangled index is imported as a closed index that cannot be open.

Closes #10215
2015-05-19 23:37:05 -04:00
Adrien Grand 4131bcbec7 Search: Make FilteredQuery a forbidden API.
This commit makes FilteredQuery a forbidden API and also removes some more usage
of the Filter API. There are some remaining code using filters for parent/child
queries but I'm not touching this as they are already being refactored in #6511.
2015-05-19 15:33:43 +02:00
Robert Muir 38cccfb057 cleanup and ban temp files going to jvm default location 2015-05-08 15:08:13 -04:00
Robert Muir 51c71c235b Ban PathUtils.get (for now, until we fix the two remaining issues) 2015-05-08 14:42:27 -04:00
Adrien Grand b72f27a410 Core: Cut over to the Lucene filter cache.
This removes Elasticsearch's filter cache and uses Lucene's instead. It has some
implications:
 - custom cache keys (`_cache_key`) are unsupported
 - decisions are made internally and can't be overridden by users ('_cache`)
 - not only filters can be cached but also all queries that do not need scores
 - parent/child queries can now be cached, however cached entries are only
   valid for the current top-level reader so in practice it will likely only
   be used on read-only indices
 - the cache deduplicates filters, which plays nicer with large keys (eg. `terms`)
 - better stats: we already had ram usage and evictions, but now also hit count,
   miss count, lookup count, number of cached doc id sets and current number of
   doc id sets in the cache
 - dynamically changing the filter cache size is not supported anymore

Internally, an important change is that it removes the NoCacheFilter infrastructure
in favour of making Query.rewrite specializing the query for the current reader so
that it will only be cached on this reader (look for IndexCacheableQuery).

Note that consuming filters with the query API (createWeight/scorer) instead of
the filter API (getDocIdSet) is important for parent/child queries because
otherwise a QueryWrapperFilter(ParentQuery) would run the wrapped query per
segment while relations might be cross segments.
2015-05-04 09:02:15 +02:00
Alexander Reelsen b25259532e Release: Fix build repositories script
Minor issue with specifying the correct version when starting the package release script.
Another issue fixed to make sure that the S3 bucket parameters act the same.
2015-04-28 10:04:30 +02:00
Alexander Reelsen 924479369f Release script: Fix wrong argument for string formatting 2015-04-27 11:09:02 +02:00
Alexander Reelsen f64739788b Build: Update package repositories when creating a release
In order to automatically sign and and upload our debian and RPM
packages, this commit incorporates signing into the build process
and adds the necessary steps to the release process. In order to do this
the pom.xml has been adapted and the RPM and jdeb maven plugins have been
updated, so the packages are signed on build. However the repositories
need to signed as well.

Syncing the repos requires downloading the current repo, adding
the new packages and syncing it back.

The following environment variables are now required as part of the build

* GPG_KEY_ID - the key ID of the key used for signing
* GPG_PASSPHRASE - your GPG passphrase
* S3_BUCKET_SYNC_TO: S3 bucket to sync new repo into

The following environment variables are optional

* S3_BUCKET_SYNC_FROM: S3 bucket to get existing packages from
* GPG_KEYRING - home of gnupg, defaults to ~/.gnupg

The following command line tools are needed

* createrepo (creates RPM repositories)
* expect (used by the maven rpm plugin)
* apt-ftparchive (creates DEB repositories)
* gpg (signs packages and repo files)
* s3cmd (syncing between the different S3 buckets)

The current approach would also work for users who want to run their
own repositories, all they need to change are a couple of environment
variables.

Minor implementation detail: Right now the branch name is used as version
for the repositories (like 1.4/1.5/1.6) - if we ever change our branch naming
scheme, the script needs to be fixed.
2015-04-26 19:05:47 +02:00
Robert Muir 270cb9f349 enable securitymanager 2015-04-22 03:04:50 -04:00
Robert Muir 69718916df actually remove this line rather than comment it out. tsts pass 2015-04-21 19:04:56 -04:00
Robert Muir 9d6b1382e7 Fix JVM isolation in tests.
Currently security manager would allow for one JVM to muck
with the files (read, write, AND delete) of another JVM.

This is unnecessary.
2015-04-21 19:02:14 -04:00
Adrien Grand d7abb12100 Replace deprecated filters with equivalent queries.
In Lucene 5.1 lots of filters got deprecated in favour of equivalent queries.
Additionally, random-access to filters is now replaced with approximations on
scorers. This commit
 - replaces the deprecated NumericRangeFilter, PrefixFilter, TermFilter and
   TermsFilter with NumericRangeQuery, PrefixQuery, TermQuery and TermsQuery,
   wrapped in a QueryWrapperFilter
 - replaces XBooleanFilter, AndFilter and OrFilter with a BooleanQuery in a
   QueryWrapperFilter
 - removes DocIdSets.isBroken: the new two-phase iteration API will now help
   execute slow filters efficiently
 - replaces FilterCachingPolicy with QueryCachingPolicy

Close #8960
2015-04-21 15:32:43 +02:00
Simon Willnauer 7ad138e17b [TEST] allow to read from lig/sigar 2015-04-20 18:15:51 +02:00
Robert Muir b09d236fc0 run tests with AssertingCodec to find bugs 2015-04-19 13:56:12 -04:00
Robert Muir 370819a98a Merge branch 'master' into mockfilesystem 2015-04-16 18:26:12 -04:00
Robert Muir 68267f4bb6 these leaks are plugged 2015-04-16 09:42:13 -04:00
Michael McCandless 399f0ccce9 Core: add only_ancient_segments to upgrade API, so only segments with an old Lucene version are upgraded
This option defaults to false, because it is also important to upgrade
the "merely old" segments since many Lucene improvements happen within
minor releases.

But you can pass true to do the minimal work necessary to upgrade to
the next major Elasticsearch release.

The HTTP GET upgrade request now also breaks out how many bytes of
ancient segments need upgrading.

Closes #10213

Closes #10540

Conflicts:
	dev-tools/create_bwc_index.py
	rest-api-spec/api/indices.upgrade.json
	src/main/java/org/elasticsearch/action/admin/indices/optimize/OptimizeRequest.java
	src/main/java/org/elasticsearch/action/admin/indices/optimize/ShardOptimizeRequest.java
	src/main/java/org/elasticsearch/action/admin/indices/optimize/TransportOptimizeAction.java
	src/main/java/org/elasticsearch/index/engine/InternalEngine.java
	src/test/java/org/elasticsearch/bwcompat/StaticIndexBackwardCompatibilityTest.java
	src/test/java/org/elasticsearch/index/engine/InternalEngineTests.java
	src/test/java/org/elasticsearch/rest/action/admin/indices/upgrade/UpgradeReallyOldIndexTest.java
2015-04-16 05:24:33 -04:00
Adrien Grand 563e704881 Mappings: Same code path for dynamic mappings updates and updates coming from the API.
We have two completely different code paths for mappings updates, depending on
whether they come from the API or are guessed based on the parsed documents.
This commit makes dynamic mappings updates execute like updates from the API.

The only change in behaviour is that a document that fails parsing can not
modify mappings anymore (useful to prevent issues such as #9851). Other than
that, this change should be fairly transparent to users but working this way
opens doors to other changes such as validating dynamic mappings updates on the
master node (#8688).

The way it works internally is that Mapper.parse now returns a Mapper instead
of being void. The returned Mapper represents a mapping update that has been
performed in order to parse the document. Mappings updates are propagated
recursively back to the root mapper, and once parsing is finished, we check
that the mappings update can be applied, and either fail the parsing if the
update cannot be merged (eg. because of a concurrent mapping update from the
API) or merge the update into the mappings.

However not all mappings updates can be applied recursively, `copy_to` for
instance can add mappings at totally different places in the tree. Because of
it I added ParseContext.rootMapperUpdates which `copy_to` fills when the
field to copy data to does not exist in the mappings yet. These mappings
updates are merged from the ones generated by regular parsing.

One particular mapping update was the `auto_boost` setting on the `all` root
mapper. Being tricky to work on, I removed it in favour of search-time checks
that payloads have been indexed.

One interesting side-effect of the change is that concurrency on ObjectMapper
is greatly simplified since we do not have to care anymore about having
concurrent dynamic mappings and API updates.
2015-04-16 10:16:59 +02:00
Robert Muir e5a699fa05 cutover to lucenetestcase 2015-04-16 00:58:02 -04:00
Robert Muir 6ac4d6daef contain filesystem access 2015-04-15 18:23:30 -04:00
Ryan Ernst a3f078985b Tests: Forbid tests from writing to CWD
Allowing tests writing to the working directory can mask problems.
For example, multiple tests running in the same jvm, and using the
same relative path, may cause issues if the first test to run
leaves data in the directory, and the second test does not remember
to cleanup the path before using it.

This change adds security manager rules to disallow tests writing
to the working directory. Instead, tests create a temp dir with
the existing test framework.

closes #10605
2015-04-15 12:45:20 -07:00
Simon Willnauer 67b48da15f [BUILD] Fix m2.repository path permission in tests.policy 2015-04-14 10:40:31 +02:00
Simon Willnauer fe411a9295 [BUILD] Restrict read permission to project.basedir/target if security manager is used 2015-04-14 09:35:40 +02:00
Simon Willnauer c13e604697 [BUILD] Restrict read permission to project.basedir
This prevents reads from anywhere outside of the elasticsearch
clone when running tests with security manager enabled.
2015-04-13 16:44:31 +02:00
Robert Muir b936ec9a25 allow reflection of MXBean for file descriptor stats 2015-04-10 11:28:30 -04:00
Ryan Ernst e575c4ce53 Tests: Add --all flag to create-bwc script to regenerate all indexes
It is currently a pain to regenerate all the bwc indexes when making
a change to the generation script.  This makes it a single command.
2015-04-07 08:32:34 -07:00
Ryan Ernst c3011cead4 Tests: Revamp static bwc test framework to use dangling indexes
The static old index tests currently take a long time to run because
each index version essentially recreates the cluster, and spins up
new nodes.  This PR instead loads each old version into the existing
cluster as a dangling index. It also removes the intermediate
"StaticIndexBackwardCompatibilityTest" which was an extra layer
with no purpose, and moves a shared version of a commonly found
function to get an http client.

The test now takes between 40 and 60 seconds for me. I also ran it
"under stress" by running all ES tests in one shell, while
simultaneously running 10 iterations of the old index tests. Each
iteration took on average about 90 seconds, which is much better
than the 20+ minutes we see in master on jenkins.

closes #10247
2015-04-03 23:21:55 -07:00
Adrien Grand 1401075070 Tests: Speed up backward-compatibility tests for 1.1.0
1.1.0 is affected by #5817 which prevents merges from keeping up with the
indexing rate. As a consequence it generates lots of segments and makes bw
compat tests slow. So I added a special case for this version to index fewer
documents.
2015-04-02 19:11:12 +02:00
Adrien Grand 08f93cf33f Add doc values support to boolean fields.
This pull request makes boolean handled like dates and ipv4 addresses: things
are stored as as numerics under the hood and aggregations add some special
formatting logic in order to return true/false in addition to 1/0.

For example, here is an output of a terms aggregation on a boolean field:
```
   "aggregations": {
      "top_f": {
         "doc_count_error_upper_bound": 0,
         "buckets": [
            {
               "key": 0,
               "key_as_string": "false",
               "doc_count": 2
            },
            {
               "key": 1,
               "key_as_string": "true",
               "doc_count": 1
            }
         ]
      }
   }
```

Sorted numeric doc values are used under the hood.

Close #4678
Close #7851
2015-04-02 15:40:46 +02:00
javanna d9628649a2 [TEST] remove needless script settings from create-bwc-index script
dynamic scripts are not needed here, can be disabled.
2015-03-26 19:56:55 +01:00
javanna f4592a17e3 [TEST] remove needless script settings from upgrade tests script
dynamic scripts are not needed in our upgrade tests, can be removed.
2015-03-26 19:56:55 +01:00
javanna d9d1e6a67a Scripting: add support for fine-grained settings
Allow to on/off scripting based on their source (where they get loaded from), the  operation that executes them and their language.

The settings cover the following combinations:

- mode: on, off, sandbox
- source: indexed, dynamic, file
- engine: groovy, expressions, mustache, etc
- operation: update, search, aggs, mapping

The following settings are supported for every engine:

script.engine.groovy.indexed.update:    sandbox/on/off
script.engine.groovy.indexed.search:    sandbox/on/off
script.engine.groovy.indexed.aggs:      sandbox/on/off
script.engine.groovy.indexed.mapping:   sandbox/on/off
script.engine.groovy.dynamic.update:    sandbox/on/off
script.engine.groovy.dynamic.search:    sandbox/on/off
script.engine.groovy.dynamic.aggs:      sandbox/on/off
script.engine.groovy.dynamic.mapping:   sandbox/on/off
script.engine.groovy.file.update:       sandbox/on/off
script.engine.groovy.file.search:       sandbox/on/off
script.engine.groovy.file.aggs:         sandbox/on/off
script.engine.groovy.file.mapping:      sandbox/on/off

For ease of use, the following more generic settings are supported too:

script.indexed: sandbox/on/off
script.dynamic: sandbox/on/off
script.file:    sandbox/on/off

script.update:  sandbox/on/off
script.search:  sandbox/on/off
script.aggs:    sandbox/on/off
script.mapping: sandbox/on/off

These will be used to calculate the more specific settings, using the stricter setting of each combination. Operation based settings have precedence over conflicting source based ones.

Note that the `mustache` engine is affected by generic settings applied to any language, while native scripts aren't as they are static by definition.

Also, the previous `script.disable_dynamic` setting can now be deprecated.

Closes #6418
Closes #10116
Closes #10274
2015-03-26 19:56:55 +01:00
Michael McCandless 442f539802 Tests: improve back compat tests by adding delete-by-query in the transaction log on upgrade
Closes #10266
2015-03-26 10:12:22 -04:00
Ryan Ernst 66669c337b Tests: Add static bwc index for 1.5.0 2015-03-23 08:04:05 -07:00
Martijn van Groningen 6a2a4bf28d change elasticsearch.org into elastic.co 2015-03-23 15:13:28 +01:00
Martijn van Groningen 5b3c0143c8 Disable marvel as it may fail the tests, because it creates indices. 2015-03-23 15:13:28 +01:00
Martijn van Groningen 911f522a0e change url to use elastic organization 2015-03-23 13:13:26 +01:00
javanna 65b9a94baa Benchmark api: removed leftovers
Closes #10182
Closes #10184
2015-03-21 10:36:05 +01:00
Clinton Gormley ef5bf9c21b Changed the release notes script to use the category labels instead of title prefix 2015-03-19 16:52:47 +01:00
Robert Muir 3a0700862a regenerate 0.90.x indexes without completion suggester 2015-03-18 14:29:41 -04:00
Robert Muir 087b107dd2 Create bigger back compat indexes. 2015-03-18 14:16:16 -04:00
Britta Weber fc5dcf189c [release script] Check for //NORELEASE in code before release
Lines in the code that should be removed before a release can be annotated with
//NORELEASE . This can be useful when debugging test failures. For example,
one might want to add additional logging that would be too verbose for production
and therfore should be removed before releasing.

closes #10141
2015-03-18 09:59:28 -07:00
Simon Willnauer 8e58c0b7a8 Add license header to signature files 2015-03-16 14:19:08 -07:00
Simon Willnauer 1f712842f7 [TEST] Ban @Seed from test 2015-03-16 14:17:51 -07:00
Tanguy Leroux c457499cb2 [Native] Use direct mapping call in Kernel32Library
This commit modifies the Kernel32Library to use direct mapping instead of a proxy class when doing native calls on Windows platforms. It also adds the "createSecurityManager" permission to the tests.policy file, and adds unit tests that should have failed when the Java security manager is enabled.

Closes #9802
2015-03-02 09:48:18 +01:00
Ryan Ernst 7181bbde26 Mappings: Remove _boost field
This has been deprecated since 1.0.0.RC1. It is finally removed here.

closes #8875
2015-02-26 15:07:07 -08:00
Robert Muir b7e49f11ed fix comment 2015-02-25 15:32:45 -05:00
Robert Muir 99714ee1bd Reduce tests.policy a bit more
Remove some more wildcard permissions and replace with just what we need.

Closes #9868.
2015-02-25 11:22:02 -05:00
Igor Motov a7f8e636ba Tests: make sure that repository is not stored when old index file is created
Otherwise the fs repository metadata that points to non-existing location is stored in the old index cluster state, which causes warnings during OldIndexBackwardsCompatibilityTests.
2015-02-24 19:28:35 -05:00
Robert Muir f71beacbdb ban deprecated lucene postings apis 2015-02-20 07:11:37 -05:00