375 Commits

Author SHA1 Message Date
Robert Muir
1994dbde15 Give a better exception when a jar contains same classfile twice.
And ignore the known issue with xmlbeans for now... though it may
cause us issues ultimately: https://issues.apache.org/jira/browse/XMLBEANS-499
2015-07-07 13:26:54 -04:00
David Pilato
d57de59158 Simplify Plugin Manager for official plugins
Plugin Manager can now use another simplified form when a user wants to install an official plugin hosted at elasticsearch download service.

The form we use is:

```sh
bin/plugin install pluginname
```

As plugins share now the same version as elasticsearch, we can automatically guess what is the exact current version of the plugin manager script.

Also, download service will now use `/org.elasticsearch.plugins/pluginName/pluginName-version.zip` URL path to download a plugin.

If the older form is provided (`user/plugin/version` or `user/plugin`), we will still use:

 * elasticsearch download service at `/user/plugin/plugin-version.zip`
 * maven central with groupIp=user, artifactId=plugin and version=version
 * github with user=user, repoName=plugin and tag=version
 * github with user=user, repoName=plugin and branch=master if no version is set

Note that community plugin providers can use other download services by using `--url` option.

If you try to use the new form with a non core elasticsearch plugin, the plugin manager will reject
it and will give you all known core plugins.

```
Usage:
    -u, --url     [plugin location]   : Set exact URL to download the plugin from
    -i, --install [plugin name]       : Downloads and installs listed plugins [*]
    -t, --timeout [duration]          : Timeout setting: 30s, 1m, 1h... (infinite by default)
    -r, --remove  [plugin name]       : Removes listed plugins
    -l, --list                        : List installed plugins
    -v, --verbose                     : Prints verbose messages
    -s, --silent                      : Run in silent mode
    -h, --help                        : Prints this help message

 [*] Plugin name could be:
     elasticsearch-plugin-name    for Elasticsearch 2.0 Core plugin (download from download.elastic.co)
     elasticsearch/plugin/version for elasticsearch commercial plugins (download from download.elastic.co)
     groupId/artifactId/version   for community plugins (download from maven central or oss sonatype)
     username/repository          for site plugins (download from github master)

Elasticsearch Core plugins:
 - elasticsearch-analysis-icu
 - elasticsearch-analysis-kuromoji
 - elasticsearch-analysis-phonetic
 - elasticsearch-analysis-smartcn
 - elasticsearch-analysis-stempel
 - elasticsearch-cloud-aws
 - elasticsearch-cloud-azure
 - elasticsearch-cloud-gce
 - elasticsearch-delete-by-query
 - elasticsearch-lang-javascript
 - elasticsearch-lang-python
```
2015-07-07 18:27:40 +02:00
Ryan Ernst
4aecd37e57 Mappings: Remove AbstractFieldMapper
AbstractFieldMapper is the only direct base class of FieldMapper.
This change moves all AbstractFieldMapper functionality into
FieldMapper, since there is no need for 2 levels of abstraction.
2015-07-07 08:43:38 -07:00
Jason Tedor
c563d68872 Failure during the fetch phase of scan should invoke the failed fetch phase handler.
This commit fixes an issue where during a failure in the fetch phase of a scan the wrong failure handler was invoked.

Closes #12086
2015-07-07 11:35:03 -04:00
Ryan Ernst
2cc0382cf0 Merge pull request #12068 from rjernst/fix/mapper-names-conflict
Mappings: Enforce field names do not contain dot
2015-07-07 08:34:04 -07:00
Martijn van Groningen
f7ac2a7e1c test: check node count on all nodes before checking if cluster state is the same on all nodes 2015-07-07 16:46:24 +02:00
Jason Tedor
b2d8a1fd1b Count scans in search stats and add metrics for scrolls
Each scroll on a scan causes a query to be executed. This commit adds support for these indirect queries to count against the search stats.
Additionally, this commit adds three new search stats: scroll_count, scroll_time_in_millis, and scroll_current. scroll_count tracks the
number of completed scrolls. scroll_time_in_millis tracks the total time that scrolls were held open. scroll_current tracks the number of
scrolls currently open.

Closes #9109
2015-07-07 10:20:45 -04:00
Alexander Reelsen
21b4f9b6f8 Plugins: Ensure logging configuration is loaded in plugin manager
This prevents log4j warnings printed out, when installing a plugin
due to the JarHell class using an ESLogger.

Closes #12064
2015-07-07 14:56:06 +02:00
Alex Ksikes
5e023848de Properly fix the default regex flag to ALL for RegexpQueryBuilder and Parser
Relates to #11896
Closes #12067
2015-07-07 14:26:46 +02:00
Boaz Leskes
67318ce7ba Tests: Faster recovery from simulated disurptions
In testing infra, one can simulate node GCs, network issues and other problems by adding a disruption to the test cluster. Those disruption are automatically removed after the test is done. At the moment each disruption indicates how long it will take the cluster to heal once the disruption is removed and the test cluster waits for this amount of time. However, more often than not this is an upper bound, causing a much longer wait than needed. Instead we should push the responsibility of healing to the disruption it self, where we can be smarter about what we wait for.

Closes #12071
2015-07-07 14:16:31 +02:00
Alex Ksikes
4f9855261a Revert "fix RegexpQueryBuilder#maxDeterminizedStates"
This reverts commit b7e26fae3ff9779caec251b7490d71eeaa297161.
2015-07-07 14:06:07 +02:00
Alex Ksikes
b7e26fae3f fix RegexpQueryBuilder#maxDeterminizedStates
Value was improperly set to `true`.

Relates to #11896
2015-07-07 13:44:05 +02:00
David Pilato
af1dc6d809 [test] awaitBusy: add a ceiling to max sleep time
When using `awaitBusy`, sometimes, you might not want to double time between two runs in an infinitive manner.

For example, let's say it will probably take 30 seconds to run a test.
When doubling all the time, you will most likely wait for a bigger time than needed:

|iteration|ms           |s            |duration (ms)|duration (s)|
|-----------|-------------|-----------|-----------|-----------|
|1|1|0,001|1|0,001|
|2|2|0,002|3|0,003|
|3|4|0,004|7|0,007|
|4|8|0,008|15|0,015|
|5|16|0,016|31|0,031|
|6|32|0,032|63|0,063|
|7|64|0,064|127|0,127|
|8|128|0,128|255|0,255|
|9|256|0,256|511|0,511|
|10|512|0,512|1023|1,023|
|11|1024|1,024|2047|2,047|
|12|2048|2,048|4095|4,095|
|13|4096|4,096|8191|8,191|
|14|8192|8,192|16383|16,383|
|15|16384|16,384|32767|32,767|
|16|32768|32,768|65535|65,535|
|17|65536|65,536|131071|131,071|
|18|131072|131,072|262143|262,143|
|19|262144|262,144|524287|524,287|
|20|524288|524,288|1048575|1048,575|
|21|1048576|1048,576|2097151|2097,151|

For example here, if the task is successful after 35 seconds, we will most likely have to wait for 32s more before the Predicate is run again.

With this patch, the maximum sleep time is now set to 1 second.
2015-07-07 12:04:16 +02:00
Colin Goodheart-Smithe
1d7fc6b4f2 Aggregations: Pipeline Aggregation to filter buckets based on a script
This pipeline aggregation runs a script on each bucket in the parent aggregation to determine whether the bucket is kept in the final aggregation tree. If the script returns true the bucket is retained, if it returns false the bucket is dropped
2015-07-07 09:51:16 +01:00
Alexander Reelsen
b612cab96a Dates: More strict parsing of ISO dates
If you are using the default date or the named identifiers of dates,
the current implementation was allowed to read a year with only one
digit. In order to make this more strict, this fixes a year to be at
least 4 digits. Same applies for month, day, hour, minute, seconds.

Also the new default is `strictDateOptionalTime` for indices created
with Elasticsearch 2.0 or newer.

In addition a couple of not exposed date formats have been exposed, as they
have been mentioned in the documentation.

Closes #6158
2015-07-07 09:34:37 +02:00
Christoph Büscher
35ddc749b1 Merge pull request #12060 from cbuescher/fix/9821
Fix: Use correct OpType on Failure in BulkItemResponse
2015-07-07 09:29:29 +02:00
Robert Muir
d732c0d19f Add symlink permissions test 2015-07-07 02:38:11 -04:00
Simon Willnauer
9e196c3a0b [TEST] Wait for test thread to join before 2015-07-07 07:24:50 +02:00
Ryan Ernst
aed1f68e49 Mappings: Enforce field names do not contain dot
Field names containing dots can cause problems. For example, @jpountz
made this recreation which cause no error, but can result in a
serialization exception if the type already exists:
https://gist.github.com/jpountz/8c66817e00a322b81f85

But this is not just a potential conflict. It also has larger problems,
since only the leaf mapper is created. The intermediate "foo" object
field would not exist if only "foo.bar" was in the mappings.

This change forbids the use of dots in field names. It also
fixes an issue with passing through the update_all_types setting,
which was always set to true whenever a type already existed (!).

I do not think we should worry about backwards compatibility here. This
should be a hard break (and added to the migration plugin).
2015-07-06 18:22:06 -07:00
Simon Willnauer
3906ff950c Don't use forbidden API in test 2015-07-06 22:56:32 +02:00
Simon Willnauer
04c5dab3d9 Add basic recovery prioritization to GatewayAllocator
This commit adds logic to prefer shards with higher priority
or from newer indicse to be allocated first if they are unallocated post API.

This commit allows users to set `index.priority` to a non-negative integer to
prioritize index recovery for certain indices. This setting is dynamically updateable
and defaults to `0`. If two indices have the same priority this change takes the creation
date into account to prioritize shards from newer indices which is important in the time-based
indices usecase.

Closes #11787
2015-07-06 22:51:34 +02:00
Robert Muir
546e99f072 Merge pull request #12061 from rmuir/plugin-integration-tests
Add integration test harness for plugins
2015-07-06 16:03:56 -04:00
David Pilato
1a3eb4d83a Upsert does not use ttl value
When running an upsert which defines a ttl value, the ttl value is set to null and ignored.

Related to #3256#issuecomment-64963409
2015-07-06 21:57:41 +02:00
Christoph Büscher
201dc87aae Fix: Use correct OpType on Failure in BulkItemResponse
When a bulk request fails on a Delete or Update request, the BulkItemResponse
reports incorrect "index" operation in the response. This PR fixes this
for the case of closed indices as reported in #9821 but also for
other failures and adds tests for the two cases covered.

Closes #9821
2015-07-06 20:20:54 +02:00
Igor Motov
177180ac94 Fix snapshot-related REST integration tests 2015-07-06 14:16:28 -04:00
Robert Muir
7595104ec3 Factor integration tests logic to separate build file 2015-07-06 13:59:16 -04:00
Martijn van Groningen
fb6daebe34 scroll: Append the shard top docs in such a way that ArrayIndexOutOfBoundsException is impossible to occur.
also added AtomicArray#setOnce() method to be sure that we fail if an shard response has already been set
2015-07-06 19:33:28 +02:00
Ryan Ernst
ff0f480938 Merge pull request #12014 from rjernst/remove/mapper-doc-values
Completely move doc values and fielddata settings to field types
2015-07-06 10:21:09 -07:00
Ryan Ernst
184b1cbd68 Fix fake field mapper to not pass null for defaultfieldtype 2015-07-06 10:20:19 -07:00
Ryan Ernst
91e3cdb187 Addressed PR comments 2015-07-06 10:02:53 -07:00
Alex Ksikes
afe9c52f07 XMoreLikeThis: fix potentially unpositioned enum 2015-07-06 18:08:06 +02:00
Robert Muir
d7dab0df72 fix test to allow integration tests 2015-07-06 11:20:30 -04:00
Robert Muir
3f4b8df00d Merge pull request #12026 from rmuir/integ_tests
add integration test harness to maven build
2015-07-06 10:16:54 -04:00
Alex Ksikes
8ef1708395 [TEST] fix testMoreLikeThisArtificialDocs
Ensure that the indexed doc hasn't all its fields empty and that the
artificial doc requested is always routed to the shard having that doc.
2015-07-06 15:27:38 +02:00
Colin Goodheart-Smithe
b93512e000 [TEST] NettyTransportMultiPortTests now checks selected random ports are free before using them 2015-07-06 12:04:22 +01:00
markharwood
f95981b977 Search enhancement: adds a special case for the “_index” field which allows queries to test the name of the index. Adding this capability means we can deprecate the specialist indices query.
IndexFieldMapper is changed to make the term query factories produce match_all or match_none queries based on tests on the index name.

Closes #3316
2015-07-06 10:55:28 +01:00
javanna
d085088793 BoolQueryBuilder#hasClauses to take into account filterClauses too 2015-07-06 11:02:09 +02:00
Boaz Leskes
d74eac94f2 ZenDiscovery: #11960 failed to remove eager reroute from node join
It is removed now :)

Closes #12019
2015-07-03 22:06:42 +02:00
Robert Muir
9d233aeaf0 use external test cluster for integration tests 2015-07-03 12:20:35 -04:00
Tanguy Leroux
9495816cb7 Remove sigar completely 2015-07-03 15:49:17 +02:00
Simon Willnauer
ac32f3d310 Don't special-case on ElasticsearchWrapperException in toXContent
the specialization can cause stack overflows if an exception is a
ElasticsearchWrapperException as well as a ElasticsearchException.
This commit just relies on the unwrap logic now to find the cause and only
renders if we the rendering exception is the cause otherwise forwards
to the generic exception rendering.

Closes #11994
2015-07-03 13:51:11 +02:00
Ryan Ernst
e83ae64ea9 Mappings: Completely move doc values and fielddata settings to field types
While MappedFieldType contains settings for doc values and fielddata,
AbstractFieldMapper had special logic in its constructor that
required setting these on the field type from there. This change
removes those settings from the AbstractFieldMapper constructor.
As a result, defaultDocValues(), defaultFieldType() and
defaultFieldDataType() are no longer needed.
2015-07-03 03:10:20 -07:00
Martijn van Groningen
13602a591f Removed forcedDateParser from hashCode() and equals() as it holds no value when it comes to equality. 2015-07-03 10:04:22 +02:00
Simon Willnauer
20d0b4f446 Promote headers to first class citizens on exceptions
This commit merges the pre-existing special exception that
allowed to associate headers with exceptions and the elasticsaerch
base class `ElasticsearchException` This allows for more generic use
of exceptions where plugins can associate meta-data with any elasticsearch
base exception to control behavior etc.

This also addds a generic SecurityException to allow plugins to pass on
information based on the RestStatus.
2015-07-03 08:58:44 +02:00
Robert Muir
80871bae2b Add simple integ testing infra 2015-07-03 02:12:01 -04:00
Boaz Leskes
3084bed194 Test: force local transport in ZenDiscoveryTests.testHandleNodeJoin_incompatibleMinVersion
Otherwise we ignore the join request due to a transport address mismatch instead of failing it.
2015-07-02 22:20:12 +02:00
Robert Muir
3171f52461 Merge pull request #12001 from rmuir/no_dont_do_that
Don't add CWD to classpath when ES_CLASSPATH isn't set.
2015-07-02 14:53:24 -04:00
Martijn van Groningen
a4b99e6291 zen: Don't join master nodes or accept join requests of old and too new nodes.
If the version of a node is lower than the minimum supported version or higher than the maximum supported version, a node shouldn't be allowed to join and nodes should join that elected master node

Closes #11924
2015-07-02 20:51:21 +02:00
Ryan Ernst
0f62950856 fix compile error after bad merge 2015-07-02 10:53:30 -07:00
Ryan Ernst
e12bb4a54a Merge pull request #11977 from rjernst/pr/mapper-names
Move short name access out of field type
2015-07-02 10:38:30 -07:00