Commit Graph

14283 Commits

Author SHA1 Message Date
Areek Zillur 4849e76275 Currently when an engine is failed, it is marked as corrupted regardless of
the failure type. This change marks the engine as corrupted only when the failure
is caused by an actual index corrruption. When an engine is failed for other
reasons, the engine is only closed without removing the shard state.

closes #11788
2015-07-07 16:48:18 -04:00
Tanguy Leroux fbcf4dbbf7 FS Stats: remove sigar specific stats from APIs:
- fs.*.disk_reads
- fs.*.disk_writes
- fs.*.disk_io_op
- fs.*.disk_read_size_in_bytes
- fs.*.disk_write_size_in_bytes
- fs.*.disk_io_size_in_bytes
- fs.*.disk_queue
- fs.*.disk_service_time
2015-07-07 22:16:39 +02:00
Robert Muir 7dbc5c7ab9 Merge pull request #12093 from rmuir/no_fucking_way
Give a better exception when a jar contains same classfile twice.
2015-07-07 16:13:12 -04:00
Zachary Tong c898dd252b [DOCS] Update section about gap_policy 2015-07-07 15:40:15 -04:00
Tanguy Leroux 30892c4129 Remove network stats & info 2015-07-07 21:16:42 +02:00
Clinton Gormley 50ec1b1f8e Merge pull request #12091 from clintongormley/plugin_rest_tests
Plugin rest tests for analysis-phonetic, cloud-*, and lang-*
2015-07-07 19:27:17 +02:00
Robert Muir 1994dbde15 Give a better exception when a jar contains same classfile twice.
And ignore the known issue with xmlbeans for now... though it may
cause us issues ultimately: https://issues.apache.org/jira/browse/XMLBEANS-499
2015-07-07 13:26:54 -04:00
Clinton Gormley 956123157b Added rest tests for language plugins 2015-07-07 19:26:31 +02:00
Clinton Gormley 7aac4d5417 Added rest tests for cloud plugins 2015-07-07 19:26:31 +02:00
Clinton Gormley eefb3cf41c Added rest tests for analysis-phonetic plugin 2015-07-07 19:26:31 +02:00
David Pilato d57de59158 Simplify Plugin Manager for official plugins
Plugin Manager can now use another simplified form when a user wants to install an official plugin hosted at elasticsearch download service.

The form we use is:

```sh
bin/plugin install pluginname
```

As plugins share now the same version as elasticsearch, we can automatically guess what is the exact current version of the plugin manager script.

Also, download service will now use `/org.elasticsearch.plugins/pluginName/pluginName-version.zip` URL path to download a plugin.

If the older form is provided (`user/plugin/version` or `user/plugin`), we will still use:

 * elasticsearch download service at `/user/plugin/plugin-version.zip`
 * maven central with groupIp=user, artifactId=plugin and version=version
 * github with user=user, repoName=plugin and tag=version
 * github with user=user, repoName=plugin and branch=master if no version is set

Note that community plugin providers can use other download services by using `--url` option.

If you try to use the new form with a non core elasticsearch plugin, the plugin manager will reject
it and will give you all known core plugins.

```
Usage:
    -u, --url     [plugin location]   : Set exact URL to download the plugin from
    -i, --install [plugin name]       : Downloads and installs listed plugins [*]
    -t, --timeout [duration]          : Timeout setting: 30s, 1m, 1h... (infinite by default)
    -r, --remove  [plugin name]       : Removes listed plugins
    -l, --list                        : List installed plugins
    -v, --verbose                     : Prints verbose messages
    -s, --silent                      : Run in silent mode
    -h, --help                        : Prints this help message

 [*] Plugin name could be:
     elasticsearch-plugin-name    for Elasticsearch 2.0 Core plugin (download from download.elastic.co)
     elasticsearch/plugin/version for elasticsearch commercial plugins (download from download.elastic.co)
     groupId/artifactId/version   for community plugins (download from maven central or oss sonatype)
     username/repository          for site plugins (download from github master)

Elasticsearch Core plugins:
 - elasticsearch-analysis-icu
 - elasticsearch-analysis-kuromoji
 - elasticsearch-analysis-phonetic
 - elasticsearch-analysis-smartcn
 - elasticsearch-analysis-stempel
 - elasticsearch-cloud-aws
 - elasticsearch-cloud-azure
 - elasticsearch-cloud-gce
 - elasticsearch-delete-by-query
 - elasticsearch-lang-javascript
 - elasticsearch-lang-python
```
2015-07-07 18:27:40 +02:00
Robert Muir 992716ac5c Merge pull request #12084 from rmuir/more_robust
make integ testing a bit more picky
2015-07-07 12:20:32 -04:00
Ryan Ernst 4aecd37e57 Mappings: Remove AbstractFieldMapper
AbstractFieldMapper is the only direct base class of FieldMapper.
This change moves all AbstractFieldMapper functionality into
FieldMapper, since there is no need for 2 levels of abstraction.
2015-07-07 08:43:38 -07:00
David Pilato 7843c63aa3 [kuromoji] move integration tests to REST tests
We can keep only unit tests in plugins instead of starting each time a local node and running tests against it.
2015-07-07 17:40:54 +02:00
Jason Tedor c563d68872 Failure during the fetch phase of scan should invoke the failed fetch phase handler.
This commit fixes an issue where during a failure in the fetch phase of a scan the wrong failure handler was invoked.

Closes #12086
2015-07-07 11:35:03 -04:00
Ryan Ernst 2cc0382cf0 Merge pull request #12068 from rjernst/fix/mapper-names-conflict
Mappings: Enforce field names do not contain dot
2015-07-07 08:34:04 -07:00
Martijn van Groningen f7ac2a7e1c test: check node count on all nodes before checking if cluster state is the same on all nodes 2015-07-07 16:46:24 +02:00
Robert Muir 023cb0b174 make integ testing a bit more picky 2015-07-07 10:21:40 -04:00
Jason Tedor b2d8a1fd1b Count scans in search stats and add metrics for scrolls
Each scroll on a scan causes a query to be executed. This commit adds support for these indirect queries to count against the search stats.
Additionally, this commit adds three new search stats: scroll_count, scroll_time_in_millis, and scroll_current. scroll_count tracks the
number of completed scrolls. scroll_time_in_millis tracks the total time that scrolls were held open. scroll_current tracks the number of
scrolls currently open.

Closes #9109
2015-07-07 10:20:45 -04:00
Clinton Gormley aaf1d14b21 Docs: Fixed bad links 2015-07-07 16:08:10 +02:00
Simon Willnauer 3ffb50828b Merge pull request #11955 from clintongormley/translog_docs
Docs: Updated the translog docs to reflect the new behaviour/settings
2015-07-07 15:37:38 +02:00
Alexander Reelsen 21b4f9b6f8 Plugins: Ensure logging configuration is loaded in plugin manager
This prevents log4j warnings printed out, when installing a plugin
due to the JarHell class using an ESLogger.

Closes #12064
2015-07-07 14:56:06 +02:00
Alex Ksikes 5e023848de Properly fix the default regex flag to ALL for RegexpQueryBuilder and Parser
Relates to #11896
Closes #12067
2015-07-07 14:26:46 +02:00
Boaz Leskes 67318ce7ba Tests: Faster recovery from simulated disurptions
In testing infra, one can simulate node GCs, network issues and other problems by adding a disruption to the test cluster. Those disruption are automatically removed after the test is done. At the moment each disruption indicates how long it will take the cluster to heal once the disruption is removed and the test cluster waits for this amount of time. However, more often than not this is an upper bound, causing a much longer wait than needed. Instead we should push the responsibility of healing to the disruption it self, where we can be smarter about what we wait for.

Closes #12071
2015-07-07 14:16:31 +02:00
Alex Ksikes 4f9855261a Revert "fix RegexpQueryBuilder#maxDeterminizedStates"
This reverts commit b7e26fae3f.
2015-07-07 14:06:07 +02:00
Alex Ksikes b7e26fae3f fix RegexpQueryBuilder#maxDeterminizedStates
Value was improperly set to `true`.

Relates to #11896
2015-07-07 13:44:05 +02:00
David Pilato af1dc6d809 [test] awaitBusy: add a ceiling to max sleep time
When using `awaitBusy`, sometimes, you might not want to double time between two runs in an infinitive manner.

For example, let's say it will probably take 30 seconds to run a test.
When doubling all the time, you will most likely wait for a bigger time than needed:

|iteration|ms           |s            |duration (ms)|duration (s)|
|-----------|-------------|-----------|-----------|-----------|
|1|1|0,001|1|0,001|
|2|2|0,002|3|0,003|
|3|4|0,004|7|0,007|
|4|8|0,008|15|0,015|
|5|16|0,016|31|0,031|
|6|32|0,032|63|0,063|
|7|64|0,064|127|0,127|
|8|128|0,128|255|0,255|
|9|256|0,256|511|0,511|
|10|512|0,512|1023|1,023|
|11|1024|1,024|2047|2,047|
|12|2048|2,048|4095|4,095|
|13|4096|4,096|8191|8,191|
|14|8192|8,192|16383|16,383|
|15|16384|16,384|32767|32,767|
|16|32768|32,768|65535|65,535|
|17|65536|65,536|131071|131,071|
|18|131072|131,072|262143|262,143|
|19|262144|262,144|524287|524,287|
|20|524288|524,288|1048575|1048,575|
|21|1048576|1048,576|2097151|2097,151|

For example here, if the task is successful after 35 seconds, we will most likely have to wait for 32s more before the Predicate is run again.

With this patch, the maximum sleep time is now set to 1 second.
2015-07-07 12:04:16 +02:00
Colin Goodheart-Smithe 1d7fc6b4f2 Aggregations: Pipeline Aggregation to filter buckets based on a script
This pipeline aggregation runs a script on each bucket in the parent aggregation to determine whether the bucket is kept in the final aggregation tree. If the script returns true the bucket is retained, if it returns false the bucket is dropped
2015-07-07 09:51:16 +01:00
Alexander Reelsen b612cab96a Dates: More strict parsing of ISO dates
If you are using the default date or the named identifiers of dates,
the current implementation was allowed to read a year with only one
digit. In order to make this more strict, this fixes a year to be at
least 4 digits. Same applies for month, day, hour, minute, seconds.

Also the new default is `strictDateOptionalTime` for indices created
with Elasticsearch 2.0 or newer.

In addition a couple of not exposed date formats have been exposed, as they
have been mentioned in the documentation.

Closes #6158
2015-07-07 09:34:37 +02:00
Christoph Büscher 35ddc749b1 Merge pull request #12060 from cbuescher/fix/9821
Fix: Use correct OpType on Failure in BulkItemResponse
2015-07-07 09:29:29 +02:00
Robert Muir d732c0d19f Add symlink permissions test 2015-07-07 02:38:11 -04:00
Robert Muir 69cd933ea3 Merge pull request #12070 from rmuir/analysis_rest_tests
add integration tests for analysis plugins
2015-07-07 01:41:10 -04:00
Robert Muir 0736f97cfb add stempel integration tests 2015-07-07 01:27:25 -04:00
Simon Willnauer 9e196c3a0b [TEST] Wait for test thread to join before 2015-07-07 07:24:50 +02:00
Robert Muir 5bcc247eda Add integration tests for smart chinese 2015-07-07 01:14:39 -04:00
Robert Muir f412c5a6ed add kuromoji integration tests 2015-07-07 00:51:52 -04:00
Robert Muir c88c12c6c8 Add rest tests for analysis-icu 2015-07-07 00:15:49 -04:00
Ryan Ernst aed1f68e49 Mappings: Enforce field names do not contain dot
Field names containing dots can cause problems. For example, @jpountz
made this recreation which cause no error, but can result in a
serialization exception if the type already exists:
https://gist.github.com/jpountz/8c66817e00a322b81f85

But this is not just a potential conflict. It also has larger problems,
since only the leaf mapper is created. The intermediate "foo" object
field would not exist if only "foo.bar" was in the mappings.

This change forbids the use of dots in field names. It also
fixes an issue with passing through the update_all_types setting,
which was always set to true whenever a type already existed (!).

I do not think we should worry about backwards compatibility here. This
should be a hard break (and added to the migration plugin).
2015-07-06 18:22:06 -07:00
David Pilato 7ec6dc9956 [build] Update to Apache Maven PMD Plugin 3.5
http://maven.apache.org/plugins/maven-pmd-plugin/

You should specify the version in your project's plugin configuration:

```xml
<plugin>
 <groupId>org.apache.maven.plugins</groupId>
 <artifactId>maven-pmd-plugin</artifactId>
 <version>3.5</version>
</plugin>
```

Release Notes - Apache Maven PMD Plugin - Version 3.5
---------------

Bug

* [MPMD-208] Warning about deprecated Rule name when using rulesets/maven.xml
* [MPMD-205] Javascript violations won't fail the build

Improvement

* [MPMD-211] Upgrade plexus-resources from 1.0-alpha-7 to 1.1
* [MPMD-209] Upgrade to PMD 5.3.2
* [MPMD-206] Make the sourceDirectories configurable

New Feature

* [MPMD-207] Support Javascript and JSP for CPD
2015-07-06 23:19:55 +02:00
Simon Willnauer 3906ff950c Don't use forbidden API in test 2015-07-06 22:56:32 +02:00
Simon Willnauer 04c5dab3d9 Add basic recovery prioritization to GatewayAllocator
This commit adds logic to prefer shards with higher priority
or from newer indicse to be allocated first if they are unallocated post API.

This commit allows users to set `index.priority` to a non-negative integer to
prioritize index recovery for certain indices. This setting is dynamically updateable
and defaults to `0`. If two indices have the same priority this change takes the creation
date into account to prioritize shards from newer indices which is important in the time-based
indices usecase.

Closes #11787
2015-07-06 22:51:34 +02:00
Robert Muir 546e99f072 Merge pull request #12061 from rmuir/plugin-integration-tests
Add integration test harness for plugins
2015-07-06 16:03:56 -04:00
David Pilato 1a3eb4d83a Upsert does not use ttl value
When running an upsert which defines a ttl value, the ttl value is set to null and ignored.

Related to #3256#issuecomment-64963409
2015-07-06 21:57:41 +02:00
Robert Muir 23dfb5f1a5 don't try to run tests for packaging=pom 2015-07-06 15:49:57 -04:00
Robert Muir d7bc274690 Merge branch 'master' into plugin-integration-tests
Conflicts:
	core/pom.xml
2015-07-06 15:15:15 -04:00
Robert Muir 9a146b9e75 Get delete by query rest tests running 2015-07-06 15:13:50 -04:00
Christoph Büscher 201dc87aae Fix: Use correct OpType on Failure in BulkItemResponse
When a bulk request fails on a Delete or Update request, the BulkItemResponse
reports incorrect "index" operation in the response. This PR fixes this
for the case of closed indices as reported in #9821 but also for
other failures and adds tests for the two cases covered.

Closes #9821
2015-07-06 20:20:54 +02:00
Robert Muir 75285cee88 factor out this helper 2015-07-06 14:19:28 -04:00
Igor Motov 177180ac94 Fix snapshot-related REST integration tests 2015-07-06 14:16:28 -04:00
Robert Muir 7595104ec3 Factor integration tests logic to separate build file 2015-07-06 13:59:16 -04:00