Commit Graph

647 Commits

Author SHA1 Message Date
Zachary Tong 7367af57cf String.join() is not available pre-Java8 2015-07-16 12:24:36 -04:00
Michael McCandless 42291e3cfd QueryString ignores maxDeterminizedStates when creating a WildcardQuery
This patch backports https://issues.apache.org/jira/browse/LUCENE-6677

Merges 3ede4c469922308d43fba9ebc63e397ffd6997fe from 1.x
2015-07-16 12:21:39 -04:00
Alex Ksikes d0256dac69 RegexpQueryParser takes a String as value like its Builder
Relates #11896
Closes #12200
2015-07-16 17:59:08 +02:00
Zachary Tong 380648ab86 Merge pull request #12280 from polyfractal/bugfix/movavg_validation
Aggregations: Add better validation of moving_avg model settings
2015-07-16 11:44:32 -04:00
Zachary Tong 702f884ba0 Aggregations: Add better validation of moving_avg model settings 2015-07-16 11:41:30 -04:00
Lee Hinman a8391fcae9 Add _replica and _replica_first as search preference.
Just like specifying `?preference=_primary`, this adds the ability to
specify `?preference=_replica` or `?preference=_replica_first` on
requests that support it.

Resolves #12222
2015-07-16 09:25:23 -06:00
Boaz Leskes 439c67ab15 Transport: Tracer should exclude "cluster:monitor/nodes/liveness" by default
This action is a liveness test added in #8763 . It should be excluded, just like the fault detection logic or things become overly chatty.

Closes #12291
2015-07-16 17:23:27 +02:00
Martijn van Groningen 211b4d9083 Merge pull request #12288 from martijnvg/upgrade_groovy
upgrade groovy from 2.4.0 to 2.4.4
2015-07-16 14:40:06 +02:00
Martijn van Groningen 9687670e8d upgraded groovy from 2.4.0 to 2.4.4 2015-07-16 13:56:32 +02:00
Adrien Grand e86ff6785a Do not assert that all shards were successful in tests.
Most of our tests call assertSearchResponse which checks whether all shards
were successful. However this is usually not necessary given that all shards
that received documents should be available given the way our indexing works.
The only shards that might not be available are those that did not index
any documents. So removing this assertion would allow us to remove most
ensureGreen/ensureYellow calls while still being able to assert on the content
of the search response since all data have been taken into account.

For now I only removed the ensureYellow/Green calls from SearchQueryTests in
order to not de-stabilize the build, but eventually we should remove most of
them.
2015-07-16 11:18:55 +02:00
Boaz Leskes f9a7e7cd84 add awaitFix the entire MinimumMasterNodesTests class. multiple tests are failing... 2015-07-16 10:47:18 +02:00
Jason Tedor d1c8703477 Add global search timeout setting
This commit adds a dynamically updatable cluster-level search timeout setting.

Closes #12149
2015-07-15 20:18:16 -05:00
Igor Motov bfbee383bd Add checksum to snapshot metadata files
This commit adds checksum to snapshot files that store global and index based metadata as well as shard information.

Closes #11589
2015-07-15 18:52:56 -04:00
Jay Modi 5091a10f3a Merge pull request #12271 from jaymode/test_zen_port_range
restrict the test unicast zen discovery to the port range of the JVM
2015-07-15 12:38:25 -04:00
Jason Tedor 0820cb66c3 Fix issues due to missing refresh after indexing test documents in SearchStatsTests.testOpenContexts 2015-07-15 11:27:48 -05:00
Simon Willnauer 901421bc14 Merge pull request #12270 from s1monw/publish_s3
Deploy artifacts to S3 as well as sonatype when running a release
2015-07-15 18:11:38 +02:00
jaymode 883d33be95 restrict the test unicast zen discovery to the port range of the JVM
Today, the unicast zen test configuration will try to find a open port starting at the internal test cluster's
base port and continuing for 1000 ports. The internal test cluster class assigns a port range of 100 ports
to each JVM. This means that the unicast zen test configuration will try ports in the range for another JVM
and can lead to port conflicts. This change uses the same value for both so that the unicast configuration
does not go into another JVM's port range.
2015-07-15 11:42:41 -04:00
Simon Willnauer 6faa19c3de Deploy artifacts to S3 as well as sonatype when running a release
This change allows when running a release build with:

```
 mvn -Prelease deploy
```

To deploy all artifacts to S3 as well as to sonatype at the same time.
Both sources will be consistent in terms of content and no further action
is required to publish our artifacts including .rpm, .deb, .zip, .tar.gz, .jar
etc. on to out S3 download service. Albeit this changes the structure of our
downloads to pretty much matching the maven repository layout, this makes
releaseing core as well as the plugins extremely simple. This will allow to
remove most of our python script used for release and it will automatically
allow to release and integrate new modules without further interaction.

This also allows us to bascially streamline our release process on CI such that
CI builds can simply run maven deploy which is all we do during a release.

With this commit only the git related modifications like tagging, version bumping
on our pom files and publishing RPM and .deb in their dedicated repository is left
for the python script.

With this change our artifact are available as follows:

```
http://download.elasticsearch.org/elasticsearch/release/org/elasticsearch/elasticsearch/2.0.0-beta1/elasticsearch-2.0.0-beta1.deb
http://download.elasticsearch.org/elasticsearch/release/org/elasticsearch/elasticsearch/2.0.0-beta1/elasticsearch-2.0.0-beta1.zip
http://download.elasticsearch.org/elasticsearch/release/org/elasticsearch/elasticsearch/2.0.0-beta1/elasticsearch-2.0.0-beta1.rpm
```

Plugins are deployed to URLs like this:

```
http://download.elasticsearch.org/elasticsearch/release/org/elasticsearch/plugin/elasticsearch-analysis-icu/2.0.0-beta1/elasticsearch-analysis-icu-2.0.0-beta1.zip
```

All artifacts like .jar as well as their checksums and gpg signatures are also available next to it.
2015-07-15 17:14:34 +02:00
Shay Banon f78db1f1d3 Unique allocation id
Add a unique allocation id for a shard, helping to uniquely identify a specific allocation taking place to a node.
A special case is relocation, where a transient relocationId is kept around to make sure the target initializing shard (when using RoutingNodes) is using it for its id, and when relocation is done, the transient relocationId becomes the actual id of it.
closes #12242
2015-07-15 17:08:25 +02:00
Lee Hinman eec3095bf6 [TEST] Remove @AwaitsFix from testRamAccountingTermsEnum()
This test has stabilized with recent ES changes.
2015-07-15 08:57:53 -06:00
Boaz Leskes ea3135f580 add await fix to multipleNodesShutdownNonMasterNodes 2015-07-15 16:25:22 +02:00
Lee Hinman 69d0952c68 [TEST] Remove @AwaitsFix from testMemoryBreaker()
Simplifies the test to make it not fail
2015-07-15 08:09:14 -06:00
Shay Banon f8ba89d9e4 Carry over shard exception failure to master node
Don't loose the shard exception failure when sending a shard failrue to the master node
2015-07-15 14:55:04 +02:00
Simon Willnauer cdcfcf4b00 Allow shards to be allocated if leftover shard from different index exists.
If an index name is reused but a leftover shard still exists on any node
we fail repeatedly to allocate the shard since we now check the index UUID
before reusing data. This commit allows to recover even if there is such a
leftover shard by deleting the leftover shard.

Closes #10677
2015-07-15 09:39:15 +02:00
Boaz Leskes 1e35bf3171 Discovery: wait on incoming joins before electing local node as master
During master election each node pings in order to discover other nodes and validate the liveness of existing nodes. Based on this information the node either discovers an existing master or, if enough nodes are found (based on `discovery.zen.minimum_master_nodes>>) a new master will be elected.

Currently, the node that is elected as master will currently update it the cluster state to indicate the result of the election. Other nodes will submit a join request to the newly elected master node. Instead of immediately processing the election result, the elected master
node should wait for the incoming joins from other nodes, thus validating the elections result is properly applied. As soon as enough nodes have sent their joins request (based on the `minimum_master_nodes` settings) the cluster state is modified.

Note that if `minimum_master_nodes` is not set, this change has no effect.

Closes #12161
2015-07-15 07:43:49 +02:00
Igor Motov f2911117ef Add index name to the upgrade exception 2015-07-14 19:05:39 -04:00
Igor Motov 24a93840d5 Add url repository whitelist
Require urls for URL repository to be listed in repositories.url.allowed_urls setting. This change ensures that only authorized URLs can be accessed by elasticsearch
2015-07-14 18:38:26 -04:00
Martijn van Groningen 0af70614ac test: removed AwaitsFix annotation as the associated issue is fixed and the failures don't reproduce any more 2015-07-14 22:50:47 +02:00
Simon Willnauer 3fe48c89cf Merge pull request #12240 from s1monw/m_structured_response
Render structured exceptions in mget / mpercolate
2015-07-14 22:03:20 +02:00
Robert Muir caab1df111 more fine-grained jython hack that also plays with java 9 2015-07-14 15:42:21 -04:00
Martijn van Groningen f5e351aaf1 parent/child: Removed AwaitsFix annotation
Closes #9461
2015-07-14 20:05:35 +02:00
Robert Muir 01601e9a3d Add permissions to all classpath elements in BootstrapForTesting
The previous strategy (target/xxx + .m2/repository) is obviously broken for multi-module
builds.

Includes hack for crazy jython, which "finds its own jar" then looks for a Lib/ beside it
2015-07-14 12:01:03 -04:00
Simon Willnauer def08bd594 Render structured exceptions in mget / mpercolate
Instead of rendering only the exception message this commit
adds structured exception rendering to mget and mpercolate
2015-07-14 17:52:28 +02:00
Simon Willnauer 09bd19b947 Don't allow fuzziness specified as a and require edits [0,2]
Lucene deprecated this in 4.0 and we only try best effort to support it.
Folks should only use edit distance rather than some length based
similarity. Yet the formular is simple enough such that users can
still do it in the client if they really need to.

Closes #10638
2015-07-14 17:10:29 +02:00
Simon Willnauer 738a0bc8a2 [TEST] Don't rely on iteration oder in tests
Closes #12238
2015-07-14 17:08:26 +02:00
Colin Goodheart-Smithe bccfcfa522 Aggregation: Removed Old Script Java API from metrics aggregations
The old script syntax has been removed from the Java API but the metrics aggregations were missed. This change removes the old script API from the ValuesSourceMetricsAggregationBuilder and removes the relevant test methods for the metrics aggregations.
2015-07-14 15:41:11 +01:00
Simon Willnauer 7db293c616 Generify Index and Shard exceptions
Today we have a intermediate hierarchy for shard and index exceptions
which makes it hard to introduce generic exceptions like ResourceNotFoundException
intoduced in this commit. This commit breaks up the hierarchy by adding index and shard
as a special internal header that gets rendered for every exception that fills that header.
This commit removes dedicated exceptions like `IndexMissingException` or
`IndexShardMissingException` in favour of `ResourceNotFoundException`
2015-07-14 16:31:49 +02:00
Shay Banon c6b110c6ef Simplify assignToNode to only do initializing
The method really only should do the move from unassigned to initializing, all the other moves have explicit methods like relocate
2015-07-14 16:05:48 +02:00
Martijn van Groningen 5cdbe60f6b percolator: Support filtering percolator queries by date using `now`
Closes #12185
2015-07-14 12:35:32 +02:00
Shay Banon e598f16b58 Default delayed allocation timeout to 1m from 0
Change the default delayed allocation timeout from 0 (no delayed allocation) to 1m. The value came from a test of having a node with 50 shards being indexed into (so beefy translog requiring flush on shutdown), then shutting it down and starting it back up and waiting for it to join the cluster. This took, on a slow machine, about 30s.
The value is conservatively low and does not try to address a virtual machine / OS restart for now, in order to not have the affect of node going away and users being concerned that shards are not being allocated to the rest of the cluster as a result of that. The setting can always be changed in order to increase the delayed allocation if needed.
closes #12166
2015-07-14 11:31:16 +02:00
Martijn van Groningen 44275f3e3e Merge pull request #12214 from martijnvg/percolator/_parent_field/npe
Fix NPE when percolating a document that has a _parent field configured in its mapping
2015-07-14 10:31:10 +02:00
Martijn van Groningen 7bc26d1edd percolator: Don't throw NPE when percolating a document that has a _parent field configured in its mapping
Closes #12192
2015-07-14 10:01:16 +02:00
Robert Muir 5d48f93557 fail plugins on version mismatch 2015-07-13 21:01:39 -04:00
Boaz Leskes d12fb5f725 Add MetaData.clusterUUID to ClusterState.toXContent
The MetaData.clusterUUID is guaranteed to be unique across clusters and is handy (which may or may not have the same human readable cluster name).

Closes #11832
2015-07-13 20:31:11 +02:00
uboness b40186652c updated the elasticsearch versioning format
Moving to from `X.Y.Z.beta1`/`X.Y.Z.RC1` to `X.Y.Z-beta1`/`X.Y.Z-rc1`
2015-07-13 20:26:37 +02:00
Boaz Leskes 17906ca7d6 Rename MetaData.uuid -> MetaData.clusterUUID and IndexMetaData.uuid-> IndexMetaData.indexUUID
As explained in #11831, we currently have uuid fields on the cluster state, meta data and index metadata. The latter two are persistent across changes are being effectively used as a persistent uuid for this cluster and a persistent uuid for an index. The first (ClusterState.uuid) is ephemeral and changes with every change to the cluster state. This is confusing,

We settled on having the following, new names:

-> ClusterState.uuid -> stateUUID (transient)
-> MetaData.uuid -> clusterUUID (persistent)
-> IndexMetaData.uuid -> indexUUID (persistent).

Closes #11914
Closes #11831
2015-07-13 20:15:54 +02:00
Simon Willnauer 8e07b4fba4 Throw LockObtainFailedException exception when we can lock index directory
Today we throw ElasticsearchException if we can't lock the index. This can cause
problems since some places where we have logic to deal with IOException on shard
deletion won't schedule a retry if we can't lock the index dir for removal. This
is the case on shadow replicas for instance if a shared FS is used. The result
of this is that the delete of an index is never acked.
2015-07-13 17:40:35 +02:00
Alex Ksikes 64f981fc32 Fix rewrite set twice in WildcardQueryParser
Closes #12207
2015-07-13 17:35:50 +02:00
Simon Willnauer 9faa3c6756 [TEST] ensure exceptions won't cause test to hang 2015-07-13 12:29:50 +02:00
Simon Willnauer 2e2bb25205 Only clear open search ctx if the index is delete or closed via API
A change in #12116 introduces closing / cleaning of search ctx even if
the index service was closed due to a relocation of it's last shard. This
is not desired since in that case it's fine to serve the pending requests from
the relocated shard. This commit adds an extra check to ensure that the index is
either removed (delete) or closed via API.
2015-07-13 11:34:34 +02:00
Simon Willnauer ca56d442bc remove dead code 2015-07-13 11:21:35 +02:00
Alexander Reelsen 195c8ec903 Term Query: Be more strict during parsing
The term query parser was too lenient during parsing and allowed to specify
more than one field, even though this expected to filter only for a single field.

This commit returns an exception if a query has been specified more than once.

Closes #12184
2015-07-13 09:59:30 +02:00
aleph-zero aba3730643 jsr166e was left out of shaded jar
The classes in com.twitter.jsr166e were not getting included in the
shaded jar due to a missing configuration line.

Closes #12193
2015-07-11 15:12:23 -07:00
Jack Conradson c1137b3b78 Add script type and script name to error messages
Modified ScriptEngineService to pass in a CompiledScript object
with newly added name and type member variables.
This can in turn be used to give better scripting error messages
with the type of script used and the name of the script.

Required slight modifications to the caching mechanism.

Note that this does not enforce good behavior in that plugins will
have to write exceptions that also output the name of the script
in order to be effective. There was no way to wrap the script
methods in a try/catch block properly further up the chain because
many have script-like objects passed back that can be run at a
later time.

closes #6653
closes #11449
2015-07-10 18:27:48 -07:00
Zachary Tong bb9c160855 Merge pull request #11196 from polyfractal/feature/aggs_2_0_diff
Aggregations: add serial differencing pipeline aggregation
2015-07-10 18:26:19 -04:00
Zachary Tong e3f9d561e4 Aggregations: add serial differencing pipeline aggregation 2015-07-10 18:22:01 -04:00
Jay Modi c9042a5d2c Merge pull request #11890 from jaymode/cors
change CORS allow origin default to allow no origins
2015-07-10 14:48:26 -04:00
Martijn van Groningen 03148dc3b0 change visibility 2015-07-10 17:42:53 +02:00
Martijn van Groningen 512239041b fix npe 2015-07-10 17:12:51 +02:00
Martijn van Groningen 9eb11267f5 Merge pull request #12150 from martijnvg/aliases/remove_strict_filter_parsing
Don't require fields in alias filters to exist in the mapping
2015-07-10 16:50:51 +02:00
Martijn van Groningen 52859e3a52 Internal: refactored MetaData to split the concrete index name resolution to IndexNameExpressionResolver.
Changes in a nutshell:
* All expression logic is now encapsulated by ExpressionResolver interface.
* MetaData#convertFromWildcards() gets replaced by WildcardExpressionResolver.
* All of the indices expansion methods are being moved from MetaData class to the new IndexNameExpressionResolver class.
* All single index expansion optimisations are removed.

The logic for resolving a concrete index name from an expression has been moved from MetaData to IndexExpressionResolver. The logic has been cleaned up and simplified were was possible without breaking bwc.

Also the notion of aliasOrIndex has been changed to index expression.

The IndexNameExpressionResolver translates index name expressions into concrete indices. The list of index name expressions are first delegated to the known ExpressionResolverS. An ExpressionResolver is responsible for translating if possible an expression into another expression (possibly but not required this can be concrete indices or aliases) otherwise the expressions are left untouched. Concretely this means converting wildcard expressions into concrete indices or aliases, but in the future other implementations could convert expressions based on different rules.

To prevent many overloading of methods, DocumentRequest extends now from IndicesRequest. All implementation of DocumentRequest already did implement IndicesRequest indirectly.
2015-07-10 15:18:28 +02:00
Simon Willnauer 75a51ede24 Allow rpm to be build as part of package phase
This allows the creation of the RPM artifact as part of the
maven package phase. The result of this is that we get checksum and
name correction for-free as it's all build an installed into the m2
repository. This also publishes the RPM together with .deb to the mvn
mirror.

Note: this will only build the RPM as part of the package phase if
`-Dpackage.rpm=true` since the binaries to build the RPM are not
availabel on all platforms.
2015-07-10 14:43:47 +02:00
Simon Willnauer 164212940b Free all pending search contexts if index is closed or removed
Today we only clear search contexts for deleted indies. Yet, we should
do the same for closed indices to ensure they can be reopened quickly.

Closes #12116
2015-07-10 14:27:53 +02:00
Colin Goodheart-Smithe fcf8d2408f Aggregations: Adds new script API to ValuesSourceMetricsAggregationBuilder
A method for the new Script API were missing in the ValuesSourceMetricsAggregationBuilder. This change adds the missing method and deprecates the old Script API methods
2015-07-10 13:09:31 +01:00
szroland 2c453820e4 Treat path object as a simple value objects instead of Iterable in XContentBuilder, using toString() to create String representation.
This addresses #11771
2015-07-10 10:55:43 +02:00
David Pilato 7c3ea748e2 [test] list of plugins should have well-defined order
We now return list of plugins sorted by name.

Closes #12174.
2015-07-10 10:25:15 +02:00
Boaz Leskes 28090b3d73 Allocation: Shard Started messages should be matched using an exact match
When a node sends a shard started message to the master, the master goes through the routing table looking for the shard to start. At the moment we validate the indexUUID, the node the shard is assigned to and the fact that the shard is initializing. This check goes wrong if a relocating replica shard finishes recovery just at the moment the source node  leaves the cluster. In this case the master will cancel the recovery and will likely assign a new initializing replica to the same target node. In this case the message from the relocation recovery can activate the new replica wrongfully.

Also, the logic for decided whether an incoming shard started message will be applied was split between ShardStateAction and the AllocationService.
This commit does the following:
1) Let ShardStateAction only filter basic stuff like index existence and indexUUID.
2) Move the trickier shard started matching logic to the AllocationService and make it stricter
3) Unify ShardStateAction filtering logic for both shard started and shard failed.
4) Add unit tests for all of the above.

For an example test failure see: http://build-us-00.elastic.co/job/es_core_16_centos/388/

Closes #11999
2015-07-10 08:49:31 +03:00
Robert Muir 5b4e86389c fix broken usage of path api 2015-07-10 01:02:13 -04:00
Simon Willnauer 0ea9adb284 [TEST] Assert against minCompatVersion instead of a static string 2015-07-09 21:47:17 +02:00
Ryan Ernst 59d6e59dec silence symlink test on windows 2015-07-09 12:41:48 -07:00
Simon Willnauer e0708813a9 Make 2.0.0.beta1-SNAPSHOT the current version.
Today everything is tight to having the next version as the latest.
In order to work towards 2.0.0.beta1 we need to fix all the usage of
2.0.0-SNAPSHOT to reflect the version we will release soon.
Usually we do this on the release branch but to simplify things I wanna
keep this on master for now and move to 2.1.0-SNAPSHOT on master once
we created a 2.0 branch.

Closes #12148
2015-07-09 21:24:32 +02:00
Igor Motov f303a1d4eb Snapshot info should contain version of elasticsearch that created the snapshot
This information was stored with the snapshot but wasn't available on the interface. Knowing the version of elasticsearch that created the snapshot can be useful to determine the minimal version of the cluster that is required in order to restore this snapshot.

Closes #11980
2015-07-09 12:02:33 -04:00
jaymode ee30cf32ab strip elasticsearch- and es- from any plugin name
#12158 only partially fixed #12143 by removing the prefix from official plugin names. This change
removes the prefixes from any plugin names.
2015-07-09 11:50:41 -04:00
jaymode 3fd9f4b82e remove elasticsearch- from name of official plugins
This change fixes the plugin manager to trim `elasticsearch-` and `es-` prefixes from plugin names
for our official plugins. This restores the old behavior prior to #11805.

Closes #12143
2015-07-09 11:09:12 -04:00
Robert Muir dfbe108902 Fix pluginmanager permissions for bin/ scripts
Today it will remove all permissions and only set execute bit:

    ---x--x--x

Instead we should preserve existing permissions, and just add
read and execute to whatever is there.

Closes #12142
2015-07-09 10:39:58 -04:00
jaymode a3e30a89a7 allow settings to be passed to client for external test cluster
This change allows custom settings to be passed to the client for the external test cluster,
which is necessary when additional settings need to be passed to the client in order to
properly communicate with the external test cluster.
2015-07-09 09:27:47 -04:00
Adrien Grand e406baf59e Merge pull request #11800 from arcz/master
Fix indentation in deb init script
2015-07-09 15:07:42 +02:00
Martijn van Groningen e39b6a5689 aliases: Don't require fields used in alias filters to exist 2015-07-09 12:02:49 +02:00
Martijn van Groningen 86f362699e test: make sure the inner hits are in the expected order 2015-07-09 11:56:23 +02:00
Shay Banon 8a52a362dd Merge pull request #12147 from kimchy/remove_double_elect
Remove double call to elect primaries
2015-07-09 11:44:17 +02:00
Shay Banon 72dc9c5b4f Remove double call to elect primaries
There is no need to call the elect logic twice, we used to need it, but no longer since we handle dangling replicas for unassigned primaries properly
2015-07-09 11:29:41 +02:00
Martijn van Groningen 5f6623c03b inner_hits: Properly support named queries for both nested and parent child inner hits.
Before inner_hits existed named queries has support to also verify if inner queries of nested query matched with returned documents. This logic was broken and became obsolete from the moment inner hits get released. #10694 fixed named queries for nested docs in the top_hits agg, but it didn't fix the named query support for nested inner hits. This commit fixes that and on top of this also adds support for parent/child inner hits.
2015-07-09 10:19:03 +02:00
Adrien Grand 8a44d938d4 [TEST] Add missing ensureYellow() calls to SearchQueryTests. 2015-07-09 09:10:29 +02:00
Adrien Grand f82b5ce201 Merge pull request #12127 from jpountz/fix/sort_merge
Clean up handling of missing values when merging shard results on the coordinating node.
2015-07-09 08:55:52 +02:00
Israel Ekpo ee73056970 BulkRequestBuilder.add() has typo in JavaDoc.
Closes #12138
2015-07-08 22:34:12 -04:00
Zachary Tong 1504b3d97d Fix value of HoltWinters seasonality param in builder
Should be `type`, not `seasonalityType`
2015-07-08 18:56:29 -04:00
Zachary Tong 531b894b5d [TESTS] Fix bad math in moving_avg unit tests 2015-07-08 18:04:36 -04:00
Zachary Tong d5f73ec8ac Merge pull request #11881 from polyfractal/feature/movavg_optim2
Aggregations: add cost minimizer to tune moving_avg parameters
2015-07-08 16:45:24 -04:00
Zachary Tong 0f76e656dd Aggregations: add cost minimizer to moving_avg aggregation 2015-07-08 16:20:34 -04:00
Igor Motov 8692292d0c Add validation of snapshot FileInfo during parsing
Making sure that the file info that we read from the snapshot is still sane.
2015-07-08 15:09:43 -04:00
jaymode ae1ed34355 do not prompt for node name twice
We allow setting the node's name a few different ways: the `name` system
property, the setting `name`, and the setting `node.name`. There is an order
of preference to these settings that gets applied, which can copy values from the
system property or `node.name` setting to the `name` setting. When setting
only `node.name` to one of the prompt placeholders, the user would be
prompted twice as the value of `node.name` is copied to `name` prior to
prompting for input. Additionally, the value entered by the user for `node.name`
would not be used and only the value entered for `name` would be used.

This fix changes the behavior to only prompt once when `node.name is set` and
`name` is not set. This is accomplished by waiting until all values have been
prompted and replaced, then the logic for determining the node's name is
executed.

Closes #11564
2015-07-08 14:03:58 -04:00
Adrien Grand ccbea2e5d2 Fix parsing of the `fields` parameter of bulk requests. 2015-07-08 19:55:24 +02:00
Igor Motov 0187508f70 The test getSwap in OsProbeTests shouldn't fail on a machine with no swap
In some cases the swap size can be reported as 0.
2015-07-08 13:34:38 -04:00
William Webber 6b40169b59 Only load term statistics if required 2015-07-08 18:43:08 +02:00
Jason Tedor b61709c716 Add support for retrieving fields in bulk updates
This commit adds support to retrieve fields when using the bulk update API. This functionality was previously available for the update API
but not for the bulk update API.

Closes #11527
2015-07-08 12:06:20 -04:00
Ryan Ernst df41d0c3ba Merge pull request #12130 from rjernst/fix/doc-mapper-in-engine
Remove mapper references from Engines
2015-07-08 08:58:27 -07:00
Tanguy Leroux 19e348a82c Update OS stats 2015-07-08 17:48:10 +02:00
Ryan Ernst 8696a81a8e Internal: Remove mapper references from Engines
This was previously necessary with MapperAnalyzer, but there are no more uses.
2015-07-08 08:45:11 -07:00
Adrien Grand da5fa6c4f3 Minor fixes to the `match` query.
Fixed documentation since the default rewrite method for fuzzy queries is to
select top terms, fixed usage of the fuzzy rewrite method, and removed unused
`rewrite` parameter.

Close #6932
2015-07-08 16:51:41 +02:00
Adrien Grand 8238f497d8 Expose Lucene's new TopTermsBlendedFreqScoringRewrite.
This rewrite method is interesting because it computes scores as if all terms
had the same frequencies, which avoids disappointments with ranking when a fuzzy
query ranks typos first given that they are less frequent than the correct term.
2015-07-08 16:01:47 +02:00
Tanguy Leroux 1c5d8efd47 Process Stats: remove sigar specific stats from APIs and add JMX implementation 2015-07-08 15:12:45 +02:00
Adrien Grand fbab48e451 Clean up handling of missing values when merging shard results on the coordinating node.
Today shards are responsible for producing one sort value per document, which
is later used on the coordinating node to resolve the global top documents.
However, this is problematic on string fields with
`missing: _first, order: desc` or `missing: _last, order: asc` given that there
is no such thing as a string that compares greater than any other string. Today
we use a string containing a single code point which is the maximum allowed code
point but this is a hack: instead we should inform the coordinating node that
the document had no value and let it figure out how it should be sorted
depending on whether missing values should be sorted first or last.

Close #9155
2015-07-08 14:48:35 +02:00
Simon Willnauer f9a45fd605 Cleanup ShardRoutingState uses and hide implementation details of ClusterInfo 2015-07-08 14:36:26 +02:00
Shay Banon 097b132238 Consolidate ShardRouting construction
Simplify and consolidate ShardRouting construction. Make sure that there is really only one place it gets created, when a shard is first created in unassigned state, and from there on, it is either copy constructed or built internally as a target for relocation.
This change helps make sure within our codebase data carries over by the ShardRouting is not lost as the shard goes through transitions, and can help simplify the addition of more data on it (like uuid).
For testing, a centralized TestShardRouting allows to create testable versions of ShardRouting, that are not needed to be as strict as the non test codebase. This can be cleanup more later on, but it is a good start.
closes #12125
2015-07-08 14:15:28 +02:00
Alex Ksikes a6c0007325 Fix FuzzyQuery to properly handle Object, number, dates or String.
This makes FuzzyQueryBuilder and Parser take an Object as a value using the
same logic as termQuery, so that numbers, dates or Strings would be properly
handled.

Relates #11865
Closes #12020
2015-07-08 10:41:03 +02:00
Simon Willnauer b5452074a3 [TEST] Only sanitly check time values in stats
Testing the actual time value even with lowerbounds is very tricky
and fails very often. We should really just sanity check the values.
2015-07-08 10:02:24 +02:00
Ryan Ernst 8d9053a841 Merge pull request #12089 from rjernst/refactor/field-mapper-collapse
Remove AbstractFieldMapper
2015-07-07 21:36:47 -07:00
Ryan Ernst 8c45c7f482 Internal: Change JarHell to operate on Path instead of URL
This converts the tracking of jars and classes in JarHell to use
Path objects, instead of URL. This makes for nicer printing
of the underlying path when an error does occur.
2015-07-07 20:14:23 -07:00
Ryan Ernst bab1323d1e Fix JarHell check to properly convert URL to Path so it can be compared
to java.home
2015-07-07 19:44:15 -07:00
Jason Tedor 83f6587e61 Default fuzzy transpositions to true
This commit defaults fuzzy_transpositions on fuzzy queries to true. This means that by default, tranpositions will now count as a single
edit.

Closes #9278
2015-07-07 20:13:56 -04:00
Ryan Ernst 6eacbf764d Merge pull request #12106 from rjernst/tests/jar-hell
Tests: Add unit tests for JarHell
2015-07-07 16:58:31 -07:00
Ryan Ernst 35b76ca081 Tests: Add unit tests for JarHell 2015-07-07 16:54:03 -07:00
Robert Muir 46c89f006d Allow use of bouncycastle 2015-07-07 17:43:35 -04:00
Robert Muir 27b8e59c24 remove temporary leniency 2015-07-07 17:35:16 -04:00
Jack Conradson 6dbf56fe99 Simplify CacheKey used for scripts
Replaced the CacheKey class with a static method that returns a String.
The class was overkill.

closes #12092
2015-07-07 14:20:03 -07:00
Areek Zillur 71a6d6d5e9 Merge branch 'master' of github.com:elasticsearch/elasticsearch 2015-07-07 17:00:04 -04:00
Tanguy Leroux 44efbf2770 Renaming FsStats to FsInfo 2015-07-07 22:50:15 +02:00
Areek Zillur 4849e76275 Currently when an engine is failed, it is marked as corrupted regardless of
the failure type. This change marks the engine as corrupted only when the failure
is caused by an actual index corrruption. When an engine is failed for other
reasons, the engine is only closed without removing the shard state.

closes #11788
2015-07-07 16:48:18 -04:00
Tanguy Leroux fbcf4dbbf7 FS Stats: remove sigar specific stats from APIs:
- fs.*.disk_reads
- fs.*.disk_writes
- fs.*.disk_io_op
- fs.*.disk_read_size_in_bytes
- fs.*.disk_write_size_in_bytes
- fs.*.disk_io_size_in_bytes
- fs.*.disk_queue
- fs.*.disk_service_time
2015-07-07 22:16:39 +02:00
Robert Muir 7dbc5c7ab9 Merge pull request #12093 from rmuir/no_fucking_way
Give a better exception when a jar contains same classfile twice.
2015-07-07 16:13:12 -04:00
Tanguy Leroux 30892c4129 Remove network stats & info 2015-07-07 21:16:42 +02:00
Robert Muir 1994dbde15 Give a better exception when a jar contains same classfile twice.
And ignore the known issue with xmlbeans for now... though it may
cause us issues ultimately: https://issues.apache.org/jira/browse/XMLBEANS-499
2015-07-07 13:26:54 -04:00
David Pilato d57de59158 Simplify Plugin Manager for official plugins
Plugin Manager can now use another simplified form when a user wants to install an official plugin hosted at elasticsearch download service.

The form we use is:

```sh
bin/plugin install pluginname
```

As plugins share now the same version as elasticsearch, we can automatically guess what is the exact current version of the plugin manager script.

Also, download service will now use `/org.elasticsearch.plugins/pluginName/pluginName-version.zip` URL path to download a plugin.

If the older form is provided (`user/plugin/version` or `user/plugin`), we will still use:

 * elasticsearch download service at `/user/plugin/plugin-version.zip`
 * maven central with groupIp=user, artifactId=plugin and version=version
 * github with user=user, repoName=plugin and tag=version
 * github with user=user, repoName=plugin and branch=master if no version is set

Note that community plugin providers can use other download services by using `--url` option.

If you try to use the new form with a non core elasticsearch plugin, the plugin manager will reject
it and will give you all known core plugins.

```
Usage:
    -u, --url     [plugin location]   : Set exact URL to download the plugin from
    -i, --install [plugin name]       : Downloads and installs listed plugins [*]
    -t, --timeout [duration]          : Timeout setting: 30s, 1m, 1h... (infinite by default)
    -r, --remove  [plugin name]       : Removes listed plugins
    -l, --list                        : List installed plugins
    -v, --verbose                     : Prints verbose messages
    -s, --silent                      : Run in silent mode
    -h, --help                        : Prints this help message

 [*] Plugin name could be:
     elasticsearch-plugin-name    for Elasticsearch 2.0 Core plugin (download from download.elastic.co)
     elasticsearch/plugin/version for elasticsearch commercial plugins (download from download.elastic.co)
     groupId/artifactId/version   for community plugins (download from maven central or oss sonatype)
     username/repository          for site plugins (download from github master)

Elasticsearch Core plugins:
 - elasticsearch-analysis-icu
 - elasticsearch-analysis-kuromoji
 - elasticsearch-analysis-phonetic
 - elasticsearch-analysis-smartcn
 - elasticsearch-analysis-stempel
 - elasticsearch-cloud-aws
 - elasticsearch-cloud-azure
 - elasticsearch-cloud-gce
 - elasticsearch-delete-by-query
 - elasticsearch-lang-javascript
 - elasticsearch-lang-python
```
2015-07-07 18:27:40 +02:00
Ryan Ernst 4aecd37e57 Mappings: Remove AbstractFieldMapper
AbstractFieldMapper is the only direct base class of FieldMapper.
This change moves all AbstractFieldMapper functionality into
FieldMapper, since there is no need for 2 levels of abstraction.
2015-07-07 08:43:38 -07:00
Jason Tedor c563d68872 Failure during the fetch phase of scan should invoke the failed fetch phase handler.
This commit fixes an issue where during a failure in the fetch phase of a scan the wrong failure handler was invoked.

Closes #12086
2015-07-07 11:35:03 -04:00
Ryan Ernst 2cc0382cf0 Merge pull request #12068 from rjernst/fix/mapper-names-conflict
Mappings: Enforce field names do not contain dot
2015-07-07 08:34:04 -07:00
Martijn van Groningen f7ac2a7e1c test: check node count on all nodes before checking if cluster state is the same on all nodes 2015-07-07 16:46:24 +02:00
Jason Tedor b2d8a1fd1b Count scans in search stats and add metrics for scrolls
Each scroll on a scan causes a query to be executed. This commit adds support for these indirect queries to count against the search stats.
Additionally, this commit adds three new search stats: scroll_count, scroll_time_in_millis, and scroll_current. scroll_count tracks the
number of completed scrolls. scroll_time_in_millis tracks the total time that scrolls were held open. scroll_current tracks the number of
scrolls currently open.

Closes #9109
2015-07-07 10:20:45 -04:00
Alexander Reelsen 21b4f9b6f8 Plugins: Ensure logging configuration is loaded in plugin manager
This prevents log4j warnings printed out, when installing a plugin
due to the JarHell class using an ESLogger.

Closes #12064
2015-07-07 14:56:06 +02:00
Alex Ksikes 5e023848de Properly fix the default regex flag to ALL for RegexpQueryBuilder and Parser
Relates to #11896
Closes #12067
2015-07-07 14:26:46 +02:00
Boaz Leskes 67318ce7ba Tests: Faster recovery from simulated disurptions
In testing infra, one can simulate node GCs, network issues and other problems by adding a disruption to the test cluster. Those disruption are automatically removed after the test is done. At the moment each disruption indicates how long it will take the cluster to heal once the disruption is removed and the test cluster waits for this amount of time. However, more often than not this is an upper bound, causing a much longer wait than needed. Instead we should push the responsibility of healing to the disruption it self, where we can be smarter about what we wait for.

Closes #12071
2015-07-07 14:16:31 +02:00
Alex Ksikes 4f9855261a Revert "fix RegexpQueryBuilder#maxDeterminizedStates"
This reverts commit b7e26fae3f.
2015-07-07 14:06:07 +02:00
Alex Ksikes b7e26fae3f fix RegexpQueryBuilder#maxDeterminizedStates
Value was improperly set to `true`.

Relates to #11896
2015-07-07 13:44:05 +02:00
David Pilato af1dc6d809 [test] awaitBusy: add a ceiling to max sleep time
When using `awaitBusy`, sometimes, you might not want to double time between two runs in an infinitive manner.

For example, let's say it will probably take 30 seconds to run a test.
When doubling all the time, you will most likely wait for a bigger time than needed:

|iteration|ms           |s            |duration (ms)|duration (s)|
|-----------|-------------|-----------|-----------|-----------|
|1|1|0,001|1|0,001|
|2|2|0,002|3|0,003|
|3|4|0,004|7|0,007|
|4|8|0,008|15|0,015|
|5|16|0,016|31|0,031|
|6|32|0,032|63|0,063|
|7|64|0,064|127|0,127|
|8|128|0,128|255|0,255|
|9|256|0,256|511|0,511|
|10|512|0,512|1023|1,023|
|11|1024|1,024|2047|2,047|
|12|2048|2,048|4095|4,095|
|13|4096|4,096|8191|8,191|
|14|8192|8,192|16383|16,383|
|15|16384|16,384|32767|32,767|
|16|32768|32,768|65535|65,535|
|17|65536|65,536|131071|131,071|
|18|131072|131,072|262143|262,143|
|19|262144|262,144|524287|524,287|
|20|524288|524,288|1048575|1048,575|
|21|1048576|1048,576|2097151|2097,151|

For example here, if the task is successful after 35 seconds, we will most likely have to wait for 32s more before the Predicate is run again.

With this patch, the maximum sleep time is now set to 1 second.
2015-07-07 12:04:16 +02:00
Colin Goodheart-Smithe 1d7fc6b4f2 Aggregations: Pipeline Aggregation to filter buckets based on a script
This pipeline aggregation runs a script on each bucket in the parent aggregation to determine whether the bucket is kept in the final aggregation tree. If the script returns true the bucket is retained, if it returns false the bucket is dropped
2015-07-07 09:51:16 +01:00
Alexander Reelsen b612cab96a Dates: More strict parsing of ISO dates
If you are using the default date or the named identifiers of dates,
the current implementation was allowed to read a year with only one
digit. In order to make this more strict, this fixes a year to be at
least 4 digits. Same applies for month, day, hour, minute, seconds.

Also the new default is `strictDateOptionalTime` for indices created
with Elasticsearch 2.0 or newer.

In addition a couple of not exposed date formats have been exposed, as they
have been mentioned in the documentation.

Closes #6158
2015-07-07 09:34:37 +02:00
Christoph Büscher 35ddc749b1 Merge pull request #12060 from cbuescher/fix/9821
Fix: Use correct OpType on Failure in BulkItemResponse
2015-07-07 09:29:29 +02:00
Robert Muir d732c0d19f Add symlink permissions test 2015-07-07 02:38:11 -04:00
Simon Willnauer 9e196c3a0b [TEST] Wait for test thread to join before 2015-07-07 07:24:50 +02:00
Ryan Ernst aed1f68e49 Mappings: Enforce field names do not contain dot
Field names containing dots can cause problems. For example, @jpountz
made this recreation which cause no error, but can result in a
serialization exception if the type already exists:
https://gist.github.com/jpountz/8c66817e00a322b81f85

But this is not just a potential conflict. It also has larger problems,
since only the leaf mapper is created. The intermediate "foo" object
field would not exist if only "foo.bar" was in the mappings.

This change forbids the use of dots in field names. It also
fixes an issue with passing through the update_all_types setting,
which was always set to true whenever a type already existed (!).

I do not think we should worry about backwards compatibility here. This
should be a hard break (and added to the migration plugin).
2015-07-06 18:22:06 -07:00
Simon Willnauer 3906ff950c Don't use forbidden API in test 2015-07-06 22:56:32 +02:00
Simon Willnauer 04c5dab3d9 Add basic recovery prioritization to GatewayAllocator
This commit adds logic to prefer shards with higher priority
or from newer indicse to be allocated first if they are unallocated post API.

This commit allows users to set `index.priority` to a non-negative integer to
prioritize index recovery for certain indices. This setting is dynamically updateable
and defaults to `0`. If two indices have the same priority this change takes the creation
date into account to prioritize shards from newer indices which is important in the time-based
indices usecase.

Closes #11787
2015-07-06 22:51:34 +02:00
Robert Muir 546e99f072 Merge pull request #12061 from rmuir/plugin-integration-tests
Add integration test harness for plugins
2015-07-06 16:03:56 -04:00
David Pilato 1a3eb4d83a Upsert does not use ttl value
When running an upsert which defines a ttl value, the ttl value is set to null and ignored.

Related to #3256#issuecomment-64963409
2015-07-06 21:57:41 +02:00
Christoph Büscher 201dc87aae Fix: Use correct OpType on Failure in BulkItemResponse
When a bulk request fails on a Delete or Update request, the BulkItemResponse
reports incorrect "index" operation in the response. This PR fixes this
for the case of closed indices as reported in #9821 but also for
other failures and adds tests for the two cases covered.

Closes #9821
2015-07-06 20:20:54 +02:00
Igor Motov 177180ac94 Fix snapshot-related REST integration tests 2015-07-06 14:16:28 -04:00
Robert Muir 7595104ec3 Factor integration tests logic to separate build file 2015-07-06 13:59:16 -04:00
Martijn van Groningen fb6daebe34 scroll: Append the shard top docs in such a way that ArrayIndexOutOfBoundsException is impossible to occur.
also added AtomicArray#setOnce() method to be sure that we fail if an shard response has already been set
2015-07-06 19:33:28 +02:00
Ryan Ernst ff0f480938 Merge pull request #12014 from rjernst/remove/mapper-doc-values
Completely move doc values and fielddata settings to field types
2015-07-06 10:21:09 -07:00
Ryan Ernst 184b1cbd68 Fix fake field mapper to not pass null for defaultfieldtype 2015-07-06 10:20:19 -07:00
Ryan Ernst 91e3cdb187 Addressed PR comments 2015-07-06 10:02:53 -07:00
Alex Ksikes afe9c52f07 XMoreLikeThis: fix potentially unpositioned enum 2015-07-06 18:08:06 +02:00
Robert Muir d7dab0df72 fix test to allow integration tests 2015-07-06 11:20:30 -04:00
Robert Muir 3f4b8df00d Merge pull request #12026 from rmuir/integ_tests
add integration test harness to maven build
2015-07-06 10:16:54 -04:00
Alex Ksikes 8ef1708395 [TEST] fix testMoreLikeThisArtificialDocs
Ensure that the indexed doc hasn't all its fields empty and that the
artificial doc requested is always routed to the shard having that doc.
2015-07-06 15:27:38 +02:00
Colin Goodheart-Smithe b93512e000 [TEST] NettyTransportMultiPortTests now checks selected random ports are free before using them 2015-07-06 12:04:22 +01:00
markharwood f95981b977 Search enhancement: adds a special case for the “_index” field which allows queries to test the name of the index. Adding this capability means we can deprecate the specialist indices query.
IndexFieldMapper is changed to make the term query factories produce match_all or match_none queries based on tests on the index name.

Closes #3316
2015-07-06 10:55:28 +01:00
javanna d085088793 BoolQueryBuilder#hasClauses to take into account filterClauses too 2015-07-06 11:02:09 +02:00
Boaz Leskes d74eac94f2 ZenDiscovery: #11960 failed to remove eager reroute from node join
It is removed now :)

Closes #12019
2015-07-03 22:06:42 +02:00
Robert Muir 9d233aeaf0 use external test cluster for integration tests 2015-07-03 12:20:35 -04:00
Tanguy Leroux 9495816cb7 Remove sigar completely 2015-07-03 15:49:17 +02:00
Simon Willnauer ac32f3d310 Don't special-case on ElasticsearchWrapperException in toXContent
the specialization can cause stack overflows if an exception is a
ElasticsearchWrapperException as well as a ElasticsearchException.
This commit just relies on the unwrap logic now to find the cause and only
renders if we the rendering exception is the cause otherwise forwards
to the generic exception rendering.

Closes #11994
2015-07-03 13:51:11 +02:00
Ryan Ernst e83ae64ea9 Mappings: Completely move doc values and fielddata settings to field types
While MappedFieldType contains settings for doc values and fielddata,
AbstractFieldMapper had special logic in its constructor that
required setting these on the field type from there. This change
removes those settings from the AbstractFieldMapper constructor.
As a result, defaultDocValues(), defaultFieldType() and
defaultFieldDataType() are no longer needed.
2015-07-03 03:10:20 -07:00
Martijn van Groningen 13602a591f Removed forcedDateParser from hashCode() and equals() as it holds no value when it comes to equality. 2015-07-03 10:04:22 +02:00
Simon Willnauer 20d0b4f446 Promote headers to first class citizens on exceptions
This commit merges the pre-existing special exception that
allowed to associate headers with exceptions and the elasticsaerch
base class `ElasticsearchException` This allows for more generic use
of exceptions where plugins can associate meta-data with any elasticsearch
base exception to control behavior etc.

This also addds a generic SecurityException to allow plugins to pass on
information based on the RestStatus.
2015-07-03 08:58:44 +02:00
Robert Muir 80871bae2b Add simple integ testing infra 2015-07-03 02:12:01 -04:00
Boaz Leskes 3084bed194 Test: force local transport in ZenDiscoveryTests.testHandleNodeJoin_incompatibleMinVersion
Otherwise we ignore the join request due to a transport address mismatch instead of failing it.
2015-07-02 22:20:12 +02:00
Robert Muir 3171f52461 Merge pull request #12001 from rmuir/no_dont_do_that
Don't add CWD to classpath when ES_CLASSPATH isn't set.
2015-07-02 14:53:24 -04:00
Martijn van Groningen a4b99e6291 zen: Don't join master nodes or accept join requests of old and too new nodes.
If the version of a node is lower than the minimum supported version or higher than the maximum supported version, a node shouldn't be allowed to join and nodes should join that elected master node

Closes #11924
2015-07-02 20:51:21 +02:00
Ryan Ernst 0f62950856 fix compile error after bad merge 2015-07-02 10:53:30 -07:00
Ryan Ernst e12bb4a54a Merge pull request #11977 from rjernst/pr/mapper-names
Move short name access out of field type
2015-07-02 10:38:30 -07:00
Robert Muir c598c4cc97 blind stab at windows 2015-07-02 13:27:39 -04:00
Robert Muir 68be95ca1a Don't add CWD to classpath when ES_CLASSPATH isn't set. 2015-07-02 13:16:20 -04:00
javanna 90f32d4d32 Internal: make sure ParseField is always used in combination with parse flags
Removed ParseField#match variant that accepts the field name only, without parse flags. Such a method is harmful as it defaults to empty parse flags, meaning that no deprecation exceptions will be thrown in strict mode, which defeats the purpose of using ParseField. Unfortunately such a method was used in a lot of places were the parse flags weren't easily accessible (outside of query parsing), and in a lot of other places just by mistake.

Parse flags have been introduced now as part of SearchContext and mappers where needed. There are a few places (e.g. java api requests) where it is not possible to retrieve them as they depend on the index settings, in that case we explicitly pass in EMPTY_FLAGS for now, but this has to be seen as an exception.

Closes #11859
2015-07-02 14:30:01 +02:00
Simon Willnauer 753376dba7 [TEST] Remove timeout - it's not needed in this test 2015-07-02 13:11:00 +02:00
Colin Goodheart-Smithe 11e851eb0c Change TODO comments for removing old Script API to 3.0 2015-07-02 11:20:17 +01:00
Adrien Grand 6765635067 Merge pull request #11988 from jpountz/fix/zlib_skip_underlying
Transport: Do not make the buffer skip while a stream is open.
2015-07-02 10:36:11 +02:00
Britta Weber 6bf543dca8 [TEST] mute IndicesStoreIntegrationTests.indexCleanup 2015-07-02 10:24:06 +02:00
Adrien Grand 7482e1362f Transport: Do not make the buffer skip while a stream is open.
This commit changes MessageChannelHandler to not skip the underlying
ChannelBuffer while a StreamInput is open on top of it. In case eg. compression
is enabled, this prevents failures due to the fact that the decompressed
stream input expects a certain structure that it can't verify if the position
of the underlying buffer is changed.
2015-07-02 09:30:28 +02:00
Robert Muir ce900b4d7a Merge pull request #11982 from rmuir/grr_bad_defaults
really ban exitVM with security policy
2015-07-01 17:58:07 -04:00
uboness 6021bd8cca Parameterized exception messages
Added dynamic arguments to `ElasticsearchException`, `ElasticsearchParseException` and `ElasticsearchTimeoutException`.

This helps keeping the exception messages clean and readable and promotes consistency around wrapping dynamic args with `[` and `]`.

This is just the start, we need to propagate this to all exceptions deriving from `ElasticsearchException`. Also, work started on standardizing on lower case logging & exception messages. We need to be consistent here...

 - Uses the same `LoggerMessageFormat` as used by our logging infrastructure.
2015-07-01 23:08:38 +02:00
Robert Muir 5aa5c98826 really ban exitVM with security policy 2015-07-01 15:40:46 -04:00
Martijn van Groningen 53874bf5a6 aliases: Parse aliases at search time and never cache parsed alias filters
The work around for resolving `now` doesn't need to be used for aliases, becuase alias filters are parsed at search time. However it can't be removed, because the percolator relies on it.

Parent/child can be specified again in alias filters, this now works again because alias filters are parsed at search time. Parent/child will also use the late query parse work around, to make sure to do the final preparations when the search context is around. This allows the aliases api to validate the parent/child queries without failing because there is no search context.

Closes #10485
2015-07-01 21:20:54 +02:00
Robert Muir 6b4c51f442 Merge pull request #11979 from rmuir/nosystem
Don't jarhell check system jars
2015-07-01 14:53:20 -04:00
Robert Muir 976ace21e0 let IDEs have jar hell in tests 2015-07-01 14:29:57 -04:00
Robert Muir 41ac191cac Add java.home to debug logging 2015-07-01 14:06:53 -04:00
Robert Muir 30f267f758 Don't jarhell check system jars 2015-07-01 13:26:00 -04:00
Martijn van Groningen 5a2a2861ab Merge pull request #11954 from martijnvg/percolator/stats/typo
Change `percolator.getTime` -> `percolator.time`
2015-07-01 19:08:42 +02:00
Ryan Ernst 41dc1b0120 Mappings: Move short name access out of field type
Eventually, the field type should not need any names, because there
will be only one name which leads to finding it (the full name, which is
also the index name). However, the short or "simple" name (using java
terminology for class names) is needed just in a couple places, for
serialization.

This change moves the simple name out of MappedFieldType.Names, into
Mapper, and makes Mapper and FieldMapper abstract classes.
2015-07-01 09:46:52 -07:00
Ryan Ernst ac8896c10e Merge pull request #11962 from rjernst/pr/root-mappers-rename
Rename "root" mappers to "metadata" mappers
2015-07-01 09:13:14 -07:00
Robert Muir b55a3f32b8 Add debug logging in case classpath is crazy 2015-07-01 10:04:36 -04:00
Simon Willnauer 93acb98bd6 Carry on rest status if exceptions are not serializable
Today we loose the RestStatus code for non-serializable exceptions.
This can be tricky if they are supposed to signal certain situations
like authentication errors etc. This commit adds support for carrying on
the exceptions in the NotSerializableExceptoinWrapper
2015-07-01 13:57:54 +02:00
Colin Goodheart-Smithe e366d0380d Aggregations: Adds other bucket to filters aggregation
The filters aggregation now has an option to add an 'other' bucket which will, when turned on, contain all documents which do not match any of the defined filters. There is also an option to change the name of the 'other' bucket from the default of '_other_'

Closes #11289
2015-07-01 10:44:04 +01:00
Colin Goodheart-Smithe ab80130c10 Aggregations: Makes SKIP Gap Policy work correctly for Bucket Script aggregation
This change means that when the skip gap policy is used, the bucket script aggregation will skip executing the script on a bucket if any of the required bucket_paths are missing for the bucket. No aggregation will be added to the bucket, and the aggregation will move to the next bucket.
2015-07-01 10:18:18 +01:00
Tanguy Leroux 737440b580 Postrm script should not fail
This commit changes the postrm script so that it prints error messages instead of failing & exiting when the deletion of a directory failed while removing a RPM/DEB package.

Closes #11373
2015-07-01 11:03:38 +02:00
Colin Goodheart-Smithe d73ce3492a Aggregations: Makes ValueFormat and ValueFormatter never null
This allows a lot of null checks to be removed where we were always falling back to the ValueFormat.RAW anyway. Now the format is set to ValueFormat.RAW when no alternative is suitable.

Closes #10594
2015-07-01 08:57:03 +01:00
szroland 68d658a30f Recovery: Fix wrong reused file bytes in Recovery API reports
Simple snapshot.restore test case that also exposes the bug mentioned in #11876

Fix #11876
Closes #11965
2015-07-01 09:08:53 +02:00
Martijn van Groningen ef9d70b9b3 field stats: added index constraints
Field stats index constraints allows to omit all field stats for indices that don't match with the constraint. An index
constraint can exclude indices' field stats based on the `min_value` and `max_value` statistic. This option is only
useful if the `level` option is set to `indices`.

For example index constraints can be useful to find out the min and max value of a particular property of your data in
a time based scenario. The following request only returns field stats for the `answer_count` property for indices
holding questions created in the year 2014:

curl -XPOST 'http://localhost:9200/_field_stats?level=indices' -d '{
   "fields" : ["answer_count"] <1>
   "index_constraints" : { <2>
      "creation_date" : { <3>
         "min_value" : { <4>
            "gte" : "2014-01-01T00:00:00.000Z",
         },
         "max_value" : {
            "lt" : "2015-01-01T00:00:00.000Z"
         }
      }
   }
}'

Closes #11187
2015-07-01 08:47:03 +02:00
Boaz Leskes 3cf78669f2 Mute testInstallPluginWithElasticsearchDownloadService & testInstallPluginWithMavenCentral
Example failure: http://build-us-00.elastic.co/job/es_core_master_oracle_6/519/testReport/
2015-07-01 08:34:47 +02:00
Robert Muir 5294b1ae40 Merge pull request #11963 from rmuir/prevent_hell
detect jar hell before installing a plugin
2015-07-01 00:00:12 -04:00
Robert Muir f312a4e30d cleanup uwe's unzip code 2015-06-30 23:59:45 -04:00
Igor Motov 2a9f26dfdc Improve repository verification failure message
Closes #11922
2015-06-30 20:08:15 -04:00
Simon Willnauer 04631d6948 Carry on headers if unknow exception is a WithRestHeadersException 2015-06-30 23:02:45 +02:00
Simon Willnauer c6837f56d9 Serialize AlreadyClosedException first before it's superclass 2015-06-30 23:02:45 +02:00
Robert Muir a839644826 detect jar hell before installing a plugin 2015-06-30 16:42:01 -04:00
Ryan Ernst 4b3dba527b Mappings: Rename "root" mappers to "metadata" mappers
"Root" is a very confusing term for meta field mappers. This change
renames "RootMapper" to "MetadataFieldMapper" and simplifies
how metadata mappers are setup.

It also requires that metadata mappers are now a FieldMapper
(MetadataFieldMapper extends from AbstractFieldMapper). The only
use of a root mapper that wasn't a field mapper was the theoretical
"external" root mapper (just a test mapper). But it doesn't make
sense to not have an actual field, and this falls inline with
the hopefully eventual collapsing of AbstractFieldMapper/FieldMapper/Mapper.
2015-06-30 13:31:17 -07:00
Boaz Leskes 182c59f5b4 Discovery: reroute after node join is processed
- shard listing actions underpinning shard allocation do not have access to that new node yet (causing errors during shard allocation see #11923
 - the very first cluster state published to a node already has shard assignments to it. This surfaced other issues we are working to fix separately

 This commit changes the reroute to be done post processing the initial join cluster state to side step these issues while we work on a longer term solution.

Closes #11960
2015-06-30 22:09:29 +02:00
Simon Willnauer dab2a76eae Cut over to writeable for TransportAddress
This will allow us to move away from reflection hacks to serialize
transport exceptions.
2015-06-30 20:59:47 +02:00
Martijn van Groningen c6ae6fc6d9 percolator: `getTime` -> `time` 2015-06-30 18:44:58 +02:00
Colin Goodheart-Smithe d9ab3cba77 Search Templates: Adds API endpoint to render search templates as a response
Closes #6821
2015-06-30 16:57:23 +01:00
Simon Willnauer bbaf4710cb FileName is optional in StacktraceElement 2015-06-30 15:51:48 +02:00
Simon Willnauer 0ac8c1bc55 Carry on stacktrace and suppressed exceptions if exception is not serializable 2015-06-30 14:51:44 +02:00
Simon Willnauer 6ee9a3d5f2 add more essential exceptions 2015-06-30 14:51:44 +02:00
Simon Willnauer 971ac0475e add IOOB to supported exceptions 2015-06-30 14:51:44 +02:00
Simon Willnauer 0148c462e0 remove leftover ObjectOutputStream references 2015-06-30 14:51:44 +02:00
Simon Willnauer 325cc6bc61 serialize as not-serializable if exception is not registered 2015-06-30 14:51:44 +02:00
Simon Willnauer 4e2621ca21 Cleanup non-serializable code 2015-06-30 14:51:44 +02:00
Simon Willnauer c4f15c2c91 add dedicated unittests for exceptions that have a writeTo method (custom serialization) 2015-06-30 14:51:44 +02:00
Simon Willnauer 62d29ff6a4 simplify exception reading / writing - now use a named exception for unknonw exceptions 2015-06-30 14:51:43 +02:00
Simon Willnauer fa016a2b09 Don't implement Serializable in core classes
We can't add this to forbidden API since literally everything implements this.
Any API that would for instance use a String would not compile anymore.
2015-06-30 14:51:43 +02:00
Simon Willnauer e7eb9cf4de Ban java serialization
We had several problems with Java Serializatin in the past. At some point
in the Java 1.7.x series JDKs where not compatible anymore when java
serialization (ObjectStream) was used to exchange objects. In elasticsearch
we used this to serialize exceptions across the wire which caused several problems
with incompatible JDKs. While causing lot of trouble this essentially prevented
users from moving forward and upgrade their JVMs. To prevent these kind of issues
this commit removes the dependency on java serialization entirely and bans the
usage of ObjectOutputStream and ObjectInputStream entirely.

Yet, we can't fully serialize all exception anymore such that this commit
is best effort and adds hand written serialization to all elasticsearch exceptions
as well to a selected set of JDK and Lucene exceptions. (see StreamOutput#writeThrowable /
StreamInput.readThrowable). Stacktraces should be preserved for all exceptions while
several names might be replaced with ElasticsearchException if there is no mapping for
the given exception.
2015-06-30 14:51:43 +02:00
Alexander Reelsen f26672c184 Release: Build two RPMS, signed and unsigned
In order to support older RPM based distributions like CentOS5,
we should have one RPM available, which is not signed.

This commit creates an unsigned RPM first, then moves it over to
target/releases during the build, then builds a signed RPM.

The unsigned one is uploaded via S3, where as the signed one is
used for the repositories.

In addition, you can now build an RPM without having to specify
any gpg credentials due to offloading this into a maven profile
that is only activated when specifying `rpm.sign` property.

Closes #11587
2015-06-30 14:22:20 +02:00
Simon Willnauer d1a29a63d2 Decode URL.getPath before resolving a real file
URL with spaces are URL-Encoded and need to be decoded before
they can be used to open files etc.
2015-06-30 13:33:40 +02:00
Boaz Leskes 4102e263df AwaitFix testTracerLog 2015-06-30 13:25:27 +02:00
Boaz Leskes 4649650ab2 Merge pull request #11928 from szroland/#11897
Create Snapshot: remove _create from POST path to match PUT

Closes #11928
2015-06-30 12:16:22 +02:00
Martijn van Groningen 1df8e56a83 Merge pull request #11929 from martijnvg/percolator/drop__own_qpc_cache
The query parse context should be fetched from the IndexQueryParseService
2015-06-30 11:04:32 +02:00
Adrien Grand ed561cd434 Merge pull request #11768 from jpountz/fix/stream_eos
Make sure messages are fully read even in case of EOS markers.
2015-06-30 09:00:59 +02:00
Alexander Reelsen 185dbac8fe Testing: Fix running of rest tests against external cluster
The TransportClient specified in the ExternalCluster was missing a
`path.home` setting and thus would not start.
2015-06-30 08:17:45 +02:00
Robert Muir f20e23e6fb jar hell check should fail, if jars require higher java version 2015-06-30 01:49:27 -04:00
Robert Muir 0b2070e350 Merge pull request #11932 from rmuir/jarhell
Fail startup (and tests) on jar hell
2015-06-30 00:58:44 -04:00
Robert Muir 4059d7c18d tests: fix test bugs 2015-06-30 00:16:39 -04:00
Robert Muir b58eb35dfa detect jar hell in tests better and fix more stuff 2015-06-30 00:12:40 -04:00
Robert Muir f8fbf1fa0c Fail startup (and tests) on jar hell 2015-06-29 23:13:45 -04:00
Martijn van Groningen 07090b8ee8 percolator: The query parse context should be fetched from the IndexQueryParseService
instead of maintaining a thread local cache in the PercolatorQueriesRegistry.
Before PercolatorQueriesRegistry had its own cache, because all the queries had to forcefully opt out of caching. Nowadays in master small segments are never cached by the query cache, so the reason for the dedicated cache is no longer valid.
2015-06-30 00:32:09 +02:00
Robert Muir 0bb65f710e tests: simple tests for things we should kick out 2015-06-29 17:21:25 -04:00
szroland e2d76aaa20 remove _create from POST to match PUT path 2015-06-29 21:55:52 +02:00
Simon Willnauer d274e585e6 Add jsr-166e license, notice and sha1 hash 2015-06-29 21:17:22 +02:00
Simon Willnauer fcbd0dddcd Don't pull the engine for access to the QueryCachingPolicy 2015-06-29 21:07:47 +02:00
Robert Muir 0ae638fbb2 Merge pull request #11918 from rmuir/plugin_manager_classpath
Load plugins into classpath in bootstrap
2015-06-29 14:45:38 -04:00
Robert Muir 3f95275f23 Add back check with a note, so anyone reviewing isn't confused. 2015-06-29 13:20:53 -04:00
Robert Muir b93af3c27c Remove try/finally, its not necessary here 2015-06-29 12:35:59 -04:00
Robert Muir f9598030fd Load plugins into classpath in bootstrap 2015-06-29 12:11:04 -04:00
Martijn van Groningen c9f21a7ecc Merge pull request #11909 from martijnvg/nested_query_min_score_mode
Added `min` score mode.
2015-06-29 17:48:48 +02:00
Robert Muir 55c33b29f6 Merge pull request #11898 from rmuir/lockdown
steps to remove dangerous security permissions
2015-06-29 11:13:16 -04:00
Simon Willnauer 2dfd6df369 Merge pull request #11911 from s1monw/no_utf_uuid
Don't convert possibly corrupted bytes to UTF-8
2015-06-29 15:00:40 +02:00
Colin Goodheart-Smithe b5e958b2e0 Aggregations: Fix bucket_script builder method name 2015-06-29 12:53:38 +01:00
Adrien Grand ccaf846f1e Store filter cache statistics at the shard level instead of index.
Today we keep track of how often filters are used at the index level in order
to decide whether they should be cached or not. This is an issue if you have
several shards of the same index on the same node as it will multiply statistics
by the number of shards that you have for this index on the node, which defeats
the purpose of waiting for a filter to be reused before caching them.
2015-06-29 13:34:31 +02:00
Simon Willnauer 1a44afed05 Don't convert possibly corrupted bytes to UTF-8
If the translog UUID is corrupted we should not convert it
to UTF-8 since it might be invalid. Instead we should compare
the UTF-8 byte representation directly.
2015-06-29 13:16:52 +02:00
Adrien Grand 21ffd931ec Fix missing renamings.
Relates to #11893
2015-06-29 12:47:10 +02:00
Martijn van Groningen 47a43e4063 nested query: Added `min` score mode.
This score mode was added with the Lucene 5.2 release, but the `nested` query parser hasn't been changed to use it.
2015-06-29 12:26:30 +02:00