Commit Graph

693 Commits

Author SHA1 Message Date
Ryan Ernst 8a587ead89 Mappings: Validate parsed document does not have trailing garbage that is invalid json
See #2315
2015-08-10 21:57:14 -07:00
Ryan Ernst 6fea6c542c Test: Fix netty integ test to ensure local mode is not used.
closes #12788
2015-08-10 20:57:55 -07:00
Igor Motov 3354a816fc Changes in unassigned info and version might not be transferred as part of cluster state diffs
The equalTo logic of ShardRouting doesn't take version and unassignedInfo into the account when compares shard routings.  Since cluster state diff relies on equal to detect the changes that needs to be sent to other cluster, this omission might lead to changes not being properly propagated to other nodes in the cluster.

Closes #12387
2015-08-10 19:58:36 -04:00
Ryan Ernst 1b3cd504a5 test: quiet netty integ test 2015-08-10 16:57:02 -07:00
Ryan Ernst 40f119d85a This method on settings loaded a class, based on a setting value, using
the default classloader. It had all kinds of leniency in how the
classname was found, and simply cannot work with plugins having isolated
classloaders.

This change removes that method. Some of the uses of it were for custom
extension points, like custom repository or discovery types. A lot were
just there to plugin mock implementations for tests. For the settings
that were legitimate, all now support plugins adding the given setting
via onModule. For those that were specific to tests for mocks, they now
use Classes.loadClass (a helper around Class.forName). This is a
temporary measure until (in a future PR) tests can change the
implementation via package private statics.

I also removed a number of unnecessary intermediate modules, added a
"jvm-example" plugin that can be filled in in the future as a smoke test
for breaking plugins, and gave some documentation to "spawn" modules
interface.

closes #12643
closes #12656
2015-08-10 14:04:45 -07:00
Christoph Büscher a269b96799 Merge pull request #12581 from cbuescher/fix/12531
Fix setting timezone on default DateTime formatter
2015-08-10 15:05:39 +02:00
Christoph Büscher cdb39cf5dc Aggregations: Fix setting timezone on default DateTime formatter
This PR prevents setting timezone on ValueFormatter.DateTime. Instead
the timezone information needed when printing buckets key-as-string
information is provided at constrution time of the ValueFormatter, making
sure we don't overwrite any constants. This, however, made it necessary to
be able to access the timezone information when resolving the format
in ValueSourseParser, so the `time_zone` parameter is now parsed alongside
the `format` parameter in ValueSourceParser rather than in DateHistogramParser.

Closes #12531
2015-08-10 15:04:46 +02:00
Simon Willnauer 9a7257c55a Remove lenient store type parsing
Today we accept store types like `nio_fs`, `nioFs`, `niofs` etc.
this commit removes the leniency and only accepts plain values without underscore.
Yet, this commit also has a BWC layer that upgrades existing indices to the new version.
Affected enums are:
 * `nio_fs`
 * `mmap_fs`
 * `simple_fs`
2015-08-10 13:39:08 +02:00
Alexander Reelsen e24b57e613 PluginManager: Do not try other URs if specific URL was configured
If an URL specified with --url on the command line cannot be reached,
the plugin manager tries URLs at download.elastic.co automatically.

This can lead to download wrong plugins and also exposes potentially
the name of an internal plugin to the download service.

This fix ensures, that the plugin manager simply aborts, if the specified
URL cannot be downloaded.
2015-08-10 13:09:02 +02:00
Clinton Gormley 205262c737 Enable the license checker over distribution/* and plugins/*
Moved the license checker config into the parent pom, and overrede
the license dir/target-to-check in distributions/pom.

Disabled the license checker explicitly for projects which run integration
tests but have no licenses dir:

* core
* distribution
* qa
* plugins/delete-by-query
* plugins/mapper-size
* plugins/site-example

Closes #12752
Closes #12754
2015-08-09 18:18:59 +02:00
Robert Muir 368c41666c WE CANT EVEN GET BIN/ELASTICSEARCH WORKING!!!! 2015-08-09 08:09:44 -04:00
Jason Tedor f8415ae7ca Avoid use of forbidden method in test 2015-08-07 19:09:52 -04:00
Jason Tedor 23b348040e Add script compilation stats
This commit adds basic support to track the number of times scripts are
compiled and compiled scripts are evicted from the script cache. These
statistics are tracked at the node level.

Closes #12673
2015-08-07 18:25:55 -04:00
javanna 56c0b46f96 [TEST] remove duplicated tests from SearchQueryIT
After merging filters and queries, tests against indices filter and indices query became exactly the same. we can remove one copy then.
2015-08-07 16:04:51 +02:00
Simon Willnauer d0b2566345 Make RoutingNodes read-only by default
ClusterState has 3 different methods to access RoutingNodes:
 * #routingNodes() - mutable version
 * #getRoutingNodes() - delegates to #getReadOnlyRoutingNodes()
 * #getReadOnlyRoutingNodes() - it's docs say `NOTE, the routing nodes are mutable, use them just for read operations`

The latter also reuses the instance that it creates. This has several problems beside the obvious:
 * creating RoutingNodes is costly and should be done only if really needed ie. use cached version as much as possible
 * the common case is ReadOnly but all kinds of things are called
 * mutable version are only needed in one place and should only be used in the AllocationService
 * RoutingNodes can freeze it's ShardRoutings but doesn't
 * RoutingNodes should check if it's read-only or not

This commit fixed all the problems and special cases the mutable case such that all accesses via ClusterState#getRoutingNodes()
is read-only and RoutingNodes enforces this.
2015-08-07 09:48:42 +02:00
Adrien Grand 137bfaaeff Merge pull request #12707 from jpountz/enhancement/filters_function_score_needs_score
Only compute scores when necessary with FiltersFunctionScoreQuery.
2015-08-07 08:53:19 +02:00
Adrien Grand 2db1db0686 Tests: disable randomly-enabled custom data paths to make tests passing against an external cluster. 2015-08-06 22:28:40 +02:00
Adrien Grand 5cad559914 Only compute scores when necessary with FiltersFunctionScoreQuery.
This was just done with FunctionScoreQuery, but FiltersFunctionScoreQuery works
in a similar way.
2015-08-06 21:42:46 +02:00
Adrien Grand 5dd5936fad Move the `_size` mapper to a plugin.
This is one of our esoteric metadata mappers so I think we should distribute
it in a plugin rather than in elasticsearch core.

This introduces one limitation: the value of the `_size` parameter is not
retrievable for documents that are only in the transaction log.
2015-08-06 20:35:22 +02:00
Adrien Grand 00093a21dc Merge pull request #12294 from jpountz/fix/multi_match_boost
`multi_match` query applies boosts too many times.
2015-08-06 19:07:44 +02:00
Adrien Grand 8d5fff37ae `multi_match` query applies boosts too many times.
The `multi_match` query groups terms that have the same analyzer together and
then applies the boost of the first query in each group. This is not necessary
given that boosts for each term are already applied another way.
2015-08-06 19:07:12 +02:00
Adrien Grand b59918f29d Merge pull request #12695 from jpountz/enhancement/script_needs_scores
Allow scripts to expose whether they use the `_score`.
2015-08-06 18:47:18 +02:00
Robert Muir d61d775f19 Merge pull request #12696 from rmuir/better_jvm_check_error
Improve jvmcheck error failure
2015-08-06 12:35:55 -04:00
Robert Muir 1ca97934b5 Improve jvmcheck error failure
Versions can be tricky with linux vendors and such. To help debug any possible issues, we should output a better version.

Today:
```
[elasticsearch] java.lang.RuntimeException: Java version: 1.7.0_55 suffers from critical bug https://bugs.openjdk.java.net/browse/JDK-8024830 which can cause data corruption.
[elasticsearch] Please upgrade the JVM, see http://www.elastic.co/guide/en/elasticsearch/reference/current/_installation.html for current recommendations.
[elasticsearch] If you absolutely cannot upgrade, please add -XX:-UseSuperWord to the JAVA_OPTS environment variable.
[elasticsearch] Upgrading is preferred, this workaround will result in degraded performance.
[elasticsearch]     at org.elasticsearch.bootstrap.JVMCheck.check(JVMCheck.java:121)
[elasticsearch]     at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:270)
[elasticsearch]     at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:28)
```

With patch:
```
java.lang.RuntimeException: Java version: Oracle Corporation 1.7.0_40 [Java HotSpot(TM) 64-Bit Server VM 24.0-b56] suffers from critical bug https://bugs.openjdk.java.net/browse/JDK-8024830 which can cause data corruption.
Please upgrade the JVM, see http://www.elastic.co/guide/en/elasticsearch/reference/current/_installation.html for current recommendations.
If you absolutely cannot upgrade, please add -XX:-UseSuperWord to the JAVA_OPTS environment variable.
Upgrading is preferred, this workaround will result in degraded performance.
	at org.elasticsearch.bootstrap.JVMCheck.check(JVMCheck.java:121)
	at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:270)
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:28)
```
2015-08-06 12:22:04 -04:00
Lee Hinman 0a1c9de075 Merge remote-tracking branch 'dakrone/avoid-extra-reroutes-FP' 2015-08-06 10:07:41 -06:00
Adrien Grand b16e2d95af Allow scripts to expose whether they use the `_score`.
This commit adds a new API to allow scripts to say whether they need scores.
In practice, only the `expression` script engine makes use of it correctly,
other engines just return `true` since they can't predict whether they'll
need scores. This should make scripted aggregations and `function_query`
faster as we'll now be able to pass needsScores=false to Query.createWeight.
2015-08-06 17:25:51 +02:00
Jason Tedor 40cd460647 Merge pull request #12691 from jasontedor/fix/12683
Do not track named queries that are null
2015-08-06 09:33:29 -04:00
Jason Tedor e5786d2d82 Do not track named queries that are null
Adding a named query that is null can lead to a NullPointerException
when copying the named queries. This is due to an implementation detail
in QueryParseContent.copyNamedQueries. In particular, this method uses
com.google.common.collect.ImmutableMap.copyOf. A documented requirement
of ImmutableMap is that none of the entries have a null key nor null
value. Therefore, we should not add such queries to the namedQueries
map. This will not change any behavior since Map.get returns null if no
entry with the given key exists anyway.

Closes #12683
2015-08-06 09:31:51 -04:00
Adrien Grand 23a3db8bb6 Speed up the `function_score` query when scores are not needed.
This change improves the `function_score` query to not compute scores at all
when they are not needed, and to not compute scores on the underlying query
when the combine function is to replace the score with the scores of the
functions.
2015-08-06 15:04:48 +02:00
javanna 6f13171d50 [TEST] add NamedWriteableRegistry argument to AbstractSimpleTransportTests#build method 2015-08-06 12:56:28 +02:00
javanna 63d18d5e05 Make FilterInputStream abstract 2015-08-06 12:55:57 +02:00
javanna e1e9e1a6e1 Transport: allow to de-serialize arbitrary objects given their name
This commit makes it possible to serialize arbitrary objects by having them extend Writeable. When reading them though, we need to be able to identify which object we have to create, based on its name. This is useful for queries once we move to parsing on the coordinating node, as well as with aggregations and so on.

Introduced a new abstraction called NamedWriteable, which is supported by StreamOutput and StreamInput through writeNamedWriteable and readNamedWriteable methods. A new NamedWriteableRegistry is introduced also where named writeable prototypes need to be registered so that we are able to retrieve the proper instance of the writeable given its name and then de-serialize it calling readFrom against it.

Closes #12393
2015-08-06 12:25:27 +02:00
Simon Willnauer d0abffc9ac Add unittest for DiskThresholdDecider#getShardSize / #sizeOfRelocatingShards
This commit adds a basic unittest for the shard size routines and simplifies
some object creation.
2015-08-06 10:35:12 +02:00
Robert Muir 0ecd7b8ccf Merge pull request #12668 from rmuir/configure_script
Add path.scripts directory
2015-08-05 22:33:30 -04:00
Lee Hinman 437c43cd7f Avoid extra reroutes of delayed shards in RoutingService
In order to avoid extra reroutes, `RoutingService` should avoid
scheduling a reroute of any shards where the delay is negative. To make
sure that we don't encounter a race condition between the
GatewayAllocator thinking a shard is delayed and RoutingService thinking
it is not, the GatewayAllocator will update the RoutingService with the
last time it checked in order to use a consistent "view" of the delay.

Resolves #12456
Relates to #12515 and #12456
2015-08-05 14:59:31 -06:00
Nik Everett 69be7f77fc Prevent DirectCandidateGenerator to reuse an unclosed analyzer
When postFilter generates a token that is identical to the input term
DirectCandidateGenerator should not preFilter this token. If postFilter
and preFilter are the same analyzer instance it would fail with :
"TokenStream contract violation: close() call missing"

This is a forward port of @nomoa's #12670
2015-08-05 15:30:13 -04:00
Simon Willnauer 0b9729af5b Use explict flag if index should be created on engine creation
Today we try to detect if there is an index existing in the directory
and if not we create one. This can be tricky and errof prone since we
rely on the filesystem without taking the context into account when the
engine gets created. We know in all situations if the index should be created
so we can just use this infromation and rely on the lucene index writer to barf
if we hit a situations where we can't append to an index while we should.
2015-08-05 20:44:56 +02:00
Simon Willnauer abf763c1c5 Rethrow exception during recovery finalization even if source if not broken
Today we miss to throw / rethrow an recovery exception if it happens during
the finalization of phase 1 if the source files are not affected. Even worse
this can cause some dataloss if the reason for this exception is a failure of
deleting a corruption marker or similar pre-existing corruptions since we continue
with the recovery and mark the target shared as started which will in-turn open
an engine with an empty index.
2015-08-05 20:44:10 +02:00
Nik Everett 804f14c68e Core: Remove nasty hack in toString
This makes the output of EsThreadPoolExecutor#toString less pretty but
we no longer have funky hacky that rely on the specific format of the
toString produced by ThreadPoolExecutor which isn't part of its API and
could change with any JVM version and break the output.
2015-08-05 13:57:28 -04:00
Nik Everett ed7d84ca5f Core: Improve toString on EsThreadPoolExecutor
Improving the toString allows for nicer error reporting. Also cleaned up
the way that EsRejectedExecutionException notices that it was rejected
from a shutdown thread pool. I left javadocs about how its not 100% correct
but good enough for most uses.

The improved toString on EsThreadPoolExecutor mean every one of them needs
a name. In most cases the name to use is obvious. In tests I use the name
of the test method and in real thread pools I use the name of the thread
pool. In non-ThreadPool executors I use the thread's name.

Closes #9732
2015-08-05 13:57:28 -04:00
Nik Everett d46fdb1638 [Tests] Cleanup EsExecutorsTests
* names prefixed with test don't need @Test
* Javadoc describing what it tests
2015-08-05 13:56:58 -04:00
Alexander Reelsen 5a701367a7 PluginManager: Add Support for basic auth
In order to support the URL notation including a user/pass combination
(like http://user:pass@host/plugin.zip) the auth info needs to be added
manually.
2015-08-05 15:56:42 +02:00
Alexander Reelsen da31fbe702 Testing: Removing leftover @AwaitsFix annotation
Since the update to HDRHistogram 2.1.6 this test was
forgotten to get the annotation removed in #12554.
2015-08-05 13:47:28 +02:00
Robert Muir d7d25fe6b5 Add path.scripts directory
Today this is "unofficial" as conf/scripts, but some people
want to share scripts across different nodes and so on. Because
they cannot configure it, they are forced to use dirty hacks
like symbolic links, which isnt going to work: we aren't going
to recursively scan conf/ and add permissions to all link targets
underneath it, thats crazy.

I really hate adding yet another configuration knob here, but
users resorting to using symlinks are going to be frustrated,
and do things in a more insecure way.
2015-08-05 06:45:52 -04:00
Alexander Reelsen 88842f3319 PluginManager: Fix elastic.co download URLs, add snapshot ones
The URL to download the main elasticsearch plugins did not match
what the S3 wageon is supposed to write to.

In addition this PR adds support for snapshots to access the
snapshot S3 bucket, so we can possibly download snapshot versions
of plugins.

The format of the URLs stems from #12270

Closes #12632
2015-08-05 11:46:20 +02:00
Igor Motov f71c9a25a1 Check for incompatible mappings while upgrading old indices
Conflicting mappings that were allowed before v2.0 can cause runaway shard failures on upgrade. This commit adds a check that prevents a cluster from starting if it contains such indices as well as restoring such indices from a snapshot into already running cluster.

Closes #11857
2015-08-04 18:13:29 -06:00
Igor Motov 8fe5d903b7 Add support for bulk delete operation in snapshot repository
Currently when we delete files belonging to deleted snapshots we issue one delete command to underlying snapshot store at a time. Some repositories can benefit from bulk deletes of multiple files.

Closes #12533
2015-08-04 14:52:28 -06:00
Robert Muir 6f9a067197 Change master branch back to 2.0-beta1 2015-08-04 15:38:21 -04:00
Simon Willnauer 8f6e75b158 Ignore EngineClosedException on IndexShard#sync
This method syncs the translog unless it's already synced. If the engine
is alreayd closed we are guaranteed to be synced already such that we can just
ignore this exception.

Closes #12603
2015-08-04 15:43:17 +02:00
Simon Willnauer 2b2657279d Use UTC instead of default timezone for creation date in CAT endpoint
this change was added recently which uses default timezone for the creation
date on CAT endpoints. We should be consistent and use UTC across the board.
This commit adds #getDefaultTimzone() to forbidden API and fixes the REST tests.

Relates to #11688
2015-08-04 14:38:23 +02:00