10791 Commits

Author SHA1 Message Date
Honza Král
750310cf94 [API] indices.upgrade's wait_for_completion param defaults to false 2014-12-11 17:29:14 +01:00
Simon Willnauer
59534391da [GATEWAY] Cleanup LocalGatewayShardsState
This commit tries to cleanup LocalGatewayShardsState to be more efficient
and easier to understand.
2014-12-11 17:17:50 +01:00
Alexander Reelsen
544ef8cb17 Packaging: Add java7/8 java-package paths to debian init script
If you use the java-package tool to create java packages, those
paths also should be added to the debian init script.

Also updated the docs, that it is ok to install java8.

Closes #7383
2014-12-11 16:15:00 +01:00
Peter Fabian Mitchell
b2bab05c29 HTTP: Add 'http.publish_port' setting to the HTTP module
This change adds a 'http.publish_port' setting to the HTTP module to configure
the port which HTTP clients should use when communicating with the node. This
is useful when running on a bridged network interface or when running behind
a proxy or firewall.

Closes #8807
Closes #8137
2014-12-11 16:10:07 +01:00
Simon Willnauer
123fb2f5db [SCRIPT] ScriptService can deadlock entire nodes if script index is recovering
we currently have operationThreaded set to false when indexing a script. This setting
means that if we are executing the operation locally that we don't spawn a new thread for
it althought incoming thread in this case is the network thread. Now sicne we are indexing here
the engine is currently recovering which sometimes locks the engine for finalization blocks on
a network call waiting for the recovery target to come back the internal lock in engine will never be
released since we are waiting with our network thread for it to be released.
2014-12-11 16:04:25 +01:00
Simon Willnauer
ba881a9b58 [ENGINE] Remove engine related command classes
Todaqy we pass structs to the engine to call optimize / refresh and flush.
This commit cleans up this logic to reduce complexity in the engine.
2014-12-11 15:47:24 +01:00
Jun Ohtani
80bd69811d Mappings: Fix Get field mapping api with pretty flag
Closes #6552
2014-12-11 22:56:54 +09:00
Michael McCandless
084d25cdbd Test: create private store for these test cases 2014-12-11 05:25:44 -05:00
Robert Muir
a2ffe494ae [core] add best_compression option for Lucene 5.0
Upgrades lucene to latest, and supports the BEST_COMPRESSION parameter
now supported (with backwards compatibility, etc) in Lucene.
This option uses deflate, tuned for highly compressible data.

index.codec::
The default value compresses stored data with LZ4 compression, but
this can be set to best_compression for a higher compression ratio,
at the expense of slower stored fields performance.

IMO its safest to implement as a named codec here, because ES already
has logic to handle this correctly, and because its unrealistic to have
a plethora of options to Lucene's default codec... we are practically
limited in Lucene to what we can support with back compat, so I don't
think we should overengineer this and add additional unnecessary plumbing.

See also:
https://issues.apache.org/jira/browse/LUCENE-5914
https://issues.apache.org/jira/browse/LUCENE-6089
https://issues.apache.org/jira/browse/LUCENE-6090
https://issues.apache.org/jira/browse/LUCENE-6100

Closes #8863
2014-12-10 22:13:09 -05:00
Nicholas Knize
aa644e3ad7 [GEO] Fix for NPE enclosed in SearchParseException for a "geo_shape" filter or query
This fix adds better error handling for parsing multipoint, linestring, and polygon GeoJSONs.  Current logic throws a NPE when parsing a multipoint, linestring, or polygon that does not comply with the GeoJSON specification. That is, if a user provides a single coordinate instead of an array of coordinates, or array of linestrings, the ShapeParser throws a NPE wrapped in a SearchParseException instead of a more useful error message.

Closes #8432
2014-12-10 16:42:36 -06:00
Simon Willnauer
f308049a90 [ENGINE] Fix updates dynamic settings in InternalEngineHolder
After the refactoring in #8784 some settings didn't get passed to the
actual engine and there exists a race if the settings are updated while
the engine is started such that the actual starting engine doesn't see
the latest settings. This commit fixes the concurrency issue as well as
adds tests to ensure the settings are reflected.
2014-12-10 23:07:13 +01:00
Simon Willnauer
788d7cb451 [TEST] Reset test logger to default level 2014-12-10 22:47:03 +01:00
Alexander Clausen
633905161a Docs: use https to download the gpg public key
Closes #8818
2014-12-10 18:14:07 +01:00
Britta Weber
57b77c6907 [TEST] wait for yellow to avoid searching while relocating
After upgrading shard might start relocating again. If there are no
replicas the cluster state of a node might not be up to data for
a few miliseconds and direct a search request to a node that does not
have the shard anymore. This result in the following test failures:

1> java.lang.AssertionError: Count is 99 but 101 was expected.  Total shards: 13 Successful shards: 12 & 0 shard failures:
1>     __randomizedtesting.SeedInfo.seed([1932F73B458703CA:6F4FAD3DAC55591C]:0)
1>     [...org.junit.*]
1>     org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertHitCount(ElasticsearchAssertions.java:184)
1>     org.elasticsearch.bwcompat.BasicBackwardsCompatibilityTest.testIndexRollingUpgrade(BasicBackwardsCompatibilityTest.java:358)

Waiting for relocation finished should fix this.
2014-12-10 17:46:32 +01:00
Simon Willnauer
905dc90eec [CORE] Drop support for state written pre 0.90
Today we have several upgrade methods that can read state written
pre 0.90 or even pre 0.19. Version 2.0 should not support these state
formats. Users of these version should upgrade to a 1.x or 0.90.x version
first.

Closes #8850
2014-12-10 16:59:07 +01:00
Simon Willnauer
e5a7eaff22 [TEST] Use private store for a test private engine
The test was using the same store as the suite level engine which caused
problems with write locks in some cases.

Closes #8880
2014-12-10 16:56:52 +01:00
Michael McCandless
624a4b87bf Core: put []'s around the values in the slow-merge log message 2014-12-10 09:22:28 -05:00
Simon Willnauer
3b585e5a5e Add more trace logging to gateway translog recovery 2014-12-10 11:10:32 +01:00
Simon Willnauer
da11bf4c1d Use Lucene version string in main rest action
This commit removes the deprecated constant for the main
version and uses the real lucene version we are running instead.
Behind the scenes the same value was used and is now obsolet.
2014-12-10 10:40:58 +01:00
Michael McCandless
de75517357 Core: also log byte size and doc count when we log slow merges
Closes #8855

Closes #8853
2014-12-09 18:16:07 -05:00
Simon Willnauer
127255f62e [TEST] Restore test logging level after test is done 2014-12-09 23:31:24 +01:00
javanna
796ebcb88b [TEST] LoggingListener to restore the initial logger levels after any modification
Modifications to LoggingListener pushed with #8820 caused the original logger levels not to be reset after modifications, as the new state was saved for restore instead of the previous one.

 Added unit tests for LoggingListener as well.

Closes #8845
2014-12-09 14:26:13 +01:00
Simon Willnauer
8ffe8e0259 [TEST] compare strings and strings - leftover from Path API refactoring 2014-12-09 14:22:15 +01:00
Adam Menges
3a3030e217 Docs: Fix the wording for inner hits a bit
Closes #8747
2014-12-09 13:36:26 +01:00
Ashraf Sarhan
24f8807cb5 Docs: Update repositories.asciidoc
1. Enable the repository using "add-apt-repository" to avoid this error "No command 'deb' found".
2. Adding "sudo" to update and install command.

Closes #8691
2014-12-09 13:23:16 +01:00
Robert Muir
39186edc86 Ban java.io.File in tests.
Restrict use of java.io.File to 5 methods (excluded), but otherwise ban.
This is a prerequisite to do any mocking here.

I don't try to do any heavy cleanup on these tests, I am not familiar with them.
So this is mostly a rote straightforward conversion.

Closes #8836
2014-12-09 05:57:48 -05:00
Simon Willnauer
c0d50f2a80 [TEST] Beef up InternalEngineTest and remove bogus timeouts 2014-12-09 09:31:33 +01:00
Ryan Ernst
0c8f5ac129 Tests: Additional test for memory stats api. 2014-12-08 15:43:31 -08:00
Ryan Ernst
fde32cc599 Stats: Add more fine grained memory stats from Lucene segment reader.
This is a start to exposing memory stats improvements from Lucene 5.0.
This adds the following categories of Lucene index pieces to index stats:
* Terms
* Stored fields
* Term Vectors
* Norms
* Doc values
2014-12-08 15:29:43 -08:00
Robert Muir
150c2203ac Add test that ES filterreader getCoreCacheKey() behaves correctly.
Closes #8831.
2014-12-08 17:58:27 -05:00
Robert Muir
380e47dbf2 fix cache key methods on leaf reader 2014-12-08 15:28:53 -05:00
Michael McCandless
b0b96af746 Test: fix this test to work in IntelliJ 2014-12-08 15:11:06 -05:00
Simon Willnauer
b28fc1afa5 [ENGINE] Add engine lifecycle store reference to EngineHolder
This commit add the engines reference to the store out of the actual
implementation into the hodler since the holder manages the actual lifcycle.
Engine internal references like per searcher or per recovery are kept inside
the actual implemenation since the have a different lifecycle.
2014-12-08 21:07:12 +01:00
tlrx
31a77185a6 Merge branch 'fix/plugins-loading' 2014-12-08 17:15:06 +01:00
tlrx
97ec8f94ae Plugins: Plugin failed to load since #8666
The method Path.endsWith(String s) doesn't work exactly the same way as String.endsWith() (see http://docs.oracle.com/javase/7/docs/api/java/nio/file/Path.html#endsWith(java.nio.file.Path)). This blocks the loading of plugins.
2014-12-08 17:13:39 +01:00
Kevin Kluge
63ac4614f4 docs: add pgp key to repositories page 2014-12-08 15:41:09 +01:00
Simon Willnauer
84066128ed [TEST] Pass class level test logging to external nodes
This commit passes the test logging annotation from the class
level to the external nodes as well.

Closes #8552
2014-12-08 13:25:03 +01:00
Boaz Leskes
83bb65a020 Internal: allow InternalEngine to be stopped and started
Once the current engine is started you can only close it once. Once closed the engine cannot be started again. This commit adds a stop method which signals the engine to free it's resources but in a way that allows restarting.

This is done by introducing InternalEngineHolder which is a wrapper around InternalEngine. This allows to add the stop() method without adding complexity the engine implementation. InternalEngineHolder also serves an entry point for listeners (incoming and outgoing) to other ES components, which removes the needs add/remove them if the engine is stopped.

Closes #8784
2014-12-08 12:40:38 +01:00
Lee Hinman
83fa7bfaba [TEST] Add unit tests for DiskThresholdDecider settings 2014-12-08 12:14:09 +01:00
Simon Willnauer
8d7ce3c558 [STORE] Expose ShardId via LeafReader rather than Direcotry API
Today we try to fetch a shard Id for a given IndexReader / LeafReader
by walking it's tree until the lucene internal SegmentReader and then
casting the directory into a StoreDirecotory. This class is fully internal
to Elasticsearch and should not be exposed outside of the Store.

This commit makes StoreDirectory a private inner class and adds dedicated
ElasticsearchDirectoryReader / ElasticserachLeafReader exposing a ShardId
getter to obtain information about the shard the index / segment belogs to.

These classes can be used to expose other segment specific information in
the future more easily.
2014-12-08 12:10:28 +01:00
Simon Willnauer
a43259eba4 Add more trace loggin to LocalIndexShardGateway.java 2014-12-08 12:07:16 +01:00
tlrx
a046ee756d Scripting: Add explicit error message when script_score script returns NaN
When a scoring script returns not a number, the current message is confusing (IllegalArgumentException[docID must be >= 0 and < maxDoc=3 (got docID=2147483647)]). This commit adds the error message ScriptException[script score function returns a wrong score: NaN].

Closes #2426
2014-12-08 10:14:01 +01:00
Alexander Reelsen
7b64a5c2c8 Packaging: Check if proc file exists before calling sysctl
The packaged init scripts could return an error, if the file
/proc/sys/vm/max_map_count was not existing and we still called
sysctl.

This is primarly to prevent confusing error messages when elasticsearch
is started under virtualized environments without a proc file system.

Closes #4978
2014-12-08 09:56:14 +01:00
Boaz Leskes
2bc48a4806 Tests: move RecoverAfterNodesTests to org.elasticsearch.gateway.local and increase BLOCK_WAIT_TIMEOUT to 10s
The tests were still in org.elasticsearch.gateway.none but the none gateway was removed.
2014-12-07 22:10:11 +01:00
Simon Willnauer
3cdf266d4d [ALLOCATION] Speed-up disk-threshold decider
Instead of iterating all shards of all indices to get all relocating
shards for a given node we can just use the RoutingNode#shardsWithState
method and fetch all INITIALIZING / RELOCATING shards and check if they
are relocating. This operation is much faster and uses pre-build
data-structures.

Relates to #6372
2014-12-06 22:03:37 +01:00
Jun Ohtani
d78d2ff93d Docs: add randomizedtesting-runner to testing-framework.asciidoc
Close #8450
2014-12-07 01:30:58 +09:00
Patrick Koenig
00275ac1d6 Clarify InternalIndexShard callbacks
This commit changes internal callback to be clear
about when they are called and also provide the
exception that was potentially thrown as a callback argument.

Closes #5945
2014-12-05 23:55:32 +01:00
Igor Motov
0b024ad2f3 Snapshot/Restore: switch to write once mode for snapshot metadata files
This commit removes creation of in-progress snapshot file and makes creation of the final snapshot file atomic.

Fixes #8696
2014-12-05 12:39:24 -05:00
Simon Willnauer
0bab17ffde [TEST] wait unitl all machines joined the cluster 2014-12-05 18:12:00 +01:00
Lee Hinman
d32f1a8ad0 [TESTS] Log what the _default_ template is in ElasticsearchIntegrationTest 2014-12-05 14:45:24 +01:00