Commit Graph

10772 Commits

Author SHA1 Message Date
Simon Willnauer a834cc0e0f Shutdown indices service last
We do wait for shards to be closed in IndicesService for 30 second.
Yet, if somebody holds on to a store reference ie. an open scroll request
the 30 seconds time-out and node shutdown takes very long. We should
release all other resources first before we shutdown IndicesService.

Closes #8940
2014-12-15 22:43:37 +01:00
Ryan Ernst 37287284e6 Settings: Remove `mapping.date.round_ceil` setting for date math parsing
The setting `mapping.date.round_ceil` (and the undocumented setting
`index.mapping.date.parse_upper_inclusive`) affect how date ranges using
`lte` are parsed.  In #8556 the semantics of date rounding were
solidified, eliminating the need to have different parsing functions
whether the date is inclusive or exclusive.

This change removes these legacy settings and improves the tests
for the date math parser (now at 100% coverage!). It also removes the
unnecessary function `DateMathParser.parseTimeZone` for which
the existing `DateTimeZone.forID` handles all use cases.

Any user previously using these settings can refer to the changed
semantics and change their query accordingly. This is a breaking change
because even dates without datemath previously used the different
parsing functions depending on context.

closes #8598
closes #8889
2014-12-15 13:13:45 -08:00
Timothy Perisho ceafde41e9 Docs: typo on "frequent"
I replaced "high frequent terms" with "high frequency terms" and "low frequent terms" with "low frequency terms".

Alternatively, we could write, "highly frequent terms" and "minimally frequent terms" (or just "rare terms").

Closes #8962
2014-12-15 19:59:50 +01:00
Clinton Gormley fcb83055de Update repositories.asciidoc
Update formatting of PGP key
2014-12-15 18:04:17 +01:00
Lee Hinman 8fbf45ef2b [TEST] Make parent breaker check less strict
In cases of heavy contention, it's possible for more than 2 threads
to race to a circuit breaking exception.

Essentially this means that if we have 3 threads all trying to add 3 and
simultaneously cause a circuit breaking exception (due to retry), when
adjusting after circuit breaking we can "rewind" past what this test
expects the child breaker to be at.

This adds leeway into the check, where it's okay to be within
NUM_THREADS from the parentLimit, because each thread should only add 1
to the breaker at a time.
2014-12-15 17:06:21 +01:00
Robert Muir 01fc84dbb3 Wire utf-8 encoding, so unicode filenames work
Sets -Dfile.encoding=UTF-8 by default.

Closes #8847
2014-12-15 10:36:43 -05:00
Simon Willnauer 3bba45289e Remove unused code 2014-12-15 16:26:48 +01:00
Simon Willnauer 1247774ff1 Remove Gateway abstraction
We only have a single gatweway since es 1.3. There is no need to keep all
these abstractsion and nested packages. We can fold most of it into simpler
structures.
2014-12-15 15:53:02 +01:00
Lee Hinman a8fa650ee6 [CORE] Remove IndexEngine
IndexEngine was an abstraction where we had index-level engines (instead
of shard-level) that could store meta information about the index. It
was never actually used by Elasticsearch, and only there for plugins.

This removes it, because it is a confusing abstraction and not needed,
no plugins should be implementing their own IndexEngines.
2014-12-15 14:30:44 +01:00
spapin ad747ba67f Docs: fix a typo in cluster stats documentation example
Closes #8898
2014-12-15 14:14:38 +01:00
Ayush 23dbecf3e7 Update percolate.asciidoc
Updating the `associated` spelling

Closes #8907
2014-12-15 14:12:03 +01:00
Simone Scarduzio fff483d612 Docs: Adding REST ACL plugin
Closes #8925
2014-12-15 14:09:22 +01:00
Boaz Leskes d62bf5f67f Discovery: concurrent node failures can cause unneeded cluster state publishing
When a node fails (or closes), the master processes the network disconnect event and removes the node from the cluster state. If multiple nodes fail (or shut down) in rapid succession, we process the events and remove the nodes one by one. During this process, the intermediate cluster states may cause the node fault detection to signal the failure of nodes that are not yet removed from the cluster state. While this is fine, it currently causes unneeded reroutes and cluster state publishing, which can be cumbersome in big clusters.

Closes #8804
Closes #8933
2014-12-15 14:01:25 +01:00
Lee Hinman 9b18c44b67 Default _cat APIs to verbose
`?v=false` can be used if the headers are not desired.

Resolves #8922

Fixes #8927
2014-12-15 12:51:59 +00:00
Simon Willnauer e47b753617 [SEARCH] close active contexts on SearchService#close()
When we close a node all pending / active search requests need to be
cleared otherwise a node will wait up to 30 sec for shutdown sicne there
could be open scroll requests. This behavior was introduces in 1.5 such that
versions <= 1.4.x are not affected.

Closes #8940
2014-12-15 09:41:31 +01:00
Boaz Leskes a63a055f63 Test: missing {} from log command in indexRandom 2014-12-13 17:24:46 +01:00
Boaz Leskes 22da975e34 Test: reduce join timeout in testFullRollingRestart
Occasionally a the join thread successfully connected to a just closed node and which causes the subsequent join request to time out. It's default timeout 60s throws the test off when it waits for a cluster to form.
2014-12-13 13:05:04 +01:00
Michael McCandless ae11c4654b Core: use compound file by default for merged segments < 10% of index size
Change Elasticsearch to use Lucene's defaults, to reduce file descriptor count.

Closes #8934

Closes #8919
2014-12-12 15:51:37 -05:00
Britta Weber 60e805cc1f [TEST] use ensureYellow() 2014-12-12 18:07:14 +01:00
Britta Weber 185521be4b [TEST] wait for yellow before searching 2014-12-12 17:34:07 +01:00
Britta Weber 2dc9392a34 [TEST] get trace logs for search packages 2014-12-12 17:06:16 +01:00
Lee Hinman 6bf18056b0 [CORE] Remove explicit .cleanUp() on cache clear
Calling cache.cleanUp() is kind of like calling System.gc(), meaning
that we should never have (non-test) things that rely on this
functionality.

For the field data and filter cache, we already have a periodic process
that runs this .cleanUp(), so there is no need to block index
closing/clearing on it. Instead, we can clean the field data cache in
InternalTestCluster before we check the circuit breaker.

This can help tests that time out because cleaning the cache is taking
too long
2014-12-12 13:24:45 +01:00
Simon Willnauer 42d9a57d0c [TEST] Wait for yellow before verifying - sometimes the shard is not even started 2014-12-12 12:40:34 +01:00
Simon Willnauer b857a1f635 Remove circular dependency between IndicesService and IndicesStore 2014-12-12 12:25:25 +01:00
Simon Willnauer 498331d16f [TEST] Remove random templates for ConcurrentDynamicTemplateTests 2014-12-12 12:18:58 +01:00
Michael McCandless c9d63d3482 Core: don't block shard stats when phase 3 of recovery is running
Today, shard stats are blocked while phase 3 of recovery (replay xlog)
is running; this change removes the engine readLock from shard stats
so it's not blocked.

Closes #8910
2014-12-12 05:18:25 -05:00
Martijn Laarman bb6e539bc7 updated (m)termvectors api specs to reflect #7530 which made it into 1.4 2014-12-12 11:04:21 +01:00
Simon Willnauer 6dacf61dfc [TEST] Add test to ensure master is not prone to #8917 2014-12-12 10:59:11 +01:00
Simon Willnauer 7b82660ffc [TEST] Remove debug leftover 2014-12-12 09:29:33 +01:00
Simon Willnauer dac520170f [TEST] Close the node env after test is done 2014-12-11 21:24:15 +01:00
Simon Willnauer 3877dc618d Remove some Internal* abstractions
We have lots of boilerplate code that is unnecessarily abstracting
services ie InternalIndexShard and IndexShard or InternalIndexService and
IndexService. It's enough to have concrete classes for these core classes.

Closes #8904
2014-12-11 17:31:01 +01:00
Honza Král 750310cf94 [API] indices.upgrade's wait_for_completion param defaults to false 2014-12-11 17:29:14 +01:00
Simon Willnauer 59534391da [GATEWAY] Cleanup LocalGatewayShardsState
This commit tries to cleanup LocalGatewayShardsState to be more efficient
and easier to understand.
2014-12-11 17:17:50 +01:00
Alexander Reelsen 544ef8cb17 Packaging: Add java7/8 java-package paths to debian init script
If you use the java-package tool to create java packages, those
paths also should be added to the debian init script.

Also updated the docs, that it is ok to install java8.

Closes #7383
2014-12-11 16:15:00 +01:00
Peter Fabian Mitchell b2bab05c29 HTTP: Add 'http.publish_port' setting to the HTTP module
This change adds a 'http.publish_port' setting to the HTTP module to configure
the port which HTTP clients should use when communicating with the node. This
is useful when running on a bridged network interface or when running behind
a proxy or firewall.

Closes #8807
Closes #8137
2014-12-11 16:10:07 +01:00
Simon Willnauer 123fb2f5db [SCRIPT] ScriptService can deadlock entire nodes if script index is recovering
we currently have operationThreaded set to false when indexing a script. This setting
means that if we are executing the operation locally that we don't spawn a new thread for
it althought incoming thread in this case is the network thread. Now sicne we are indexing here
the engine is currently recovering which sometimes locks the engine for finalization blocks on
a network call waiting for the recovery target to come back the internal lock in engine will never be
released since we are waiting with our network thread for it to be released.
2014-12-11 16:04:25 +01:00
Simon Willnauer ba881a9b58 [ENGINE] Remove engine related command classes
Todaqy we pass structs to the engine to call optimize / refresh and flush.
This commit cleans up this logic to reduce complexity in the engine.
2014-12-11 15:47:24 +01:00
Jun Ohtani 80bd69811d Mappings: Fix Get field mapping api with pretty flag
Closes #6552
2014-12-11 22:56:54 +09:00
Michael McCandless 084d25cdbd Test: create private store for these test cases 2014-12-11 05:25:44 -05:00
Robert Muir a2ffe494ae [core] add best_compression option for Lucene 5.0
Upgrades lucene to latest, and supports the BEST_COMPRESSION parameter
now supported (with backwards compatibility, etc) in Lucene.
This option uses deflate, tuned for highly compressible data.

index.codec::
The default value compresses stored data with LZ4 compression, but
this can be set to best_compression for a higher compression ratio,
at the expense of slower stored fields performance.

IMO its safest to implement as a named codec here, because ES already
has logic to handle this correctly, and because its unrealistic to have
a plethora of options to Lucene's default codec... we are practically
limited in Lucene to what we can support with back compat, so I don't
think we should overengineer this and add additional unnecessary plumbing.

See also:
https://issues.apache.org/jira/browse/LUCENE-5914
https://issues.apache.org/jira/browse/LUCENE-6089
https://issues.apache.org/jira/browse/LUCENE-6090
https://issues.apache.org/jira/browse/LUCENE-6100

Closes #8863
2014-12-10 22:13:09 -05:00
Nicholas Knize aa644e3ad7 [GEO] Fix for NPE enclosed in SearchParseException for a "geo_shape" filter or query
This fix adds better error handling for parsing multipoint, linestring, and polygon GeoJSONs.  Current logic throws a NPE when parsing a multipoint, linestring, or polygon that does not comply with the GeoJSON specification. That is, if a user provides a single coordinate instead of an array of coordinates, or array of linestrings, the ShapeParser throws a NPE wrapped in a SearchParseException instead of a more useful error message.

Closes #8432
2014-12-10 16:42:36 -06:00
Simon Willnauer f308049a90 [ENGINE] Fix updates dynamic settings in InternalEngineHolder
After the refactoring in #8784 some settings didn't get passed to the
actual engine and there exists a race if the settings are updated while
the engine is started such that the actual starting engine doesn't see
the latest settings. This commit fixes the concurrency issue as well as
adds tests to ensure the settings are reflected.
2014-12-10 23:07:13 +01:00
Simon Willnauer 788d7cb451 [TEST] Reset test logger to default level 2014-12-10 22:47:03 +01:00
Alexander Clausen 633905161a Docs: use https to download the gpg public key
Closes #8818
2014-12-10 18:14:07 +01:00
Britta Weber 57b77c6907 [TEST] wait for yellow to avoid searching while relocating
After upgrading shard might start relocating again. If there are no
replicas the cluster state of a node might not be up to data for
a few miliseconds and direct a search request to a node that does not
have the shard anymore. This result in the following test failures:

1> java.lang.AssertionError: Count is 99 but 101 was expected.  Total shards: 13 Successful shards: 12 & 0 shard failures:
1>     __randomizedtesting.SeedInfo.seed([1932F73B458703CA:6F4FAD3DAC55591C]:0)
1>     [...org.junit.*]
1>     org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertHitCount(ElasticsearchAssertions.java:184)
1>     org.elasticsearch.bwcompat.BasicBackwardsCompatibilityTest.testIndexRollingUpgrade(BasicBackwardsCompatibilityTest.java:358)

Waiting for relocation finished should fix this.
2014-12-10 17:46:32 +01:00
Simon Willnauer 905dc90eec [CORE] Drop support for state written pre 0.90
Today we have several upgrade methods that can read state written
pre 0.90 or even pre 0.19. Version 2.0 should not support these state
formats. Users of these version should upgrade to a 1.x or 0.90.x version
first.

Closes #8850
2014-12-10 16:59:07 +01:00
Simon Willnauer e5a7eaff22 [TEST] Use private store for a test private engine
The test was using the same store as the suite level engine which caused
problems with write locks in some cases.

Closes #8880
2014-12-10 16:56:52 +01:00
Michael McCandless 624a4b87bf Core: put []'s around the values in the slow-merge log message 2014-12-10 09:22:28 -05:00
Simon Willnauer 3b585e5a5e Add more trace logging to gateway translog recovery 2014-12-10 11:10:32 +01:00
Simon Willnauer da11bf4c1d Use Lucene version string in main rest action
This commit removes the deprecated constant for the main
version and uses the real lucene version we are running instead.
Behind the scenes the same value was used and is now obsolet.
2014-12-10 10:40:58 +01:00