Commit Graph

10178 Commits

Author SHA1 Message Date
Simon Willnauer 0a4e041ee5 [RECOVERY] Throw IndexShardClosedException if shard is closed
Today we throw a generic ElasticsearchException when a recovery is cancled. This
causes verbose logging and send shard failures and additional unnecessary cluster state
events. We can just throw IndexShardClosedException which prevents the send shard failures
2014-11-25 14:35:06 +01:00
Matt Hughes afba977e80 Docs: Added swift openstack repository
Closes #8583
2014-11-25 13:49:15 +01:00
David Haney 2c429452e9 Typo: changed "5% or the real words" to "5% of the real words"
Closes #8582
2014-11-25 13:15:33 +01:00
Simon Willnauer 35b278fc68 [TRANSLOG] Cut over to Path API
This commit moves all the Translog related code over to the
NIO2 Path API. It also make transaction logs write once since it
never reuses a translog file.

Closes #8611
2014-11-25 12:43:57 +01:00
Lee Hinman 6749b2c306 [TEST] Reduce stringency of breaker assertions
While in a perfect world we should only ever have 2 circuit breaker
trips, it's possible to get a race condition between the child and the
parent breaker with many threads. Since multiple breaking exceptions are
not actually a bad thing, it's okay to relax the constraints in the
test.

The race conditions are due to no locking inside the breaker logic, to
ensure that it is as low overhead as possible. Even though no locking is
used, we use atomic counters internally to ensure that the "estimated"
numbers for the breakers are never out of sync (which this test still
checks with no leeway).
2014-11-25 11:53:29 +01:00
Colin Goodheart-Smithe c420a17f7d Aggregations: Added getProperty method to Aggregations
This allows arbitrary properties to be retrieved from an aggregation tree. The property is specified using the same syntax as the
order parameter in the terms aggregation. If a property path contians a multi-bucket aggregation the property values from each bucket will be returned in an array.
2014-11-25 10:07:42 +00:00
Michael McCandless 856b294441 Core: let Lucene kick off merges
Today, Elasticsearch has a separate merge thread pool checking once
per second (by default) if any merges are necessary, but this is no
longer necessary since we can and do now tell Lucene's
ConcurrentMergeScheduler never to "hard pause" threads when merges
fall behind, since we do our own index throttling.

This change goes back to letting Lucene launch merges as needed, and
removes these two expert settings:

  index.merge.force_async_merge
  index.merge.async_interval

Now merges kick off immediately instead of waiting up to 1 second
before running.

Closes #8643
2014-11-25 04:13:57 -05:00
unknown c4e2f63b17 Aggregations: Make size parsing consistent for strings.
Close #6061
2014-11-25 08:55:24 +01:00
Igor Motov 668dbe498a Snapshot/Restore: fix compilation issue for JDK 1.7 2014-11-24 21:17:03 -05:00
Igor Motov 1aff8631ed Snapshot/Restore: restore with wait_for_completion=true should wait for succesfully restored shards to get started
This commit ensures that restore operation with wait_for_completion=true doesn't return until all successfully restored shards are started. Before it was returning as soon as restore operation was over, which cause some shards to be unavailable immediately after restore completion.

Fixes #8340
2014-11-24 19:37:43 -05:00
Sebastian Ziebell e0d6e0a6d3 Spec: Fixes a few locations of the params array in JSON specification
A few locations of the `params` object in the JSON specification were not located in the `url` block.

Closes #8607
2014-11-24 20:43:33 +01:00
Adrien Grand d60500f22e Fielddata: Fix iterator over global ordinals.
Our iterator over global ordinals is currently incorrect since it does NOT
return -1 (NO_MORE_ORDS) when all ordinals have been consumed. This bug does
not strike immediately with elasticsearch since we always consume ordinals in
a random-access fashion. However it strikes when consuming ordinals through
Lucene helpers such as DocValues#docsWithField.

Close #8580
2014-11-24 19:42:53 +01:00
Martijn van Groningen 13b9e07522 Core: Fields defined in the `_default_` mapping of an index template should be picked up when an index alias filter is parsed if a new index is introduced when a document is indexed into an index that doesn't exist yet.
Closes #8473
2014-11-24 18:25:31 +01:00
Simon Willnauer 043b37d932 [GATEWAY] Remove _state directory if index has been deleted
Today we try to delete the index directory if all shard locks have been
acquired. Yet, if this fails due to still running recoveries etc. We might
re-import the index as dangeling which also can happen if the node is restarted.
In contrast to the shard direcotries we can safely delete the metastate which is used
to import dangling indices while leaving the shard directories untouched.
2014-11-24 16:00:45 +01:00
Reuben Sutton fda1576d55 Fix SearchRequest.templateParams so that it is a Map<String, Object> so that it can take more data-types than just strings, to support Arrays. 2014-11-24 14:46:48 +00:00
Nicholas Knize 08bbfac7eb Updating comments in BasePolygonBuilder to explain "pairwise" intersection 2014-11-24 08:34:10 -06:00
Nicholas Knize 82f5252c14 Removing IntersectionOrder.SENTINEL and adding Edge.MAX_COORDINATE for code readability. 2014-11-24 08:33:56 -06:00
Nicholas Knize fc955551d4 [GEO] Fix for geo_shape query with polygon from -180/90 to 180/-90
This fix adds a simple consistency check that intersection edges appear pairwise. Polygonal boundary tests were passing (false positive) on the Eastern side of the dateline simply due to the initial order (edge direction) of the intersection edges.  Polygons in the Eastern hemispehere (which were not being tested) were correctly failing inside of JTS due to an attempt to connect incorrect intersection edges (that is, edges that were not even intersections). While this patch fixes issue/8467 (and adds broader test coverage) it is not intented as a long term solution.  The mid term fix (in work) will refactor all geospatial computational geometry to use ENU / ECF coordinate systems for higher accuracy and eliminate brute force mercator checks and conversions.

Closes #8467
2014-11-24 08:31:36 -06:00
Martijn van Groningen 1d7cdd7d22 Applied PR, changed the way defaults are handled and updated the docs.
Closes #4452
2014-11-24 13:32:41 +01:00
Brusic 5a001e1357 Cleaner error handling. Pre without post (or vice versa) is not an error condition. Set appropriate defaults for pre/post. 2014-11-24 13:28:55 +01:00
Brusic 94074b4028 Use static int for easier to read code 2014-11-24 13:28:55 +01:00
Brusic 7c10b445d4 Expose dist/pre/post options for SpanNotQuery 2014-11-24 13:28:54 +01:00
Lee Hinman 45408844e7 Remove NoneGateway, NoneGatewayAllocator, & NoneGatewayModule
Always use the LocalGateway* equivalents

We already check in the LocalGateway whether a node is a client node, or
is not master-eligible, and skip writing the state there. This allows us
to remove this code that was previously used only for tribe nodes (which
are not master eligible anyway and wouldn't write state) and in
tests (which can shake more bugs out)
2014-11-24 12:22:05 +01:00
dw ad408eee85 Docs: Reword note regarding _source for accuracy
Previously it suggested _source was always present, when that is not the case.

Closes #8491
2014-11-24 12:19:44 +01:00
Laurent Broudoux feb465f26f Docs: Update plugins.asciidoc on river plugins section
Adding links to Amazon S3 and Google Drive river plugins

Closes #8544
2014-11-24 12:15:12 +01:00
Michael McCandless dfb6d6081c Core: upgrade to current Lucene 5.0.0 snapshot
Elasticsearch no longer unlocks the Lucene index on startup (this was
dangerous, and could possibly lead to corruption).

Added the new serbian_normalization TokenFilter from Lucene.

NoLockFactory is no longer supported (index.store.fs.fs_lock = none),
and if you have a typo in your fs_lock you'll now hit a StoreException
instead of silently using NoLockFactory.

Closes #8588
2014-11-24 05:08:42 -05:00
Uwe Schindler 866571f4d7 Switch to forbidden-apis 1.7 2014-11-24 09:58:44 +01:00
Adrien Grand 8346e92ebb Core: Fix script fields to be returned as a multivalued field when they produce a list.
This change is essentially the same as #3015 but on script fields.

Close #8592
2014-11-24 09:41:16 +01:00
mdzor bc52ccfd33 Docs: Update update-settings.asciidoc
Inconsistent indentation

Closes #8525
2014-11-23 14:45:56 +01:00
barbasa fd6c41bfbf Missing quote in the example 2014-11-23 14:03:58 +01:00
Alban Perillat-Merceroz 54466938da Fix error in documentation
Indexation does not fail if no timestamp provided when there is a default value defined in mapping.
2014-11-23 14:02:51 +01:00
dw bb81055c33 Docs: Remove reference to imaginary "no_docs_query"
No reference to it in the source code except this file.

Closes #8566
2014-11-23 13:56:33 +01:00
Mariam Hakobyan 4a1ab6543c Docs: Added new elasticsearch-river-kafka plugin to the documentation (uses latest version of Kafka, EL Bulk API, and supports concurrent requests)
Closes #8518
2014-11-23 12:28:45 +01:00
Simon Willnauer b6b3382a8b [STORE] Use Lucene checksums if segment version is >= 4.9.0
We started to use the lucene CRC32 checksums instead of the legacy Adler32
in `v1.3.0` which was the first version using lucene `4.9.0`. We can safely
assume that if the segment was written with this version that checksums
from lucene can be used even if the legacy checksum claims that it has a Adler32
for a given file / segment.

Closes #8587

Conflicts:
	src/main/java/org/elasticsearch/index/store/Store.java
	src/test/java/org/elasticsearch/index/store/StoreTest.java
2014-11-21 22:35:21 +01:00
ananich 95a396159c Added utility method to CountRequestBuilder
Allows to add a query to CountRequestBuilder as a XContentBuilder.

Closes #8594
2014-11-21 21:04:01 +01:00
Simon Willnauer 1c00790213 [TEST] all tests should extend ElasticsearchTestCase 2014-11-21 20:27:52 +01:00
Andreas Kohn 3205dce549 Fix a typo in a javadoc comment in MapperService.
Close #8575
2014-11-21 19:28:34 +01:00
Damien Alexandre 80295458ad Fix wrong error messages in MultiMatchQueryParser.
Also fix a typo in the comment

Close #8597
2014-11-21 19:28:34 +01:00
Ryan Ernst 40598a5692 Fix test failures caused by #8556 2014-11-21 10:22:13 -08:00
Ryan Ernst 4a3afcc0d3 Fix compile error from bad merge in #8556 2014-11-21 09:33:36 -08:00
Ryan Ernst fae9dcaed7 DateMath: Fix semantics of rounding with inclusive/exclusive ranges.
Date math rounding currently works by rounding the date up or down based
on the scope of the rounding.  For example, if you have the date
`2009-12-24||/d` it will round down to the inclusive lower end
`2009-12-24T00:00:00.000` and round up to the non-inclusive date
`2009-12-25T00:00:00.000`.

The range endpoint semantics work as follows:
* `gt` - round D down, and use > that value
* `gte` - round D down, and use >= that value
* `lt` - round D down, and use <
* `lte` - round D up, and use <=

There are 2 problems with these semantics:
* `lte` ends up including the upper value, which should be non-inclusive
* `gt` only excludes the beginning of the date, not the entire rounding scope

This change makes the range endpoint semantics symmetrical.  First, it
changes the parser to round up and down using the first (same as before)
and last (1 ms less than before) values of the rounding scope.  This
makes both rounded endpoints inclusive. The range endpoint semantics
are then as follows:
* `gt` - round D up, and use > that value
* `gte` - round D down, and use >= that value
* `lt` - round D down, and use < that value
* `lte` - round D up, and use <= that value

closes #8424
closes #8556
2014-11-21 09:28:30 -08:00
Alex Ksikes 1959275622 Term Vectors: More consistent naming for term vector[s]
We speak of the term vectors of a document, where each field has an associated
stored term vector. Since by default we are requesting all the term vectors of
a document, the HTTP request endpoint should rather be called `_termvectors`
instead of `_termvector`. The usage of `_termvector` is now deprecated, as
well as the transport client call to termVector and prepareTermVector.

Closes #8484
2014-11-21 14:06:44 +01:00
Simon Willnauer a6e6c4efc4 [CORE] Ensure shards are deleted under lock on close
Today there is a race condition between the actual deletion of
the shard and the release of the lock in the store. This race can cause
rare imports of dangeling indices if the cluster state update loop
tires to import the dangeling index in that particular windonw. This commit
adds more safety to the import of dangeling indices and removes the race
condition by holding on to the lock on store closing while the listener
is notified.
2014-11-21 12:20:58 +01:00
Adrien Grand abc0bc4c7f Aggregations: Fix geohash grid doc counts computation on multi-valued fields.
Close #8512
2014-11-21 11:02:04 +01:00
Thomas Cucchietti 98641ff39f Update DiskThresholdDecider javadoc
Since v1.3.0, and issue #6201, the default values in code and documentation have been updated to 85% and 90% for low and high watermarks. However, the related javadoc still contains the initial values : this commit fix this.
2014-11-21 10:24:24 +01:00
Robert Muir 9ef69f9f36 Disable bloom filters.
make the "es090" postings format read-only, just to support old segments. There is a test version that subclasses it with write-capability for testing.

Closes #8571
2014-11-20 21:03:23 -05:00
Simon Willnauer 3e1b7c7a34 [BLOOM] Fix Bloom filter ram usage calculation
BloomFilter actually returned the size of the bitset as the
size in bytes so off by factor 8 plus a constant :)

Closes #8564
2014-11-20 22:45:28 +01:00
Simon Willnauer afa62e90d9 [CORE] Free pending search contexts if index is closed
Today we hold on to search context reference if they are not cleaned
up for a while until a reaper thread trashes them if they timed out.
This commit removes all pending contexts once the index is closed to release
resources and filehandles immediatly once the index is closed.
2014-11-20 16:13:35 +01:00
Simon Willnauer d5d5dece56 [INDEX] Add before/after indexDeleted callbacks to IndicesLifecycle
In order to implement #8551 correctly without causing problems of relocating
shards we need to be informed if an index is actually deleted. This commit adds
more callbacks to the listener and makes deleteIndex a dedicated method on IndicesService
2014-11-20 15:37:35 +01:00
Simon Willnauer 26b4ebcd00 [TEST] Delete index in test to release file handles 2014-11-20 15:35:08 +01:00