Commit Graph

10728 Commits

Author SHA1 Message Date
Colin Goodheart-Smithe e2949d7df1 make InternalAggregation.reduce(ReduceContext) use template pattern
sub-classes of InternalAggregation now implement doReduce(ReduceContext) that is called from InternalAggregation.reduce(ReduceContext) which is now final
2015-02-12 13:23:12 +00:00
Adrien Grand de41981373 Aggs: Refactor aggregations to use lucene5-style collectors.
Aggregators now return a new collector instance per segment, like Lucene 5 does
with its oal.search.Collector API. This is important for us because things like
knowing whether the field is single or multi-valued is only known at a segment
level.

In order to do that I had to change aggregators to notify their sub aggregators
of new incoming segments (pretty much in the spirit of #6477) while everything
used to be centralized in the AggregationContext class. While this might slow
down a bit deeply nested aggregation trees, this also makes the children
aggregation and the `breadth_first` collection mode much better options since
they can now only replay what they need while they used to have to replay the
whole aggregation tree.

I also took advantage of this big refactoring to remove some abstractions that
were not really required like ValuesSource.MetaData or BucketAnalysisCollector.
I also splitted Aggregator into Aggregator and AggregatorBase in order to
separate the Aggregator API from implementation helpers.

Close #9544
2015-02-12 14:13:31 +01:00
gseng d1deb6bd1e Update update-settings.asciidoc
Updating to the fields as mentioned on http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-fielddata.html

Closes #9657
2015-02-12 13:09:16 +01:00
Clinton Gormley 856b0fa1a0 Docs: Fixed explanation of how the query string query is rewritten 2015-02-12 12:46:44 +01:00
javanna 4e94be8a37 [TEST] Introduce basic validation of our REST spec
Whenever we have an api that supports GET with a body, we always support the POST method too, as well as providing the body as a query_string parameter called `source`. Our REST spec should reflect this convention. FIxed them and introduced a hard check at parse time in our Java REST tests runner, which will cause the tests to fail if spec are not compliant.

Closes #9629
2015-02-12 22:25:17 +11:00
Clinton Gormley 20ece4acb5 Update core-types.asciidoc
Provide an example of how to disable norms

Closes #9641
2015-02-12 12:10:11 +01:00
Alexander Reelsen dfc2e6381b CliTool: CheckFileCommand checks for file existence
As a CliTool command could potentially also delete files, the
CheckFileCommand needs to check if those files exist, before
trying to get permissions/owners/groups from that path.
2015-02-12 11:36:47 +01:00
Alexander Reelsen 9cd14a5c29 CliTool: Add command to warn on permission/owner change
When using the CLI tool infrastructure, a command can potentially write
a new file. In case it overwrites an existing one, you may want to ensure
that the permissions, the owner and the group are kept the same and do not
accidentally change when overwriting those files.

This PR introduces a command that allows you to execute this check per path.

It also adds a new testing dependency, namely jimfs, which allows you to create
in-memory filesystems with certain properties (like supporting or not posix permissions
on this filesystem), so that you can test those features, without executing
tests on a certain operating system.
2015-02-12 10:10:11 +01:00
Alexander Reelsen 30a9d97a71 FileSystemUtils: Only create backup copies if files differ
The FileSystemUtils class has a helper method to create files with
a .new suffix, in case the file, which should be created already
exists. If you install plugins and those have configuration files,
even without changes, you will end up with tons of .new files.

This commit checks the file size and sha-256 sum, and only if those
differ, a .new file is actually being created.
2015-02-12 10:08:14 +01:00
Igor Motov 9b75d3ef98 Test: wait for the cluster to recover in ClusterServiceTests before waiting for update state task results
On CI machines node recovery sometimes takes up to 2 seconds. When it happens an update cluster state task gets stuck behind the recovery and tests fail with 1 second timeout. This commit makes sure that we wait for recovery to complete before starting the clock.
2015-02-11 19:11:00 -05:00
Ryan Ernst f735baf306 Core: Remove ability to run optimize and upgrade async
This has been very trappy. Rather than continue to allow buggy behavior
of having upgrade/optimize requests sidestep the single shard per node
limits optimize is supposed to be subject to, this removes
the ability to run the upgrade/optimize async.

closes #9638
2015-02-11 11:30:27 -08:00
Clinton Gormley faae98c5d8 Updated latest version in docs 2015-02-11 19:25:10 +01:00
Martijn van Groningen 26b9d14443 Added 1.3.9-SNAPSHOT and 1.4.4-SNAPSHOT versions 2015-02-11 18:02:01 +01:00
Clinton Gormley 57a4646776 Docs: Added note about groovy sandbox vulnerability to modules/scripting 2015-02-11 17:54:53 +01:00
Clinton Gormley 6fadeeca56 Updated doc annotations for 1.4.3 2015-02-11 17:54:53 +01:00
Ryan Ernst 54c1813920 Tests: Add forgotten files for static bwc tests 2015-02-11 07:07:40 -08:00
Ryan Ernst 7328aa1c15 Tests: Add static bwc tests for new releases 1.3.8 and 1.4.3 2015-02-11 07:06:17 -08:00
Martijn van Groningen 173cfc14d6 Marked 1.4.3 as released 2015-02-11 15:58:39 +01:00
Simon Willnauer 764fda6420 [TEST] make sandbox settings explicit in Tests 2015-02-11 13:21:43 +01:00
Simon Willnauer 0b0cd1c46c [ENGINE] Fix deadlock problems when API flush and finish recovery happens concurrently
Unfortunately the lock order is important in the current flush codehe. We have to acquire the readlock fist otherwise
if we are flushing at the end of the recovery while holding the write lock we can deadlock if:
 * Thread 1: flushes via API and gets the flush lock but blocks on the readlock since Thread 2 has the writeLock
 * Thread 2: flushes at the end of the recovery holding the writeLock and blocks on the flushLock owned by Thread 2

This commit acquires the read lock first which would be done further down anyway for the time of the flush.
As a sideeffect we can now safely flush on calling close() while holding the writeLock.
2015-02-11 13:17:44 +01:00
Simon Willnauer 2f0d158692 [CORE] Consolidate index / shard deletion in IndicesService
Today the logic related to deleting an index is spread across several
classes which makes changes to this rather delicate part of the code-base
very difficult. This commit consolidates this logic into the IndicesService
and moves the handling of ack-ing the delete to the master entirely into
`IndicesClusterStateService`.
2015-02-11 09:05:20 +01:00
Simon Willnauer d3762d6427 [TEST] Make tests pass while flying 2015-02-11 09:05:18 +01:00
Igor Motov 00b5c6431c Test: testSortMinValueScript - use unmappedType to handle slow propagation of mapping 2015-02-10 19:59:51 -05:00
javanna 9c847db8af Percolate api: support encoded body as query string param consistently
The percolate api doesn't parse the encoded body provided as `source` query string parameter, when percolating an existing document. Fixed and added REST test that would have caught this since we randomly use GET + encoded `source` param instead of GET + request body in our java runner (the perl runner does the same too).

Closes #9628
2015-02-11 08:53:04 +11:00
Ryan Ernst b3474f6b25 Mappings: Remove ability to set path for _id and _routing on 2.0+ indexes
_id and _routing now no longer support the 'path' setting on indexes
created with 2.0.  Indexes created before 2.0 still support this
setting for backcompat.

closes #6730
2015-02-10 10:53:44 -08:00
Igor Motov 6544890e14 Internal: promptly cleanup updateTask timeout handler
Improve cleanup of updateTask timeout handlers. The timeout handlers should be removed as soon as a corresponding update task is processed. Otherwise, timeout handlers might keep old updateTasks and all objects that they are pointing to in memory for the duration of timeout (15 minutes by default).

Fixes #9621
2015-02-10 13:00:40 -05:00
Nicholas Knize 5b96595854 [GEO] Updating javadoc for XShapeCollection
XShapeCollection has an incorrect description left over from when the relate method was overridden. This one line commit corrects the description.
2015-02-10 07:23:00 -06:00
Simon Willnauer 401e6c6b06 [ENGINE] Factor out settings updates from Engine
The engine is already pretty complex, it's still confulated with
code that doesn't necessarily belong there. Updateing the settings from
the settings service can be done on the level above. This commit cleans up
the settings code in the engine and moves it to the IndexShard.
2015-02-10 12:59:12 +01:00
Nicholas Knize c9893ba0c2 [GEO] Correct bounding box logic for GeometryCollection type
"The OpenGIS Abstract Specification: An Object Model for Interoperable Geoprocessing" published by the OGC defines "The boundary of a geometric object is a set of geometric objects of the next lower dimension." The bounding box of a GeometryCollection is therefore the set of bounding rectangles derived from the geometric objects of the next lower dimension. This commit updates the computeBoundingBox and relate methods for the ShapeCollection base class to correctly determine the prefixTree detail level used in Lucene's FilterCellIterator.

closes #9360
2015-02-09 17:40:05 -06:00
Simon Willnauer de7461efd0 [ENGINE] Close Engine immediately if a tragic event strikes.
Until lately we couldn't close the engine in a tragic event due to
some the lock order and all it's complications. Now that the engine
is much more simplified in terms of having a single IndexWriter etc.
we don't necessarily need the write-lock on close anymore and can
easily just close and continue.
2015-02-09 23:21:53 +01:00
Lee Hinman 622d2c8e42 [CORE] Refactor InternalEngine into AbstractEngine and classes
InternalEngine contains a number of inner classes that it uses, however,
this makes the class overly large and hard to extend. In order to be
able to easily add other Engines (such as the ShadowEngine), these
helping methods have been extracted into an AbstractEngine class. The
classes that were previously in `InternalEngine` have been moved to
separate classes, which will allow for better unit testing as well.

None of the functionality of InternalEngine has been changed, this is
only refactoring.

Note that this is a change I originally made on my shadow-replica
branch, however it is easier to review piecemeal so I extracted it into
a separate PR.
2015-02-09 13:28:55 -07:00
Igor Motov dcc15a6460 Test: add wait for nodes to restorePersistentSettingsTest
Sometimes by the time update settings is called the second node is not in the cluster yet. As a result change of minimum master node settings to 2 is ignored making this test to fail.
2015-02-09 12:51:48 -05:00
Alfredo Serafini e607e53591 Update span-multi-term-query.asciidoc
added wildcard to the list of possible nested queries

Closes #9586
2015-02-09 16:01:46 +01:00
Christoph Büscher d2f852a274 Aggregations: Add 'offset' option to date_histogram, replacing 'pre_offset' and 'post_offset'
Add offset option to 'date_histogram' replacing and simplifying the previous 'pre_offset' and 'post_offset' options.
This change is part of a larger clean up task for `date_histogram` from issue #9062.
2015-02-09 14:03:28 +01:00
Simon Willnauer 93df178469 Remove unneeded bwc code 2015-02-09 11:45:56 +01:00
Alexander Reelsen 98a2482825 Testing: Add test rule to repeat tests on binding exceptions
Due to the possibility of ports being already used when choosing a
random port, it makes sense to simply repeat a unit test upon a bind
exception.

This commit adds a junit rule, which does exactly this and does not
require you to change the test code and add loops.

Closes #9010
2015-02-09 11:18:00 +01:00
Simon Willnauer b3b1a11a64 Mapping update task back references already closed index shard
In the ShardRecoveryHandler we issue cluster update tasks to update the
mapping. The annonymous inner class backreferences the ShardRecoveryHandler
which holds a potentially large IndexShard object (which references buffers & caches etc)
If the queue of update tasks piles up and recoveries get cancled and/or shards are closed
the ShardRecoveryHandler can't be GCed. This commit moves the update task into a static
inner class to allos the GC to do its job.
2015-02-09 09:21:55 +01:00
Boaz Leskes 1167beed48 Test: testRelocationWithBusyClusterUpdateThread - listener should wait for replicas to be created 2015-02-07 10:54:21 +01:00
Boaz Leskes e684d7fde4 Logging: improve logging messages added in #9562 & #9562
Closes #9603
2015-02-06 22:26:16 +01:00
Robert Muir 66b5ed86f7 fix typo 2015-02-06 09:07:08 -05:00
Robert Muir 9c9b5c27d3 Upgrade to Lucene r1657571.
Closes #9587

Squashed commit of the following:

commit 23ac91dca4b949638ca1d3842fd6db2e00ee1d36
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Feb 5 18:42:28 2015 +0100

    Do not compute scores if aggregations do not need it (like top_hits) or use a script (which might compute scores).

commit 51262fe2681c067337ca41ab88096ef80a2e8ebb
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Feb 5 15:58:38 2015 +0100

    Fix more compile errors.

commit a074895d55b8b3c898d23f7f5334e564d5271a56
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Feb 5 09:31:22 2015 -0500

    fix a few more obvious ones

commit 399c41186cb3c9be70107f6c25b51fc4844f8fde
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Feb 5 09:28:32 2015 -0500

    fix some collectors and queries

commit 5f46c2f846c5020d5749233b71cbe66ae534ba51
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Feb 5 09:24:24 2015 -0500

    upgrade to lucene r1657571
2015-02-06 08:53:20 -05:00
Boaz Leskes 487ef80c35 Test: testRelocationWithBusyClusterUpdateThread - CountDownLatch.countDown should be await 2015-02-06 14:31:19 +01:00
Boaz Leskes 45ecb49a09 Test: testRelocationWithBusyClusterUpdateThread - use cluster state listener instead of assertBusy 2015-02-06 14:27:27 +01:00
Boaz Leskes 1b7920f202 Test: add document indexing back to testCancellationCleansTempFiles
It was lost during a merge conflict in 796aa5c3fe2424390a8edee604cd292b8afdf514
2015-02-06 12:54:30 +01:00
Christoph Büscher dfc0496fc0 Add warning to settings documentation about setting number_of_replicas on a closed index
Issue #9566 raises the point that setting the number of shards on a closed index can lead to this index not beeing able to open again. This change in documentation is ment to warn the user about this issue.
2015-02-06 12:09:24 +01:00
Boaz Leskes 23022227d4 Recovery: add a timeout to local mapping change check
After phase1 of recovery is completed, we check that all pending mapping changes have been sent to the master and processed by the other nodes. This is needed in order to make sure that the target node has the latest mapping (we just copied over the corresponding lucene files). To make sure we do not miss updates, we do so under a local cluster state update task. At the moment we don't have a timeout when waiting on the task to be completed. If the local node update thread is very busy, this may stall the recovery for too long. This commit adds a timeout (equal to `indices.recovery.internal_action_timeout`) and upgrade the task urgency to `IMMEDIATE`. If we fail to perform the check, we fail the recovery.

Closes #9575
2015-02-06 10:06:47 +01:00
Ryan Ernst c6968883a7 Mappings: Remove support for new indexes using path setting in
object/nested fields or index_name in any field

Backcompat is still here for indexes created before 2.0.

closes #6677
2015-02-05 12:44:43 -08:00
Boaz Leskes 9362ba200d Gateway: add logging around gateway shard allocation
This commit adds more logs around the gateway shard allocation. Any errors while reaching out to nodes to list the local shards are logged in `WARN`. Shard info loading time is logged under DEBUG. Also, we log a `WARN` message if an exception forces a full checksum check during reading the store metadata

Closes #9562
2015-02-05 18:06:33 +01:00
Adrien Grand 95f46f1212 Docs: Use the new experimental annotation.
We now have a very useful annotation to mark features or parameters as
experimental. Let's use it! This commit replaces some custom text warnings with
this annotation and adds this annotation to some existing features/parameters:
 - inner_hits (unreleased yet)
 - terminate_after (released in 1.4)
 - per-bucket doc count errors in the terms agg (released in 1.4)

I also tagged with this annotation settings which should either be not needed
(like the ability to evict entries from the filter cache based on time) or that
are too deep into the way that Elasticsearch works like the Directory
implementation or merge settings.

Close #9563
2015-02-05 15:29:45 +01:00
Adrien Grand 3a486066fd Docs: Remove the experimental status of the cardinality and percentiles(-ranks) aggregations
These aggregations are not experimental anymore but some of their parameters
still are:
 - `precision_threshold` and `rehash` on `cardinality`
 - `compression` on percentiles(-ranks)

Close #9560
2015-02-05 15:18:40 +01:00