Commit Graph

9951 Commits

Author SHA1 Message Date
Zachary Tong f5b2dfd052 Aliases: Throw exception if index is null or missing when creating an alias
Fixes a bug where alias creation would allow `null` for index name, which thereby
applied the alias to _all_ indices.  This patch makes the validator throw an
exception if the index is null.

```bash
POST /_aliases
{
   "actions": [
      {
         "add": {
            "alias": "empty-alias",
            "index": null
         }
      }
   ]
}
```
```json
{
   "error": "ActionRequestValidationException[Validation Failed: 1: Alias action [add]: [index] may not be null;]",
   "status": 400
}
```

The reason this bug wasn't caught by the existing tests is because
the old test for nullness only validated against a cluster which had
zero indices.  The null index is translated into "_all", and since
there are no indices, this fails because the index doesn't exist.
 So the test passes.

However, as soon as you add an index, "_all" resolves and you get the
situation described in the original bug report:  null index is
accepted by the alias, resolves to "_all" and gets applied to everything.

The REST tests, otoh, explicitly tested this bug as a real feature and therefore
passed.  The REST tests were modified to change this behavior.

Fixes #7863
2014-10-27 14:39:01 -04:00
Alex Ksikes 0be5c60bce MLT Query: use ParseField#withAllDeprecated for percent_terms_to_match
Also the parameter was deprecated but not removed so we keep it in the doc and
mark it as deprecated ...

Closes #8241
2014-10-27 17:35:06 +01:00
Alex Ksikes 991f3e2cd3 Docs: fix tags for dfs and new like parameter 2014-10-27 15:42:44 +01:00
Clinton Gormley fbd0403a6f Documented that HTTP pipelining is not supported 2014-10-27 14:49:48 +01:00
Simon Willnauer 9a5264660d Remove unused imports 2014-10-27 14:26:04 +01:00
Simon Willnauer 5190cee7fc [CORE] remove usage of Directory#fileExists
This commit removes the usage of #fileExists which has been
deprecated long ago an can be a source of race conditions.
2014-10-27 14:21:25 +01:00
Martijn van Groningen 35ede09511 Core: Cleanup non nested filter to not flip the bits in the FixedBitSet returned by the wrapped filter.
If the filter producing the FBS were to be cached then it would flip bits each time the NonNestedDocsFilter was executed.
In our case we cache the NonNestedDocsFilter itself so that didn't happen each time this filter was used.
However the hashcode of NonNestedDocsFilter and NestedDocsFilter was the same, since NonNestedDocsFilter directly used NestedDocsFilter#hashCode()

Closes #8227
Closes #8232
2014-10-27 13:57:52 +01:00
Boaz Leskes da70b5c847 Test: remove wait on events from multipleNodesShutdownNonMasterNodes
The test validates that minimum master node is honored after shutting down nodes. When nodes are restarted we may go through a couple of master election of a master is elected and  discovers some of the old nodes left before all new node have joined. Processing that node failure the master de-elects it self, which is fine because we will start a new master election. Test however runs a clusterHealth call with a  wait for event. This is implemented using a cluster state update task which fails when the master steps down. Longer term fix requires a rewrite of the cluster health API code but a quick fix is not check for events (not needed for this test).
2014-10-27 13:31:27 +01:00
Simon Willnauer a12b34c36c [TEST] Beef up test and replace deprecated API 2014-10-27 13:05:12 +01:00
Adrien Grand 7ea490dfd1 Aggregations: Return the sum of the doc counts of other buckets.
This commit adds a new field to the response of the terms aggregation called
`sum_other_doc_count` which is equal to the sum of the doc counts of the buckets
that did not make it to the list of top buckets. It is typically useful to have
a sector called eg. `other` when using terms aggregations to build pie charts.

Example query and response:

```json
GET test/_search?search_type=count
{
  "aggs": {
    "colors": {
      "terms": {
        "field": "color",
        "size": 3
      }
    }
  }
}
```

```json
{
   [...],
   "aggregations": {
      "colors": {
         "doc_count_error_upper_bound": 0,
         "sum_other_doc_count": 4,
         "buckets": [
            {
               "key": "blue",
               "doc_count": 65
            },
            {
               "key": "red",
               "doc_count": 14
            },
            {
               "key": "brown",
               "doc_count": 3
            }
         ]
      }
   }
}
```

Close #8213
2014-10-27 12:11:26 +01:00
tlrx 96e62b3c1b [TESTS Fix wrong assertion in test introduced by #8128 2014-10-27 11:11:54 +01:00
tlrx e1c75bae87 Cat API: Add node name to _cat/recovery
Add source_node and target_node fields to the recovery cat API. Also fixed and updated the documentation which was not complete concerning fields names.

Closes #8041
2014-10-27 09:47:26 +01:00
Simon Willnauer baa1e020ce [TEST] Add test for #8226 2014-10-27 09:19:29 +01:00
Nirmal Chidambaram b8d2e2cd29 Core: Validates bool values in yaml for node settings
- Added parseBooleanExact in booleans which throws exception in case of
  parse failure
- Used ParseExact in static's to make it consistent

+ code review fixes

- Added unit test
- Changed Exception Type to ElasticSearchIllegalArg from Parse
- used isExplicit*

Closes #8097
2014-10-27 11:27:41 +09:00
Boaz Leskes 7f14674ff5 Recovery: RecoveriesCollection.findRecoveryByShard should call recoveryStatus.tryIncRef before accessing fields
The function iterates over a snapshot of on going recoveries and thus may access RecoveryStatus objects that are already finished.

Closes #8231
2014-10-26 21:10:17 +01:00
Simon Willnauer bb7ad4ab96 [TEST] NettyTransportMultiPortIntegrationTests used wrong random to
initialize ports.

JUnit uses an instance per test which caused the prot range to be
initialized twice since suite level tests are not configured in a
different context.
2014-10-26 13:07:32 +01:00
Simon Willnauer 58c292dc14 add missing imports 2014-10-26 10:09:18 +01:00
Simon Willnauer 75595bd0e6 [TEST] `Scope.SUITE` is not reproducible due to late cluster initialization
The cluster for `Scope.SUITE` tests must be initialize in a static manner
before the first test runs otherwise the random context used to initialize
the cluster is taken from tests randomness rather than the suites randomness.
This means test clusters will have different setups if only a single test is
executed or even the test might have a entirely different random sequence.
2014-10-26 09:42:35 +01:00
Boaz Leskes b82e2c7083 Test: testRecoverFromPreviousVersion should refresh from new nodes if running against v<1.3.0
This is due to a bug in older version causing refreshes to potentially be missed due to relocations #6545

Also:
- Changed test to keep track of ids and report missing ones.
- Removed total count check from assertSearchHits in order to enable per id checks in cased of a mismatch
- Added a printable unique id part the ids of dummy documents added by indexRandom. The current random unicode id
sometimes prints as ???? to the logs making them hard to trace
2014-10-25 23:08:42 +02:00
Alex Ksikes 4da407a869 MLT Query: versatile 'like' parameter
The MLT query has a lot of parameters. For example, a set of documents is
specified with either `like_text`, `ids` or `docs`, with at least one
parameter required. This commit groups all the document specification
parameters under one called `like`. The syntax is described below and could
easily be extended to allow for new means of specifying document input. The
`like_text`, `ids` and `docs` parameters are deprecated.

As a single piece text:

{
  "query": {
    "more_like_this": {
      "like": "some text here"
    }
  }
}

As a single item:

{
  "query": {
    "more_like_this": {
      "like": {
        "_index": "imdb",
        "_type": "movies",
        "_id": "88247"
      }
    }
  }
}

Or as a mixture of all:

{
  "query": {
    "more_like_this": {
      "like": [
        "Some random text ...",
        {
          "_index": "imdb",
          "_type": "movies",
          "_id": "88247"
        },
        {
          "_index": "imdb",
          "_type": "movies",
          "doc": {
            "title": "Document with an artificial title!"
          }
        }
      ]
    }
  }
}

Closes #8039
2014-10-25 11:04:51 +02:00
Simon Willnauer c95545a64e Catch both FileNotFound & NoSuchFileException 2014-10-25 09:34:18 +02:00
David Pilato 62d8b7ab97 Docs: rolling upgrade process seems incorrect
When reading the [rolling upgrade process](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-upgrade.html#rolling-upgrades), you can see that we wrote:

* disable allocation
* upgrade node1
* upgrade node2
* upgrade node3
* ...
* enable allocation

That won't work as after a node has been removed and restarted, no shard will be allocated anymore.
So closing node2 and remaining nodes, won't help to serve index and search request anymore.

We should write:

* disable allocation
* upgrade node1
* enable allocation
* wait for shards being recovered on node1
* disable allocation
* upgrade node2
* enable allocation
* wait for shards being recovered on node2
* disable allocation
* upgrade node3
* enable allocation
* wait for shards being recovered on node3
* disable allocation
* ...
* enable allocation

I think this documentation update should go in 1.3, 1.4, 1.x and master branches.

Closes #8218
Closes #7973.
2014-10-24 16:45:42 +02:00
Marcin Mikosik ed86d925cd Docs: fixed typo in documentation
Closes #8205
2014-10-24 15:27:31 +02:00
markharwood d12ae196af Bulk indexing: Fix 8125 hanged request when auto create index is off.
If a bulk request contains a mix of indexing requests for an existing index and one that needs to be auto-created but a cluster configuration prevents the auto-create of the new index the ingest process hangs. The exception for the failure to create an index was not caught or reported back properly. Added a Junit test to recreate the issue and the associated fix is in TransportBulkAction.

Closes #8125
2014-10-24 13:50:37 +01:00
Alex Ksikes a87a23447f ParseField: Support for when all fields are deprecated
Closes #8067
2014-10-24 14:35:38 +02:00
Alex Ksikes 437c5d0d3b Term Vectors: removed Nullable annotation on a primitive type 2014-10-24 13:49:42 +02:00
Simon Willnauer c09af6df61 [STORE] Don't catch FNF/NSF exception when reading metadata
When reading metadata we do catch FileNotFound and NoSuchFileExceptions
today, log the even and return an empty metadata object. Yet, in some cases
this might be the wrong thing todo ie. if a commit point is provided these
situations are actually an error and should be rethrown. This commit
pushes the responsiblity to the caller to handle this exception.

Closes #8207
2014-10-24 12:19:30 +02:00
Simon Willnauer 4a14c635c8 [CACHE] Expose concurrency_level on all caches
The concurrency level allows to configure the cache internal segments
used to cache data. This can have direct impact on evicition rates since
memory bound caches are equally divided into segments which can cause
early evictions if cache entries are not well balanced.

Relates to #7836
2014-10-24 11:45:13 +02:00
Simon Willnauer 347ce36654 [UTILITIES] Introduce a RefCounted interface and basic impl
We already have two places duplicating this rather hairy logic, this
commit intorduces a new RefCoutned interace and an abstract implementation
that can be used for delegation. It factors out all the reference counting
and adds single and multithreaded test for it.

Closes #8210
2014-10-24 10:33:54 +02:00
Boaz Leskes f72e0c89f7 test: RelocationTests.testCancellationCleansTempFiles may fail due to io errors while check temp files are deleted
Windows can throw NoSuchFileException when using File.walkFileTree and deleting files concurrently. This commit changes IO exceptions into assertion error so that assertBusy will wait for them as well.
2014-10-24 09:49:18 +02:00
Michael McCandless ec3a473a00 Test: additional logging 2014-10-23 11:30:57 -04:00
Boaz Leskes 24bc8d331e Recovery: refactor RecoveryTarget state management
This commit rewrites the state controls in the RecoveryTarget family classes to make it easier to guarantee that:
- recovery resources are only cleared once there are no ongoing requests
- recovery is automatically canceled when the target shard is closed/removed
- canceled recoveries do not leave temp files behind when canceled.

Highlights of the change:
1) All temporary files are cleared upon failure/cancel (see #7315 )
2) All newly created files are always temporary
3) Doesn't list local files on the cluster state update thread (which throw unwanted exception)
4) Recoveries are canceled by a listener to IndicesLifecycle.beforeIndexShardClosed, so we don't need to explicitly call it.
5) Simplifies RecoveryListener to only notify when a recovery is done or failed. Removed subtleties like ignore and retry (they are dealt with internally)

Closes #8092 , Closes #7315
2014-10-23 15:06:02 +02:00
Lee Hinman 1557c34f2c [TESTS] Change test node watermarks for DiskThresholdDecider
Ensures that we can still run tests if the machine running them does not
have enough free disk space to be below the high watermark
2014-10-23 14:42:06 +02:00
Simon Willnauer d5c0a49620 [ROUTING] Add rebalance enabled allocation decider
This commit adds the ability to enable / disable relocations
on an entire cluster or on individual indices for either:

 * `primaries` - only primaries can rebalance
 * `replica` - only replicas can rebalance
 * `all` - everything can rebalance (default)
 * `none` - all rebalances are disabled

similar to the allocation enable / disable functionality.

Relates to #7288
2014-10-23 14:07:13 +02:00
Simon Willnauer ed798296a5 [SEARCH] Passing fieddata_fields as a non array causes OOM
If `fielddata_fields` are passed as a simple value instead of an array
we end up in an infinite loop createing parsed elements with null
values.
This commit validates the incoming token

Closes #8203
2014-10-23 14:04:42 +02:00
Alex Ksikes c13f5f21de Term Vectors: support for distributed frequencies
Adds distributed frequencies support for the Term Vectors API. A new parameter
called `dfs` is introduced which defaults to `false`.

Closes #8144
2014-10-23 13:59:59 +02:00
Lee Hinman 19514a2ef4 Enable ClusterInfoService by default
Since we enabled the disk threshold decider by default, we need to
enable the cluster info service so that disk usages and shard sizes can
be gathered also.

Adds a test that checks that we are gathering information by default.
2014-10-23 13:48:48 +02:00
Randy D. Wallace Jr 4539e7adc0 Add EnhancedPatternLayout to logging.yml options 2014-10-23 11:42:51 +01:00
Brian Murphy fb4a32a398 TEST : Disable throttleStats test
This test has been intermittently failing, disabling while I dig into it.
2014-10-23 09:11:10 +01:00
Ryan Ernst 87f41b4c4e Tests: Improve range tests to check inclusive/exclusive on ends.
closes #8199
2014-10-22 15:06:00 -07:00
Ron DuPlain 1664355f32 Startup: Add ES_HOME to ES_INCLUDE search path
With this change, the elasticsearch script can be linked to another path
without having to set ES_INCLUDE to match the installation path.
Previously, the elasticsearch would find ES_HOME correctly even if linked
but could not find the include script, and finding it would be expected
behavior to me based on its current search path.

Closes #4958
2014-10-22 09:37:09 -07:00
Clinton Gormley 5b5c032dee Docs: Fixed typo in MissingFilterBuilder
Closes #8185
2014-10-22 17:34:58 +02:00
Brian Murphy 2ebcbbc66b TEST : Force throttle test to use tiered merge
This commit fixes the failing throttle test by forcing the use of the tiered merge policy.
2014-10-22 16:25:39 +01:00
Brian Murphy 7333694830 Stats : Add time in index throttle to stats.
This commit adds throttle stats to the indexing stats and uses a call back from InternalEngine to manage the stats.
Also includes updates the IndexStatsTests to test for these new stats.
Stats added :
```
throttle_time_in_millis
is_throttled
```

Closes #7861
2014-10-22 14:07:28 +01:00
Simon Willnauer 40945aebcf [CORE] Don't handle FNF exceptions when reading snapshot
We used to handle FNF exceptions in the store when reading a snapshot.
For instance if we can't open a segments file for a given commit point
we just return an empty metadata object and tracelog the even. This can
cause shards to be false marked as corrupted if a shard is forcefully
removed while a recovery started at the same time. We should in general
bubble up these exceptions and let the caller decided how to handle the
IOExceptions.
2014-10-22 13:02:16 +02:00
Clinton Gormley a8b21f2cd5 Update update-settings.asciidoc
Removed deprecated `cluster.routing.allocation.disable` settings
2014-10-22 12:46:33 +02:00
Clinton Gormley 2d0c440b09 Update cluster.asciidoc
Fixed asciidoc syntax
2014-10-22 12:45:10 +02:00
Lee Hinman f7d227e4d5 Make "noop" request breaker a non-dynamic setting
The issue with making it dynamic is that in the event a cluster is
switched from a noop to a concrete implementation, there may be
in-flight requests, once these requests complete we adjust the breaker
with a negative number and trip an assertion.

This also rarely uses noop breakers in InternalTestCluster
2014-10-22 10:53:53 +02:00
Lee Hinman 26bc940101 Make simple_query_string leniency more fine-grained
Previously, the leniency was on a per-query basis, with each query being
parsed into multiple queries, one for each field. If any one of these
queries failed, the entire query was discarded in the name of being
lenient.

Now query parts will only be discarded if they fail for a particular
field, the entire query is not discarded. This helps when performing a
query over a numeric and string field, as only the sub-queries that are
invalid due to format exceptions will be discarded.

Also moves the `simple_query_string` queries out of SimpleQueryTests and
into a dedicated SimpleQueryStringTests class.

Fixes #7967
2014-10-22 10:31:34 +02:00
Martijn van Groningen ec86d2cd3e Aggregations: the `children` agg didn't take deleted document into account.
The live docs that is passed down was ignored by the filter impl. Now the children filter gets wrapped with ApplyAcceptedDocsFilter, so live docs are actually applied.

Closes #8180
2014-10-22 10:20:25 +02:00