Commit Graph

9424 Commits

Author SHA1 Message Date
Boaz Leskes 246b2583a3 [Test] ElasticsearchIntegrationTest.clearDisruptionScheme should test if the current cluster is internal
When running on a non-internal cluster the function is a noop.
2014-09-01 21:14:30 +02:00
javanna 0d49a8ec76 [TEST] remove global scope mention from ElasticsearchIntegrationTest#buildTestCluster
The global cluster gets created from a static block and shared through all tests in the same jvm. The `buildTestCluster` method can't get called passing in `Scope.GLOBAL`, hence removed its mention from it as it might be misleading. The only two scopes supported within the `buildTestCluster` method are `SUITE` and `TEST`.
2014-09-01 18:34:32 +02:00
Boaz Leskes 598854dd72 [Discovery] accumulated improvements to ZenDiscovery
Merging the accumulated work from the feautre/improve_zen branch. Here are the highlights of the changes:

__Testing infra__
- Networking:
    - all symmetric partitioning
    - dropping packets
    - hard disconnects
    - Jepsen Tests
- Single node service disruptions:
    - Long GC / Halt
    - Slow cluster state updates
- Discovery settings
    - Easy to setup unicast with partial host list

__Zen Discovery__
- Pinging after master loss (no local elects)
- Fixes the split brain issue: #2488
- Batching join requests
- More resilient joining process (wait on a publish from master)

Closes #7493
2014-09-01 16:13:57 +02:00
Boaz Leskes 34f4ca763c [Cluster] Refactored ClusterStateUpdateTask protection against execution on a non master
Previous implementation used a marker interface and had no explicit failure call back for the case update task was run on a non master (i.e., the master stepped down after it was submitted). That lead to a couple of instance of checks.

This approach moves ClusterStateUpdateTask from an interface to an abstract class, which allows adding a flag to indicate whether it should only run on master nodes (defaults to true). It also adds an explicit onNoLongerMaster call back to allow different error handling for that case. This also removed the need for the  NoLongerMaster.

Closes #7511
2014-09-01 15:57:07 +02:00
Boaz Leskes 596a4a0735 [Internal] Extract a common base class for (Master|Nodes)FaultDetection
They share a lot of settings and some logic.

Closes #7512
2014-09-01 15:51:26 +02:00
Britta Weber 889db1c824 [TEST]: remove field_value_factor , was only added 1.2 2014-09-01 15:08:45 +02:00
Britta Weber 40d86a630b Tests: wait for yellow instead of green 2014-09-01 12:26:14 +02:00
javanna ab57d4a002 [TEST] Unify the randomization logic for number of shards and replicas
We currently have two ways to randomize the number of shards and replicas: random index template, that stays the same for all indices created under the same scope, and the overridable `indexSettings` method, called by `createIndex` and `prepareCreate` which returns different values each time.

Now that the `randomIndexTemplate` method is not static anymore, we can easily apply the same logic to both. Especially for number of replicas, we used to have slightly different behaviours, where more than one replicas were only rarely used through random index template, which gets now applied to the `indexSettings` method too (might speed up the tests a bit)

Side note: `randomIndexTemplate` had its own logic which didn't depend on `numberOfReplicas` or `maximumNumberOfReplicas`, which was causing bw comp tests failures since in some cases too many copies of the data are requested, which cannot be allocated to older nodes, and the write consistency quorum cannot be met, thus indexing times out.

Closes #7522
2014-09-01 12:04:24 +02:00
Britta Weber 3f0288fc59 fix typo in class name 2014-09-01 11:43:52 +02:00
Britta Weber c5ff70bf43 function_score: add optional weight parameter per function
Weights can be defined per function like this:

```
"function_score": {
    "functions": [
        {
            "filter": {},
            "FUNCTION": {},
            "weight": number
        }
        ...
```
If `weight` is given without `FUNCTION` then `weight` behaves like `boost_factor`.
This commit deprecates `boost_factor`.

The following is valid:

```
POST testidx/_search
{
  "query": {
    "function_score": {
      "weight": 2
    }
  }
}
POST testidx/_search
{
  "query": {
    "function_score": {
      "functions": [
        {
          "weight": 2
        },
        ...
      ]
    }
  }
}
POST testidx/_search
{
  "query": {
    "function_score": {
      "functions": [
        {
          "FUNCTION": {},
          "weight": 2
        },
        ...
      ]
    }
  }
}
POST testidx/_search
{
  "query": {
    "function_score": {
      "functions": [
        {
          "filter": {},
          "weight": 2
        },
        ...
      ]
    }
  }
}
POST testidx/_search
{
  "query": {
    "function_score": {
      "functions": [
        {
          "filter": {},
          "FUNCTION": {},
          "weight": 2
        },
        ...
      ]
    }
  }
}
```

The following is not valid:

```
POST testidx/_search
{
  "query": {
    "function_score": {
      "weight": 2,
      "FUNCTION(including boost_factor)": 2
    }
  }
}

POST testidx/_search
{
  "query": {
    "function_score": {
      "functions": [
        {
          "weight": 2,
          "boost_factor": 2
        }
      ]
    }
  }
}
````

closes #6955
closes #7137
2014-09-01 11:04:40 +02:00
Britta Weber 9750375412 mappings: keep parameters in mapping for _timestamp, _index and _size even if disabled
Settings that are not default for _size, _index and _timestamp were only build in
toXContent if these fields were actually enabled.
_timestamp, _index and _size can be dynamically enabled or disabled.
Therfore the settings must be kept, even if the field is disabled.
(Dynamic enabling/disabling was intended, see TimestampFieldMapper.merge(..)
and SizeMappingTests#testThatDisablingWorksWhenMerging
but actually never worked, see below).

To avoid that _timestamp is overwritten by a default mapping
this commit also adds a check to mapping merging if the type is already
in the mapping. In this case the default is not applied anymore.
(see
SimpleTimestampTests#testThatUpdatingMappingShouldNotRemoveTimestampConfiguration)

As a side effect, this fixes
- overwriting of paramters from the _source field by default mappings
  (see DefaultSourceMappingTests).
- dynamic enabling and disabling of _timestamp and _size ()
  (see SimpleTimestampTests#testThatTimestampCanBeSwitchedOnAndOff and
  SizeMappingIntegrationTests#testThatTimestampCanBeSwitchedOnAndOff )

Tests:

Enable UpdateMappingOnClusterTests#test_doc_valuesInvalidMappingOnUpdate again
The missing settings in the mapping for _timestamp, _index and _size caused a the
failure: When creating a mapping which has settings other than default and the
field disabled, still empty field mappings were built from the type mappers.
When creating such a mapping, the mapping source on master and the rest of the cluster
can be out of sync for some time:

1. Master creates the index with source _timestamp:{_store:true}
   mapper classes are in a correct state but source is _timestamp:{}
2. Nodes update mapping and refresh source which then completely misses _timestamp
3. After a while source is refreshed again also on master and the _timestamp:{}
   vanishes there also.

The test UpdateMappingOnCusterTests#test_doc_valuesInvalidMappingOnUpdate failed
because the cluster state was sampled from master between 1. and 3. because the
randomized testing injected a default mapping with disabled _size and _timestamp
fields that have settings which are not default.

The test
TimestampMappingTests#testThatDisablingFieldMapperDoesNotReturnAnyUselessInfo
must be removed because it actualy expected the timestamp to remove
parameters when it was disabled.

closes #7137
2014-09-01 10:39:33 +02:00
Boaz Leskes 0e6bb1f28b [Rest] Add the cluster name to the "/" endpoint
The root endpoint returns basic information about this node, like it's name and ES version etc. The cluster name is an important information that belongs in that list.

Closes #7524
2014-09-01 10:05:11 +02:00
Britta Weber 92937c7770 Docs: fix formula typo in documentation of the gauss decay in function_score
closes #7506
2014-09-01 09:46:13 +02:00
Areek Zillur 9df10a07b0 Improved Suggest Client API:
- Added SuggestBuilders (analogous to QueryBuilders)
 - supporting term, phrase, completion and fuzzyCompletion suggestion builders
- Added suggest(SuggestionBuilder) to SuggestRequest
   - previously only suggest(BytesReference) was supported

closes #7435
2014-08-31 21:55:03 -04:00
Boaz Leskes 7fb9e5e28e [Test] make testNoMasterActions more resilient 2014-08-30 18:34:20 +02:00
Martijn van Groningen 2ba4e35cde Aggregations: The nested aggregator should iterate over the child doc ids in ascending order.
The reverse_nested aggregator requires that the emitted doc ids are always in ascending order, which is already enforced on the scorer level,
but this also needs to be enforced on the nested aggrgetor level otherwise incorrect counts are a result.

Closes #7505
Closes #7514
2014-08-29 23:04:17 +02:00
Boaz Leskes d8a5ff0047 [Internal] introduce ClusterState.UNKNOWN_VERSION constant
Used as null value for cluster state versions.
2014-08-29 22:57:23 +02:00
Boaz Leskes 75795e44c1 [Tests] add different node name prefix for the different cluster type
During a test run we have a global shared cluster and potentially a suite level or even a test level cluster running. All of those share the same node name pattern (node_#). This can be confusing if you're debugging discovery related tests where those nodes from the different clusters potentially interact (and reject each other). This commit gives each cluster type a unique prefix to make tracing and log filtering simpler.

Closes #7518
2014-08-29 21:33:54 +02:00
Simon Willnauer 4473cdc503 [TEST] Remove unused plugin isolation leftover 2014-08-29 21:29:48 +02:00
Simon Willnauer 0d07917e99 [TEST] Stabelize SimpleRecoveryLocalGatewayTests#testReusePeerRecovery 2014-08-29 21:29:01 +02:00
Lee Hinman 1e21f27874 [TEST] fix off-by-one error in BigArrays tests
Comparisons for the BigArrays breaker use "greater than" instead of
"greater than or equal", which was never an issue before because the
test size was not right on a page boundary. A test with an exactly
divisible page boundary (4mb exactly in this case) caused the sizes to
be equal to, but not exceed, the limit, and never break.

The limit should be smaller than the test increments the breaker anyway.
2014-08-29 17:17:03 +02:00
Boaz Leskes ed5b2e0e35 Add an assertion to ZenDiscovery checking that local node is never elected if pings indicate an active master 2014-08-29 17:07:24 +02:00
Boaz Leskes 680fb36637 [Discovery] Add try/catch around repetitive onSuccess calls 2014-08-29 17:03:08 +02:00
Adrien Grand 172a40c55e Docs: Add javadocs to the client-side aggregation APIs. 2014-08-29 16:36:43 +02:00
markharwood 536d3ffed0 Highlighter Javadocs 2014-08-29 16:26:41 +02:00
Martijn van Groningen f416ed4949 Docs: added missing jdocs for the percolate client classes.
Also made constructors were possible package protected
and removed some useless getters in percolator source builder.
2014-08-29 16:26:41 +02:00
Simon Willnauer c10ef110ae [DOCS] Added JavaDocs for ClusterAdminClient, IndicesAdminClient and Warmer API 2014-08-29 16:26:41 +02:00
markharwood 1687c5ad51 Completion suggestion javadocs 2014-08-29 16:26:41 +02:00
Simon Willnauer 1bb0677df7 [CORE] Don't update indexShard if it has been removed before
Today we have logic that removes a shard from the indexservice if
the shard has changed ie. from replica to primary or if it's recovery
source vanished etc. This can cause shards from been not allocated at
all on a nodes causeing delete requests to timeout since we were waiting
for shards on nodes that got dropped due to a IndexShardMissingException

Closes #7509
2014-08-29 15:16:22 +02:00
markharwood c0aef4adc4 Suggest API - bugs with encoding multiple levels of geo precision.
1) One issue reported by a user is due to the truncation of the geohash string. Added Junit test for this scenario
2) Another suspect piece of code was the “toAutomaton” method that only merged the first of possibly many precisions into the result.

Closes #7368
2014-08-29 13:41:35 +01:00
Adrien Grand 8e1d3d56b3 Docs: Replace added[1.4.0] with coming[1.4.0] since 1.4 is not released yet. 2014-08-29 11:57:22 +02:00
Simon Willnauer 88aec9e3c0 [TEST] Fix per-segment / per-commit exclude logic in CorruptFileTest 2014-08-29 11:43:52 +02:00
Lee Hinman b2827a09a9 [TEST] add AwaitsFix for testTranslogChecksums since it may cause OOME
if the size is corrupted
2014-08-29 10:11:50 +02:00
Boaz Leskes d15909716b [Internal] moved ZenDiscovery setting to use string constants 2014-08-29 09:46:28 +02:00
Michael Brackx 0fd3ef6df0 Client: Make the query builder nullable in filteredQuery.
Close #7398
2014-08-29 09:40:38 +02:00
Simon Willnauer d7a068d02c [TEST] Exclude per commit files rather than only segments_N
When we corrupt a file in the snapshot/restore case we have to corrupt
a per-segment file. The .del file might change with the commit / flush
that is triggered by the snapshot operation.
2014-08-29 09:22:03 +02:00
Boaz Leskes 183ca37dfa Code style improvement 2014-08-29 09:01:05 +02:00
Simon Willnauer 91b8498cec [TEST] Port can have more or less than 4 digits 2014-08-29 08:57:35 +02:00
Martijn van Groningen c55341bf51 Core: Remove the warmer listener when the FixedBitSetFilterCache gets closed. 2014-08-28 20:58:34 +02:00
Martijn van Groningen 4c690fae47 Scan: Use ConcurrentHashMap instead of HashMap, because the readerStates is accessed by multiple threads during the entire scroll session.
Closes #7499
Closes #7478
2014-08-28 16:36:17 +02:00
Philip Wills a3c4137079 Aggregations: Encapsulate AggregationBuilder name and make getter public
Close #7425
2014-08-28 16:34:41 +02:00
Brian Murphy c165e640fc Indexed Scripts/Templates : Change the default auto_expand to 0-all
This commit changes the auto_expand_replicas setting for the ````.scripts```` index to
0-all from 1-all.
2014-08-28 15:31:44 +01:00
Brian Murphy f44bb502ee Indexed Scripts/Templates : Fix .script index template.
This commit makes the default number of shards for the .scripts index to ````1````, it also
forces the auto_expand replicas to ````1-all````. This change means that script index GET requests to load
scripts from the index should always use the local copy of the scripts index, preventing any network traffic or calls
on script GET.
2014-08-28 14:54:24 +01:00
javanna 88839ec546 [TEST] apply default settings by calling super.nodeSettings method when providing test specific methods 2014-08-28 15:35:35 +02:00
javanna a0e9532dca [TEST] make default settings don't override test specific settings 2014-08-28 15:35:34 +02:00
javanna 645db6867b [TEST] apply default settings before test specific ones to external nodes in bw comp tests, otherwise the defaults win all the time 2014-08-28 15:35:34 +02:00
Britta Weber 44dbd9b0c9 test: write heap dump to log folder
Per default the heap dump is written to target/JX/pidXYZ.hprof
In order to keep them when a new test is is started, they
should be written to log folder which is not cleared in a new
test run.
Heap dump location can be set with -Dtests.heapdump.path=/path/to/heapdump

closes #7452
2014-08-28 14:51:10 +02:00
Lee Hinman 09816fdf57 Validate create index requests' number of primary/replica shards
Fixes #7495
2014-08-28 14:20:32 +02:00
Simon Willnauer cc37ae13bc [CORE] Make network interface iteration order consistent
Today the iteration order of the interfaces might change across JVMs
this commit cleans up the NetworkUtils class and attempts to ensure
consistent iteration order across JVMs.
2014-08-28 12:35:56 +02:00
Simon Willnauer c93e6e3f67 [TEST] Fix RandomScoreFunctionTests#testConsistentHitsWithSameSeed 2014-08-28 12:31:47 +02:00