Commit Graph

6180 Commits

Author SHA1 Message Date
Christoph Büscher cdc77648a1 Move analysis helper methods to DirectCandidateGenerator 2016-08-10 16:20:29 +02:00
Christoph Büscher d6e16b6e74 Move getDirectSpellChecker to DirectSpellcheckerSettings 2016-08-10 16:06:05 +02:00
javanna a13dbc12e2 SuggestUtils#analyze: assign success variable a value 2016-08-10 12:57:24 +02:00
javanna a0e32e9dfe move SuggestUtils methods to their respective callers
These methods are called only once, they are then moved to the classes that call them, and become private.
2016-08-10 12:54:38 +02:00
javanna ae78394c03 Remove redundant generics type declaration 2016-08-10 12:28:06 +02:00
javanna 297b2d6739 remove unused methods from SuggestUtils
Parsing code was moved to the builder objects, these methods were left behind unused
2016-08-10 12:28:06 +02:00
javanna 2c44278ce8 [TEST] use ParseField instead of plain strings in query tests 2016-08-10 12:21:25 +02:00
javanna 0a98b5e56e [TEST] make AbstractQueryTestCase#testUnknownObjectException more accurate
testUnknownObjectException used to generate malformed json objects in some cases, due to the existence of arrays as it was not closing the injected object correctly. That is why the test was catching JsonParseException among the exception that are expected to be thrown. That is fixed by tracking where the new object is placed and placing its end object marker to the right level rather than always at the end.

Also introduced a mechanism to explicitly declare objects that won't cause any exception when they get additional objects injected, so that there is no need to override the method anymore as that caused copy pasting of the whole test method. This also makes sure that changes are reflected in tests, as those inner objects are not skipped but we actually check that what is declared is true (no exceptions get thrown when an additional object is added within them.
2016-08-10 11:48:51 +02:00
javanna f221b0ce52 [TEST] inner_hits is now parsed on the coord node, no need to skip such objects in testUnknownObjectException 2016-08-10 11:48:51 +02:00
javanna 57b90cb6ce rename local loop variable ingore->ignore 2016-08-10 10:17:54 +02:00
Adrien Grand 42725e9339 Fix expectations of GeoPointFieldMapperTests.
Closes #19895
2016-08-10 09:30:39 +02:00
Ryan Ernst 38d4382565 Mappings: Support dots in field names in mapping parsing
This change adds support for treating dots in field names found in
mappings as path separators, like was previously done for dynamic
mappings and document parsing.

closes #19443
2016-08-09 14:35:35 -07:00
Ryan Ernst 6efbe54255 Remove alpha5 bwc indexes
We don't have bwc indexes for alpha releases.
2016-08-09 13:25:16 -07:00
Ali Beyad 601602b364 Check restores in progress before deleting a snapshot (#19853)
Currently, when attempting to delete a snapshot, we check
if a snapshot is in progress before proceeding with the
delete. However, we do not check if a restore is taking
place before deleting. This can lead to concurrency issues
where a restore is in progress but the snapshotted files
for the restore are being deleted underneath.

This commit first checks if a restore is in progress and
if so, it prevents the deletion of a snapshot with an
exception.

Note that this is not a complete solution because it is
still possible that a restore of the same snapshot is
started after the deletion commenced but before the
deletion finished. But there is a much smaller window
for this to occur and this commit is a quick way to
check for the common case.
2016-08-09 15:07:09 -05:00
Areek Zillur 16d93e5a53 Merge pull request #19877 from areek/fix/remove_completion_payload
Remove payload option from completion suggester
2016-08-09 15:27:29 -04:00
David Pilato 90dbce9682 Merge branch 'fix/19772-toString' 2016-08-09 20:37:27 +02:00
Lee Hinman 5849c488b5 Merge remote-tracking branch 'dakrone/compliation-breaker' 2016-08-09 11:57:26 -06:00
David Pilato 8bc15039cd Fix after review 2016-08-09 19:44:42 +02:00
Clinton Gormley eac14f6e3d Bumped version to 5.0.0-alpha6 and added bwc indices for alpha5 2016-08-09 18:31:27 +02:00
Lee Hinman 2be52eff09 Circuit break the number of inline scripts compiled per minute
When compiling many dynamically changing scripts, parameterized
scripts (<https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-scripting-using.html#prefer-params>)
should be preferred. This enforces a limit to the number of scripts that
can be compiled within a minute. A new dynamic setting is added -
`script.max_compilations_per_minute`, which defaults to 15.

If more dynamic scripts are sent, a user will get the following
exception:

```json
{
  "error" : {
    "root_cause" : [
      {
        "type" : "circuit_breaking_exception",
        "reason" : "[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead",
        "bytes_wanted" : 0,
        "bytes_limit" : 0
      }
    ],
    "type" : "search_phase_execution_exception",
    "reason" : "all shards failed",
    "phase" : "query",
    "grouped" : true,
    "failed_shards" : [
      {
        "shard" : 0,
        "index" : "i",
        "node" : "a5V1eXcZRYiIk8lecjZ4Jw",
        "reason" : {
          "type" : "general_script_exception",
          "reason" : "Failed to compile inline script [\"aaaaaaaaaaaaaaaa\"] using lang [painless]",
          "caused_by" : {
            "type" : "circuit_breaking_exception",
            "reason" : "[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead",
            "bytes_wanted" : 0,
            "bytes_limit" : 0
          }
        }
      }
    ],
    "caused_by" : {
      "type" : "general_script_exception",
      "reason" : "Failed to compile inline script [\"aaaaaaaaaaaaaaaa\"] using lang [painless]",
      "caused_by" : {
        "type" : "circuit_breaking_exception",
        "reason" : "[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead",
        "bytes_wanted" : 0,
        "bytes_limit" : 0
      }
    }
  },
  "status" : 500
}
```

This also fixes a bug in `ScriptService` where requests being executed
concurrently on a single node could cause a script to be compiled
multiple times (many in the case of a powerful node with many shards)
due to no synchronization between checking the cache and compiling the
script. There is now synchronization so that a script being compiled
will only be compiled once regardless of the number of concurrent
searches on a node.

Relates to #19396
2016-08-09 10:26:27 -06:00
Yannick Welsch 6abcd42a05 Simplify RoutingNodes interface (#19870)
Slims the public interface of RoutingNodes down to 4 methods to update routing entries:
- initializeShard() -> initializes an unassigned shard
- startShard() -> starts an initializing shard / completes relocation of a shard
- relocateShard() -> starts relocation of a started shard
- failShard() -> fails/cancels an assigned shard

In the spirit of PR #19743, where deassociateDeadNodes was moved to its own public method to be only called when nodes have actually left the cluster and not on every reroute step, this commit also removes electPrimariesAndUnassignedDanglingReplicas from AllocationService and folds it into the shard failure logic. This means that an active replica is promoted to primary in the same method where the primary was failed. Previously we would scan in each reroute iteration for active replicas to be promoted to primary.
2016-08-09 17:07:13 +02:00
David Pilato 9b10bb7693 Fix toString method
See https://github.com/elastic/elasticsearch/pull/19773#issuecomment-238564524

Was introduced with #18939
2016-08-09 16:32:05 +02:00
David Pilato d28cc73046 Fix after merge 2016-08-09 12:34:52 +02:00
David Pilato 2a05030e22 Fix after merge 2016-08-09 12:14:50 +02:00
David Pilato 4d272cc9b2 Merge branch 'master' into fix/19772-toString
# Conflicts:
#	core/src/test/java/org/elasticsearch/action/admin/cluster/node/tasks/TransportTasksActionTests.java
2016-08-09 11:53:29 +02:00
Luca Cavanna af5fbcddfc Merge pull request #19871 from javanna/fix/short_query_multiple_fields
Throw exception when multiple field names are provided as part of query short syntax
2016-08-09 11:15:36 +02:00
Adrien Grand c44679d952 Save one utf8 conversion in KeywordFieldMapper. #19867
If a `keyword` field is both indexed and doc-valued, then we will convert the
input string to utf8 bytes twice: once for indexing/storing, and once for doc
values. This commit changes `keyword` fields to compute the utf8 representation
up-front and then feed both the inverted index and doc values with it.

Rather than adding version-based bw compat logic, I broke the `keyword` field
(they are now indexed/stored as a binary field rather than string), which is
fine since we are still on alpha releases for 5.0.
2016-08-09 10:06:30 +02:00
javanna f9a40344b2 Modify term query error when multiple fields are provided to comply with all other queries 2016-08-09 10:01:56 +02:00
javanna 0f54cb69ab Throw parsing error if span term query contains multiple fields in its short version 2016-08-09 09:53:03 +02:00
javanna d4db987825 Add common method that throws exception whenever multiple fields are provided in a query that support one field only
This makes sure that error messages are unified, and makes us save a few lines of code too.
2016-08-09 09:52:28 +02:00
javanna bbf40ca0cf [TEST] test that term query short syntax throws error when multiple fields are provided 2016-08-09 09:50:12 +02:00
Jason Tedor 1aba907ea2 Remove dead OOM handling in engine
Previously, the engine would catch an out of memory error and would try
to handle the error (it would try to fail the engine, and then it would
swallow the out of memory error). Catching the out of memory errors was
removed in 3343ceeae4 so this code path is
not effectively dead. This commit removes this dead code from the
engine.

Relates #19881
2016-08-08 21:59:49 -04:00
Areek Zillur d107141bf6 Remove payload option from completion suggester
The payload option was introduced with the new completion
suggester implementation in v5, as a stop gap solution
to return additional metadata with suggestions.

Now we can return associated documents with suggestions
(#19536) through fetch phase using stored field (_source).
The additional fetch phase ensures that we only fetch
the _source for the global top-N suggestions instead of
fetching _source of top results for each shard.
2016-08-08 16:04:06 -04:00
javanna f547886a9b [TEST] remove AwaitsFix that was fixed with #16615 2016-08-08 20:39:55 +02:00
javanna 9beb82b036 [TEST] remove unused argument from GeoPolygonQueryBuilderTests#randomPolygon 2016-08-08 20:39:55 +02:00
javanna 27a6983646 Throw parsing error if wildcard query contains multiple fields in its short version 2016-08-08 19:42:48 +02:00
javanna 796bc74163 Throw parsing error if regexp query contains multiple fields in its short version 2016-08-08 19:42:37 +02:00
javanna 8f485b3614 Throw parsing error if prefix query contains multiple fields in its short version 2016-08-08 19:42:26 +02:00
javanna 040f9c6be6 Throw parsing error if match query contains multiple fields in its short version 2016-08-08 19:42:14 +02:00
javanna d5316b2783 Throw parsing error if match phrase query contains multiple fields in its short version 2016-08-08 19:42:01 +02:00
javanna cb41f304f2 Throw parsing error if match phrase prefix query contains multiple fields in its short version 2016-08-08 19:41:45 +02:00
javanna 5d238e86f6 Throw parsing error if fuzzy query contains multiple fields in its short version 2016-08-08 19:40:54 +02:00
javanna 1db3c67e31 Throw parsing error if common terms query contains multiple fields in its short version 2016-08-08 19:40:23 +02:00
Colin Goodheart-Smithe bf0e42aaeb #19855 Throw exception when maxBounds greater than minBounds
Throw exception when maxBounds greater than minBounds
2016-08-08 13:17:25 +01:00
Colin Goodheart-Smithe 4735e0a9d3 Throw exception when maxBounds greater than minBounds
The recent changes to the Histogram Aggregator introduced a bug where
an exception would not be thrown if the maxBound of the extended bounds
is less that the minBound. This change fixes that bug.

Closes #19833
2016-08-08 12:09:43 +01:00
Yannick Welsch 180eff14dd Fix issue when relocation source and target routings are failed in same batch update
PR #19715 made AllocationService less lenient, requiring ShardRouting instances that are passed to its applyStartedShards and
applyFailedShards methods to exist in the routing table. As primary shard failures also fail initializing replica shards,
concurrent replica shard failures that are treated in the same cluster state update might not reference existing replica entries
in the routing table anymore. To solve this, PR #19715 ordered the failures by first handling replica before
primary failures. There are other failures that influence more than one routing entry, however. When we have a failed shard entry
for both a relocation source and target, then, depending on the order, either one or the other might point to an out-dated shard
entry. As finding a good order is more difficult than applying the failures, this commit re-adds parts of the ShardRouting
re-resolve logic so that the applyFailedShards method can properly treat shard failure batches.
2016-08-08 11:46:48 +02:00
Nicholas Knize ab0a0cd4d4 fix rogue license header 2016-08-05 23:21:16 -05:00
Nicholas Knize 2d590af593 Deprecate GeoDistance enumerators and remove geo distance script helpers
GeoDistance is implemented using a crazy enum that causes issues with the scripting modules. This commit moves all distance calculations to arcDistance and planeDistance static methods in GeoUtils. It also removes unnecessary distance helper methods from ScriptDocValues.GeoPoints.
2016-08-05 18:42:06 -05:00
Areek Zillur 469eb2546d Merge pull request #19536 from areek/enhancement/completion_suggester_documents
Add support for returning documents with completion suggester
2016-08-05 18:55:08 -04:00
Areek Zillur fee013c07c Add support for returning documents with completion suggester
This commit enables completion suggester to return documents
associated with suggestions. Now the document source is returned
with every suggestion, which respects source filtering options.

In case of suggest queries spanning more than one shard, the
suggest is executed in two phases, where the last phase fetches
the relevant documents from shards, implying executing suggest
requests against a single shard is more performant due to the
document fetch overhead when the suggest spans multiple shards.
2016-08-05 17:51:45 -04:00
Christoph Büscher fbbb633d81 Merge pull request #19825 from cbuescher/register-namedWritables-transportClient
Add NamedWriteables from plugins to TransportClient
2016-08-05 22:51:04 +02:00
Christoph Büscher 6ccb70e1ab Avoid using injector and more test to TransportClientTests 2016-08-05 21:39:44 +02:00
Christoph Büscher 37c433aace Merge pull request #19837
Ensure PutMappingRequest.buildFromSimplifiedDef input are pairs
2016-08-05 20:31:49 +02:00
Christoph Büscher e57f76aa2d Ensure PutMappingRequest.buildFromSimplifiedDef fails when input isn't pairs
The method requires pairs of fieldnames and property arguments and will fail if
the varargs input is an uneven number. We should check this and fail with an
appropriate IllegalArgumentException instead.
2016-08-05 19:25:20 +02:00
Britta Weber 981478e4a9 mute test 2016-08-05 19:10:13 +02:00
Britta Weber 899cddefb6 make ctors protected (#19831)
This is useful if we need an acknowledged instance in a test
2016-08-05 17:13:26 +02:00
Nik Everett 8bebf2599e Add note explaining analysis caching for plugins
```
Elasticsearch doesn't have any automatic mechanism to share these
components between indexes. If any component is heavy enough to
warrant such sharing then it is the Pugin's responsibility to do
it in their {@link AnalysisProvider} implementation. We recommend
against doing this unless absolutely necessary because it can be
difficult to get the caching right given things like behavior
changes across versions.
```

Closes #19814
2016-08-05 11:11:53 -04:00
Christoph Büscher e162935656 Add test to check that plugin NamedWriteables are registerd with TransportClient 2016-08-05 17:08:59 +02:00
Luca Cavanna 4c1a3b9a53 Merge pull request #19791 from javanna/fix/multiple_fields_queries
Query parsers to throw exception when multiple field names are provided
2016-08-05 15:53:35 +02:00
Ali Beyad f59ca9083b Snapshot repository cleans up empty index folders (#19751)
This commit cleans up indices in a snapshot repository when all
snapshots containing the index are all deleted. Previously, empty
indices folders would lay around after all snapshots containing
them were deleted.
2016-08-05 09:39:02 -04:00
Adrien Grand 284b9794c0 Do not parse the created version from the settings every time a field is parsed. #19824
I found it while looking at some jfr telemetry reports from Rally.
2016-08-05 15:35:53 +02:00
Christoph Büscher c32a4324b0 Add NamedWriteables from plugins to TransportClient
Plugins provide NamedWriteables that are added to the
NamedWriteableRegistry. Those are added on Nodes already, the same mechanism is
added to the setup for TransportClient.
2016-08-05 14:11:01 +02:00
javanna 7f0bd56094 [TEST] use expectThrows wherever possible in query builder unit tests 2016-08-05 13:55:18 +02:00
Tanguy Leroux 841d5a210e Update to Jackson 2.8.1
This commit updates Jackson to the 2.8.1 version, which is more strict when it comes to build objects. It also adds the snakeyaml dependency that was previously shaded in jackson libs.

It also closes #18076
2016-08-05 12:26:06 +02:00
javanna 6a5c44a271 fix line length in FuzzyQueryBuilder 2016-08-05 10:58:19 +02:00
javanna 0ac7dd6137 Make query parsing stricter by requiring each parser to stop at END_OBJECT token
Instead of being lenient in QueryParseContext#parseInnerQueryBuilder we check that the token where the parser stopped reading was END_OBJECT, and throw error otherwise. This is a best effort to verify that the parsers read a whole object rather than stepping out in the middle of it due to malformed queries.
2016-08-05 10:58:19 +02:00
javanna 43fee1d7fa Throw parsing error if fuzzy query contains multiple fields
Fuzzy Query, like many other queries, used to parse even when the query referred to multiple fields and the first one would win. We rather throw an exception now instead.
Also added test for short prefix query variant and modified the parsing code to consume the whole query object.
2016-08-05 10:58:19 +02:00
javanna 6d228bb09c [TEST] test that term query throws error when made against multiple fields 2016-08-05 10:58:19 +02:00
javanna 389bd06846 [TEST] check validation error messages in AbstractTermQueryTestCase 2016-08-05 10:58:19 +02:00
javanna 1bcf0722c4 Throw parsing error if span_term query contains multiple fields
Span term Query, like many other queries, used to parse even when the query referred to multiple fields and the first one would win. We rather throw an exception now instead.
Also modified the parsing code to consume the whole query object.
2016-08-05 10:58:19 +02:00
javanna c3dfe0846c Throw parsing error if common terms query contains multiple fields
Common Terms Query, like many other queries, used to parse even when the query referred to multiple fields and the first one would win. We rather throw an exception now instead.
Also added test for short prefix query variant and modified the parsing code to consume the whole query object.
2016-08-05 10:58:19 +02:00
javanna 1e45fd5850 Throw parsing error if match query contains multiple fields
Match Query, like many other queries, used to parse even when the query referred to multiple fields and the first one would win. We rather throw an exception now instead.
Also added test for short prefix query variant and modified the parsing code to consume the whole query object.
2016-08-05 10:58:19 +02:00
javanna f7b3dce4bc Throw parsing error if match_phrase_prefix query contains multiple fields
Match phrase prefix Query, like many other queries, used to parse even when the query referred to multiple fields and the first one would win. We rather throw an exception now instead.
Also added test for short prefix query variant and modified the parsing code to consume the whole query object.
2016-08-05 10:58:19 +02:00
javanna ad8f5e7e4b Throw parsing error if geo_distance query contains multiple fields
Geo distance Query, like many other queries, used to parse even when the query referred to multiple fields and the last one would win. We rather throw an exception now instead.
2016-08-05 10:58:19 +02:00
javanna 195320f2d6 [TEST] check validation error messages in IdsQueryBuilderTests 2016-08-05 10:58:19 +02:00
javanna f56333048a Throw parsing error if match_phrase query contains multiple fields
Match phrase Query, like many other queries, used to parse even when the query referred to multiple fields and the first one would win. We rather throw an exception now instead.
Also added test for short prefix query variant and modified the parsing code to consume the whole query object.
2016-08-05 10:58:19 +02:00
javanna 51ea913248 Throw parsing error if wildcard query contains multiple fields
Wildcard Query, like many other queries, used to parse even when the query referred to multiple fields and the first one would win. We rather throw an exception now instead.
Also added test for short prefix query variant and modified the parsing code to consume the whole query object.
2016-08-05 10:58:19 +02:00
javanna 003a7b6eb3 Throw parsing error if regexp query contains multiple fields
Regexp Query, like many other queries, used to parse even when the query referred to multiple fields and the last one would win. We rather throw an exception now instead.
Also added test for short prefix query variant.
2016-08-05 10:58:19 +02:00
javanna 69c2deedc7 Throw parsing error if prefix query contains multiple fields
Prefix Query, like many other queries, used to parse when the query refers to multiple fields and the last one would win. We rather throw an exception now instead.
Also added tests for short prefix quer variant.
2016-08-05 10:58:19 +02:00
javanna 11e4b0168b Throw parsing error if range query contains multiple fields
Range Query, like many other queries, used to parse when the query refers to multiple fields and the last one would win. We rather throw an exception now instead.

Closes #19547
2016-08-05 10:58:19 +02:00
Colin Goodheart-Smithe a01475a20b #19781 Refactored Rounding simplify Date Histogram code
Refactored Rounding simplify Date Histogram code
2016-08-05 09:28:38 +01:00
Boaz Leskes 609a199bd4 Upon being elected as master, prefer joins' node info to existing cluster state (#19743)
When we introduces [persistent node ids](https://github.com/elastic/elasticsearch/pull/19140) we were concerned that people may copy data folders from one to another resulting in two nodes competing for the same id in the cluster. To solve this we elected to not allow an incoming join if a different with same id already exists in the cluster, or if some other node already has the same transport address as the incoming join. The rationeel there was that it is better to prefer existing nodes and that we can rely on node fault detection to remove any node from the cluster that isn't correct any more, making room for the node that wants to join (and will keep trying).

Sadly there were two problems with this:
1) One minor and easy to fix - we didn't allow for the case where the existing node can have the same network address as the incoming one, but have a different ephemeral id (after node restart). This confused the logic in `AllocationService`, in this rare cases. The cluster is good enough to detect this and recover later on, but it's not clean.
2) The assumption that Node Fault Detection will clean up is *wrong* when the node just won an election (it wasn't master before) and needs to process the incoming joins in order to commit the cluster state and assume it's mastership. In those cases, the Node Fault Detection isn't active. 

This PR fixes these two and prefers incoming nodes to existing node when finishing an election. 
On top of the, on request by @ywelsch , `AllocationService` synchronization between the nodes of the cluster and it's routing table is now explicit rather than something we do all the time. The same goes for promotion of replicas to primaries.
2016-08-05 08:58:03 +02:00
Jason Tedor 3f6a3c01da Merge pull request #19803 from elastic/fix/transportClientTests
Fix PreBuiltTransportClientTests to run and pass
2016-08-04 16:55:08 -04:00
Simon Willnauer e08f11dabc Remove BWC serialization logic for pre 2.2 nodes (#19810)
This change removes all pre 2.2 logic from InternalSearchResponse serialization. It's unneeded in 5.0 since we require full cluster restart
2016-08-04 22:47:39 +02:00
Daniel Mitterdorfer 4598c36027 Fix various concurrency issues in transport (#19675)
Due to various issues (most notably a missing happens-before edge
between socket accept and channel close in MockTcpTransport),
MockTcpTransportTests sometimes did not terminate.

With this commit we fix various concurrency issues that led to
this hanging test.

Failing example build: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-os-compatibility/os=oraclelinux/835/console
2016-08-04 21:00:59 +02:00
Boaz Leskes 7010082112 Add checksumming and versions to the Translog's Checkpoint files (#19797)
This prepares the infrastructure to be able to extend the checkpoint file to store more information.
2016-08-04 20:42:12 +02:00
javanna cd9388ce66 [TEST] parse query alternate versions in strict mode
AbstractQueryTestCase parses the main version of the query in strict mode, meaning that it will fail if any deprecated syntax is used. It should do the same for alternate versions (e.g. short versions). This is the way it is because the two alternate versions for ids query are both deprecated. Moved testing for those to a specific test method that isolates the deprecations and actually tests that the two are deprecated.
2016-08-04 19:49:43 +02:00
Colin Goodheart-Smithe b6ef99195d Remove offset rounding
This is in favour of doing the offset calculations in the date histogram
2016-08-04 16:24:19 +01:00
Colin Goodheart-Smithe c14155e4a8 Remove TimeZoneRounding abstraction
Because the Rounding class now only deals with date based rounding of
values we can remove the TimeZoneRounding abstraction to simplify the
code.
2016-08-04 16:24:19 +01:00
Colin Goodheart-Smithe 5ab5cc69b8 Remove unused rounding code
Factor rounding and Interval rounding (the non-date based rounding)
was no longer used so it has been removed. Offset rounding has been
retained for no since both date based rounding classes rely on it
2016-08-04 16:24:19 +01:00
Ali Beyad 34bb150863 [TEST] Fixes primary term in TransportReplicationActionTests#testReplicaProxy 2016-08-04 10:18:48 -04:00
Colin Goodheart-Smithe b0730bb214 Fix PreBuiltTransportClientTests to run and pass
This change does three things:

1. Makes PreBuiltTransportClientTests run since it was silently
failing on a missing dependency
2. Makes PreBuiltTransportClientTests pass
3. Removes the http.type and transport.type from being set in the
transport clients additional settings since these are set to `netty4` by
default anyway.
2016-08-04 14:15:28 +01:00
Ali Beyad 8bbc312fdd Fixes issue with dangling index being deleted instead of re-imported (#19666)
Fixes an issue where a node that receives a cluster state
update with a brand new cluster UUID but without an
initial persistence block could cause indices to be wiped out,
preventing them from being reimported as dangling indices.
This commit only removes the in-memory data structures and
thus, are subsequently reimported as dangling indices.
2016-08-04 08:47:46 -04:00
Yannick Welsch ede78ad231 Use primary terms as authority to fail shards (#19715)
A primary shard currently instructs the master to fail a replica shard that it fails to replicate writes to before acknowledging the writes to the client. To ensure that the primary instructing the master to fail the replica is still the current primary in the cluster state on the master, it submits not only the identity of the replica shard to fail to the master but also its own shard identity. This can be problematic however when the primary is relocating. After primary relocation handoff but before the primary relocation target is activated, the primary relocation target is replicating writes through the authority of the primary relocation source. This means that the primary relocation target should probably send the identity of the primary relocation source as authority. However, this is not good enough either, as primary shard activation and shard failure instructions can arrive out-of-order. This means that the relocation target would have to send both relocation source and target identity as authority. Fortunately, there is another concept in the cluster state that represents this joint authority, namely primary terms. The primary term is only increased on initial assignment or when a replica is promoted. It stays the same however when a primary relocates.

This commit changes ShardStateAction to rely on primary terms for shard authority. It also changes the wire format to only transmit ShardId and allocation id of the shard to fail (instead of the full ShardRouting), so that the same action can be used in a subsequent PR to remove allocation ids from the active allocation set for which there exist no ShardRouting in the cluster anymore. Last but not least, this commit also makes AllocationService less lenient, requiring ShardRouting instances that are passed to its applyStartedShards and applyFailedShards methods to exist in the routing table. ShardStateAction, which is calling these methods, now has the responsibility to resolve the ShardRouting objects that are to be started / failed, and remove duplicates.
2016-08-04 12:00:37 +02:00
Boaz Leskes d327dd46b1 Recovery: don't log an error when listing an empty folder 2016-08-04 10:23:36 +02:00
Jason Tedor 533412e36f Improve cat thread pool API
Today, when listing thread pools via the cat thread pool API, thread
pools are listed in a column-delimited format. This is unfriendly to
command-line tools, and inconsistent with other cat APIs. Instead,
thread pools should be listed in a row-delimited format.

Additionally, the cat thread pool API is limited to a fixed list of
thread pools that excludes certain built-in thread pools as well as all
custom thread pools. These thread pools should be available via the cat
thread pool API.

This commit improves the cat thread pool API by listing all thread pools
(built-in or custom), and by listing them in a row-delimited
format. Finally, for each node, the output thread pools are sorted by
thread pool name.

Relates #19721
2016-08-03 23:02:13 -04:00
David Pilato 54603903f3 Remove ListTasksResponse#setDiscoveryNodes 2016-08-04 02:02:51 +02:00
Ali Beyad be87d50f32 Fixes CreateIndexIT test that assumes an index create propogated
before calling delete.
2016-08-03 16:24:24 -04:00
Ryan Ernst c3a5e4fa48 Merge pull request #19765 from rjernst/metadata_mapper_dup
Mappings: Fix detection of metadata fields in documents
2016-08-03 11:58:24 -07:00
Ryan Ernst ef425f4b7c Merge pull request #19770 from rjernst/script_service_component
Add ScriptService to dependencies available for plugin components
2016-08-03 11:57:58 -07:00
javanna 4805250ecf Throw ParsingException if a query is wrapped in an array
Our parsing code accepted up until now queries in the following form (note that the query starts with `[`:

```
{
    "bool" : [
        {
          "must" : []
        }
    ]
}
```

This would lead to a null pointer exception as most parsers assume that the field name ("must" in this example) is the first thing that can be found in a query if its json is valid, hence always non null while parsing. Truth is that the additional array layer doesn't make the json invalid, hence the following code fragment would cause NPE within ParseField, because null gets passed to `parseContext.isDeprecatedSetting`:

```
if (token == XContentParser.Token.FIELD_NAME) {
    currentFieldName = parser.currentName();
} else if (parseContext.isDeprecatedSetting(currentFieldName)) {
    // skip
} else if (token == XContentParser.Token.START_OBJECT) {
```

We could add null checks in each of our parsers in lots of places, but we rely on `currentFieldName` being non null in all of our parsers, and we should consider it a bug when these unexpected situations are not caught explicitly. It would be best to find a way to prevent such queries altogether without changing all of our parsers.

The reason why such a query goes through is that we've been allowing a query to start with either `[` or `{`. The only reason I found is that we accept `match_all : []`. This seems like an undocumented corner case that we could drop support for. Then we can be stricter and accept only `{` as start token of a query. That way the only next token that the parser can encounter if the json is valid (otherwise the json parser would barf earlier) is actually a field_name, hence the assumption that all our parser makes hold.

The downside of this is simply dropping support for `match_all : []`

Relates to #12887
2016-08-03 17:05:14 +02:00
javanna 51bbe2c5c4 [TEST] fix log statement in ESIndexLevelReplicationTestCase 2016-08-03 16:56:19 +02:00
Clinton Gormley 39081af9d6 Added version 2.3.5 with bwc indices 2016-08-03 15:50:47 +02:00
David Pilato a1633d6444 ListTasksResponse#toString() should not group by nodes
We just overwrite `toString()` method so it calls toXContent
with `group_by` = "whatever" so we don't try to group by nodes
which does not make sense in a toString() method.

We keep the old behavior for `toXContent()` method which
means that there is no impact in the REST layer but
only in logs and tests (where we call `toString()`).

Closes #19772.
2016-08-03 14:56:09 +02:00
Robert Muir ef5debc6ce Merge pull request #19754 from rmuir/docker_seccomp
ignore some docker craziness in seccomp environment checks
2016-08-03 05:50:25 -04:00
Britta Weber abcb4c8a97 [Test] move methods from bwc test to test package for use in plugins (#19738)
* [Test] move methods from bwc test to test package for use in other plugins
2016-08-03 11:41:46 +02:00
Adrien Grand 0e64117512 package-info.java should be in src/main only. 2016-08-03 11:11:25 +02:00
Ryan Ernst 18f242b069 Merge pull request #19764 from rjernst/writeable_registry
Make NamedWriteableRegistry immutable and add extension point for named writeables
2016-08-03 01:36:38 -07:00
Ryan Ernst fe823c857b Plugins: Add ScriptService to dependencies available for plugin components 2016-08-03 00:43:04 -07:00
Adrien Grand a0818d3b87 Split regular histograms from date histograms. #19551
Currently both aggregations really share the same implementation. This commit
splits the implementations so that regular histograms can support decimal
intervals/offsets and compute correct buckets for negative decimal values.

However the response API is still the same. So for intance both regular
histograms and date histograms will produce an
`org.elasticsearch.search.aggregations.bucket.histogram.Histogram`
aggregation.

The optimization to compute an identifier of the rounded value and the
rounded value itself has been removed since it was only used by regular
histograms, which now do the rounding themselves instead of relying on the
Rounding abstraction.

Closes #8082
Closes #4847
2016-08-03 08:39:48 +02:00
Boaz Leskes f6aeb35ce8 Tighten up concurrent store metadata listing and engine writes (#19684)
In several places in our code we need to get a consistent list of files + metadata of the current index. We currently have a couple of ways to do in the `Store` class, which also does the right things and tries to verify the integrity of the smaller files. Sadly, those methods can run into trouble if anyone writes into the folder while they are busy. Most notably, the index shard's engine decides to commit half way and remove a `segment_N` file before the store got to checksum (but did already list it). This race condition typically doesn't happen as almost all of the places where we list files also happen to be places where the relevant shard doesn't yet have an engine. There  is however an exception (of course :)) which is the API to list shard stores, used by the master when it is looking for shard copies to assign to.

I already took one shot at fixing this in #19416 , but it turns out not to be enough - see for example https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-os-compatibility/os=sles/822.

The first inclination to fix this was to add more locking to the different Store methods and acquire the `IndexWriter` lock, thus preventing any engine for accessing if if the a shard is offline and use the current index commit snapshotting logic already existing in `IndexShard` for when the engine is started. That turned out to be a bad idea as we create more subtleties where, for example, a store listing can prevent a shard from starting up (the writer lock doesn't wait if it can't get access, but fails immediately, which is good). Another example is running on a shared directory where some other engine may actually hold the lock.

Instead I decided to take another approach:
1) Remove all the various methods on store and keep one, which accepts an index commit (which can be null) and also clearly communicates that the *caller* is responsible for concurrent access. This also tightens up the API which is a plus.
2) Add a `snapshotStore` method to IndexShard that takes care of all the concurrency aspects with the engine, which is now possible because it's all in the same place. It's still a bit ugly but at least it's all in one place and we can evaluate how to improve on this later on. I also renamed the  `snapshotIndex` method to `acquireIndexCommit` to avoid confusion and I think it communicates better what it does.
2016-08-03 08:34:09 +02:00
Ryan Ernst 7bfe1bd628 Check inner field with metadata field name is ok 2016-08-02 17:03:21 -07:00
Ryan Ernst 4e48154130 Mappings: Fix detection of metadata fields in documents
In 2.0, the ability to specify metadata fields like _routing and _ttl
inside a document was removed. However, the ability to break through
this restriction has lingered, and the check that enforced it is
completely broken.

This change fixes the check, and adds a parsing test.
2016-08-02 16:54:44 -07:00
Ryan Ernst df8dc64e9b Plugins: Make NamedWriteableRegistry immutable and add extenion point for named writeables
Currently any code that wants to added NamedWriteables to the
NamedWriteableRegistry can do so via guice injection of the registry,
and registering at construction time. However, this makes the registry
complex: it has both get and register methods synchronized, and there is
likely contention on the read side from multiple threads.  The
registration has mostly already been contained to guice modules at node
construction time.

This change makes the registry immutable, taking all of the
NamedWriteable readers at construction time. It also allows plugins to
added arbitrary named writables that it may use in its own transport
actions.
2016-08-02 15:56:25 -07:00
Lee Hinman a9b2e172fa [TEST] Increase time waiting for all shards to move off/on to a node 2016-08-02 16:18:39 -06:00
Ali Beyad c28eee77df Fixes the active shard count check in the case of (#19760)
ActiveShardCount.ALL by checking for active shards,
not just started shards, as a shard could be active
but in the relocating state (i.e. not in the started
state).
2016-08-02 18:00:39 -04:00
Igor Motov 22e63b4783 Fixes cat tasks operation in detailed mode
Currently the cat tasks operation fails in the detailed mode.

Closes #19755
2016-08-02 15:21:31 -04:00
Robert Muir f77e8a512c ignore some docker craziness in scccomp environment checks 2016-08-02 12:19:38 -04:00
Ali Beyad c4ae23f5d8 Enables implementations of the BlobContainer interface to (#19749)
conform with the requirements of the writeBlob method by
throwing a FileAlreadyExistsException if attempting to write
to a blob that already exists. This change means implementations
of BlobContainer should never overwrite blobs - to overwrite a
blob, it must first be deleted and then can be written again.

Closes #15579
2016-08-02 09:48:21 -04:00
Nik Everett 42fe2f0aca Add docs for a few packages
This'll make javadocs slightly more useful....
2016-08-02 09:30:30 -04:00
Ali Beyad 456ea56527 Cleans up the BlobContainer interface by removing the (#19727)
writeBlob method takes a BytesReference in favor of just
the writeBlob method that takes an InputStream.

Closes #18528
2016-08-02 09:21:43 -04:00
Ali Beyad 3d2a105825 Merge pull request #19454 from abeyad/remove-write-consistency-level
Removes write consistency level across replication action APIs in favor of wait_for_active_shards
2016-08-02 09:01:11 -04:00
Daniel Mitterdorfer 419e9e090e Document and enforce cancellation policy of CancellableThreads (#19712)
With this commit we add documentation and additional checks to
enforce the cancellation policy of CancellableThreads (which is
disallow `Thread#interrupt()` on any of the threads managed by
it).
2016-08-02 08:46:38 +02:00
Ali Beyad 4923da93c8 Refactors wait_for_active_shards index settings tests 2016-08-01 19:14:37 -04:00
Lee Hinman f9fd64fc78 Revert to older exception message
If the uuidBytes and ref are converted to utf8, it's possible they can
trip an assertion related to valid UTF-8/UTF-16 ranges, so display them
as hex, not as strings.
2016-08-01 11:51:39 -06:00
Ali Beyad 6a7d005081 Makes the index.write.wait_for_active_shards setting index-level and
dynamically updatable for both index creation and write operations.
2016-08-01 13:37:05 -04:00
Ali Beyad 4a51ea8c8e Before, transport replication actions implemented a checkWriteConsistency()
method to determine if a write consistency check should be performed
before proceeding with the action.  This commit removes this method from
the transport replication actions in favor of setting the ActiveShardCount
on the request, with setting the value to ActiveShardCount.NONE if the
transport action's checkWriteConsistency() method returned false.
2016-08-01 13:35:30 -04:00
Ali Beyad d93f7d6085 Refactors ActiveShardCount 2016-08-01 13:35:29 -04:00
Ali Beyad 25d8eca62d Removes the notion of write consistency level across all APIs in
favor of waiting for active shard copy count (wait_for_active_shards).
2016-08-01 13:35:29 -04:00
Ali Beyad 9f88a8194a Merge pull request #19706 from elastic/enhancement/snapshot-blob-handling
More resilient blob handling in snapshot repositories
2016-08-01 12:03:53 -04:00
Tanguy Leroux 386902903e [TEST] Kill remaining lang-groovy messy tests
After #13834 many tests that used Groovy scripts (for good or bad reason) in their tests have been moved in the lang-groovy module and the issue #13837 has been created to track these messy tests in order to clean them up.

The work started with #19280, #19302 and #19336 and this PR moves the remaining messy tests back in core, removes the dependency on Groovy, changes the scripts in order to use the mocked script engine, and change the tests to integration tests.

It also moves IndexLookupIT test back (even if it has good chance to be removed soon) and fixes its tests.

It also changes AbstractQueryTestCase to use custom script plugins in tests.

closes #13837
2016-08-01 16:59:47 +02:00
Alexander Lin 9ac6389e43 Rename operation to result and reworking responses
* Rename operation to result and reworking responses
* Rename DocWriteResponse.Operation enum to DocWriteResponse.Result

These are just easier to interpret names.

Closes #19664
2016-08-01 10:42:58 -04:00
Nik Everett 12fd4ed8f8 Add description to org.elasticsearch.tasks package (#19700)
Yet more readable docs!
2016-08-01 07:43:32 -04:00
Nik Everett aefc36bfaa Add descriptions for o.e.search.suggest packages (#19699)
Let's have readable javadoc!
2016-08-01 07:43:13 -04:00
Boaz Leskes 7c6527ed09 make election stop not be a failure (#19705)
During our master elections, nodes "vote" for a master being issuing a join request to it. Since this is done in an async fashion, joins may arrive before the master itself has realized it had won the election. Therefore we start accumulating node joins on every node at election start (we don't know the result yet). When the election finish nodes that did not become the master (i.e., joined another node which won the election) need to potentially process and fail any incoming join request they may have received during the election. This is currently achieved by always issuing a cluster state update task that is doomed to fail, even if no pending joins are actually there. That aspect results in confusing (debug) log messages, making it seems like something is wrong. For example (note that `NotMasterException`)

```
[2016-07-30 22:25:53,040][DEBUG][cluster.service          ] [node_t1] processing [zen-disco-process-pending-joins [{node_t0}{4SqBTyYNQ82J9c75Cs7jtg}{kutaNSYbTZCSybvqczgWCA}{127.0.0.1}{127.0.0.1:9400} elected]]: execute
[2016-07-30 22:25:53,041][DEBUG][transport                ] [node_t1] connected to node [{node_t0}{4SqBTyYNQ82J9c75Cs7jtg}{kutaNSYbTZCSybvqczgWCA}{127.0.0.1}{127.0.0.1:9400}]
[2016-07-30 22:25:53,045][DEBUG][cluster.service          ] [node_t1] cluster state update task [zen-disco-process-pending-joins [{node_t0}{4SqBTyYNQ82J9c75Cs7jtg}{kutaNSYbTZCSybvqczgWCA}{127.0.0.1}{127.0.0.1:9400} elected]] failed
NotMasterException[Node [{node_t1}{eAQts270TiGFpoCDE-0PQQ}{or5bsv2ET220su78DLJk5g}{127.0.0.1}{127.0.0.1:9401}] not master for join request]
[2016-07-30 22:25:53,048][DEBUG][cluster.service          ] [node_t1] processing [zen-disco-process-pending-joins [{node_t0}{4SqBTyYNQ82J9c75Cs7jtg}{kutaNSYbTZCSybvqczgWCA}{127.0.0.1}{127.0.0.1:9400} elected]]: took [7ms] no change in cluster_state
```

This commit cleans up  the logic a bit to only use failure where there are actual joins that are failed. The result is cleaner logs as well:

```
[2016-07-30 22:23:12,880][DEBUG][cluster.service          ] [node_t1] processing [zen-disco-election-stop [{node_t0}{jMR5HCpOQnOM4pGeFkUjng}{B5WIZQAdQk2cWbjGZ21mvQ}{127.0.0.1}{127.0.0.1:9400} elected]]: execute
[2016-07-30 22:23:12,881][DEBUG][cluster.service          ] [node_t1] processing [zen-disco-election-stop [{node_t0}{jMR5HCpOQnOM4pGeFkUjng}{B5WIZQAdQk2cWbjGZ21mvQ}{127.0.0.1}{127.0.0.1:9400} elected]]: took [0s] no change in cluster_state
[2016-07-30 22:23:12,881][DEBUG][transport                ] [node_t1] connected to node [{node_t0}{jMR5HCpOQnOM4pGeFkUjng}{B5WIZQAdQk2cWbjGZ21mvQ}{127.0.0.1}{127.0.0.1:9400}]
```
2016-08-01 13:08:50 +02:00
Tanguy Leroux 737db98bd7 /_cat/shards should support wilcards for indices
closes #19634
2016-08-01 11:09:48 +02:00
Christoph Büscher 87a4995bed Merge pull request #19665 from cbuescher/missing-field-MultiMatchQuery
`multi_match` query should produce MatchNoDocs query on unknown field
2016-08-01 10:59:52 +02:00
Tanguy Leroux 7d4f557aa3 Allow routing table to be filtered by index pattern
Before this commit when an index pattern is used to filter the cluster state, only indices metadata are populated and routing table is just empty. This commit aligns the behavior of the filtering of cluster state's routing table with the filtering of cluster state's metadata so that coherent data are returned for both routing table & metadata when index pattern is requested.
2016-08-01 09:22:12 +02:00
chengpohi 8aa1eb6aa4 Fix EquivalenceIT#testRandomRanges failed with -Dtest.seed A4648847991E5C27
Set double value to double type mapping in EquivalenceIT.

Closes #19697
2016-07-31 12:49:28 -04:00
Ali Beyad 0f335ac873 Removes legacy format in RepositoryData 2016-07-30 18:46:58 -04:00
Nik Everett 303c9faca5 Squash o.e.rest.action.admin.cluster
In an effort to reduce the number of tiny packages we have in the
code base this moves all the files that were in subdirectories of
`org.elasticsearch.rest.action.admin.cluster` into
`org.elasticsearch.rest.action.admin.cluster`.

Also fixes line length in these packages.
2016-07-29 20:31:24 -04:00
Michael McCandless 71166c020a Merge pull request #19554 from mikemccand/negative_usable_space
Guard against negative result from FileStore.getUsableSpace when picking data path for a new shard
2016-07-29 20:26:30 -04:00
Mike McCandless 59181c8a66 use mockito instead 2016-07-29 17:13:01 -04:00
Nik Everett bdebd02d8c Only write forced_refresh if we forced a refresh
Otherwise it just adds noise to the response.

Closes #19629
2016-07-29 15:00:30 -04:00
Christoph Büscher 0d7c289f4c Adressing review comments 2016-07-29 20:28:17 +02:00
Alexander Lin 119026b4fb Remove isCreated and isFound from the Java API
This is cleanup work from #19566, where @nik9000 suggested trying to nuke the isCreated and isFound methods. I've combined nuking the two methods with removing UpdateHelper.Operation in favor of DocWriteResponse.Operation here.

Closes #19631.
2016-07-29 14:21:43 -04:00
Christoph Büscher 4450039cf6 Try catching potential null query results and convert to MatchNoDocsQuery 2016-07-29 18:29:48 +02:00
Martijn van Groningen a91bb29585 ingest: Made the response format of the get pipeline api match with the response format of the index template api
Closes #19585
2016-07-29 17:58:30 +02:00
Mike McCandless 37e0e63a65 add defense to selectNewPathForShard 2016-07-29 11:51:33 -04:00
Nik Everett ad028f3f9c Squash o.e.rest.action.admin.indices
In an effort to reduce the number of tiny packages we have in the
code base this moves all the files that were in subdirectories of
`org.elasticsearch.rest.action.admin.indices` into
`org.elasticsearch.rest.action.admin.indices`.

It also adds a `package-info.java` file explaining what the files in
the package *do*.

Also fixes line length in these packages. It makes a single non-checkstyle
change: implementing `ToXContent` on `GetIndexTemplatesResponse`. I did
this because it was the right thing to do and it fixed a line length
violation.
2016-07-29 10:08:03 -04:00
Martijn van Groningen 81112508ea test: fix type in test name 2016-07-29 14:52:24 +02:00
Martijn van Groningen 72e0d422e9 Plain highlighter should ignore parent/child queries.
The plain highligher fails when it tries to select the fragments based on a query containing either a `has_child` or `has_parent` query.

The plain highligher should just ignore parent/child queries as it makes no sense to highligh a parent match with a has_child as the child documents are not available at highlight time. Instead if child document should be highlighed inner hits should be used.

Parent/child queries already have no effect when the `fvh` or `postings` highligher is used. The test added in this commit verifies that.

Closes #14999
2016-07-29 12:41:11 +02:00
Christoph Büscher 757de805d3 `multi_match` query should produce MatchNoDocs query on unknown fieldname
Currently when the `fields` parameter used in a `multi_match` query contains a
wildcard expression that doesn't resolve to any field name in the target index,
MultiMatchQueryBuilder produces a `null` query. This change changes it to be a
MatchNoDocs query, since returning no documents for this case is already the
current behaviour. Also adding missing field names (with and without wildcards)
to the unit and integration test.
2016-07-29 10:56:37 +02:00
Colin Goodheart-Smithe f1257bfb86 Added JavaDocs and comments to ParseField 2016-07-29 09:39:38 +01:00
Colin Goodheart-Smithe cd88b7724e Undeprecates `aggs` in the search request
This change adds a second ParseField for the `aggs` field in the search
request so both `aggregations` and `aggs` are undeprecated allowed
fields in the search request

Closes #19504
2016-07-29 09:14:32 +01:00
Adrien Grand dcc598c414 Make the heuristic to compute the default shard size less aggressive.
The current heuristic to compute a default shard size is pretty aggressive,
it returns `max(10, number_of_shards * size)` as a value for the shard size.
I think making it less aggressive has the benefit that it would reduce the
likelyness of running into OOME when there are many shards (yearly
aggregations with time-based indices can make numbers of shards in the
thousands) and make the use of breadth-first more likely/efficient.

This commit replaces the heuristic with `size * 1.5 + 10`, which is enough
to have good accuracy on zipfian distributions.
2016-07-29 09:59:29 +02:00
Ali Beyad 58d6b9dcd1 This commit first reads the repository data and only
upgrades if it determines the read data is in the legacy
format. It writes the upgraded version if it is not a
read-only repository and caches the repository data if
it is a read-only repository.
2016-07-28 22:09:01 -04:00
Nik Everett e04f06258f Assert we return Location header with 201 CREATED
Add an assertion to the most popular way of turning the response object
into the actual http response. As it stands all places we return
`201 CREATED` we return the `Location` header. This will help to keep it
that way, though it won't catch all uses.

Followup to #19509
2016-07-28 16:13:58 -04:00
Mike McCandless ed5e5db188 merge master 2016-07-28 11:55:16 -04:00
Areek Zillur 69941931c7 Merge pull request #19610 from areek/enhancement/19484
Add zero-padding to auto-generated rollover index name increment
2016-07-28 11:44:50 -04:00
Mike McCandless ef15e1b91f work around JDK bug: if FileStore.getXXXSpace APIs return negative value, change that to Long.MAX_VALUE instead 2016-07-28 11:31:16 -04:00
David Pilato 0d2ccf0989 Merge branch 'pr/15724-gce-network-host-master' 2016-07-28 16:59:18 +02:00
David Pilato 7b9ce1212f Merge branch 'fix/npe-simulate-pipeline-no-id' 2016-07-28 14:55:07 +02:00
Colin Goodheart-Smithe bab3e766c7 #19649 Makes `m` case sensitive in TimeValue
Makes `m` case sensitive in TimeValue
2016-07-28 13:00:57 +01:00
David Pilato d406b88857 Fix NPE when simulating a pipeline with no id
When you simulate a pipeline without specifying an id against a node where the request is redirected to a master node,
the request and the response is throwing a NPE:

```
java.lang.NullPointerException
    at __randomizedtesting.SeedInfo.seed([3B9536AC6AA23C06:DD62280CF765DA1F]:0)
    at org.elasticsearch.common.io.stream.StreamOutput.writeString(StreamOutput.java:300)
    at org.elasticsearch.action.ingest.SimulatePipelineRequest.writeTo(SimulatePipelineRequest.java:92)
    at org.elasticsearch.transport.local.LocalTransport.sendRequest(LocalTransport.java:222)
    at org.elasticsearch.test.transport.AssertingLocalTransport.sendRequest(AssertingLocalTransport.java:95)
    at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:470)
    at org.elasticsearch.action.TransportActionNodeProxy.execute(TransportActionNodeProxy.java:51)
    at org.elasticsearch.client.transport.support.TransportProxyClient.lambda$execute$441(TransportProxyClient.java:63)
    at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:233)
    at org.elasticsearch.client.transport.support.TransportProxyClient.execute(TransportProxyClient.java:63)
    at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:309)
    at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403)
    at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:67)
    at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403)
    at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:710)
    at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:80)
    at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:54)
    at org.elasticsearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:62)
    at org.elasticsearch.ingest.bano.BanoProcessorIntegrationTest.testSimulateProcessorConfigTarget(BanoProcessorIntegrationTest.java:139)
```

This patch fixes this and adds some random tests.
2016-07-28 13:28:24 +02:00
Britta Weber 105dce0e07 fix explain in function_score if no function filter matches (#19185)
* fix explain in function_score if no function filter matches

When each function in function_score has a filter but none of them matches
we always assume 1 for the combined functions and then combine that with the
sub query score.
But the explanation did not reflect that because in case no function matched
we did not even use the actual score that was computed in the explanation.
2016-07-28 13:14:08 +02:00
Colin Goodheart-Smithe eab5ceb9de Makes `m` case sensitive in TimeValue
The reason for this change is that currently if a user specifies e.g.`2M`
meaning 2 months as a time value instead of throwing an exception
explaining that time units in months are not supported (due to months
having variable time spans) we instead will parse this to 2 minutes.
This could be surprising to a user and could mean put a lot of load on
the cluster performing a task that was never intended and whose results
will be useless anyway.

It is generally accepted that `m` indicates minutes and `M` indicates
months with time values so this is consistent with the expectations a
user might have around specifying time units.

A concrete example of where this causes issues is in the decay score
function which uses TimeValue to parse the scale and offset parameters
of the decay into millisecond values to use in the calculation.

Relates to #19619
2016-07-28 11:27:24 +01:00
Lee Hinman 9fa33b6d07 [TEST] throw correct error within assertBusy in TruncateTranslogIT 2016-07-27 16:40:49 -06:00
Ryan Ernst dcf42b8d64 Merge pull request #19638 from rjernst/filewatcher_interface
Change file changes listener for resource watcher to an interface
2016-07-27 15:33:14 -07:00
Nik Everett 56ee49255b Only log running out of slots when out of slots (#19637)
We were logging on every `refresh=wait_for`.
2016-07-27 18:26:09 -04:00
Ryan Ernst 95499c45a5 Change file changes listener for resource watcher to an interface
Currently to use the ResourceWatcherService to watch files, you
implement a FileChangesListener. However, this is a class, not an
interface, even though it has no base state or anything like that, just
defining a few methods. This change converts FileChangesListener to an
interface.
2016-07-27 15:25:24 -07:00
Nik Everett fb45f6a8a8 Add authentication to reindex-from-remote
The tests for authentication extend ESIntegTestCase and use a mock
authentication plugin. This way the clients don't have to worry about
running it. Sadly, that means we don't really have good coverage on the
REST portion of the authentication.

This also adds ElasticsearchStatusException, and exception on which you
can set an explicit status. The nice thing about it is that you can
set the RestStatus that it returns to whatever arbitrary status you like
based on the status that comes back from the remote system.
reindex-from-remote then uses it to wrap all remote failures, preserving
the status from the remote Elasticsearch or whatever proxy is between us
and the remove Elasticsearch.
2016-07-27 14:17:41 -04:00
Areek Zillur 4e3602a790 Add zero-padding to auto-generated rollover index name increment
closes #19484
2016-07-27 10:50:47 -04:00
David Pilato 9cb1e79e84 Fix comments and method name 2016-07-27 13:35:58 +02:00
David Pilato 3d9f2bf531 Revert last change and make generateCustomNameResolvers private in Node class 2016-07-27 12:19:08 +02:00
David Pilato e949101cc7 Move generateCustomNameResolvers to DiscoveryPlugin interface 2016-07-27 11:36:06 +02:00
David Pilato e9339a1960 Merge branch 'master' into pr/15724-gce-network-host-master 2016-07-27 11:24:53 +02:00
David Pilato b62bb47663 Move registerCustomNameResolvers to Node class and rename it 2016-07-27 11:23:25 +02:00
Martijn van Groningen 24d7fa6d54 ingest: Change the `foreach` processor to use the `_ingest._value` ingest metadata attribute to store the current array element being processed.
Closes #19592
2016-07-27 09:35:09 +02:00
Ali Beyad 21ff90fed3 Fixes debug logging on index creation waiting for shards to be started (#19612) 2016-07-26 19:17:02 -04:00
Lee Hinman 0876247bca [TEST] Assert that shard has been released before running truncate tool
It's possible that the shard has been closed but the resources
associated with it have not yet been released. This waits until the
index lock can be obtained before running the tool.
2016-07-26 14:14:04 -06:00
Igor Motov 7275291f35 Tests: add more logging to testCorruptFileThenSnapshotAndRestore
This test fails because of an unknown exceptions in FsService.stats() method, which causes no stats to be returned. With this change the exception that is causing this issue is going to be logged.

Related to #19591 and #17964
2016-07-26 15:08:19 -04:00
Nik Everett 9270e8b22b Rename client yaml test infrastructure
This makes it obvious that these tests are for running the client yaml
suites. Now that there are other ways of running tests using the REST
client against a running cluster we can't go on calling the shared
client yaml tests "REST tests". They are rest tests, but they aren't
**the** rest tests.
2016-07-26 13:53:44 -04:00
Chris Earle 0553ba9151 [Ingest] Add REST _ingest/pipeline to get all pipelines
This adds an extra REST handler for "_ingest/pipeline" so that users do not need to supply "_ingest/pipeline/*" to get all of them.

- Also adds a teardown section to related REST-tests for ingest.
2016-07-26 13:48:15 -04:00
David Pilato 0d3edee928 Merge branch 'master' into pr/15724-gce-network-host-master 2016-07-26 18:51:01 +02:00
David Pilato fde15ae470 Move custom name resolvers to NetworkService CTOR
Instead of using NetworkModule we can directly inject them in NetworkService CTOR.

See https://github.com/elastic/elasticsearch/pull/15765#issuecomment-235307974
2016-07-26 18:26:30 +02:00
Christoph Büscher e1415d6519 Merge pull request #19595 from cbuescher/fix-19422
Allow empty json object in request body in `_count` API.
2016-07-26 18:17:52 +02:00
Boaz Leskes 8151224883 add `Socket closed` variant to NetworkExceptionHelper.isCloseConnectionException 2016-07-26 18:01:57 +02:00
Lee Hinman e538c1c6d6 Merge remote-tracking branch 'dakrone/translog-cli' 2016-07-26 09:39:11 -06:00
Nik Everett a182e356d3 Fix unit test build failure
We didn't catch the failure because we tested against the fork instead
of master. I think.
2016-07-26 11:35:17 -04:00
Alexander Lin 8f2882a442 Add _operation field to index, update, delete responses
Performing the bulk request shown in #19267 now results in the following:
```
{"_index":"test","_type":"test","_id":"1","_version":1,"_operation":"create","forced_refresh":false,"_shards":{"total":2,"successful":1,"failed":0},"status":201}
{"_index":"test","_type":"test","_id":"1","_version":1,"_operation":"noop","forced_refresh":false,"_shards":{"total":2,"successful":1,"failed":0},"status":200}
```
2016-07-26 11:16:19 -04:00
Lee Hinman ac53c90ff4 Add 'elasticsearch-translog' CLI tool with 'translog' command
This adds the `bin/elasticsearch-translate` bin file that will be used
for CLI tasks pertaining to Elasticsearch. Currently it implements only
a single sub-command, `truncate-translog`, that creates a truncated
translog for a given folder.

Here's what running the tool looks like:

```
λ bin/elasticsearch-translog truncate -d data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/
Checking existing translog files
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!   WARNING: Elasticsearch MUST be stopped before running this tool   !
!                                                                     !
!   WARNING:    Documents inside of translog files will be lost       !
!                                                                     !
!   WARNING:          The following files will be DELETED!            !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
--> data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-10.tlog
--> data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-18.tlog
--> data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-21.tlog
--> data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-12.ckp
--> data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-25.ckp
--> data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-29.tlog
--> data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-2.tlog
--> data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-5.tlog
--> data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-41.ckp
--> data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-6.ckp
--> data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-37.ckp
--> data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-24.ckp
--> data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-11.ckp

Continue and DELETE files? [y/N] y
Reading translog UUID information from Lucene commit from shard at [data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/index]
Translog Generation: 3
Translog UUID      : AxqC4rocTC6e0fwsljAh-Q
Removing existing translog files
Creating new empty checkpoint at [data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog.ckp]
Creating new empty translog at [data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-3.tlog]
Done.
```

It also includes a `-b` batch operation that can be used to skip the
confirmation diaglog.

Resolves #19123
2016-07-26 08:34:07 -06:00
Christoph Büscher 4bac61425c Adding unit tests for QueryParseContext 2016-07-26 15:27:25 +02:00
Colin Goodheart-Smithe 2c12c3e628 Add _bucket_count option to buckets_path
This change adds a new special path to the buckets_path syntax
`_bucket_count`. This new option will return the number of buckets for a
multi-bucket aggregation, which can then be used in pipeline
aggregations.

Closes #19553
2016-07-26 09:28:21 +01:00
Christoph Büscher b861ec1cc0 Allow empty json object in request body in `_count` API
When the request body is missing, all documents in the target index are counted.
As mentioned in #19422, the same should happen when the request body is an empty
json object. This is also the behaviour for the `_search` endpoint and the two
APIs should behave in the same way.
2016-07-26 09:54:05 +02:00
Martijn van Groningen c7c0faa54d aggs: Changed how `nested` and `reverse_nested` aggs know about their nested depth level.
Before the aggregation tree was traversed to figure out what the parent level is, this commit
changes that by using `NestedScope` to figure out the nested depth level. The big upsides
are that this cleans up `NestedAggregator` (it used a hack to lazily figure out the nested parent filter)
 and this is also what `nested` query uses and therefor the `nested` query can be included inside `nested`
 aggregation and work correctly.

Closes #11749
Closes #12410
2016-07-26 09:04:51 +02:00
Nik Everett a95d4f4ee7 Add Location header and improve REST testing
This adds a header that looks like `Location: /test/test/1` to the
response for the index/create/update API. The requirement for the header
comes from https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

https://tools.ietf.org/html/rfc7231#section-7.1.2 claims that relative
URIs are OK. So we use an absolute path which should resolve to the
appropriate location.

Closes #19079

This makes large changes to our rest test infrastructure, allowing us
to write junit tests that test a running cluster via the rest client.
It does this by splitting ESRestTestCase into two classes:
* ESRestTestCase is the superclass of all tests that use the rest client
to interact with a running cluster.
* ESClientYamlSuiteTestCase is the superclass of all tests that use the
rest client to run the yaml tests. These tests are shared across all
official clients, thus the `ClientYamlSuite` part of the name.
2016-07-25 17:02:40 -04:00
Lee Hinman 1623cff6c0 Merge remote-tracking branch 'dakrone/bucket-circuit-breaker' 2016-07-25 13:37:26 -06:00
Lee Hinman 124a9fabe3 Circuit break on aggregation bucket numbers with request breaker
This adds new circuit breaking with the "request" breaker, which adds
circuit breaks based on the number of buckets created during
aggregations. It consists of incrementing during AggregatorBase creation

This also bumps the REQUEST breaker to 60% of the JVM heap now.

The output when circuit breaking an aggregation looks like:

```json
{
  "shard" : 0,
  "index" : "i",
  "node" : "a5AvjUn_TKeTNYl0FyBW2g",
  "reason" : {
    "type" : "exception",
    "reason" : "java.util.concurrent.ExecutionException: QueryPhaseExecutionException[Query Failed [Failed to execute main query]]; nested: CircuitBreakingException[[request] Data too large, data for [<agg [otherthings]>] would be larger than limit of [104857600/100mb]];",
    "caused_by" : {
      "type" : "execution_exception",
      "reason" : "QueryPhaseExecutionException[Query Failed [Failed to execute main query]]; nested: CircuitBreakingException[[request] Data too large, data for [<agg [myagg]>] would be larger than limit of [104857600/100mb]];",
      "caused_by" : {
        "type" : "circuit_breaking_exception",
        "reason" : "[request] Data too large, data for [<agg [otherthings]>] would be larger than limit of [104857600/100mb]",
        "bytes_wanted" : 104860781,
        "bytes_limit" : 104857600
      }
    }
  }
}
```

Relates to #14046
2016-07-25 11:33:37 -06:00
Martijn van Groningen a784055db1 Cleaned up the tests in lang-mustache.
Messy tests with mustache were either moved to core, moved to a rest test or remained untouched if they actually tested mustache.
Also removed tests that were redundant.
2016-07-25 17:57:39 +02:00
Jim Ferenczi 5fc503342a Merge pull request #19579 from jimferenczi/docvalue_fields_fetch
Rename FieldDataFieldsContext and FieldDataFieldsFetchSubPhase in DocValueFieldsContext and DocValueFieldsFetchSubPhase
2016-07-25 17:20:27 +02:00
Tanguy Leroux f745c96949 Clean up more messy tests
After #13834 many tests that used Groovy scripts (for good or bad reason) in their tests have been moved in the lang-groovy module and the issue #13837 has been created to track these messy tests in order to clean them up.

This commit moves more tests back in core, removes the dependency on Groovy, changes the scripts in order to use the mocked script engine, and change the tests  to integration tests.
2016-07-25 17:02:49 +02:00
Jim Ferenczi 33461a8432 Rename FieldDataFieldsContext and FieldDataFieldsFetchSubPhase in DocValueFieldsContext and DocValueFieldsFetchSubPhase
This change renames the package org.elasticsearch.search.fetch.fielddata in org.elasticsearch.search.fetch.docvalues and renames the
FieldData* classes in DocValue*. This is a follow up of the renaming that happened in #18943
2016-07-25 16:20:59 +02:00
Ali Beyad 299b8a7a52 Removes unnecessary blobExists() check before reading a blob in the
Azure and Google cloud blob containers, as the APIs for both return
a 404 in the case of a missing object, which we already handle through
a NoSuchFileFoundException.
2016-07-23 23:24:56 -04:00
Ali Beyad a6f5e0b0fe Remove IndexMeta and addresses code review comments 2016-07-23 23:24:56 -04:00
Boaz Leskes cd596772ee Persistent Node Names (#19456)
With #19140 we started persisting the node ID across node restarts. Now that we have a "stable" anchor, we can use it to generate a stable default node name and make it easier to track nodes over a restarts. Sadly, this means we will not have those random fun Marvel characters but we feel this is the right tradeoff.

On the implementation side, this requires a bit of juggling because we now need to read the node id from disk before we can log as the node node is part of each log message. The PR move the initialization of NodeEnvironment as high up in the starting sequence as possible, with only one logging message before it to indicate we are initializing. Things look now like this:

```
[2016-07-15 19:38:39,742][INFO ][node                     ] [_unset_] initializing ...
[2016-07-15 19:38:39,826][INFO ][node                     ] [aAmiW40] node name set to [aAmiW40] by default. set the [node.name] settings to change it
[2016-07-15 19:38:39,829][INFO ][env                      ] [aAmiW40] using [1] data paths, mounts [[ /(/dev/disk1)]], net usable_space [5.5gb], net total_space [232.6gb], spins? [unknown], types [hfs]
[2016-07-15 19:38:39,830][INFO ][env                      ] [aAmiW40] heap size [1.9gb], compressed ordinary object pointers [true]
[2016-07-15 19:38:39,837][INFO ][node                     ] [aAmiW40] version[5.0.0-alpha5-SNAPSHOT], pid[46048], build[473d3c0/2016-07-15T17:38:06.771Z], OS[Mac OS X/10.11.5/x86_64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_51/25.51-b03]
[2016-07-15 19:38:40,980][INFO ][plugins                  ] [aAmiW40] modules [percolator, lang-mustache, lang-painless, reindex, aggs-matrix-stats, lang-expression, ingest-common, lang-groovy, transport-netty], plugins []
[2016-07-15 19:38:43,218][INFO ][node                     ] [aAmiW40] initialized
```

Needless to say, settings `node.name` explicitly still works as before.

The commit also contains some clean ups to the relationship between Environment, Settings and Plugins. The previous code suggested the path related settings could be changed after the initial Environment was changed. This did not have any effect as the security manager already locked things down.
2016-07-23 22:46:48 +02:00
Jason Tedor 2d1b0587dd Introduce Netty 4
This commit adds transport-netty4, a transport and HTTP implementation
based on Netty 4.

Relates #19526
2016-07-22 22:26:35 -04:00
Mike McCandless 98c39533d7 Guard against negative result from FileStore.getUsableSpace when picking data path for a new shard 2016-07-22 15:02:31 -04:00
Ali Beyad d9ec959dfc Index folder names now use a UUID (not the index UUID but one specific
to snapshot/restore) and the index to UUID mapping is stored in the
repository index file.
2016-07-22 13:59:13 -04:00
Ali Beyad a0a4d67eae All snapshot metadata files use UUID for the blob ID 2016-07-22 13:52:13 -04:00
Ali Beyad 630218a16f Change the BlobContainer interface to throw a NoSuchFileFoundException
for reads and deletes if the blob does not exist.
2016-07-22 13:49:25 -04:00
Ali Beyad abaf8443e5 More robust handling of snapshot deletions
Makes deleting snapshots more robust by first deleting the
snapshot from the index generational file, then handling
individual deletion file errors with log messages instead of
failing the entire operation.
2016-07-22 13:49:25 -04:00
gfyoung 6a9f488b17 Caught exceptions during compromised snapshot deletion 2016-07-22 13:48:45 -04:00
gfyoung 95a118d9c6 Changed Files.deleteIfExists to Files.delete in FsBlobContainer 2016-07-22 13:48:45 -04:00
gfyoung dfcdadb59f Added HdfsBlobStoreContainer tests
Added BlobContainer tests for HDFS storage
and caught a bug at the same time in which
deleteBlob was not raising an IOException
when the blobName did not exist.
2016-07-22 13:48:45 -04:00
gfyoung b02a6da8fd Properly raise IOException for Azure, Fs, Hdfs, and S3 2016-07-22 13:48:45 -04:00
gfyoung 0620a3d6c2 Raised IOException on deleteBlob
Closes gh-18530.
2016-07-22 13:48:45 -04:00
Jason Tedor c27237be9f Revert "Allow to listen on virtual interfaces"
This reverts commit 4cb8b620c3.
2016-07-22 13:30:05 -04:00
Michael Nitschinger 4cb8b620c3 Allow to listen on virtual interfaces
Previously when trying to listen on virtual interfaces during
bootstrap the application would stop working - the interface
couldn't be found by the NetworkUtils class.

The NetworkUtils utilize the underlying JDK NetworkInterface
class which, when asked to lookup by name only takes physical
interfaces into account, failing at virtual (or subinterfaces)
ones (returning null).

Note that when interating over all interfaces, both physical and
virtual ones are taken into account.

This changeset asks for all known interfaces, iterates over them
and matches on the given name as part of the loop, allowing it
to catch both physical and virtual interfaces.

As a result, elasticsearch can now also serve on virtual
interfaces.

A test case has been added which at least makes sure that all
iterable interfaces can be found by their respective name. (It's
not easily possible in a unit test to "fake" virtual interfaces).

Relates #19537
2016-07-22 12:33:21 -04:00
Ali Beyad 2b9cfff90f Fixes CORS handling so that it uses the defaults
Fixes CORS handling so that it uses the defaults for http.cors.allow-methods
and http.cors.allow-headers if none are specified in the config.

Closes #19520
2016-07-22 12:25:28 -04:00
Boaz Leskes bd574d92ae Verify lower level transport exceptions don't bubble up on disconnects (#19518)
#19096 introduced a generic TCPTransport base class so we can have multiple TCP based transport implementation. These implementations can vary in how they respond internally to situations where we concurrently send, receive and handle disconnects and can have different exceptions. However, disconnects are important events for the rest of the code base and should be distinguished from other errors (for example, it signals TransportMasterAction that it needs to retry and wait for the a (new) master to come back).  Therefore, we should make sure that all the implementations do the proper translation from their internal exceptions into ConnectTransportException which is used externally. 

Similarly we should make sure that the transport implementation properly recognize errors that were caused by a disconnect as such and deal with them correctly. This was, for example, the source of a build failure at https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-intake/1080 , where a concurrency issue cause SocketException to bubble out of MockTcpTransport.

This PR adds a tests which concurrently simulates connects, disconnects, sending and receiving and makes sure the above holds. It also fixes anything (not much!) that was found it.
2016-07-22 14:35:47 +02:00
Tal Levy 19e7b1c737 fix: no other processors should be executed after on_failure is called in a compound processor (#19545) 2016-07-21 14:27:04 -07:00
Ali Beyad 9765b4a6ff Fixes the ActiveShardsObserverIT tests that have a very short index (#19540)
creation timeout so they process the index creation cluster state update
before the test finishes and attempts to cleanup. Otherwise, the index
creation cluster state update could be processed after the test finishes
and cleans up, thereby leaking an index in the cluster state that could
cause issues for other tests that wouldn't expect the index to exist.

Closes #19530
2016-07-21 11:47:21 -04:00
Yannick Welsch d4771b993f Use executor's describeTasks method to log task information in cluster service (#19531)
This fixes the log output in some places of ClusterService where the executor's describeTasks wasn't used to log task information.
2016-07-21 14:32:37 +02:00
David Pilato 5e57febe53 Add DiscoveryPlugin interface
So we have a Pull interface easier to use which reduce the need of Guice.

See 2a9d7f68a1 (commitcomment-18335161)
2016-07-21 11:35:29 +02:00
David Pilato 2a9d7f68a1 Move custom name resolver registration to the NetworkModule
As explained in https://github.com/elastic/elasticsearch/pull/15765#discussion_r65804713
2016-07-21 10:27:38 +02:00
Simon Willnauer 302c7a521a Fix analyzer alias processing (#19506)
In the lack of tests the analyzer.alias feature was pretty much not working
at all on current master. Issues like #19163 showed some serious problems for users
using this feature upgrading to an alpha version.
This change fixes the processing order and allows aliases to be set for
existing analyzers like `default`. This change also ensures that if `default`
is aliased the correct analyzer is used for `default_search` etc.

Closes #19163
2016-07-21 09:32:47 +02:00
Jun Ohtani cebad703fe Analyze: Specify anonymous char_filters/tokenizer/token_filters in the analyze API
Add parser for anonymous char_filters/tokenizer/token_filters
Using Settings in AnalyzeRequest for anonymous definition
Add breaking changes document

Closed #8878
2016-07-21 11:06:36 +09:00
Tal Levy f7cd86ef6d rethrow script compilation exceptions into ingest configuration exceptions (#19318)
* rethrow script compilation exceptions into ingest configuration exceptions
* update readProcessor to rethrow any exception as an ElasticsearchException
2016-07-20 10:37:56 -07:00
Nik Everett 3a82c613e4 Migrate query registration from push to pull
Remove `ParseField` constants used for names where there are no deprecated
names and just use the `String` version of the registration method instead.

This is step 2 in cleaning up the plugin interface for extending
search time actions. Aggregations are next.

This is breaking for plugins because those that register a new query should
now implement `SearchPlugin` rather than `onModule(SearchModule)`.
2016-07-20 12:33:51 -04:00
Yannick Welsch 2cf94d2d8a Fix race in testCreateIndexWaitsForAllActiveShards
When index creation is not acknowledged (due to a very low request timeout) it is possible that the index is still created.
If a subsequent index-exists request completes before the cluster state of the index creation has been fully applied, it
might miss the newly created index.
2016-07-20 18:28:28 +02:00
Nik Everett fc4b439635 Remove AggregationStreams and friends
* Remove outdated aggregation registration method
* Remove AggregationStreams
* Adds StreamInput#readNamedWriteableList and
StreamOutput#writeNamedWriteableList convenience methods. We strive to
make the reading and writing from the streams terse so they are easier
to scan visually.
* Remove PipelineAggregatorStreams
* Remove stream info from InternalAggreation.Type
* Remove InternalAggregation#type
* Remove Streamable from PipelineAggregator
* Remove Streamable from MultiBucketsAggregation.Bucket
2016-07-20 09:46:04 -04:00
Daniel Mitterdorfer a4f09d2b81 Restore parameter name auto_generate_phrase_queries (#19514)
During query refactoring the query string query parameter
'auto_generate_phrase_queries' was accidentally renamed
to 'auto_generated_phrase_queries'.

With this commit we restore the old name.

Closes #19512
2016-07-20 13:13:57 +02:00
Martijn van Groningen 9b1a477120 Fix ClusterInfo serialization 2016-07-20 09:16:27 +02:00
Ryan Ernst 0f2d7a84a8 Add tests for disabling positions and copy the check to text fields 2016-07-19 19:07:56 -07:00
Ryan Ernst c85cb37cc4 Mappings: Fix not_analyzed string fields to error when position_increment_gap is set
Currently if a string field is not_analyzed, but a
position_increment_gap is set, it will lookup the default analyzer and
set it, along with the position_increment_gap, before the code which
handles setting the keyword analyzer for not_analyzed fields has a
chance to run. This change adds a parsing check and test for that case.
2016-07-19 17:54:13 -07:00
Jason Tedor 128f0276d9 Fix Javadocs for ThreadPool#schedule
This commit fixes an issue with an @throws tag on ThreadPool#schedule
not containing a description.
2016-07-19 18:35:30 -04:00
Jason Tedor 770186f6cf Catch the right rejected execution exception
ThreadPool#schedule can throw a rejected execution exception. Yet, the
rejected execution exception that it throws comes from the EsAbortPolicy
which throws an EsRejectedExecutionException. This exception does not
inherit from RejectedExecutionException so instead we must catch the
former instead of the latter.
2016-07-19 16:45:12 -04:00
Jason Tedor 720b53b018 Handle rejected execution exception on reschedule
A self-rescheduling runnable can hit a rejected execution exception but
this exception goes uncaught. Instead, this exception should be caught
and passed to the onRejected handler. Not catching handling this
rejected execution exception can lead to test failures. Namely, a race
condition can arise between the shutting down of the thread pool and
cancelling of the rescheduling of the task. If another reschedule fires
right as the thread pool is being terminated, the rescheduled task will
be rejected leading to an uncaught exception which will cause a test
failure. This commit addresses these issues.

Relates #19505
2016-07-19 15:35:51 -04:00
Nik Everett 9e2221cae5 Migrate remaining aggregations to NamedWriteable
After this we'll be able to remove AggregationStreams and
PipelineAggregatorStreams.
2016-07-19 14:43:29 -04:00
jaymode 11389638f9 Require executor name when calling scheduleWithFixedDelay
The ThreadPool#scheduleWithFixedDelay method does not make it clear that all scheduled runnable instances
will be run on the scheduler thread. This becomes problematic if the actions being performed include
blocking operations since there is a single thread and tasks may not get executed due to a blocking task.

This change includes a few different aspects around trying to prevent this situation. The first is that
the scheduleWithFixedDelay method now requires the name of the executor that should be used to execute
the runnable. All existing calls were updated to use Names.SAME to preserve the existing behavior.

The second aspect is the removal of using ScheduledThreadPoolExecutor#scheduleWithFixedDelay in favor of
a custom runnable, ReschedulingRunnable. This runnable encapsulates the logic to deal with rescheduling a
runnable with a fixed delay and mimics the behavior of executing using a ScheduledThreadPoolExecutor and
provides a ScheduledFuture implementation that also mimics that of the typed returned by a
ScheduledThreadPoolExecutor.

Finally, an assertion was added to BaseFuture to detect blocking calls that are being made on the scheduler
thread.
2016-07-19 12:47:47 -04:00
Adrien Grand 0854b03f13 Elasticsearch should reject dynamic templates with unknown `match_mapping_type`. #17285
When looking at the logstash template, I noticed that it has definitions for
dynamic temilates with `match_mapping_type` equal to `byte` for instance.
However elasticsearch never tries to find templates that match the byte type
(only long or double as far as numbers are concerned). This commit changes
template parsing in order to ignore bad values of `match_mapping_type` (given
how the logstash template is popular, this would break many upgrades
otherwise). Then I hope to fail the parsing on bad values in 6.0.
2016-07-19 15:38:00 +02:00
Nik Everett a2a7ea1f17 Make ExtendedBounds immutable
We used to mutate it as part of building the aggregation. That
caused assertVersionSerializable to fail because it assumes that
requests aren't mutated after they are sent.

Closes #19481
2016-07-19 08:48:14 -04:00
Yannick Welsch c4fe8e7bf2 Fix replica-primary inconsistencies when indexing during primary relocation with ongoing replica recoveries (#19287)
Primary relocation violates two invariants that ensure proper interaction between document replication and peer recoveries, ultimately leading to documents not being properly replicated.

Invariant 1: Document writes must be replicated based on the routing table of a cluster state that includes all shards which have ongoing or finished recoveries. This is ensured by the fact that do not start a recovery that is not reflected by the cluster state available on the primary node and we always sample a fresh cluster state before starting to replicate write operations.

Invariant 2: Every operation that is not part of the snapshot taken for phase 2, must be succesfully indexed on the target replica (pending shard level errors which will cause the target shard to be failed). To ensure this, we start replicating to the target shard as soon as the recovery start and open it's engine before we take the snapshot. All operations that are indexed after the snapshot was taken are guaranteed to arrive to the shard when it's ready to index them. Note that this also means that the replication doesn't fail a shard if it's not yet ready to recieve operations - it's a normal part of a recovering shard.

With primary relocations, the two invariants can be possibly violated. Let's consider a primary relocating while there is another replica shard recovering from the primary shard.

Invariant 1 can be violated if the target of the primary relocation is so lagging on cluster state processing that it doesn't even know about the new initializing replica. This is very rare in practice as replica recoveries take time to copy all the index files but it is a theoretical gap that surfaces in testing scenarios.

Invariant 2 can be violated even if the target primary knows about the initializing replica. This can happen if the target primary replicates an operation to the intializing shard and that operation arrives to the initializing shard before it opens it's engine but arrives to the primary source after it has taken the snapshot of the translog. Those operations will be currently missed on the new initializing replica.

The fix to reestablish invariant 1 is to ensure that the primary relocation target has a cluster state with all replica recoveries that were successfully started on primary relocation source. The fix to reestablish invariant 2 is to check after opening engine on the replica if the primary has been relocated in the meanwhile and fail the recovery.

Closes #19248
2016-07-19 14:07:58 +02:00
Simon Willnauer f79fb4ada7 Create RecoveryTarget once we reset the source
RecoveryTarget increments a reference on the store once it's
created. If we fail to return the instance from the reset method
we leak a reference causing shard locks to not be released. This
change creates the reference in the return statement to ensure no
references are leaked
2016-07-19 12:27:11 +02:00
Martijn van Groningen 52b1b3e31f allocation explain: Also serialize `includeDiskInfo` field. 2016-07-19 11:54:43 +02:00
Yannick Welsch 79ab6d19af Fix NPE when initializing replica shard has no unassignedInfo (#19491)
An initializing replica shard might not have an UnassignedInfo object, for example when it is a relocation target. The method allocatedPostIndexCreate does not account for this situation.
2016-07-19 11:30:57 +02:00
Simon Willnauer 5b07f81fcf Move `reset recovery` into RecoveriesCollection (#19466)
Today when we reset a recovery because of the source not being
ready or the shard is getting removed on the source (for whatever reason)
we wipe all temp files and reset the recovery without respecting any
reference counting or locking etc. all streams are closed and files are
wiped. Yet, this is problematic since we assert that some files are on disk
etc. when we finish writing a file. These assertions don't hold anymore if we
concurrently wipe the tmp files.

This change moves the logic out of RecoveryTarget into RecoveriesCollection which
basically clones the RecoveryTarget on reset instead which allows in-flight operations
to finish gracefully. This means we now have a single path for cleanups in RecoveryTarget
and can safely use assertions in the class since files won't be removed unless the recovery
is either canceled, failed or finished.

Closes  #19473
2016-07-19 10:23:02 +02:00
Adrien Grand 37e20c6f34 Automatically created indices should honor `index.mapper.dynamic`. #19478
Today they don't because the create index request that is implicitly created
adds an empty mapping for the type of the document. So to Elasticsearch it
looks like this type was explicitly created and `index.mapper.dynamic` is not
checked.

Closes #17592
2016-07-19 09:02:31 +02:00
Nik Everett 7861548786 Migrate serial_diff aggregation to NamedWriteable
This is the last migration before AggregationStreams and
PipelineAggregatorStreams can be removed to remove redundant
code.
2016-07-18 13:00:06 -04:00