Commit Graph

24593 Commits

Author SHA1 Message Date
Ryan Ernst e2854bb3a4 Build: Remove maven central snapshots from repositories (#20567)
We were using maven snapshots during heavy development, but this should
not be something generally available (we should never release depending
on a snapshot version in maven). This change removes the snapshot repo.
If we ever need it temporarily for some reason, we can add it if/when
it is necessary.

relates #20559
2016-09-19 15:06:05 -07:00
Nik Everett 156393be0e Fail build if new doc snippets aren't `// CONSOLE`
This tracks the snippets that probably should be converted to
`// CONSOLE` or `// TESTRESPONSE` and fails the build if the list
of files with such snippets doesn't match the list in `docs/build.gradle`.
Setting the file looks like
```
/* List of files that have snippets that probably should be converted to
 * `// CONSOLE` and `// TESTRESPONSE` but have yet to be converted. Try and
 * only remove entries from this list. When it is empty we'll remove it
 * entirely and have a party! There will be cake and everything.... */
buildRestTests.expectedUnconvertedCandidates = [
  'plugins/discovery-azure-classic.asciidoc',
...
  'reference/search/suggesters/completion-suggest.asciidoc',
]
```

This list is in `build.gradle` because we expect it to be fairly
temporary. In a few months we'll have converted all of the docs and won't
ned it any more.

From now on if you add now docs that contain a snippet that shows an
interaction with elasticsearch you have three choices:
1. Stick `// CONSOLE` on the interactions and `// TESTRESPONSE` on the
responses. The build (specifically (`gradle docs:check`) will test that
these interactions "work". If there isn't a `// TESTRESPONSE` snippet
then "work" just means "Elasticsearch responds with a 200-level response
code and no `WARNING` headers. This is way better than nothing.

2. Add `// NOTCONSOLE` if the snippet isn't actually interacting with
Elasticsearch. This should only be required for stuff like javascript
source code or `curl` against an external service like AWS or GCE. The
snippet will not get "OPEN IN CONSOLE" or "COPY AS CURL" buttons or be
tested.

3. Add `// TEST[skip:reason]` under the snippet. This will just skip the
snippet in the test phase. This should really be reserved for snippets
where we can't test them because they require an external service that
we don't have at testing time.

Please, please, please, please don't add more things to the list. After
all, it sais there'll be cake when we remove it entirely!

Relates to #18160
2016-09-19 16:43:43 -04:00
Ali Beyad 50584c4103 Merge pull request #20532 from rjernst/rolling_upgrades
This PR introduces backward compatibility index tests to test the rolling upgrade process amongst Elasticsearch instances within the same major version. The test executes in three phases. In the first phase, we form a cluster of 2 ES instances on an old version. In the second phase, we keep one of the nodes from the old cluster, kill the other node, but preserve its data directory and start an instance of the current version of ES using the same data directory as the killed instance. In the third phase, we kill the other old version ES instance from the first phase and launch a new instance, using the same data directory as the killed instance. Therefore, during phase 3, we have fully migrated and have all current versions of ES running. In each phase, we run REST tests that index documents and search them, ensuring at each stage that the documents from the previous phase are still there.

Note that because we haven't released a GA yet of 5.0, the tests currently don't start an old version cluster in the first phase. Once GA is released, this will be changed to make the backward compatibility version 5.0, while the current version in the cluster will be 5.x.
2016-09-19 16:14:38 -04:00
Simon Willnauer ee8d14798f Unguice Transport and friends (#20526)
This change removes all guice interaction from Transport, HttpServerTransport,
HttpServer and TransportService. All these classes as well as their subclasses
or extended version configured via plugins are now created by using plain old
bloody java constructors. YAY!
2016-09-19 22:10:47 +02:00
Ali Beyad 8afc83047f Change the timeout of the rolling upgrades test from 40 mins to 5 mins
to still allow accounting for slow VMs
2016-09-19 15:45:41 -04:00
David Pilato ed4d0881b1 Add profile and explain parameters to template API
We can now run templates using `explain` and/or `profile` parameters.
Which is interesting when you have defined a complicated profile but want to debug it in an easier way than running the full query again.

You can use `explain` parameter when running a template:

```js
GET /_search/template
{
  "file": "my_template",
  "params": {
    "status": [ "pending", "published" ]
  },
  "explain": true
}
```

You can use `profile` parameter when running a template:

```js
GET /_search/template
{
  "file": "my_template",
  "params": {
    "status": [ "pending", "published" ]
  },
  "profile": true
}
```
2016-09-19 17:52:13 +02:00
Simon Willnauer bd93b3dce6 Call onNewEngine before the engine is actually published 2016-09-19 17:00:17 +02:00
Simon Willnauer 817d55cf93 Take refresh IOExceptions into account when catching ACE in InternalEngine (#20546)
Since #19975 we are aggressively failing with AssertionError when we catch an ACE
inside the InternalEngine. We treat everything that is neither a tragic even on
the IndexWriter or the Translog as a bug and throw an AssertionError. Yet, if the
engine hits an IOException on refresh of some sort and the IW doesn't realize it since
it's not fully under it's control we fail he engine but neither IW nor Translog are marked
as failed by tragic event while they are already closed.
This change takes the `failedEngine` exception into account and if it's set we know
that the engine failed by some other even than a tragic one and can continue.

This change also uses the `ReferenceManager#RefreshListener` interface in the engine rather
than it's concrete implementation.

Relates to #19975
2016-09-19 15:15:51 +02:00
David Pilato 4c9a77e6c0 Add link to "offline install" and show the full URL 2016-09-19 15:04:29 +02:00
David Pilato 9e58ca835c Merge branch 'master' into doc/plugins-offline-url
# Conflicts:
#	docs/plugins/mapper-attachments.asciidoc
2016-09-19 14:51:55 +02:00
David Pilato a530399ae5 Merge branch 'pr/remove-mapper-attachments' 2016-09-19 14:47:14 +02:00
Jason Tedor 7874463db0 Merge pull request #20553 from jasontedor/node-attributes
Node attributes and REST API spec README
2016-09-19 08:31:09 -04:00
Daniel Mitterdorfer 6f628557b9 Upgrade from JMH 1.12 to JMH 1.14 2016-09-19 14:28:16 +02:00
Daniel Mitterdorfer 9d8961aeb9 Provide log4j2 logging config for microbenchmarks 2016-09-19 14:28:16 +02:00
Boaz Leskes 2ee9ab25d9 Remove `RoutingAllocation.Result` (#20538)
Currently all the reroute-like methods of `AllocationService` return a result object of type `RoutingAllocation.Result`. The result object contains the new `RoutingTable` and `MetaData` plus an indication whether those were changed. The caller is then responsible of updating a cluster state with these. These means that things can easily go wrong and one can take one of these but not the other causing inconsistencies. We already have a utility method on the `ClusterState` builder that does but no one forces you to do so. Also 99% of the callers do the same thing: i.e., check if the result was changed and if so update the very same cluster state that was passed to `AllocationService`.  This PR folds this pattern into `AllocationService` and changes almost all it's methods to return a new cluster state (potentially the original one).  This saves some 500 lines of code.

The one exception here is the reroute API which executes allocation commands and potentially returns an explanation as well (next to the routing table and metadata). That API now returns a `CommandsResult` object which encapsulate a cluster state and the explanation.
2016-09-19 13:54:35 +02:00
Alexander Lin 7cd0316b51 Fix minhash docs level
Relates #20547
2016-09-19 07:54:04 -04:00
Jason Tedor 9082c604c1 Fix command-line arguments in REST API spec
The command-line arguments for Elasticsearch must now be specified using
-E. This commit fixes the usage of command-line arguments in the REST
API spec README.
2016-09-19 07:47:02 -04:00
Jason Tedor b456823434 Fix leftover node attributes usage
Previously node attributes could be set via node.* but this now requires
using node.attr.*. This commit fixes some leftover usages of the old
way.
2016-09-19 07:45:31 -04:00
Tanguy Leroux 656596c2a9 [DOC] Remove obsolete node names from documentation
Funny node names have been removed in #19456 and replaced by UUID. This commit removes these obsolete node names and replace them by real UUIDs in the documentation.

closes #20065
2016-09-19 11:56:28 +02:00
Tanguy Leroux 1894489832 [DOC] Update /_cat/nodes doc
closes #20162
2016-09-19 09:31:48 +02:00
David Pilato dfd1eebdd0 Remove mapper attachments plugin
We now have in 5.0.0 `ingest-attachment` plugin.
We can remove `mapper-attachments` plugin for 6.0.

Closes #18837.
2016-09-19 09:01:16 +02:00
Alexander Guz 135a19e7a1 Added wildcards imports configuration for IntelliJ IDEA
Closes #20534
2016-09-17 17:29:24 -04:00
Alexander Guz c1e8b6a8ba Fixed aggregation by "gender" request example.
`gender.keyword` should be used instead of just `gender` or we
get an error `Fielddata is disabled on text fields by default. Set
fielddata=true on [gender] in order to load fielddata in memory by
uninverting the inverted index. Note that this can however use

Closes #20535
significant memory.`
2016-09-17 17:23:27 -04:00
Ali Beyad 98230d035a Adds a preserveIndicesUponCompletion method to ESRestTestCase
that can be overridden by subclasses if the test must not
delete indices it created after exiting.
2016-09-16 19:21:26 -04:00
Ali Beyad 83adc87015 Removes stopNodeUponCompletion in favor of moving the stop
nodes task to the final part of the cluster test task execution
graph.
2016-09-16 16:32:43 -04:00
Ali Beyad f608e6c6cf Improves the documentation for the (#20531)
`cluster.routing.allocation.cluster_concurrent_rebalance` setting,
clarifying in which shard allocation situations the rebalance limit
takes effect.

Closes #20529
2016-09-16 16:06:18 -04:00
Luca Cavanna 7fd66a4a41 Merge pull request #20521 from javanna/test/random_search_ext
introduce test plugin to inject random search ext elements in search request tests
2016-09-16 21:05:55 +02:00
javanna 7097f4943c [TEST] delete specific index boost serialization test, already covered by testSerialization
indexBoost is already randomly set, we don't need a specific test for it in SearchSourceBuilderTests
2016-09-16 21:03:27 +02:00
javanna 20badcdbab [TEST] set back the visibility of our search ext plugin in FetchSubPhasePluginIT to private
This plugin is not used in other tests anymore, it should be private to make sure its usage doesn't spread again
2016-09-16 21:03:27 +02:00
javanna 4c726311e2 [TEST] introduce test plugin to inject random search ext elements in search request tests
A few of our unit tests generate a random search request body nd run tests against it. The source can optionally contain ext elements under the ext sections, which can be parsed by plugins. With this commit we introduce a plugin so that the tests don't use the one from FetchSubPhasePluginIT anymore. They rather generate multiple search ext elements. The plugin can parse and deal with all those. This extends the test coverage as we may have multiple elements with random names.

Took the chance to introduce a common test base class for search requests, called AbstractSearchTestCase, given that the setup phase is the same for all three tests around search source. Then we can have the setup isolated to the base class and the subclasses relying on it.

Closes #17685
2016-09-16 21:03:27 +02:00
Luca Cavanna 629e2b2aff Throw error if query element doesn't end with END_OBJECT (#20528)
* Throw error if query element doesn't end with END_OBJECT

Followup to #20515 where we added validation that after we parse a query within a query element, we should not get a field name. Truth is that the only token allowed at that point is END_OBJECT, as our DSL allows only one single query within the query object:

```
{
  "query" : {
    "term" : { "field" : "value" }
  }
}
```

We can then check that after parsing of the query we have an end_object that closes the query itself (which we already do). Following that we can check that the query object is immediately closed, as there are no other tokens that can be present in that position.

Relates to #20515
2016-09-16 21:02:12 +02:00
Nik Everett 697adfb3c4 Fix up tasks integ test
I'd made some mistakes that hadn't caused the test to fail but did
slow it down and partially invalidate some of the assertions. This
fixes those mistakes.
2016-09-16 12:18:41 -04:00
Alexander Guz 8ec94a4ba0 Edited response structure on indexing a document (#20517)
Added "_shards" and "result" keys to the response.
2016-09-16 10:16:40 -06:00
Ali Beyad 22e6bc8359 Disables unit tests for rolling upgrades, as there are only
rest integration tests.
2016-09-16 11:32:28 -04:00
Ali Beyad ce86ed1fdd Merge remote-tracking branch 'upstream/master' into rolling_upgrades 2016-09-16 10:43:38 -04:00
Ali Beyad 56f97500c6 In the rolling upgrades tests, we do not want to stop nodes
automatically between tasks, as we want some of the nodes from
the previous task to continue running in the next task. This
commit enables a cluster configuration setting to not stop
nodes automatically after a task runs, but instead the creator
of the test task must stop the running nodes explicitly in a
cleanup phase.
2016-09-16 10:36:55 -04:00
Jim Ferenczi d0f4bc16ca Fix FieldStats deserialization of `ip` field (#20522)
* Fix FieldStats deserialization of `ip` field

Add missing readBytes in `ip` field deserialization
Add (de)serialization tests for all types
This change also removes the ability to set FieldStats.minValue or FieldStats.maxValue to null.
This is not required anymore since the stats are built on fields with values only.

Fixes #20516
2016-09-16 16:15:46 +02:00
Nicholas Knize 01a6b7c408 [TEST] Refactor Geo test names to follow naming and style convention
This commit closes a stale issue where GeoJson parsing tests used a combination of underscore and camel case.

closes #8998
2016-09-16 09:13:38 -05:00
Simon Willnauer 3aabda6752 Ensure elasticsearch doesn't start with unuspported indices (#20514)
If an index was created with pre 2.0 we should not treat it as supported
even if all segments have been upgraded to a supported lucene version.

Closes #20512
2016-09-16 16:11:53 +02:00
Ali Beyad ba072ec18e When Elasticsearch nodes are started in gradle to form a
cluster, we wait for the cluster health to indicate the
necessary nodes have formed a cluster.  This check was an
exact value (equality) check.  However, if we are trying to
connect the nodes in the cluster to nodes from a previously
formed cluster (of the same name), then we will have more
nodes returned by the cluster health check than the current
task's configured number of nodes.  Hence, this check needs
to be a >= check.  This commit fixes it.
2016-09-16 09:41:26 -04:00
Simon Willnauer f5daa165f1 Remove ability to plug-in TransportService (#20505)
TransportService is such a central part of the core server, replacing
it's implementation is risky and can cause serious issues. This change removes the ability to
plug in TransportService but allows registering a TransportInterceptor that enables
plugins to intercept requests on both the sender and the receiver ends. This is a commonly used
and overwritten functionality but encapsulates the custom code in a contained manner.
2016-09-16 09:47:53 +02:00
Boaz Leskes 577dcb3237 Add current cluster state version to zen pings and use them in master election (#20384)
During a networking partition, cluster states updates (like mapping changes or shard assignments)
are committed if a majority of the masters node received the update correctly. This means that the current master has access to enough nodes in the cluster to continue to operate correctly. When the network partition heals, the isolated nodes catch up with the current state and get the changes they couldn't receive before. However, if a second partition happens while the cluster
is still recovering from the previous one *and* the old master is put in the minority side, it may be that a new master is elected which did not yet catch up. If that happens, cluster state updates can be lost.

This commit fixed 95% of this rare problem by adding the current cluster state version to `PingResponse` and use them when deciding which master to join (and thus casting the node's vote). 

Note: this doesn't fully mitigate the problem as a cluster state update which is issued concurrently with a network partition can be lost if the partition prevents the commit message (part of the two phased commit of cluster state updates) from reaching any single node in the majority side *and* the partition does allow for the master to acknowledge the change. We are working on a more comprehensive fix but that requires considerate work  and is targeted at 6.0.
2016-09-15 23:39:11 +02:00
Areek Zillur 0556c93920 Merge pull request #20488 from areek/docs/clarify_plugins_installation
[DOC] Add note requiring full cluster restart for installing plugins with custom metadata
2016-09-15 17:05:27 -04:00
Areek Zillur 9b17242b5d [DOC] Add note for full cluster restart for installing plugins using custom metadata
Currently, we check if a node has the same set of custom metadata as the master
before joining the cluster. This implies freshly installing a plugin that has its
custom metadata requires a full cluster restart.
2016-09-15 17:04:45 -04:00
Simon Willnauer 1e2ef192c9 [TEST] Reduce the number of docs per indexRandom in FieldSortIT#testIssue6614 2016-09-15 22:56:26 +02:00
Areek Zillur 8c12b7c3b6 Merge pull request #20515 from areek/fix/query_validation
Fix silently accepting malformed queries
2016-09-15 16:53:17 -04:00
Areek Zillur bdad62a2f8 Fix silently accepting malformed queries
Currently, we silently accept malformed query where more
than one key is defined at the top-level for query object.
If all the keys have a valid query body, only the last query
is executed, besides throwing off parsing for additional suggest,
aggregation or highlighting defined in the search request.

This commit throws a parsing exception when we encounter a query
with multiple keys.

closes #20500
2016-09-15 16:32:28 -04:00
Boaz Leskes 2233d48235 add a reduced TCP_CONNECT_TIMEOUT setting to DiscoveryWithServiceDisruptionsIT
The default of 30s causes some tests to timeout when running ensureGreen and similar. This is because network delays simulation blocks connect until either the connect timeout expires or the disruption configured time stops. We do *not* immediately connect when the disruption is stopped.
2016-09-15 21:50:24 +02:00
Ali Beyad ec7e383783 Better checks for the cluster being up in the rolling upgrades tests. 2016-09-15 15:08:11 -04:00
Nik Everett d0be96df7b Clean up snapshots after each REST test
The only repository we can be sure is safe to clean is `fs` so we clean
any snapshots in those repositories after each test. Other repositories
like url and azure tend to throw exceptions rather than let us fetch
their contents during the REST test. So we clean what we can....

Closes #18159
2016-09-15 14:49:11 -04:00