Commit Graph

3684 Commits

Author SHA1 Message Date
Martijn van Groningen 3fcb95b814 percolator: Fail indexing percolator queries containing either a has_child or has_parent query.
Closes #2960
2016-08-31 07:46:17 +02:00
Jason Tedor abf8a1a3f0 Avoid allocating log parameterized messages
This commit modifies the call sites that allocate a parameterized
message to use a supplier so that allocations are avoided unless the log
level is fine enough to emit the corresponding log message.
2016-08-30 18:17:09 -04:00
Jason Tedor 7da0cdec42 Introduce Log4j 2
This commit introduces Log4j 2 to the stack.
2016-08-30 13:31:24 -04:00
Jack Conradson c8f57df5cf Merge branch 'master' into deprecate 2016-08-30 09:11:18 -07:00
Jack Conradson 7930233527 Deprecate Groovy, Python, and Javascript scripts. 2016-08-30 09:06:18 -07:00
Jason Tedor 0df92a8da8 Upgrade to Netty 4.1.5
This commit upgrades the Netty dependencies from version 4.1.4 to
version 4.1.5. This upgrade brings several bug fixes including the
removal of a obnoxious and scary-looking log message when unsafe is
explicitly disabled.

Relates #20222
2016-08-30 05:46:40 -04:00
Martijn van Groningen 1925813e09 ingest: Fix rename processor change rename leaf fields into branch fields
Instead of get, set and remove we do get, remove and then set to avoid type conflicts in IngestDocument.
If the set still fails we try to restore the original field in ingest document.

Closes #19892
2016-08-30 07:38:01 +02:00
Tanguy Leroux 9727f123b9 Rename Netty TCP transports thread factories from http_* to transport_*
Netty3/4 TcpTransport implementations are creating thread factories with a "http_server" thread prefix whereas it should start with "transport_server" and let the "http_server" prefix for the HttpServerTransport implementations.
2016-08-29 13:49:52 +02:00
Jun Ohtani 2a00c9dc46 Merge pull request #19860 from johtani/fix/validate_empty_field_name
Validate blank field name
2016-08-29 11:52:18 +09:00
Martijn van Groningen 48926b4d66 ingest: don't render template twice for append processor 2016-08-26 18:07:32 +02:00
Chris Earle bd0b06440e Add "Async" to the end of each Async RestClient method
This makes it much harder to accidentally miss the Response.
2016-08-26 10:51:33 -04:00
Jun Ohtani 450f47d5b5 Validate blank field name
add validation and validate only 5.0+
Add tests before 5.0

Closes #19251
2016-08-26 20:10:33 +09:00
Jason Tedor bc136a90d5 Add network types to cluster stats
The network types in use on a cluster can be useful information to have,
so this commit adds aggregate metrics for the network types in use in a
cluster to the cluster stats.

Relates #20144
2016-08-25 21:08:05 -04:00
Jason Tedor 5a48ad661d Address race condition in HTTP pipeline tests
The Netty 4 HTTP server pipeline tests contains two different test
cases. The general idea behind these tests is to submit some requests to
a Netty 4 HTTP server, one test with pipelining enabled and another test
with pipelining disabled. These requests are submitted to two endpoints,
one with a path like /{id} and another with a path like /slow with a
query string parameter sleep. This parameter tells the request handler
how long to sleep for before replying. The idea is that in the case of
the pipelining enabled tests, the requests should come back exactly in
the order submitted, even with some of the requests hitting the slow
endpoint with random sleep durations; this is the guarantee that
pipelining provides. And in the case of the pipelining disabled tests,
requests were randombly submitted to /{id} and /slow with sleep
parameters starting at 600ms and increasing by 100ms for each slow
request constructed. We would expect the requests to come back with the
all the responses to the /{id} requests first because these requests
will execute instantaneously, and then the responses to the /slow
requests. Further, it was expected that the slow requests would come
back ordered by the length of the sleep, the thinking being that 100ms
should be enough of a difference between each request that we would
avoid any race conditions. Sadly, this is not the case, the threads do
sometimes hit race conditions.

This commit modifies the HTTP server pipelining tests to address this
race condition. The modification is that the query string parameter on
the /slow endpoint is removed in favor of just submitting requests to
the path /slow/{id}, where id just used a marker to distinguish each
request. The server chooses a random sleep of at least 500ms for each
request on the slow path. The assertion here then is that the /{id}
responses arrive first, then then /slow responses. We can not make an
assertion on the order of the responses, but we can assert that we did
see every expected response.

Relates #19845
2016-08-25 14:34:11 -04:00
Jack Conradson 0fdadf4737 Merge branch 'master' into break 2016-08-25 09:26:04 -07:00
Michael McCandless 1fe3e36934 Merge pull request #20147 from mikemccand/lucene_620_upgrade
Upgrade to Lucene 6.2.0
2016-08-25 06:03:34 -04:00
Jack Conradson 3deea3dbde Made for/each break tests more robust in Painless. 2016-08-24 15:17:18 -07:00
Mike McCandless 0ccfe69789 Upgrade to Lucene 6.2.0 2016-08-24 17:26:28 -04:00
Jack Conradson c60885b5d4 Fix break bug in for/foreach loops. 2016-08-24 14:25:54 -07:00
Igor Motov b36fbc4452 Add support for parameters to the script ingest processor
The script processor should support `params` to be consistent with all other script consumers.
2016-08-24 16:49:48 -04:00
Jim Ferenczi 4682fc34ae Add the ability to disable the retrieval of the stored fields entirely
This change adds a special field named _none_ that allows to disable the retrieval of the stored fields in a search request or in a TopHitsAggregation.

To completely disable stored fields retrieval (including disabling metadata fields retrieval such as _id or _type) use _none_ like this:

````
POST _search
{
   "stored_fields": "_none_"
}
````
2016-08-24 16:40:08 +02:00
Clinton Gormley abc025e18b Fixed the reindex_rethrottle REST tests
The API was renamed from reindex.rethrottle to reindex_rethrottle
2016-08-24 14:55:02 +02:00
Areek Zillur 80ca78479f Make bulk item-level requests implement DocumentRequest interface
Currently, bulk item requests can be any ActionRequest, this commit
restricts bulk item requests to DocumentRequest. This simplifies
handling failures during bulk requests. Additionally, a new enum
is added to DocumentRequest to represent the intended operation
to be performed by a document request. Now, index operation type
also uses the new enum to specify whether the request should
create or index a document.
2016-08-23 10:33:37 -04:00
Jack Conradson 131e370a16 Make Painless the default scripting language.
Closes #20017
2016-08-22 17:38:02 -07:00
Nik Everett 312a7d45ba Wait for task to start in reindex test
`RethrottleTests#testReindex` fail in CI:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-intake/1274/console

I was unable to reproduce it locally but it *looks* like a race to start
the task. So I've added a wait for it to start just in case.
2016-08-17 12:08:55 -04:00
Nik Everett 39d8f5f123 Reindex tests should expect the right failure
Reindex intentionally tries to fail the search operation to make sure
that the exception flows back. The exception message changed so we
should catch the appropriate exception.
2016-08-17 10:25:38 -04:00
Nik Everett 34bbd27f84 Fix _update_by_query's ingest pipeline support
It wasn't being serialized so it wasn't working with the transport
client.
2016-08-16 16:03:18 -04:00
Nik Everett 862843ec90 Suppress failing test
This test was failing in the presence of transport clients. This turns
off transport clients while I fix the test so it doesn't fail for
everyone in the mean time.
2016-08-16 15:12:40 -04:00
Ryan Ernst 743d9fd008 Merge branch 'master' into search_parser 2016-08-16 11:28:59 -07:00
Nik Everett fdd50612ae Fix reindex under the transport client
The big change here is cleaning up the `TaskListResponse` so it doesn't
have a breaky `toString` implementation. That was causing the reindex
tests to break.

Also removed `NetworkModule#registerTaskStatus` which is part of the
Plugin API. Use `Plugin#getNamedWriteables` instead.
2016-08-16 12:15:15 -04:00
Nik Everett 46bf8baf2e Switch aggregation registration for push to pull
Adds `getAggregations` to `SearchPlugin` which can be used to register
aggregations.

Fixup MockNode which wasn't createing MockBigArrays.
2016-08-16 09:08:36 -04:00
Ryan Ernst 7fde410586 Internal: Consolidate search parser registries
Parsing a search request is currently split up among a number of
classes, using multiple public static methods, which take multiple
regstries of elements that may appear in the search request like query
parsers and aggregations. This change begins consolidating all this code
by collapsing the registries normally used for parsing search requests
into a single SearchRequestParsers class. It is also made available to
plugin services to enable templating of search requests.  Eventually all
of the actual parsing logic should move to the class, and the registries
should be hidden, but for now they are at least co-located to reduce the
number of objects that must be passed around.
2016-08-16 01:59:24 -07:00
Nik Everett 1452ab4b9f Squash the rest of o.e.rest.action
Squashes all the subpackages of `org.elasticsearch.rest.action` down to
the following:
* `o.e.rest.action.admin` - Administrative actions
* `o.e.rest.action.cat` - Actions that make tables for `grep`ing
* `o.e.rest.action.document` - Actions that act on documents
* `o.e.rest.action.ingest` - Actions that act on ingest pipelines
* `o.e.rest.action.search` - Actions that search

I'm tempted to merge `search` into `document` but the `document`
package feels fairly complete as is and `Suggest` isn't actually always
about documents either....

I'm also tempted to merge `ingest` into `admin.cluster` because the
latter contains the actions for dealing with stored scripts.

I've moved the `o.e.rest.action.support` into `o.e.rest.action`.

I've also added `package-info.java`s to all packges in `o.e.rest`. I
figure if the package is too small to deserve a `package-info.java` file
then it is too small to deserve to be a package....

Also fixes checkstyle in all moved classes.
2016-08-15 21:06:32 -04:00
Tal Levy 84bf24b1e9 remove ability to set field value in script-processor configuration (#19981) 2016-08-15 10:57:39 -07:00
Igor Motov 10a766704e Rename Task Persistence into Storing Task Results
The term persisted task was used to indicate that a task should store its results upon its completion. We would like to use this term to indicate that a task can survive restart of nodes instead. This commit removes usages of the term "persist" when it means store results.
2016-08-15 10:02:43 -04:00
Nik Everett cf6e1a4362 Move all FetchSubPhases to `o.e.search.fetch.subphase`
As the most complicated `FetchSubPhase` highlighting gets its own package
(`o.e.seach.fetch.subphase.highlight`. No other `FetchSubPhase`s get their
own package. Instead they all reside together in `o.e.search.fetch.subphase`.

Add package descriptions to `o.e.search.fetch` and subpackages.
2016-08-12 18:21:15 -04:00
Lee Hinman f832eaa6d2 Merge remote-tracking branch 'dakrone/handle-groovy-errors-better' 2016-08-12 13:07:13 -06:00
Lee Hinman aab8c1f032 Catch AssertionError and NoClassDefFoundError in groovy scripts
Previously, we only caught subclasses of Exception, however, there are
some cases when Errors are thrown instead of Exceptions. These two cases
are `assert` and when a class cannot be found.

Without this change, the exception would bubble up to the
`uncaughtExceptionHandler`, which would in turn, exit the JVM
(related: #19923).

A note of difference between regular Java asserts and Groovy asserts,
from http://docs.groovy-lang.org/docs/latest/html/documentation/core-testing-guide.html

"Another important difference from Java is that in Groovy assertions are
enabled by default. It has been a language design decision to remove the
possibility to deactivate assertions."

In the event that a user uses an assert such as:

```groovy
def bar=false; assert bar, "message";
```

The GroovyScriptEngineService throws a NoClassDefFoundError being unable
to find the `java.lang.StringBuffer` class. It is *highly* recommended
that any Groovy scripting user switch to using regular exceptions rather
than unconfiguration Groovy assertions.

Resolves #19806
2016-08-12 13:06:49 -06:00
Nik Everett 9f8f2ea54b Remove ESIntegTestCase#pluginList
It was a useful method in 1.7 when javac's type inference wasn't as
good, but now we can just replace it with `Arrays.asList`.
2016-08-11 15:44:02 -04:00
Luca Cavanna a80a35ebc4 Merge pull request #19961 from javanna/fix/reindex_repleaceable
update and delete by query requests to implement IndicesRequest.Replaceable
2016-08-11 21:10:58 +02:00
javanna 4424d2263f UpdateByQueryRequest to implement IndicesRequest.Replaceable rather than CompositeIndicesRequest
Update by query is a shortcut to search + index. UpdateByQueryRequest gets serialized on the transport layer only when the transport client is used. Given that the request supports wildcards and allows to set its indices, it should implement IndicesRequest.Repleaceable. implementing CompositeIndicesRequest makes little sense as the indices that the request works against depend entirely on the inner search request.
2016-08-11 18:11:26 +02:00
javanna 11d770dde3 DeleteByQueryRequest to implement IndicesRequest.Replaceable
Delete by query is a shortcut to search + delete. DeleteByQueryRequest gets serialized on the transport layer only when the transport client is used. Given that the request supports wildcards and allows to set its indices, it should implement IndicesRequest.Repleaceable
2016-08-11 18:11:26 +02:00
Nik Everett e07e5d66fa Make reindex and lang-javascript compatible
Fixes two issues:
1. lang-javascript doesn't support `executable` with a `null` `vars`
parameters. The parameter is quite nullable.
2. reindex didn't support script engines who's `unwrap` method wasn't
a noop. This didn't come up for lang-groovy or lang-painless because
both of those `unwrap`s were noops. lang-javascript copys all maps that
it `unwrap`s.

This adds fairly low level unit tests for these fixes but dosen't add
an integration test that makes sure that reindex and lang-javascript
play well together. That'd make backporting this difficult and would
add a fairly significant amount of time to the build for a fairly rare
interaction. Hopefully the unit tests will be enough.
2016-08-11 09:54:03 -04:00
Jack Conradson f38d64039b Catch OutOfMemory and StackOverflow errors in Painless since it's safe
to do so.
2016-08-10 18:09:45 -07:00
Ali Beyad 04e8272607 Fixes netty4 module's CORS config to use defaults (#19874)
Fixes netty4 CORS config to use defaults for `http.cors.allow-methods`
and `http.cors.allow-headers` when none are specified.
2016-08-10 16:47:21 -04:00
Adrien Grand 0d6ac57acf Collapse o.e.index.mapper packages. #19921
I also reduced the visibility of a couple classes and renamed/consolidated some
test classes for consistency, eg. removing the `Simple` prefix or using the
`<Type>FieldMapperTests` convention for testing field mappers.
2016-08-10 17:51:11 +02:00
Tanguy Leroux cefc346d37 [Test] Add test for Netty4 HTTP support of header 100-continue (#19908)
* [Test] Add test for Netty4 HTTP support of header 100-continue

Related to #19904

* Update after David comments
2016-08-10 07:22:18 -04:00
Jason Tedor f025c83a17 Fix expect 100 continue header handling
Due to a misordering of the HTTP handlers, the Netty 4 HTTP server
mishandles Expect: 100-continue headers from clients. This commit fixes
this issue by ordering the handlers correctly.

Relates #19904
2016-08-10 07:21:22 -04:00
javanna 2c44278ce8 [TEST] use ParseField instead of plain strings in query tests 2016-08-10 12:21:25 +02:00
javanna 0a98b5e56e [TEST] make AbstractQueryTestCase#testUnknownObjectException more accurate
testUnknownObjectException used to generate malformed json objects in some cases, due to the existence of arrays as it was not closing the injected object correctly. That is why the test was catching JsonParseException among the exception that are expected to be thrown. That is fixed by tracking where the new object is placed and placing its end object marker to the right level rather than always at the end.

Also introduced a mechanism to explicitly declare objects that won't cause any exception when they get additional objects injected, so that there is no need to override the method anymore as that caused copy pasting of the whole test method. This also makes sure that changes are reflected in tests, as those inner objects are not skipped but we actually check that what is declared is true (no exceptions get thrown when an additional object is added within them.
2016-08-10 11:48:51 +02:00
David Pilato 90dbce9682 Merge branch 'fix/19772-toString' 2016-08-09 20:37:27 +02:00
Lee Hinman 5849c488b5 Merge remote-tracking branch 'dakrone/compliation-breaker' 2016-08-09 11:57:26 -06:00
Lee Hinman 2be52eff09 Circuit break the number of inline scripts compiled per minute
When compiling many dynamically changing scripts, parameterized
scripts (<https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-scripting-using.html#prefer-params>)
should be preferred. This enforces a limit to the number of scripts that
can be compiled within a minute. A new dynamic setting is added -
`script.max_compilations_per_minute`, which defaults to 15.

If more dynamic scripts are sent, a user will get the following
exception:

```json
{
  "error" : {
    "root_cause" : [
      {
        "type" : "circuit_breaking_exception",
        "reason" : "[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead",
        "bytes_wanted" : 0,
        "bytes_limit" : 0
      }
    ],
    "type" : "search_phase_execution_exception",
    "reason" : "all shards failed",
    "phase" : "query",
    "grouped" : true,
    "failed_shards" : [
      {
        "shard" : 0,
        "index" : "i",
        "node" : "a5V1eXcZRYiIk8lecjZ4Jw",
        "reason" : {
          "type" : "general_script_exception",
          "reason" : "Failed to compile inline script [\"aaaaaaaaaaaaaaaa\"] using lang [painless]",
          "caused_by" : {
            "type" : "circuit_breaking_exception",
            "reason" : "[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead",
            "bytes_wanted" : 0,
            "bytes_limit" : 0
          }
        }
      }
    ],
    "caused_by" : {
      "type" : "general_script_exception",
      "reason" : "Failed to compile inline script [\"aaaaaaaaaaaaaaaa\"] using lang [painless]",
      "caused_by" : {
        "type" : "circuit_breaking_exception",
        "reason" : "[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead",
        "bytes_wanted" : 0,
        "bytes_limit" : 0
      }
    }
  },
  "status" : 500
}
```

This also fixes a bug in `ScriptService` where requests being executed
concurrently on a single node could cause a script to be compiled
multiple times (many in the case of a powerful node with many shards)
due to no synchronization between checking the cache and compiling the
script. There is now synchronization so that a script being compiled
will only be compiled once regardless of the number of concurrent
searches on a node.

Relates to #19396
2016-08-09 10:26:27 -06:00
David Pilato 4d272cc9b2 Merge branch 'master' into fix/19772-toString
# Conflicts:
#	core/src/test/java/org/elasticsearch/action/admin/cluster/node/tasks/TransportTasksActionTests.java
2016-08-09 11:53:29 +02:00
Nicholas Knize 2d590af593 Deprecate GeoDistance enumerators and remove geo distance script helpers
GeoDistance is implemented using a crazy enum that causes issues with the scripting modules. This commit moves all distance calculations to arcDistance and planeDistance static methods in GeoUtils. It also removes unnecessary distance helper methods from ScriptDocValues.GeoPoints.
2016-08-05 18:42:06 -05:00
javanna 2437226802 [TEST] restore tests repeatability in AbstractQueryTestCase
Some random operations were conditionally performed in the before test, which made tests not repeatable. For instance take the seed chain to repeat a specific iteration and try to reproduce it, this conditional code would get executed in both cases when trying to isolate the failure, but not among the different iterations (as only the first method/iteration executes it), hence the failure will not reproduce.

Moved the random operations to beforeClass and left the non random part in the before method, which is needed as it depends on some method that can be overridden by subclasses.
2016-08-05 22:38:31 +02:00
Jason Tedor a62740bbd2 Avoid early initializing Netty
Today when we load the Netty plugins, we indirectly cause several Netty
classes to initialize. This is because we attempt to load some classes
by name, and loading these classes is done in a way that triggers a long
chain of class initializers within Netty. We should not do this, this
can lead to log messages before the logger is loader, and it leads to
initialization in cases when the classes would never be needed (for
example, Netty 3 class initialization is never needed if Netty 4 is
used, and vice versa). This commit avoids this early initialization of
these classes by removing the need for the early loading.

Relates #19819
2016-08-05 14:58:33 -04:00
Tanguy Leroux 841d5a210e Update to Jackson 2.8.1
This commit updates Jackson to the 2.8.1 version, which is more strict when it comes to build objects. It also adds the snakeyaml dependency that was previously shaded in jackson libs.

It also closes #18076
2016-08-05 12:26:06 +02:00
Nik Everett 1e587406d8 Fail yaml tests and docs snippets that get unexpected warnings
Adds `warnings` syntax to the yaml test that allows you to expect
a `Warning` header that looks like:
```
    - do:
        warnings:
            - '[index] is deprecated'
            - quotes are not required because yaml
            - but this argument is always a list, never a single string
            - no matter how many warnings you expect
        get:
            index:    test
            type:    test
            id:        1
```

These are accessible from the docs with:
```
// TEST[warning:some warning]
```

This should help to force you to update the docs if you deprecate
something. You *must* add the warnings marker to the docs or the build
will fail. While you are there you *should* update the docs to add
deprecation warnings visible in the rendered results.
2016-08-04 15:23:05 -04:00
Jason Tedor 7d750d2811 Increase Netty 3 REST test suite timeout
This commit increases the Netty 3 REST test suite timeout to thirty
minutes. This is to address these tests running slowly after increasing
the number of nodes in the tests to two. This has surfaced that the
tests are heavily impacted by excessive fsyncs from most tests using the
default number of shards of five.
2016-08-03 21:31:48 -04:00
David Pilato 54603903f3 Remove ListTasksResponse#setDiscoveryNodes 2016-08-04 02:02:51 +02:00
Jason Tedor e74d02138f Merge branch 'master' into netty4
* master:
  Fix REST test documentation
  [Test] move methods from bwc test to test package for use in plugins (#19738)
  package-info.java should be in src/main only.
  Split regular histograms from date histograms. #19551
  Tighten up concurrent store metadata listing and engine writes (#19684)
  Plugins: Make NamedWriteableRegistry immutable and add extenion point for named writeables
  Add documentation for the 'elasticsearch-translog' tool
  [TEST] Increase time waiting for all shards to move off/on to a node
  Fixes the active shard count check in the case of (#19760)
  Fixes cat tasks operation in detailed mode
  ignore some docker craziness in scccomp environment checks
2016-08-03 09:16:18 -04:00
Ryan Ernst 18f242b069 Merge pull request #19764 from rjernst/writeable_registry
Make NamedWriteableRegistry immutable and add extension point for named writeables
2016-08-03 01:36:38 -07:00
Adrien Grand a0818d3b87 Split regular histograms from date histograms. #19551
Currently both aggregations really share the same implementation. This commit
splits the implementations so that regular histograms can support decimal
intervals/offsets and compute correct buckets for negative decimal values.

However the response API is still the same. So for intance both regular
histograms and date histograms will produce an
`org.elasticsearch.search.aggregations.bucket.histogram.Histogram`
aggregation.

The optimization to compute an identifier of the rounded value and the
rounded value itself has been removed since it was only used by regular
histograms, which now do the rounding themselves instead of relying on the
Rounding abstraction.

Closes #8082
Closes #4847
2016-08-03 08:39:48 +02:00
Ryan Ernst df8dc64e9b Plugins: Make NamedWriteableRegistry immutable and add extenion point for named writeables
Currently any code that wants to added NamedWriteables to the
NamedWriteableRegistry can do so via guice injection of the registry,
and registering at construction time. However, this makes the registry
complex: it has both get and register methods synchronized, and there is
likely contention on the read side from multiple threads.  The
registration has mostly already been contained to guice modules at node
construction time.

This change makes the registry immutable, taking all of the
NamedWriteable readers at construction time. It also allows plugins to
added arbitrary named writables that it may use in its own transport
actions.
2016-08-02 15:56:25 -07:00
Jason Tedor 0461e12663 Simplify Netty 4 transport implementations
The Netty 4 transport implementations have an unnecessary dependency on
SocketChannels, and can instead just use plain Channels.
2016-08-02 17:07:30 -04:00
jaymode 6def10c5d9 make netty4 http request public 2016-08-02 14:46:44 -04:00
Jason Tedor 669daccfbb Allow for Netty 4 HTTP extensions
This commit enables the Netty 4 HTTP server implementation to allow for
extensions.
2016-08-02 14:10:40 -04:00
jaymode 00ca6c417e add javadocs 2016-08-02 14:06:29 -04:00
jaymode 8498470781 update transport to allow for extensions 2016-08-02 13:45:41 -04:00
Jason Tedor 227463c356 Default transport and HTTP to Netty 4
This commit sets the default transport and HTTP implementation to use
transport-netty4.

Relates #19563
2016-08-02 12:19:19 -04:00
Ali Beyad 3d2a105825 Merge pull request #19454 from abeyad/remove-write-consistency-level
Removes write consistency level across replication action APIs in favor of wait_for_active_shards
2016-08-02 09:01:11 -04:00
Nicholas Knize 63f33e0f1e Serialize doc counts in Matrix-Stats module
This fixes a bug in the RunningStats class for the matrix stats aggregation module. doc counts were not being searlized which means they were only computed the first time the aggregation was computed. This was causing incorrect results when the aggregation was pulled from cache.
2016-08-01 18:05:10 -05:00
Ali Beyad 6a7d005081 Makes the index.write.wait_for_active_shards setting index-level and
dynamically updatable for both index creation and write operations.
2016-08-01 13:37:05 -04:00
Ali Beyad 25d8eca62d Removes the notion of write consistency level across all APIs in
favor of waiting for active shard copy count (wait_for_active_shards).
2016-08-01 13:35:29 -04:00
Tanguy Leroux 386902903e [TEST] Kill remaining lang-groovy messy tests
After #13834 many tests that used Groovy scripts (for good or bad reason) in their tests have been moved in the lang-groovy module and the issue #13837 has been created to track these messy tests in order to clean them up.

The work started with #19280, #19302 and #19336 and this PR moves the remaining messy tests back in core, removes the dependency on Groovy, changes the scripts in order to use the mocked script engine, and change the tests to integration tests.

It also moves IndexLookupIT test back (even if it has good chance to be removed soon) and fixes its tests.

It also changes AbstractQueryTestCase to use custom script plugins in tests.

closes #13837
2016-08-01 16:59:47 +02:00
Alexander Lin 9ac6389e43 Rename operation to result and reworking responses
* Rename operation to result and reworking responses
* Rename DocWriteResponse.Operation enum to DocWriteResponse.Result

These are just easier to interpret names.

Closes #19664
2016-08-01 10:42:58 -04:00
Nik Everett 303c9faca5 Squash o.e.rest.action.admin.cluster
In an effort to reduce the number of tiny packages we have in the
code base this moves all the files that were in subdirectories of
`org.elasticsearch.rest.action.admin.cluster` into
`org.elasticsearch.rest.action.admin.cluster`.

Also fixes line length in these packages.
2016-07-29 20:31:24 -04:00
Nik Everett 6f24866902 Reindex: Only ask for _version we need it
`_reindex` only needs the `_version` if the `dest` has
`"version_type": "external"`. So it shouldn't ask for it unless it does.

`_update_by_query` and `_delete_by_query` always need the `_version`.

Closes #19135
2016-07-29 17:13:27 -04:00
Alexander Lin 119026b4fb Remove isCreated and isFound from the Java API
This is cleanup work from #19566, where @nik9000 suggested trying to nuke the isCreated and isFound methods. I've combined nuking the two methods with removing UpdateHelper.Operation in favor of DocWriteResponse.Operation here.

Closes #19631.
2016-07-29 14:21:43 -04:00
Nik Everett c9790a1257 Use fewer threads when reindexing-from-remote
Reindex from remote uses the Elasticsearch client which uses apache
httpasyncclient which spins up 5 thread by default, 1 as a dispatcher
and 4 more to handle IO. This changes Reindex's usage so it only spins
up two thread - 1 dispatcher and one to handle io. It also renames the
threads to "es-client-$taskid-$thread_number". That way if we see any
thread sticking around we can trace it back to the task.
2016-07-29 14:13:10 -04:00
Martijn van Groningen a91bb29585 ingest: Made the response format of the get pipeline api match with the response format of the index template api
Closes #19585
2016-07-29 17:58:30 +02:00
Alexander Reelsen 30b497622b Dependencies: Upgrade to netty 4.1.4 (#19689)
Removes our internal snapshot repo we used until 4.1.4 was released.
2016-07-29 17:53:12 +02:00
Martijn van Groningen 7b36a72ccb fixed compiler warnining and removed unused imports 2016-07-29 15:06:49 +02:00
David Pilato 0d2ccf0989 Merge branch 'pr/15724-gce-network-host-master' 2016-07-28 16:59:18 +02:00
Jack Conradson 9d87314105 Refactored Painless link nodes into expression nodes to simplify
load/store operations of variable/method chains.

Closes #19459
2016-07-27 14:01:08 -07:00
Nik Everett fb45f6a8a8 Add authentication to reindex-from-remote
The tests for authentication extend ESIntegTestCase and use a mock
authentication plugin. This way the clients don't have to worry about
running it. Sadly, that means we don't really have good coverage on the
REST portion of the authentication.

This also adds ElasticsearchStatusException, and exception on which you
can set an explicit status. The nice thing about it is that you can
set the RestStatus that it returns to whatever arbitrary status you like
based on the status that comes back from the remote system.
reindex-from-remote then uses it to wrap all remote failures, preserving
the status from the remote Elasticsearch or whatever proxy is between us
and the remove Elasticsearch.
2016-07-27 14:17:41 -04:00
Martijn van Groningen 7e3d5b21bb test: fix generic type 2016-07-27 14:56:17 +02:00
David Pilato e9339a1960 Merge branch 'master' into pr/15724-gce-network-host-master 2016-07-27 11:24:53 +02:00
David Pilato 1c03d0bb2a Fix after merge 2016-07-27 11:23:18 +02:00
Martijn van Groningen d1b7a335cb removed useless comment and make sure all public constructors delegate to the package protected constructor. 2016-07-27 10:01:36 +02:00
Martijn van Groningen 2fdf79d8d4 Deprecate template query.
Closes #19390
2016-07-27 09:50:44 +02:00
Martijn van Groningen 24d7fa6d54 ingest: Change the `foreach` processor to use the `_ingest._value` ingest metadata attribute to store the current array element being processed.
Closes #19592
2016-07-27 09:35:09 +02:00
Nik Everett 9270e8b22b Rename client yaml test infrastructure
This makes it obvious that these tests are for running the client yaml
suites. Now that there are other ways of running tests using the REST
client against a running cluster we can't go on calling the shared
client yaml tests "REST tests". They are rest tests, but they aren't
**the** rest tests.
2016-07-26 13:53:44 -04:00
Chris Earle 0553ba9151 [Ingest] Add REST _ingest/pipeline to get all pipelines
This adds an extra REST handler for "_ingest/pipeline" so that users do not need to supply "_ingest/pipeline/*" to get all of them.

- Also adds a teardown section to related REST-tests for ingest.
2016-07-26 13:48:15 -04:00
David Pilato 0d3edee928 Merge branch 'master' into pr/15724-gce-network-host-master 2016-07-26 18:51:01 +02:00
David Pilato fde15ae470 Move custom name resolvers to NetworkService CTOR
Instead of using NetworkModule we can directly inject them in NetworkService CTOR.

See https://github.com/elastic/elasticsearch/pull/15765#issuecomment-235307974
2016-07-26 18:26:30 +02:00
Alexander Lin 8f2882a442 Add _operation field to index, update, delete responses
Performing the bulk request shown in #19267 now results in the following:
```
{"_index":"test","_type":"test","_id":"1","_version":1,"_operation":"create","forced_refresh":false,"_shards":{"total":2,"successful":1,"failed":0},"status":201}
{"_index":"test","_type":"test","_id":"1","_version":1,"_operation":"noop","forced_refresh":false,"_shards":{"total":2,"successful":1,"failed":0},"status":200}
```
2016-07-26 11:16:19 -04:00
Daniel Mitterdorfer 94bc489275 Temporarily increase Netty4RestIT timeout
Recently, we experience timeouts on our Windows build slaves for
Netty4RestIT. Until we have figured out what's going on, we
increase this test suite's timeout temporarily to ensure this
timeout does not mask other problems.
2016-07-26 08:27:47 +02:00
Daniel Mitterdorfer d7d3b1a874 Remove duplicate dependency declaration for http client (#19580)
We disable transitive dependencies in our build plugin
for all dependencies except for the group `org.elasticsearch`.
However, in the reindex plugin we depend on the REST client
and declare its dependencies again which is not necessary
(and led to problems with conflicting versions in #19281).

With this PR we remove the duplicate declaration.
2016-07-26 06:29:04 +02:00
Nik Everett a95d4f4ee7 Add Location header and improve REST testing
This adds a header that looks like `Location: /test/test/1` to the
response for the index/create/update API. The requirement for the header
comes from https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

https://tools.ietf.org/html/rfc7231#section-7.1.2 claims that relative
URIs are OK. So we use an absolute path which should resolve to the
appropriate location.

Closes #19079

This makes large changes to our rest test infrastructure, allowing us
to write junit tests that test a running cluster via the rest client.
It does this by splitting ESRestTestCase into two classes:
* ESRestTestCase is the superclass of all tests that use the rest client
to interact with a running cluster.
* ESClientYamlSuiteTestCase is the superclass of all tests that use the
rest client to run the yaml tests. These tests are shared across all
official clients, thus the `ClientYamlSuite` part of the name.
2016-07-25 17:02:40 -04:00
Martijn van Groningen a9ab095b8c Moved all mustache classes into one package.
No need for multiple packages inside a small module.
2016-07-25 18:50:12 +02:00
Martijn van Groningen a784055db1 Cleaned up the tests in lang-mustache.
Messy tests with mustache were either moved to core, moved to a rest test or remained untouched if they actually tested mustache.
Also removed tests that were redundant.
2016-07-25 17:57:39 +02:00
Tanguy Leroux f745c96949 Clean up more messy tests
After #13834 many tests that used Groovy scripts (for good or bad reason) in their tests have been moved in the lang-groovy module and the issue #13837 has been created to track these messy tests in order to clean them up.

This commit moves more tests back in core, removes the dependency on Groovy, changes the scripts in order to use the mocked script engine, and change the tests  to integration tests.
2016-07-25 17:02:49 +02:00
Jason Tedor 2d1b0587dd Introduce Netty 4
This commit adds transport-netty4, a transport and HTTP implementation
based on Netty 4.

Relates #19526
2016-07-22 22:26:35 -04:00
javanna 36becd9bfe Merge branch 'master' into feature/async_rest_client 2016-07-22 22:22:03 +02:00
Ali Beyad 2b9cfff90f Fixes CORS handling so that it uses the defaults
Fixes CORS handling so that it uses the defaults for http.cors.allow-methods
and http.cors.allow-headers if none are specified in the config.

Closes #19520
2016-07-22 12:25:28 -04:00
javanna d13a3d3761 Reindex from remote: add fallback in case content-type header is not set
We better read the header, but who knows what can happen, maybe headers are filtered out for some reasons and we don't want to run into an NPE, then we fallback to auto-detection.
2016-07-22 16:46:17 +02:00
Simon Willnauer 6adfeb4dde [TEST] Add Netty3RestIT to actually run REST tests on netty3 2016-07-22 16:29:14 +02:00
javanna db8beeba3b Merge branch 'master' into feature/async_rest_client 2016-07-22 15:51:03 +02:00
Tal Levy 19e7b1c737 fix: no other processors should be executed after on_failure is called in a compound processor (#19545) 2016-07-21 14:27:04 -07:00
Tal Levy f7cd86ef6d rethrow script compilation exceptions into ingest configuration exceptions (#19318)
* rethrow script compilation exceptions into ingest configuration exceptions
* update readProcessor to rethrow any exception as an ElasticsearchException
2016-07-20 10:37:56 -07:00
Nik Everett 3a82c613e4 Migrate query registration from push to pull
Remove `ParseField` constants used for names where there are no deprecated
names and just use the `String` version of the registration method instead.

This is step 2 in cleaning up the plugin interface for extending
search time actions. Aggregations are next.

This is breaking for plugins because those that register a new query should
now implement `SearchPlugin` rather than `onModule(SearchModule)`.
2016-07-20 12:33:51 -04:00
Nik Everett fc4b439635 Remove AggregationStreams and friends
* Remove outdated aggregation registration method
* Remove AggregationStreams
* Adds StreamInput#readNamedWriteableList and
StreamOutput#writeNamedWriteableList convenience methods. We strive to
make the reading and writing from the streams terse so they are easier
to scan visually.
* Remove PipelineAggregatorStreams
* Remove stream info from InternalAggreation.Type
* Remove InternalAggregation#type
* Remove Streamable from PipelineAggregator
* Remove Streamable from MultiBucketsAggregation.Bucket
2016-07-20 09:46:04 -04:00
javanna bce54cf38d reindex from remote to read content-type header rather than guessing content type based on content 2016-07-19 15:16:45 +02:00
javanna 54fa997545 Reindex from remote: remove async client in favour of using RestClient performRequest async method 2016-07-19 15:16:45 +02:00
javanna 1fbec71243 Rest client: introduce async performRequest method and use async client under the hood for sync requests too
The new method accepts the usual parameters (method, endpoint, params, entity and headers) plus a response listener and an async response consumer. Shortcut methods are also added that don't require params, entity and the async response consumer optional.

There are a few relevant api changes as a consequence of the move to async client that affect sync methods:
- Response doesn't implement Closeable anymore, responses don't need to be closed
- performRequest throws Exception rather than just IOException, as that is the the exception that we get from the FutureCallback#failed method in the async http client
- ssl configuration is a bit simpler, one only needs to call setSSLStrategy from a custom HttpClientConfigCallback, that doesn't end up overridng any other default around connection pooling (it used to happen with the sync client and make ssl configuration more complex)

Relates to #19055
2016-07-19 15:15:58 +02:00
javanna 118a14fbe3 Build: upgrade httpcore version to 4.4.5
Closes #19127
2016-07-19 15:11:40 +02:00
Nik Everett d573541f66 Support requests_per_second=-1 to mean no throttling in reindex
This is entirely on the REST level, Float.POSITIVE_INFINITY is still
how you get no throttling over the transport api.

Closes #19089
2016-07-18 13:05:06 -04:00
Simon Willnauer 8394544548 Add a dedicated client/transport project for transport-client (#19435)
The `client/transport` project adds a new jar build project that
pulls in all dependencies and configures all required modules.

Preinstalled modules are:
 * transport-netty
 * lang-mustache
 * reindex
 * percolator

The `TransportClient` classes are still in core
while `TransportClient.Builder` has only a protected construcutor
such that users are redirected to use the new `TransportClientBuilder`
from the new jar.

Closes #19412
2016-07-18 15:42:24 +02:00
Martijn van Groningen e0ebf5da1c Template cleanup:
* Removed `Template` class and unified script & template parsing logic. Templates are scripts, so they should be defined as a script. Unless there will be separate template infrastructure, templates should share as much code as possible with scripts.
* Removed ScriptParseException in favour for ElasticsearchParseException
* Moved TemplateQueryBuilder to lang-mustache module because this query is hard coded to work with mustache only
2016-07-18 10:16:01 +02:00
Ali Beyad dddbe4acad Fix MatrixAggregationRestIT test 2016-07-15 19:41:47 -04:00
Ali Beyad 19d0dbcd17 Removes waiting for yellow cluster health upon index (#19460)
creation in the REST tests, as we no longer need it due
to index creation now waiting for active shard copies
before returning (by default, it waits for the primary of
each shard, which is the same as ensuring yellow health).

Relates #19450
2016-07-15 17:18:34 -04:00
Ali Beyad 6acb8b31fc Removes ensureYellow() calls after index creation in the (#19452)
integration tests, as they are no longer needed with
index creation now waiting for shards to be started before
returning from the index creation call (by default, it waits
for the primary of each shard to be started before returning,
which is what ensureYellow() was ensuring anyway).

Closes #19452
Relates #19450
2016-07-15 15:37:35 -04:00
Martijn van Groningen d0069f0fbb Provide access to ThreadContext in ingest plugins
Also introduced a `Processor.Parameters` class that is holder for several services processors rely on,
the  IngestPlugin#getProcessors(...) method has been changed to accept `Processor.Parameters` instead
of each service seperately.
2016-07-15 08:16:15 +02:00
Jason Tedor 31c648eee8 Rename transport-netty to transport-netty3
This commit renames the Netty 3 transport module from transport-netty to
transport-netty3. This is to make room for a Netty 4 transport module,
transport-netty4.

Relates #19439
2016-07-14 22:03:14 -04:00
Ryan Ernst b5b0f04e34 Merge pull request #19431 from rjernst/client_plugin
Add flag for plugins/modules which can be used in the transpory client
2016-07-14 08:56:46 -07:00
Simon Willnauer 5616251f22 Remove `node.mode` and `node.local` settings (#19428)
Today `node.mode` and `node.local` serve almost the same purpose, they
are a shortcut for `discovery.type` and `transport.type`. If `node.local: true`
or `node.mode: local` is set elasticsearch will start in _local_ mode which means
only nodes within the same JVM are discovered and a non-network based transport
is used. The _local_ mode it only really used in tests or if nodes are embedded.
For both, embedding and tests explicit configuration via `discovery.type` and `transport.type`
should be preferred.

This change removes all the usage of these settings and by-default doesn't
configure a default transport implemenation since netty is now a module. Yet, to make
the user expericence flawless, plugins or modules can set a `http.type.default` and
`transport.type.default`. Plugins set this via `PluginService#additionalSettings()`
which enforces _set-once_ which prevents node startup if set multiple times. This means
that our distributions will just startup with netty transport since it's packaged as a
module unless `transport.type` or `http.transport.type` is explicitly set.

This change also found a bunch of bugs since several NamedWriteables were not registered if a
transport client is used. Now that we don't rely on the `node.mode` leniency which is inherited
instead of using explicit settings, `TransportClient` uses `AssertingLocalTransport` which detects these problems since it serializes all messages.

Closes #16234
2016-07-14 13:21:10 +02:00
Ryan Ernst 9b63fc37c8 Make module client jars go to org.elasticsearch.plugin groupId 2016-07-14 02:58:27 -07:00
Ryan Ernst c36850f114 Add client flag for percolator module build 2016-07-14 02:41:58 -07:00
Ryan Ernst 20b038d976 Build: Add flag for plugins/modules which can be used in the transport client
This changes adds a flag which can be set in the esplugin closure in
build.gradle for plugins and modules which contain pieces that must be
published to maven, for use in the transport client. The jar/pom and
source/javadoc jars are moved to a new name that has the suffix
"-client".

I enabled this for the two modules that I know definitely need this;
there may be more. One open question is which groupId to use for the
generated pom.

closes #19411
2016-07-14 02:07:52 -07:00
Tal Levy ed768b101f show ignored errors in verbose simulate result (#19404)
Closes #19319.
2016-07-13 13:32:10 -07:00
Tal Levy 8fd01554bc update foreach processor to only support one applied processor. (#19402)
Closes #19345.
2016-07-13 13:13:00 -07:00
Nik Everett 2422b969c1 Migrate matrix_stats to NamedWriteable
This is the last consumer of the old style register method so I removed
the method.
2016-07-13 10:48:20 -04:00
Martijn van Groningen 2c3165d080 Removed deprecated 1.x script and template syntax
Closes #13729
2016-07-13 15:07:36 +02:00
Simon Willnauer c463083537 minor cleanups and an additional BogusPlugin for HttpSmokeTestCase 2016-07-12 17:55:05 +02:00
Simon Willnauer 9cb247287f consolidate security code in on place an allow test based on the jar dependency to opt out of netty internal property setting assertion 2016-07-12 17:41:21 +02:00
Simon Willnauer 4fb79707bd Fix remaining tests that either need access to the netty module or require explict configuration
Some tests still start http implicitly or miss configuring the transport clients correctly.
This commit fixes all remaining tests and adds a depdenceny to `transport-netty` from
`qa/smoke-test-http` and `modules/reindex` since they need an http server running on the nodes.

This also moves all required permissions for netty into it's module and out of core.
2016-07-12 16:29:57 +02:00
Ryan Ernst 93aebbef0f Merge branch 'master' into modularize_netty 2016-07-11 23:49:00 -07:00
Nik Everett 8263873783 Switch search extension from push to pull
Switches most search behavior extensions from push (`onModule(SearchModule)`)
to pull (`implements SearchPlugin`). This effort in general gives plugin
authors a much cleaner view of how to extend Elasticsearch and starts to
set up portions of Elasticsearch as "the plugin API". This commit in
particular does that for search-time behavior like customized suggesters,
highlighters, score functions, and significance heuristics.

It also switches most such customization to being done at search module
construction time which is much, much easier to reason about from a testing
perspective. It also helps significantly in the process of de-guice-ing
Elasticsearch's startup.

There are at least two major search time extensions that aren't covered in
this commit that will simply have to wait for the next commit on the topic
because this one has already grown large: custom aggregations and custom
queries. These will likely live in the same SearchPlugin interface as well.
2016-07-11 18:49:05 -04:00
Simon Willnauer 048e4416e7 Move netty transport and http into a module
This moves all netty code and it's dependency into a module.
2016-07-11 22:21:29 +02:00
Martijn van Groningen ff5527f037 percolator: Forbid the usage or `range` queries with a range based on the current time
If there are percolator queries containing `range` queries with ranges based on the current time then this can lead to incorrect results if the `percolate` query gets cached.  These ranges are changing each time the `percolate` query gets executed and if this query gets cached then the results will be based on how the range was at the time when the `percolate` query got cached.

The ExtractQueryTermsService has been renamed `QueryAnalyzer` and now only deals with analyzing the query (extracting terms and deciding if the entire query is a verified match) . The `PercolatorFieldMapper` is responsible for adding the right fields based on the analysis the `QueryAnalyzer` has performed, because this is highly dependent on the field mappings. Also the `PercolatorFieldMapper` is responsible for creating the percolate query.
2016-07-08 14:20:56 +02:00
Martijn van Groningen 7b8ae54f0f percolator: Also support query term extract for queries wrapped inside a FunctionScoreQuery
Additionally for highlighting percolator hits, also extract percolator query from FunctionScoreQuery and DisjunctionMaxQuery
2016-07-08 10:51:48 +02:00
Nik Everett 81fcdfcee9 Expose task information from NodeClient
This exposes a method to start an action and return a task from
`NodeClient`. This allows reindex to use the injected `Client` rather
than require injecting `TransportAction`s
2016-07-07 18:02:09 -04:00
Nik Everett fe0f28965a Clean up serialization of terms aggregation results
Move to NamedWriteable and remove a lot of duplication.
2016-07-07 17:01:09 -04:00
Tanguy Leroux 3267fc4e0c Clean up more messy tests
After #13834 many tests that used Groovy scripts (for good or bad reason) in their tests have been moved in the lang-groovy module and the issue #13837 has been created to track these messy tests in order to clean them up.

This commit moves more tests back in core, removes the dependency on Groovy, changes the scripts in order to use the mocked script engine, and change the tests  to integration tests.
2016-07-07 17:50:23 +02:00
Tanguy Leroux b58f2eb5c2 Move back some messy tests from Groovy plugin to core
This commit moves back some messy tests that have been placed in lang-groovy module in https://github.com/elastic/elasticsearch/pull/13834. It removes the dependency on Groovy plugin as well as change back the tests to integration tests (IT suffix).

It also changes the current MockScriptEngine and MockScriptPlugin to make it easier to use.
2016-07-07 15:26:36 +02:00
Ryan Ernst 4ea5f51a9c Fix reindex NPE when http is disabled 2016-07-05 23:59:29 -07:00
Ryan Ernst f815799bd4 Fix reindex action to depend on HttpServer instead of NodeService for
http info
2016-07-05 22:16:22 -07:00
Ryan Ernst 2fc41adeb5 Merge branch 'master' into ingest_plugin_api 2016-07-05 20:53:03 -07:00