Commit Graph

26240 Commits

Author SHA1 Message Date
Nik Everett 8abd4101eb Add tests for reducing top hits
Also adds many `equals` and `hashCode` implementations and moves
the failure printing in `MatchAssertion` into a common spot and
exposes it over `assertEqualsWithErrorMessageFromXContent` which
does an object equality test but then uses `toXContent` to print
the differences.

Relates to #22278
2017-01-27 12:32:17 -05:00
Nik Everett 1baa884ab7 Fix TophitsAggregatorTests
It needs a DirectoryReader so it has to be careful.

Closes #22818
2017-01-26 14:08:30 -05:00
Christoph Büscher 33fd1a606b Remove unneeded checkstyle LineLength check suppression for more classes 2017-01-26 19:40:49 +01:00
Nik Everett f8c28711be Merge some equivalent interfaces (#22816)
Remove `FromXContent` and use `CheckedFunction` instead.
Remove `FromXContentWithContext` and use `ContentParser` instead.
2017-01-26 13:15:29 -05:00
David Pilato 2abe948cd7 Remove non needed import 2017-01-26 17:43:59 +01:00
David Pilato 59b34ba71c Merge branch 'pr/22566-ec2-meta-host' 2017-01-26 17:37:53 +01:00
David Pilato 6e7aee0c5a use expectThrows instead of manually testing exception 2017-01-26 17:33:26 +01:00
David Pilato d97750b52c Fix checkstyle and a test 2017-01-26 17:20:27 +01:00
David Pilato 17930930a7 Update after review 2017-01-26 17:10:37 +01:00
David Pilato 3804bfcc60 Read ec2 discovery address from aws instance tags
This PR adds a new option for `host_type`: `tag:TAGNAME` where `TAGNAME` is the tag field you defined for your ec2 instance.

For example if you defined a tag `my-elasticsearch-host` in ec2 and set it to `myhostname1.mydomain.com`, then
setting `host_type: tag:my-elasticsearch-host` will tell Discovery Ec2 plugin to read the host name from the
`my-elasticsearch-host` tag. In this case, it will be resolved to `myhostname1.mydomain.com`.

Closes #22566.
2017-01-26 17:10:37 +01:00
Simon Willnauer a475323aa1 Invalidate cached query results if query timed out (#22807)
Today we cache query results even if the query timed out. This is obviously
problematic since results are not complete. Yet, the decision if a query timed
out or not happens too late to simply not cache the result since if we'd just throw
an exception all currently waiting requests with the same request / cache key would
fail with the same exception without the option to access the result or to re-execute.
Instead, this change will allow the request to enter the cache but invalidates it immediately.
Concurrent request might not get executed and return the timed out result which is not absolutely
correct but very likely since identical requests will likely timeout as well. As a side-effect
we won't hammer the node with concurrent slow searches but rather only execute one of them
and return shortly cached result.

Closes #22789
2017-01-26 16:45:29 +01:00
Nik Everett a383bc1be0 Add remaining generated painless API
This was generated by the generated added in
8a2d424d68 but not committed in that
change for review purposes.
2017-01-26 10:43:22 -05:00
Nik Everett 8a2d424d68 Generate reference links for painless API (#22775)
Adds "Appending B. Painless API Reference", a reference of all classes
and methods available from Painless. Removes links to java packages
because they contain methods that we don't expose and don't contain
methods that we do expose (the ones in Augmentation). Instead this
generates a list of every class and every exposed method using the same
type information available to the
interpreter/compiler/whatever-we-call-it. From there you can jump to
the relevant docs.

Right now you build all the asciidoc files by running
```
gradle generatePainlessApi
```

These files are expected to be committed because we build the docs
without running `gradle`.

Also changes the output of `Debug.explain` so that it is easy to
search for the class in the generated reference documentation.

You can also run it in an IDE safely if you pass the path to the
directory in which to generate the docs as the first parameter. It'll
blow away the entire directory an recreate it from scratch so be careful.

And then you can build the docs by running something like:
```
../docs/build_docs.pl --out ../built_docs/ --doc docs/reference/index.asciidoc --open
```

That is, if you have checked out https://github.com/elastic/docs in
`../docs`. Wait a minute or two and your browser will pop open in with
all of Elasticsearch's reference documentation. If you go to
`http://localhost:8000/painless-api-reference.html` you can see this
list. Or you can get there by following the links to `Modules` and
`Scripting` and `Painless` and then clicking the link in the paragraphs
below titled `Appendix B. Painless API Reference`.

I like having these in asciidoc because we can deep link to them from the
rest of the guide with constructs like
`<<painless-api-reference-Object-hashCode-0>>` and
`<<painless-api-reference->>` and we get link checking. Then the only
brittle link maintenance bit is the link generation for javadoc. Which
sucks. But I think it is important that we link to the methods directly
so they are easy to find.

Relates to #22720
2017-01-26 10:39:19 -05:00
Tanguy Leroux 1fa2734566 [TEST] Fix ElasticsearchExceptionTests
Some test failures can happen in ElasticsearchExceptionTests, this commit fixes them.
2017-01-26 16:33:56 +01:00
Tanguy Leroux be96278c95 Add parsing method for ElasticsearchException.generateThrowableXContent() (#22783)
The output of the ElasticsearchException.generateThrowableXContent() method can be parsed back by the ElasticsearchException.fromXContent() method.

This commit adds unit tests in the style of the to-and-from-xcontent tests we already have for other parsing methods. It also relax the strict parsing of the ElasticsearchException.fromXContent() so that it does not throw an exception when custom metadata and headers are parsed, as long as they are either strings or arrays of strings. Every other type is ignored at parsing time.
2017-01-26 15:17:07 +01:00
Simon Willnauer f128b7a7fe Improve connection closing in `RemoteClusterConnection` (#22804)
Some tests verify that all connection have been closed but due to the
async / concurrent nature of `RemoteClusterConnection` there are situations
where we notify listeners that trigger tests to finish before we actually
closed all connections. The race is very very small and has no impact on the
code correctness. This commit documents and improves the way we close
connections to ensure test won't fail with false positives.

Closes #22803
2017-01-26 13:58:26 +01:00
Clinton Gormley f5a0d18c4c Docs: Cluster allocation explain should be on one page 2017-01-26 11:38:19 +01:00
Simon Willnauer 281250dec9 Remove DFS_QUERY_AND_FETCH as a search type (#22787)
This commit removes the search type `dfs_query_and_fetch` without a
replacement. We don't allow to use this type via REST since 2.x
but still keep it around for no particular reason. There we no users
complaining about the availability. This should now be removed from the
codebase. `query_and_fetch` is still used internally to safe a roundtrip
if there is only one shard but it can't be used via the rest interface.
2017-01-26 09:14:44 +01:00
Tim Brooks 719e75bb3f Add repository-url module and move URLRepository (#22752)
This is related to #22116. URLRepository requires SocketPermission
connect. This commit introduces a new module called "repository-url"
where URLRepository will reside. With the new module, permissions can
be removed from core.
2017-01-25 17:09:25 -06:00
Tal Levy e9a68b3287 fix date-processor to a new default year for every new pipeline execution. (#22601)
Beforehand, the DateProcessor constructs its joda pattern formatter during processor
construction. This led to newly ingested documents being defaulted to
the year that the pipeline was constructed, not that of processing.

Fixes #22547.
2017-01-25 15:09:07 -08:00
Nik Everett d704a880e7 Add tests for top_hits aggregation (#22754)
Add unit tests for `TopHitsAggregator` and convert some snippets in
docs for `top_hits` aggregation to `// CONSOLE`.

Relates to #22278
Relates to #18160
2017-01-25 16:15:50 -05:00
Martijn van Groningen f6ed39aa08 Merge branch 'pr/22772' 2017-01-25 17:15:24 +01:00
Martijn van Groningen 81e40e3139 [TEST] Added this for 93a28b0acf submitted via #22772 2017-01-25 17:08:17 +01:00
David Pilato 98f799f6d5 Merge branch 'pr/ingest-attachment-mime4j' 2017-01-25 16:52:38 +01:00
Jason Tedor cb822b4670 Fix typo in comment in OsProbe.java
This commit fixes a silly typo in a comment relating to cgroups in
OsProbe.java.
2017-01-25 06:30:51 -05:00
Anil Bawa-Cavia 86b3ff229c Add new ruby search library to community clients doc (#22765)
* Add new Ruby search lib from Artsy.

* update ruby library reference following rename
2017-01-25 11:49:39 +01:00
Colin Goodheart-Smithe a9135cd636 RangeQuery WITHIN case now normalises query (#22431)
Previous to his change when the range query was rewritten to an unbounded range (`[* TO *]`) it maintained the timezone and format for the query. This means that queries with different timezones and format which are rewritten to unbounded range queries actually end up as different entries in the search request cache.

This is inefficient and unnecessary so this change nulls the timezone and format in the rewritten query so that regardless of the timezone or format the rewritten query will be the same.

Although this does not fix #22412 (since it deals with the WITHIN case rather than the INTERSECTS case) it is born from the same arguments
2017-01-25 10:37:15 +00:00
Boaz Leskes ed94f75a15 Remove EngineClosedException
All usage has been removed in https://github.com/elastic/elasticsearch/pull/22631, which is back ported to 5.x. This means 6.x will never get it on the wire and we can remove it
2017-01-25 11:00:50 +01:00
David Pilato ee3d73dc3d Add test-outlook.msg and test-outlook2003.msg files 2017-01-25 08:53:44 +01:00
Yannick Welsch e37a7d73a2 Remove leftover build files and simplify ProgressLogger usage
These files should have been removed in an earlier commit. This commit also simplifies usage of ProgressLoggerWrapper by using the Groovy delegation instead of using explicit delegation.
2017-01-24 20:44:14 +01:00
javanna 5103b76610 update version checks in ElasticsearchException serialization code
5.3.0 is the first version that contains the split from headers to metadata, updated the check to reflect that. It was previously after to be able to commit to master first, and only after that backport the change. Otherwise master tests would have failed until the change was backported.
2017-01-24 20:40:17 +01:00
Lee Hinman 304296ea6a Fix BulkItemResponse serialization for 6.x <-> 5.3.x
Previously the behavior where the `OpType` byte was serialized was only in
master, but it was recently backported to 5.x, so the serialization version
checks need to be updated as well.
2017-01-24 12:04:11 -07:00
srgclr 93a28b0acf skip parentid if child document is an orphan
#22770
2017-01-24 17:49:53 +00:00
Chris Earle f0f75b187a Support Preemptive Authentication with RestClient (#21336)
This adds the necessary `AuthCache` needed to support preemptive authorization. By adding every host to the cache, the automatically added `RequestAuthCache` interceptor will add credentials on the first pass rather than waiting to do it after _each_ anonymous request is rejected (thus always sending everything twice when basic auth is required).
2017-01-24 11:34:05 -05:00
Luca Cavanna 47c0e13a3b Stop returning "es." internal exception headers as http response headers (#22703)
move "es." internal headers to separate metadata set in ElasticsearchException and stop returning them as response headers

Closes #17593

* [TEST] remove ESExceptionTests, move its methods to ElasticsearchExceptionTests or ExceptionSerializationTests
2017-01-24 16:12:45 +01:00
Clinton Gormley 12f5309041 Docs: Made cross-cluster search experimental 2017-01-24 12:44:01 +01:00
Jason Tedor bcffc6fa49 Add hack for Docker cgroups
Docker cgroups are mounted in the wrong place (i.e., inconsistently with
/proc/self/cgroup). This commit adds an undocumented hack for working
around, for now.

Relates #22757
2017-01-24 06:36:03 -05:00
Christoph Büscher 59aefe5a38 Include human readable responses in response parsing tests (#22717)
As a follow up to #22649, this changes the resent tests for parsing parts of search 
responses to randomly set the humanReadable() flag of the XContentBuilder that 
is used to render the responses. This should help to test that we can parse back 
thoses classes if the user specifies `?human=true` in the request url.
2017-01-24 11:17:58 +01:00
Yannick Welsch 12b6ff5233 Use Gradle-version specific source sets
Instead of using Gradle-version specific compilation options, use distinct source sets. This also allows compilation of buildSrc/build-tools under IDEs that
don't understand the version-specific compilation options.

Relates to #22669
2017-01-24 11:09:57 +01:00
Yannick Welsch 36198e0275 Make build Gradle 2.14 / 3.x compatible (#22669)
This changes build files so that building Elasticsearch works with both Gradle 2.13 as well as higher versions of Gradle (tested 2.14 and 3.3), enabling a smooth transition from Gradle 2.13 to 3.x.
2017-01-24 11:09:57 +01:00
David Pilato 8701f7a3ce Add missing mime4j library
In some cases (apparently with outlook files), mime4j library is needed.
We removed it in the past which can cause elasticsearch to crash when you are using ingest-attachment (and probably mapper-attachments as well in 2.x series) with a file which requires this library.

 Similar problem as the one reported at #22077.
2017-01-24 10:25:02 +01:00
Jim Ferenczi b0c2a5da30 Remove unused field in CollapseBuilder 2017-01-24 09:26:23 +01:00
Jim Ferenczi 868b12b548 Add BWC tests for field collapsing
Field collapsing is supported from version 5.3
2017-01-24 08:34:16 +01:00
Tim Brooks 7f20b93051 Use generic interfaces for checking socket access (#22753)
This commit replaces specialized functional interfaces in various
plugins with generic options. Instead of creating `StorageRunnable`
interfaces in every plugin we can just use `Runnable` or `CheckedRunnable`.
2017-01-23 16:34:24 -06:00
Nik Everett 28cfc533e2 Generate javadoc jar for painless's public API (#22704)
The simplest way to do that is to move the public API into a
new package and generate javadoc for that package.
2017-01-23 17:16:20 -05:00
Nik Everett 2e399e5505 Rename constant
It deserves a new name after the cleanup in #22749
2017-01-23 16:30:42 -05:00
javanna 8065531236 [TEST] add test to verify that the SMILE format works within the _bulk api 2017-01-23 19:40:24 +01:00
Nik Everett ee264c6957 Fix parsing for `max_determinized_states` (#22749)
There was a typo in the `ParseField` declaration. I know
we want to port these parsers to `ObjectParser` eventually
but I don't have the energy for that today and want to get
this fixed.

Closes #22722
2017-01-23 11:57:43 -05:00
William Webber f1a902865f Update span-multi-term-query.asciidoc (#22733)
"term" is not actually a multi-term query (perhaps confusion with "term range")
2017-01-23 17:33:40 +01:00
William Webber abaf728882 "from" => "gte", "to" => "lte" in bool example (#22735) 2017-01-23 17:29:00 +01:00