Commit Graph

28208 Commits

Author SHA1 Message Date
Tal Levy 8cf0528001 update ingest-user-agent regexes.yml (#25608)
This new regexes are from:

3153c2f2ae/regexes.yaml
2017-07-10 08:43:11 -07:00
Yannick Welsch 7836bbf4d4 Fix tribe node cluster state version increments (#25629)
With #24236, tribe nodes submit cluster state changes to their MasterService, making it unnecessary to explicitly update the cluster state version. This PR fixes the double-incrementing of cluster state versions on tribe nodes, which are not harmful, but unnecessary.
2017-07-10 16:25:11 +02:00
Colin Goodheart-Smithe 3a5a54e83e Collapses package structure for some bucket aggs (#25579)
This change collapses some of the packages for the bucket aggregations into their parent packages. This was done for the following aggregations:
* The variants of the range aggregation (geo_distance, date and ip) were moved into the `o.e.s.a.bucket.range` package
* The `o.e.s.a.bucket.terms.support` package was removed and the classes were moved to `o.e.s.a.bucket.terms`
* The filter aggregation was moved to `o.e.s.a.bucket.filter`

Since this PR is already relatively large with only the above changes subsequent PRs will do similar operations on relevant metric and pipeline aggregations

Relates to #22868
2017-07-10 15:08:15 +01:00
Boaz Leskes e93e10f93b Close Translog trimming task when IndexService is closed
Relates to https://github.com/elastic/elasticsearch/pull/25622
2017-07-10 14:40:23 +02:00
Yannick Welsch b5521872bb [TEST] Use correct StreamInput version to deserialize in testSnapshotDeletionsInProgressSerialization
The test is currently serializing the cluster state using an older ES version format, but then deserializes those same bytes by
assuming they are of the current ES version.
2017-07-10 14:03:12 +02:00
Clinton Gormley e85871cfe9 Update cross-cluster-search.asciidoc
Increased the required min version of CCS in the docs to 5.5
2017-07-10 12:04:05 +02:00
Boaz Leskes a6db0ea908 Run Translog retention yaml tests with no replicas
Initializing replicas change the translog retention logic and confuses the test.

Switch to the solution suggested in https://github.com/elastic/elasticsearch/issues/25623, if implemented
2017-07-10 11:06:57 +02:00
Luca Cavanna a932591007 Treat aliases as unavailable indices in delete index and update aliases api (#25524)
When resolving wildcards, aliases should be treated as unavailable indices when the `ignoreAliases` option is set to `true` (currently enabled with delete index api and update aliases api). This way the `allow_no_indices` and `ignore_unavailable` options can be honoured, otherwise WildcardExpressionResolver ends up treating aliases differently and there is no way to control when an error is thrown.

The default behaviour for the delete index api, which has `ignore_unavailable` set to `false` and `allow_no_indices` set to `true` by default, is to throw an error when executed against an alias, same as when it's executed against an index that does not exist.
2017-07-10 10:58:00 +02:00
Boaz Leskes 09378f48e4 Add a scheduled translog retention check (#25622)
We currently check whether translog files can be trimmed whenever we create a new translog generation or close a view. However #25294 added a long translog retention period (12h, max 512MB by default), which means translog files should potentially be cleaned up long after there isn't any indexing activity to trigger flushes/the creation of new translog files. We therefore need a scheduled background check to clean up those files once they are no longer needed.

Relates to #10708
2017-07-10 10:28:39 +02:00
Jason Tedor c084542731 Bump version to 6.0.0-beta1
This commit does two things:
 - bumps the version from 6.0.0-alpha3 to 6.0.0-beta1
 - renames the 6.0.0-alpha3 version constant to 6.0.0-beta1

Relates #25621
2017-07-09 18:12:50 -04:00
Jason Tedor c75ddd2c85 Fix scaling thread pool test bug
This commit adjusts the expectation for the max number of threads in the
scaling thread pool configuration test. The reason that this expectation
is incorrect is because we removed the limitation that the number of
processors maxes out at 32, instead letting it be the true number of
logical processors on the machine. However, when we removed this
limitation, this test was never adjusted to reflect the new reality yet
it never arose since our tests were not running on machines with
incredibly high core counts.

Relates #20874
2017-07-09 08:00:27 -04:00
Boaz Leskes 1f4d8a05d1 testConcurrentWriteViewsAndSnapshot: writers should expose the local checkpoint to readers before trimming the translog 2017-07-09 12:26:54 +02:00
Jason Tedor cb3674c5ee Add reason to global checkpoint updates on replica
Updating the global checkpoint on a replica can occur for a few
different reasons:
 - from inlined global checkpoint updates
 - from a primary term transition
 - from finalizing recovery

Yet, the trace logging for a global checkpoint update does not present
this information that can be useful when tracing test failures. This
commit adds a reason for the global checkpoint update on a replica so
that we can trace these updates.

Relates #25612
2017-07-08 17:05:24 -04:00
Jason Tedor fe04b666ea Add console appender to deprecated settings test
This commit enables the console appender for the settings deprecation
logger used in the deprecated settings logging test. This output will be
useful if this test fails again (it failed once mysteriously).
2017-07-08 07:59:35 -04:00
Boaz Leskes 215bffb08b Enable bwc testing
#25512 & #25511 have been merged
2017-07-08 11:57:22 +02:00
Boaz Leskes 40ae134f5a Move `BulkItemRequest` BWC to 5.x (#25511)
The current BWC code in `BulkItemRequest` mutates the underlying `DocWriteRequests` which causes test failures and unexpected state (our test infra checks bwc serialization on the fly). This PR removes this logic from master. Another PR will add a BWC layer to 5.x only.

This PR contains the logic in https://github.com/elastic/elasticsearch/pull/25510 , which is needed to run the tests.
2017-07-08 11:42:57 +02:00
Boaz Leskes f189e819be testRecoveryAfterPrimaryPromotion: seqNo recovery doesn't require some initial indexing
Previously the primary didn't update it's own local checkpoint (and thus the global checkpoint) before some indexing occurred. With recent changes the primary now properly initializes it self and thus ops recovery is possible even if no indexing has occurred.
2017-07-08 10:05:05 +02:00
Boaz Leskes b40b002c1d Disable bwc tests in preparation of merging #25512 & #25511 2017-07-08 09:25:43 +02:00
DeDe Morton baa1858f56 [DOCS] Fix link (#25616) 2017-07-07 20:40:44 -07:00
DeDe Morton a4fedb213e Fix attribute reference on redirects page (#25614) 2017-07-07 20:15:42 -07:00
Jason Tedor 8148e25087 Fix disk allocator docs
This commit fixes the disk allocator docs which were broken due to the
inadvertent removal of some docs snippet markup.
2017-07-07 22:11:09 -04:00
Jason Tedor bc22c1c286 Add disk threshold settings validation
This commit adds cross-settings validation for the low/high/flood stage
disk watermark settings. This validation was enabled by the introduction
of multiple settings validation.

Relates #25600
2017-07-07 19:54:36 -04:00
Nik Everett 794257c421 Drop current from the list of released versions (#25187)
It hasn't been released....
2017-07-07 15:59:57 -04:00
Jason Tedor 93311ab717 Restore local checkpoint tracker on promotion
When a shard is promoted to replica, it's possible that it was
previously a replica that started following a new primary. When it
started following this new primary, the state of its local checkpoint
tracker was reset. Upon promotion, it's possible that the state of the
local checkpoint tracker has not yet restored from a successful
primary-replica re-sync. To account for this, we must restore the state
of the local checkpoint tracker when a replica shard is promoted to
primary. To do this, we stream the operations in the translog, marking
the operations that are in the translog as completed. We do this before
we fill the gaps on the newly promoted primary, ensuring that we have a
primary shard with a complete history up to the largest maximum sequence
number it has ever seen.

Relates #25553
2017-07-07 14:38:35 -04:00
matarrese 2eafbaf759 Document aggregating by day of the week (#25602)
Add documentation for aggregating by day of the week.

Closes #24660
2017-07-07 14:16:53 -04:00
Yannick Welsch baa87db5d1 Harden global checkpoint tracker
This commit refactors the global checkpont tracker to make it more
resilient. The main idea is to make it more explicit what state is
actually captured and how that state is updated through
replication/cluster state updates etc. It also fixes the issue where the
local checkpoint information is not being updated when a shard becomes
primary. The primary relocation handoff becomes very simple too, we can
just verbatim copy over the internal state.

Relates #25468
2017-07-07 14:04:28 -04:00
olcbean 2ba9fd2aec Remove deprecated created and found from index, delete and bulk (#25516)
The created and found fields in index and delete responses became obsolete after the introduction of the result field in index, update and delete responses (#19566).

After deprecating the created and found fields in 5.x (#19633), now they are removed.

Fixes #19630
2017-07-07 13:58:46 -04:00
Boaz Leskes efb29031f1 fix testEnsureVersionCompatibility for 5.5.0 release 2017-07-07 19:04:12 +02:00
Boaz Leskes 1d2a10bad8 fix Version.v6_0_0 min compatibility version to 5.5.0 2017-07-07 19:04:12 +02:00
Boaz Leskes ad1b9feb20 Add bwc indices for 5.5.0 2017-07-07 19:04:12 +02:00
Boaz Leskes e05c817e57 Add v5_5_1 constant 2017-07-07 19:04:12 +02:00
Luca Cavanna 4f4f9e0af1 [DOCS] revise high level client Search Scroll API docs (#25599)
Moved the full example at the end of the page, reduced the number of bullet points for it, and added smaller examples at the beginning of the page.
2017-07-07 17:48:58 +02:00
Lee Hinman 8aa0a5c111 Improve REST error handling when endpoint does not support HTTP verb, add OPTIONS support (#24437)
* Improved REST endpoint exception handling, see #15335

Also improved OPTIONS http method handling to better conform with the
http spec.

* Tidied up formatting and comments

See #15335

* Tests for #15335

* Cleaned up comments, added section number

* Swapped out tab indents for space indents

* Test class now extends ESSingleNodeTestCase

* Capture RestResponse so it can be examined in test cases

Simple addition to surface the RestResponse object so we can run tests
against it (see issue #15335).

* Refactored class name, included feedback

See #15335.

* Unit test for REST error handling enhancements

Randomizing unit test for enhanced REST response error handling. See
issue #15335 for more details.

* Cleaned up formatting

* New constructor to set HTTP method

Constructor added to support RestController test cases.

* Refactored FakeRestRequest, streamlined test case.

* Cleaned up conflicts

* Tests for #15335

* Added functionality to ignore or include path wildcards

See #15335

* Further enhancements to request handling

Refactored executeHandler to prioritize explicit path matches. See
#15335 for more information.

* Cosmetic fixes

* Refactored method handlers

* Removed redundant import

* Updated integration tests

* Refactoring to address issue #17853

* Cleaned up test assertions

* Fixed edge case if OPTIONS method randomly selected as invalid method

In this test, an OPTIONS method request is valid, and should not return
a 405 error.

* Remove redundant static modifier

* Hook the multiple PathTrie attempts into RestHandler.dispatchRequest

* Add missing space

* Correctly retrieve new handler for each Trie strategy

* Only copy headers to threadcontext once

* Fix test after REST header copying moved higher up

* Restore original params when trying the next trie candidate

* Remove OPTIONS for invalidHttpMethodArray so a 405 is guaranteed in tests

* Re-add the fix I already added and got removed during merge :-/

* Add missing GET method to test

* Add documentation to migration guide about breaking 404 -> 405 changes

* Explain boolean response, pull into local var

* fixup! Explain boolean response, pull into local var

* Encapsulate multiple HTTP methods into PathTrie<MethodHandlers>

* Add PathTrie.retrieveAll where all matching modes can be retrieved

Then TrieMatchingMode can be package private and not leak into RestController

* Include body of error with 405 responses to give hint about valid methods

* Fix missing usageService handler addition

I accidentally removed this :X

* Initialize PathTrieIterator modes with Arrays.asList

* Use "== false" instead of !

* Missing paren :-/
2017-07-07 09:01:23 -06:00
joachimdraeger 1ff2c13472 Avoid SecurityException in repository-S3 on DefaultS3OutputStream.flush() (#25254)
Moved SocketAccess.doPrivileged up the stack to DefaultS3OutputStream in repository-S3 plugin to avoid SecurityException by Streams.copy(). A plugin is only allowed to use its own jars when performing privileged operations. The S3 client might open a new Socket on close(). #25192
2017-07-07 09:26:50 -05:00
Christoph Büscher 0e8d7582ec [Tests] Add tests for CompletionSuggestionBuilder#build() (#25575)
This adds a unit test that checks the CompletionSuggestionContext that is 
the output of CompletionSuggestionBuilder#build.
2017-07-07 16:18:25 +02:00
Jason Tedor 5762bce4b8 Enable cross-setting validation
This commit introduces a framework for settings validation and enables
cross-setting validation.

Relates #25560
2017-07-07 10:15:52 -04:00
Viral-Inc f3569debb6 [Docs] Fix typo in bootstrap-checks.asciidoc (#25597) 2017-07-07 15:37:40 +02:00
Adrien Grand 40bb1663ee Index ids in binary form. (#25352)
Indexing ids in binary form should help with indexing speed since we would
have to compare fewer bytes upon sorting, should help with memory usage of
the live version map since keys will be shorter, and might help with disk
usage depending on how efficient the terms dictionary is at compressing
terms.

Since we can only expect base64 ids in the auto-generated case, this PR tries
to use an encoding that makes the binary id equal to the base64-decoded id in
the majority of cases (253 out of 256). It also specializes numeric ids, since
this seems to be common when content that is stored in Elasticsearch comes
from another database that uses eg. auto-increment ids.

Another option could be to require base64 ids all the time. It would make things
simpler but I'm not sure users would welcome this requirement.

This PR should bring some benefits, but I expect it to be mostly useful when
coupled with something like #24615.

Closes #18154
2017-07-07 14:22:47 +02:00
Boaz Leskes 17a587e709 bwc checkout should fetch from all remotes
This is a regression introduced in #25510, which removed the explicit fetching of upstream. Sadly this doesn't work if you don't have any local branch referring to `upstream` as an upstream branch.
2017-07-07 13:39:24 +02:00
Boaz Leskes 006e8cd1ea IndexingIT should check for global checkpoints regardless of master version
In the past global checkpoint syncing was done in the background based an interval set by an index setting. In order to set that setting something reasonable for a test, the master needed to know about the setting. Therefore the test didn't check global checkpoints if the master was old. These days the global checkpoint sync is inlined with indexing operations and that restriction is not needed.
2017-07-07 12:55:42 +02:00
Christoph Büscher 870d63d0cd [Tests] Add tests for PhraseSuggestionBuilder#build() (#25571)
This adds a unit test that checks the PhraseSuggestionContext output 
of PhraseSuggestionBuilder#build.
2017-07-07 12:53:06 +02:00
Christoph Büscher abe80b9ccb Remove unused class MinimalMap (#25590) 2017-07-07 12:51:38 +02:00
Tanguy Leroux b06a744b05 [Docs] Document Scroll API for Java High Level REST Client (#25554)
This commit adds documentation for _search/scroll and clear scroll methods of the high level Java REST client
2017-07-07 12:19:33 +02:00
Yu 2e5e45161e Disable date field mapping changing (#25285)
Make date field mapping unchangeable.

Closes #25271
2017-07-07 11:49:09 +02:00
Boaz Leskes f79c2cb8c0 Allow BWC Testing against a specific branch (#25510)
Some times we need a fix / change to have two parts in two different branches (corresponding to two different ES releases). In order to be able to test these cases you need to run the BWC tests against a local branch rather than then using a branch from `github.com/elastic/elasticsearch`.

This commit adds a system property called `tests.bwc.refspec` that allows you to do it. Note that I've chosen to go with the simplest code change for now, at the expense of some user friendliness.
2017-07-07 11:18:03 +02:00
Simon Willnauer d368d7cb9f [TEST] Remove test trace logging 2017-07-07 11:03:07 +02:00
Christoph Büscher 31f73cc06c
[Tests] Fixing test failure in CompletionSuggesterBuilderTests 2017-07-07 10:39:58 +02:00
Martijn van Groningen 00a9811f02
test: re-enable `Reindex from remote with parent join field` test no that join field type no longer always emits its fields inside search hits.
Closes #25363
2017-07-07 10:18:13 +02:00
Martijn van Groningen 6db708ef75
Move more token filters to analysis-common module
The following token filters were moved: common grams, limit token, pattern capture and pattern raplace.

Relates to #23658
2017-07-07 10:02:52 +02:00
Christoph Büscher d71feceb23 [Tests] Add tests for TermSuggestionBuilder#build() (#25558)
Adds a unit test that checks the TermSuggestionContext contents that is the result 
of TermSuggestionBuilder#build vs. the values the original builder contains.
2017-07-07 09:47:21 +02:00