Commit Graph

1413 Commits

Author SHA1 Message Date
Ryan Ernst a2c941806b
Tests: Add support for custom contexts to mock scripts (#34100)
This commit adds the ability to plug in compilation of custom contexts
in mock script engine. This is needed for testing plugins which add
custom contexts like watcher.
2018-09-27 12:23:59 -07:00
Jake Landis 73ee721b29
ingest: correctly measure chained pipeline stats (#33912)
Prior to this change when a pipeline processor called another
pipeline, only the stats for the first processor were recorded.
The stats for the subsequent pipelines were ignored. This change
properly accounts for pipelines irregardless if they are the first
or subsequently called pipelines.

This change moves the state of the stats from the IngestService
to the pipeline itself. Cluster updates are safe since the pipelines
map is atomically swapped, and if a cluster update happens
while iterating over stats (now read directly from the pipeline)
a slightly stale view of stats may be shown.
2018-09-27 13:54:26 -05:00
Jason Tedor 899a7c7d99
Fix remote cluster seeds fallback (#34090)
Recently we introduced the settings cluster.remote to take the place of
search.remote for configuring remote cluster connections. We made this
change due to the fact that we have generalized the remote cluster
infrastructure to also be used within cross-cluster replication and not
only cross-cluster search. For backwards compatibility, when we made this
change, we allowed that cluster.remote would fallback to
search.remote. Alas, the initial change for this contained a bug for
handling the proxy and seeds settings. The bug for the seeds settings
arose because we were manually iterating over the concrete settings only
for cluster.remote seeds but not for search.remote seeds. This commit
addresses this by iterating over both cluster.remote seeds and
search.remote seeds. Additionally, when checking for existence of proxy
settings, we have to not only check cluster.remote proxy settings, but
also fallback to search.remote proxy settings. This commit addresses
both issues, and adds tests for these situations.
2018-09-27 09:47:51 -04:00
Jim Ferenczi 269ae0bc15
Handle MatchNoDocsQuery in span query wrappers (#34106)
* Handle MatchNoDocsQuery in span query wrappers

This change adds a new SpanMatchNoDocsQuery query that replaces
MatchNoDocsQuery in the span query wrappers.
The `wildcard` query now returns MatchNoDocsQuery if the target field is not
in the mapping (#34093) so we need the equivalent span query in order to
be able to pass it to other span wrappers.

Closes #34105
2018-09-27 14:19:08 +02:00
Christoph Büscher cb4cdf17f0 Update MovAvgIT AwaitsFix bug url 2018-09-27 11:11:21 +02:00
Simon Willnauer bda7bc145b
Fold EngineSearcher into Engine.Searcher (#34082)
EngineSearcher can be easily folded into Engine.Searcher which removes
a level of inheritance that is necessary for most of it's subclasses.
This change folds it into Engine.Searcher and removes the dependency on
ReferenceManager.
2018-09-27 09:06:04 +02:00
Armin Braun acd80a1e07
TESTS: Enable DEBUG Logging in Flaky Test (#34091)
* This should surface what errors are thrown on CI
and in org.elasticsearch.transport.RemoteClusterConnection.ConnectHandler#collectRemoteNodes
(the sequence of caught error in the last catch block and moving on to the next seed node
seems to be the only path by which the errors logged in #33756 could come about)
* Relates #33756
2018-09-27 06:02:24 +02:00
Nhat Nguyen ea9b33527e TEST: Add engine is closed as expected failure msg
This commit adds "engine is closed" as an expected failure message.
This change is due to #33967 in which we might access a closed engine on
promotion.

Relates #33967
2018-09-26 22:38:55 -04:00
Nhat Nguyen 12d94e44b8 Adjust bwc version for max_seq_no_of_updates
Relates #33967
Relates #33842
2018-09-26 22:12:19 -04:00
Simon Willnauer ae8e54493d
Build DocStats from SegmentInfos in ReadOnlyEngine (#34079)
This change is related to #33903 that ports the DocStats
simplification to the master branch. This change builds the docStats
in the ReadOnlyEngine from the last committed segment infos rather than
the reader.

Co-authored-by: Tanguy Leroux <tlrx.dev@gmail.com>
2018-09-27 00:16:17 +02:00
Julie Tibshirani 1d08f63eff
When creating wildcard queries, use MatchNoDocsQuery when the field type doesn't exist. (#34093) 2018-09-26 15:08:35 -07:00
Simon Willnauer 2b730d1b9d Mute MovAvgIT#testHoltWintersNotEnoughData
Relates to #34098
2018-09-26 23:50:31 +02:00
Mayya Sharipova 80c5d30f30
XContentBuilder to handle BigInteger and BigDecimal (#32888)
Although we allow to index BigInteger and BigDecimal into a keyword
field, source filtering on these fields would fail
as XContentBuilder was not able to deserialize BigInteger and BigDecimal
to json.

This modifies XContentBuilder to allow to handle BigInteger and
BigDecimal.

Closes #32395
2018-09-26 14:24:31 -04:00
Julie Tibshirani de8bfb908f
Delegate wildcard query creation to MappedFieldType. (#34062)
* Delegate wildcard query creation to MappedFieldType.
* Disallow wildcard queries on collation fields.
* Disallow wildcard queries on non-string fields.
2018-09-26 09:36:41 -07:00
Nik Everett ddce9704d4
Logging: Drop two deprecated methods (#34055)
This drops two deprecated methods from `ESLoggerFactory`, switching all
calls to those methods to calls to methods of the same name on
`LogManager`.
2018-09-26 11:20:52 -04:00
Ryan Ernst 7800b4fa91
Core: Abstract DateMathParser in an interface (#33905)
This commits creates a DateMathParser interface, which is already
implemented for both joda and java time. While currently the java time
DateMathParser is not used, this change will allow a followup which will
create a DateMathParser from a DateFormatter, so the caller does not
need to know the internals of the DateFormatter they have.
2018-09-26 07:56:25 -07:00
Zachary Tong 25d74bd0cb
Prefer mapped aggs to lead reductions (#33528)
Previously, unmapped aggs try to delegate reduction to a sibling agg that is
mapped.  That delegated agg will run the reductions, and also
reduce any pipeline aggs.  But because delegation comes before running
pipelines, the unmapped agg _also_ tries to run pipeline aggs.

This causes the pipeline to run twice, and potentially double it's output
in buckets which can create invalid JSON (e.g. same key multiple times)
and break when converting to maps.

This fixes by sorting the list of aggregations ahead of time so that mapped
aggs appear first, meaning they preferentially lead the reduction.  If all aggs
are unmapped, the first unmapped agg simply creates a new unmapped object
and returns that for the reduction.

This means that unmapped aggs no longer defer and there is no chance for 
a secondary execution of pipelines (or other side effects caused by deferring
execution).

Closes #33514
2018-09-26 10:09:31 -04:00
Nik Everett 1871e7f7e9
Search: Simply SingleFieldsVisitor (#34052)
`SingleFieldsVisitor` is meant to load a single stored field but it
manages to be quite complex to reason about because it inherits from our
"basic" `FieldsVisitor` which is designed to load many fields. This
breaks that inheritance and adds logic to `SingleFieldsVisitor` so it can
be properly stand alone. While this amounts to more lines of code they
ought to be significantly easier to reason about.
2018-09-26 09:48:15 -04:00
David Roberts 1413ace74f Mute testSplitFromOneToN and testCreateShrinkIndexToN on Windows
Relates #34080
2018-09-26 14:02:14 +01:00
Christoph Büscher ba3ceeaccf
Clean up "unused variable" warnings (#31876)
This change cleans up "unused variable" warnings. There are several cases were we 
most likely want to suppress the warnings (especially in the client documentation test
where the snippets contain many unused variables). In a lot of cases the unused
variables can just be deleted though.
2018-09-26 14:09:32 +02:00
Jim Ferenczi a255880497
Add nested and object fields to field capabilities response (#33803)
This commit adds nested and object fields to the field capabilities response.

Closes #33237
2018-09-26 08:59:41 +02:00
Ryan Ernst be8475955e
Scripting: Use ParameterMap for deprecated ctx var in update scripts (#34065)
This commit removes the sysprop controlling whether ctx is in params for
update scripts and replaces it with use of the new ParameterMap, which
outputs a deprecation warning whenever params.ctx is used.
2018-09-25 22:08:02 -07:00
Nhat Nguyen 8a56369f5b
Move max_unsafe_auto_id_timestamp constant to Engine (#34025)
We should not access InternalEngine in other classes.
2018-09-25 19:20:00 -04:00
Jim Ferenczi 0f878eff19
Add a limit for graph phrase query expansion (#34031)
Today query parsers throw TooManyClauses exception when a query creates
too many clauses. However graph phrase queries do not respect this limit.
This change adds a protection against crazy expansions that can happen when
building a graph phrase query. This is a temporary copy of the fix available
in https://issues.apache.org/jira/browse/LUCENE-8479 but not merged yet.
This logic will be removed when we integrate the Lucene patch in a future
release.
2018-09-25 21:38:47 +02:00
Igor Motov 1e6780d703 Mute AckClusterUpdateSettingsIT
Tracked by #33673
2018-09-25 14:16:47 -04:00
Armin Braun 0ba1855740
INGEST: Tests for Drop Processor (#33430)
* INGEST: Tests for Drop Processor

* UT for behavior of dropped callback
and drop processor
   * Moved drop processor to `server`
project to enable this test
* Simple IT
* Relates #32278
2018-09-25 19:29:22 +02:00
Christoph Büscher ecc087a5bb
Remove Join utility class (#34037)
The functionality can be replaces with String.join in new Java versions.
2018-09-25 15:25:54 +02:00
David Turner 7c63f5455b
Use a threadsafe map in SearchAsyncActionTests (#33700)
Today `SearchAsyncActionTests#testFanOutAndCollect` uses a simple `HashMap` for
the `nodeToContextMap` variable, which is then accessed from multiple threads
without, apparently, explicit synchronisation. This provides an explanation for
the test failure identified in #29242 in which `.toString()` returns `"[]"`
just before `.isEmpty` returns `false`, without any concurrent modifications.

This change converts `nodeToContextMap` to a `newConcurrentMap()` so that this
cannot occur. It also fixes a race condition in the detection of double-calling
the subsequent search phase.

Closes #29242.
2018-09-25 13:58:05 +01:00
Nhat Nguyen 5166dd0a4c
Replicate max seq_no of updates to replicas (#33967)
We start tracking max seq_no_of_updates on the primary in #33842. This
commit replicates that value from a primary to its replicas in replication 
requests or the translog phase of peer-recovery.

With this change, we guarantee that the value of max seq_no_of_updates
on a replica when any index/delete operation is performed at least the
max_seq_no_of_updates on the primary when that operation was executed.

Relates #33656
2018-09-25 08:07:57 -04:00
Luca Cavanna 970407c663
[DOCS] add comment to clarify cluster name resolution (#34014)
We currently fallback to local indices whenever a remote cluster is not found, as there may still be indices / aliases with the same name. Such behaviour is lenient but needs to be kept for backwards compatibility. Clarified that in the code so we don't forget.

Relates to #26247
2018-09-25 14:03:07 +02:00
Adrien Grand 612201aee0
Fix created version for similarity validation. (#33890)
It mistakenly uses the Elasticsearch major version instead of the Lucene major
version. I noticed it when backporting, it is not noticeable on master because
the only two Lucene versions that are supported, 7 and 8, encode norms the same
way, unlike Lucene 6.
2018-09-25 13:48:25 +02:00
Hendrik Muhs bf6cf6b6d9
refactor CompositeValuesSourceParserHelper for reusage by making it public (#33945)
refactor CompositeValuesSourceParserHelper for reusage by making it public and moving toXContent into it
2018-09-25 09:15:52 +02:00
David Turner 3af8fc74c7
Make TransportService more test-friendly (#33869)
Today, TransportService uses System.currentTimeMillis() to get the current time
to report on things like timeouts, and enqueues lambdas for future execution.
However, in tests it is useful to be able to fake out the current time and to
see what all these enqueued lambdas are really for. This change alters the
situation so that we can obtain the time from the more easily-faked
ThreadPool#relativeTimeInMillis(), and implements some friendlier toString()
methods on the various Runnables so we can see what they are later.
2018-09-25 07:50:18 +01:00
Armin Braun 25bc8c4b5a
Fix typo `NodeEnvironment#assertPathsDoNotExist` (#33996)
* We want to check the individual paths here one by one to
get a better to interpret assertion message
2018-09-24 17:57:27 +02:00
Julie Tibshirani 8e8bd56cc7
In MatchQuery, remove a check for fragile search analyzers. (#33927)
As far as I can tell this guard against fragile analyzers is no longer relevant, since
we stopped setting special analyzers on numeric fields (3bf6f4). Instead of removing
the guard completely, I opted to keep a check for untokenized + unnormalized fields
to avoid going through the analysis process unnecessarily.

My motivation for simplifying this check is that I'd like to add support for
`split_queries_on_whitespace` to the new 'queryable object' fields. As it stands, I would
have to add a dedicated instanceof check for the new mapper, which is not optimal.
2018-09-24 08:56:13 -07:00
Tim Brooks 78e483e8d8
Introduce abstract security transport testcase (#33878)
This commit introduces an AbstractSimpleSecurityTransportTestCase for
security transports. This classes provides transport tests that are
specific for security transports. Additionally, it fixes the tests referenced in
#33285.
2018-09-24 09:44:44 -06:00
Ignacio Vera df333ca305
TESTS: Make score Float#NaN when there is no max score (#33997)
* TESTS: Make score Float#NaN when there is no max score

Fixes test failure due to maxScore set to Float#MinValue instead
on Float#NaN. In addition the initial value for maxScore is set to
Float#NEGATIVE_INFINITY so it is an illegal value.

Closes #33993
2018-09-24 17:36:48 +02:00
Luca Cavanna e389d9e296
Clarify RemoteClusterService#groupIndices behaviour (#33899)
When executing a cross-cluster search, we need to search against all local indices (and no remote indices) in case no indices are specified. Also, if only remote indices are specified, no local indices will be queried. We previously added empty local indices whenever they were not present in the map of the grouped indices, then we would act differently later based on the extracted remote indices. Instead, we now add the empty array for local indices only in case we need to search all local indices; the entry for local indices is not added when local indices should not be searched. This way the grouped indices reflect reality and provide a better indication of what indices will be searched.
2018-09-24 11:45:33 +02:00
Christophe Bismuth 47ed6c79ee [TEST] Add validate query tests for empty and malformed queries (#33862)
Relates to #33095
2018-09-24 11:21:47 +02:00
Simon Willnauer 7d703c2f92
Fix AutoQueueAdjustingExecutorBuilder settings validation (#33922)
Settings validation in AutoQueueAdjustingExecutorBuilder always checked against
a default value which means that we never can change a max queue size that is lower
than the default. This change adds tests and fixes this validation.
2018-09-24 07:45:50 +02:00
Nhat Nguyen 432e61c971 Adjust bwc for resync request (#33964)
Relates #33964
2018-09-22 19:29:38 -04:00
Nhat Nguyen f2f08dd6c5 Adjust bwc for recovery request (#33693)
Relates #33693
2018-09-22 19:28:20 -04:00
Nhat Nguyen e7ae2f9d36
Propagate auto_id_timestamp in primary-replica resync (#33964)
A follow-up of #33693 to propagate max_seen_auto_id_timestamp in a
primary-replica resync.

Relates #33693
2018-09-22 11:40:10 -04:00
Nhat Nguyen 7944a0cb25
Track max seq_no of updates or deletes on primary (#33842)
This PR is the first step to use seq_no to optimize indexing operations.
The idea is to track the max seq_no of either update or delete ops on a
primary, and transfer this information to replicas, and replicas use it
to optimize indexing plan for index operations (with assigned seq_no).

The max_seq_no_of_updates on primary is initialized once when a primary
finishes its local recovery or peer recovery in relocation or being
promoted. After that, the max_seq_no_of_updates is only advanced internally
inside an engine when processing update or delete operations.

Relates #33656
2018-09-22 08:02:57 -04:00
Vladimir Dolzhenko 9c0316869b
Store: keep IndexFormatTooOldException and IndexFormatTooNewException in corruption marker (#33920)
Closes #33916
2018-09-21 14:00:02 +02:00
Nik Everett cac93949fe
API: Drop deprecated methods from Retry (#33925)
We deprecated the `Retry.withBackoff` flavors with `Settings` in 6.5
because they were no longer needed. This drops them form 7.0.
2018-09-21 07:55:50 -04:00
Christoph Büscher b654d986d7
Add OneStatementPerLineCheck to Checkstyle rules (#33682)
This change adds the OneStatementPerLineCheck to our checkstyle precommit
checks. This rule restricts the number of statements per line to one. The
resoning behind this is that it is very difficult to read multiple statements on
one line. People seem to mostly use it in short lambdas and switch statements in
our code base, but just going through the changes already uncovered some actual
problems in randomization in test code, so I think its worth it.
2018-09-21 11:52:31 +02:00
Nhat Nguyen 5f7f793f43
Propagate max_auto_id_timestamp in peer recovery (#33693)
Today we don't store the auto-generated timestamp of append-only
operations in Lucene; and assign -1 to every index operations
constructed from LuceneChangesSnapshot. This looks innocent but it
generates duplicate documents on a replica if a retry append-only
arrives first via peer-recovery; then an original append-only arrives
via replication. Since the retry append-only (delivered via recovery)
does not have timestamp, the replica will happily optimizes the original
request while it should not.

This change transmits the max auto-generated timestamp from the primary
to replicas before translog phase in peer recovery. This timestamp will
prevent replicas from optimizing append-only requests if retry
counterparts have been processed.

Relates #33656 
Relates #33222
2018-09-20 19:53:30 -04:00
Vladimir Dolzhenko dbe6405354 mute RemoveCorruptedShardDataCommandTests.testCorruptedIndex 2018-09-20 21:30:40 +02:00
Nhat Nguyen 76a1a863e3
TEST: stop assertSeqNos if shards movement (#33875)
Currently, assertSeqNos assumes that the cluster is stable at the end of
the test (i.e., no more shard movement). However, this assumption does
not always hold. In these cases, we can stop the assertion instead of
failing a test.

Closes #33704
2018-09-20 13:44:26 -04:00