Commit Graph

857 Commits

Author SHA1 Message Date
Christoph Büscher ba3ceeaccf
Clean up "unused variable" warnings (#31876)
This change cleans up "unused variable" warnings. There are several cases were we 
most likely want to suppress the warnings (especially in the client documentation test
where the snippets contain many unused variables). In a lot of cases the unused
variables can just be deleted though.
2018-09-26 14:09:32 +02:00
Nhat Nguyen 48c169e065
CCR: replicates max seq_no of updates to follower (#34051)
This commit replicates the max_seq_no_of_updates on the leading index
to the primaries of the following index via ShardFollowNodeTask. The
max_seq_of_updates is then transmitted to the replicas of the follower
via replication requests (that's BulkShardOperationsRequest).

Relates #33656
2018-09-26 08:00:10 -04:00
Martijn van Groningen eae5487477
[CCR] set minimum version to 6.5.0 2018-09-26 09:31:36 +02:00
Martijn van Groningen 96b3417985
[CCR] Don't auto follow follow indices in the same cluster. (#33944) 2018-09-26 07:34:51 +02:00
Marios Trivyzas 5840be6a6b
SQL: Prevent StackOverflowError when parsing large statements (#33902)
Implement circuit breaker logic in the parser which catches expressions
that can blow up the tree and result in StackOverflowError being thrown.

Co-authored-by: Costin Leau <costin.leau@gmail.com>
2018-09-25 19:20:25 +02:00
Ed Savage cc70352b3f
[ML] Modify thresholds for normalization triggers (#33663)
[ML] Modify thresholds for normalization triggers

The (arbitrary) threshold factors used to judge if scores have
changed significantly enough to trigger a look-back renormalization have
been changed to values that reduce the frequency of such
renormalizations.

Added a clause to treat changes in scores as a 'big change' if it would
result in a change of severity reported in the UI.

Also altered the clause affecting small scores so that a change should
be considered big if scores have changed by at least 1.5. 

Relates https://github.com/elastic/machine-learning-qa/issues/263
2018-09-25 15:30:10 +01:00
Nhat Nguyen 5166dd0a4c
Replicate max seq_no of updates to replicas (#33967)
We start tracking max seq_no_of_updates on the primary in #33842. This
commit replicates that value from a primary to its replicas in replication 
requests or the translog phase of peer-recovery.

With this change, we guarantee that the value of max seq_no_of_updates
on a replica when any index/delete operation is performed at least the
max_seq_no_of_updates on the primary when that operation was executed.

Relates #33656
2018-09-25 08:07:57 -04:00
David Roberts dfe5af0411
[ML] Return both Joda and Java formats from structure finder (#33900)
Previously the timestamp_formats field in the response
from the find_file_structure endpoint contained Joda
timestamp formats.  This change makes that clear by
renaming the field to joda_timestamp_formats, and also
adds a java_timestamp_formats field containing the
equivalent Java time format strings.
2018-09-25 12:52:51 +01:00
Marios Trivyzas 2ca177be41
SQL: Fix query translation of GroupBy with Having (#34010)
Two issues are resolved:
1. The `value_type` should be either long or double in case of numeric.
2. The key label for the aggregate filter (having) was duplicate of an aggr key label.

Fixes: #33520
2018-09-25 12:17:14 +02:00
David Kyle 7bc7624caf
Leniently parse ml persistent task parameters (#33950) 2018-09-25 10:18:30 +01:00
Martijn van Groningen 793b2a94b4
[CCR] Expose auto follow stats to monitoring (#33886) 2018-09-25 07:19:46 +02:00
Shaunak Kashyap 9b86e9aed8
[Monitoring] Add cluster metadata to cluster_stats docs (#33860)
* WIP

* Adding cluster metadata to cluster stats monitoring doc

* Fixing checkstyle errors

* Adding missing license header

* Updating tests

* Getting cluster settings from cluster state

* Removing more unnecessary changes

* Adding cluster metadata settings to cluster_stats docs

* Updating test to include cluster metadata

* Fixing checkstyle

* Guarding against NPE

* Updating test fixture
2018-09-24 18:52:26 -07:00
Nhat Nguyen 6ec36b1273
CCR: Make AutoFollowMetadata immutable (#33977)
We should make AutoFollowMetadata immutable to avoid being inconsistent
when one thread modifies it while other reads it.
2018-09-24 17:47:10 -04:00
Armin Braun 018714f938
TESTS: Stabilize Renegotiation Test (#33943)
* TESTS: Stabilize Renegotiation Test

* The second `startHandshake` is not synchronous and a read of only
50ms may fail to trigger it entirely (the failure can be reproduced reliably by setting the socket timeout to `1`)
=> fixed by retrying the read until the handshake finishes (a longer timeout would've worked too,
but retrying  seemed more stable)
* Closes #33772
2018-09-24 23:40:29 +02:00
Benjamin Trent 74d7be805a
Make certain ML node settings dynamic (#33565) (#33961)
* Make certain ML node settings dynamic (#33565)

* Changing to pull in updating settings and pass to constructor

* adding note about only newly opened jobs getting updated value
2018-09-24 12:54:32 -07:00
Martijn van Groningen 2795ef561f
[CCR] Add get auto follow pattern api (#33849)
Relates to #33007
2018-09-24 20:26:13 +02:00
Tim Brooks 78e483e8d8
Introduce abstract security transport testcase (#33878)
This commit introduces an AbstractSimpleSecurityTransportTestCase for
security transports. This classes provides transport tests that are
specific for security transports. Additionally, it fixes the tests referenced in
#33285.
2018-09-24 09:44:44 -06:00
Daniel Mitterdorfer 5a53193db0 Remove public modifier in SourceOnlySnapshotTests
The public modifier is reported as redundant by checkstyle. With this
commit we remove it to avoid the checkstyle error.
2018-09-24 10:27:43 +02:00
Simon Willnauer 738d9f1a3a Drop all fully deleted segments for comparison
The source only snapshot drops fully deleted segments before snapshotting
them. In order to compare them we need to drop all fully deleted segments
in the test as well.

Closes #33755
2018-09-24 08:32:32 +02:00
Nhat Nguyen ddd5ce5740
TEST: Avoid invalid ranges in ShardChangesActionTests (#33976)
If numWrites is between 2 and 9, we will issue an invalid range because
the from_seq_no is negative. This commit makes sure that numWrites is at
least 10, and adds an explicit test to verify invalid request ranges.
2018-09-23 22:28:41 -04:00
Tim Vernum df521c1d3e
[TEST] Increase processing delay in LDAP tests (#33410)
SearchGroupsResolverInMemoryTests was (rarely) fail in a way that
suggests that the server-side delay (100ms) was not enough to trigger
the client-side timeout (5ms).

The server side delay has been increased to try and overcome this.

Resolves: #32913
2018-09-24 10:37:22 +10:00
David Roberts b89551c452
[ML] Display integers without .0 in file structure field stats (#33947)
Previously numeric values in the field_stats created by the
find_file_structure endpoint were always output with a
decimal point.  This looked unfriendly and unnatural for
fields that clearly store integer values.  This change
converts integer values to type Integer before output in
the file structure field stats.
2018-09-22 15:48:59 +01:00
Nhat Nguyen 7944a0cb25
Track max seq_no of updates or deletes on primary (#33842)
This PR is the first step to use seq_no to optimize indexing operations.
The idea is to track the max seq_no of either update or delete ops on a
primary, and transfer this information to replicas, and replicas use it
to optimize indexing plan for index operations (with assigned seq_no).

The max_seq_no_of_updates on primary is initialized once when a primary
finishes its local recovery or peer recovery in relocation or being
promoted. After that, the max_seq_no_of_updates is only advanced internally
inside an engine when processing update or delete operations.

Relates #33656
2018-09-22 08:02:57 -04:00
Costin Leau 17605bf1c6
SQL: Move away internally from JDBCType to SQLType (#33913)
Replace JDBCType enum with SQLType interface for extensibility

Fix #33094
2018-09-22 11:03:38 +03:00
Martijn van Groningen e1e5f40727
[CCR] Move headers from auto follow pattern to auto follow metadata (#33846)
This ensures that we will not serialize the headers as part of the
auto follow pattern in the to be added get auto follow api.
2018-09-21 18:08:29 +02:00
Alexander Reelsen 1de2a925ce
Watcher: Ensure that execution triggers properly on initial setup (#33360)
This commit reverts most of #33157 as it introduces another race
condition and breaks a common case of watcher, when the first watch is
added to the system and the index does not exist yet.

This means, that the index will be created, which triggers a reload, but
during this time the put watch operation that triggered this is not yet
indexed, so that both processes finish roughly add the same time and
should not overwrite each other but act complementary.

This commit reverts the logic of cleaning out the ticker engine watches
on start up, as this is done already when the execution is paused -
which also gets paused on the cluster state listener again, as we can be
sure here, that the watches index has not yet been created.

This also adds a new test, that starts a one node cluster and emulates
the case of a non existing watches index and a watch being added, which
should result in proper execution.

Closes #33320
2018-09-21 14:22:34 +02:00
Benjamin Trent e17bd8e913
Removing poor randomization for node name (#33918) 2018-09-21 04:49:20 -07:00
Christoph Büscher b654d986d7
Add OneStatementPerLineCheck to Checkstyle rules (#33682)
This change adds the OneStatementPerLineCheck to our checkstyle precommit
checks. This rule restricts the number of statements per line to one. The
resoning behind this is that it is very difficult to read multiple statements on
one line. People seem to mostly use it in short lambdas and switch statements in
our code base, but just going through the changes already uncovered some actual
problems in randomization in test code, so I think its worth it.
2018-09-21 11:52:31 +02:00
Dimitris Athanasiou 8e3a0fad9d
[ML] Refactor job deletion logic into the transport action (#33891)
The job deletion logic was scattered around a few places:
the transport action, the job manager and the deletion task.
Overloading the task with deletion logic also meant extra
dependencies in the core package which should be unnecessary.

This commit consolidates all this logic into the transport action
and replaces the deletion task with a plain one that needs not be
aware of deletion logic.
2018-09-20 15:48:42 +01:00
Andrei Stefan 6fb7e49b22
SQL: TRUNCATE and ROUND functions (#33779)
* Added TRUNCATE function, modified ROUND to accept two parameters instead of one. Made the second parameter optional for both functions.
* Added documentation for both functions.
2018-09-20 15:29:53 +03:00
Marios Trivyzas 168a0b23fe
SQL: Fix ANTL4 Grammar ambiguities. (#33854)
Removed rules in the grammar that were superfluous, as they
are already "caught" other rules in the same context.

Also switched to exact ambig detection for debug mode

Fixes: #31885
2018-09-20 14:26:07 +02:00
Martijn van Groningen 384ce58535
removed unused fields 2018-09-20 08:56:23 +02:00
Tim Vernum 3ca53437a2 Mute SimpleSecurityNetty4TransportTests.testRenegotiation
This test has failed 3 times in the last 3 days

Relates: #33772
2018-09-20 16:24:41 +10:00
Martijn van Groningen 44c7c4b166
[CCR] Add auto follow stats api (#33801)
GET /_ccr/auto_follow/stats

Returns:

```
{
   "number_of_successful_follow_indices": ...
   "number_of_failed_follow_indices": ...
   "number_of_failed_remote_cluster_state_requests": ...
   "recent_auto_follow_errors": [
      ...
   ]
}
```

Relates to #33007
2018-09-20 07:16:20 +02:00
Benjamin Trent 4767a016a5
Adding node_count to ML Usage (#33850) (#33863) 2018-09-19 13:35:09 -07:00
Nik Everett 26c4f1fb6c
Core: Default node.name to the hostname (#33677)
Changes the default of the `node.name` setting to the hostname of the
machine on which Elasticsearch is running. Previously it was the first 8
characters of the node id. This had the advantage of producing a unique
name even when the node name isn't configured but the disadvantage of
being unrecognizable and not being available until fairly late in the
startup process. Of particular interest is that it isn't available until
after logging is configured. This forces us to use a volatile read
whenever we add the node name to the log.

Using the hostname is available immediately on startup and is generally
recognizable but has the disadvantage of not being unique when run on
machines that don't set their hostname or when multiple elasticsearch
processes are run on the same host. I believe that, taken together, it
is better to default to the hostname.

1. Running multiple copies of Elasticsearch on the same node is a fairly
advanced feature. We do it all the as part of the elasticsearch build
for testing but we make sure to set the node name then.
2. That the node.name defaults to some flavor of "localhost" on an
unconfigured box feels like it isn't going to come up too much in
production. I expect most production deployments to at least set the
hostname.

As a bonus, production deployments need no longer set the node name in
most cases. At least in my experience most folks set it to the hostname
anyway.
2018-09-19 15:21:29 -04:00
Alan Woodward 5107949402
Allow TokenFilterFactories to rewrite themselves against their preceding chain (#33702)
We currently special-case SynonymFilterFactory and SynonymGraphFilterFactory, which need to 
know their predecessors in the analysis chain in order to correctly analyze their synonym lists. This
special-casing doesn't work with Referring filter factories, such as the Multiplexer or Conditional
filters. We also have a number of filters (eg the Multiplexer) that will break synonyms when they
appear before them in a chain, because they produce multiple tokens at the same position.

This commit adds two methods to the TokenFilterFactory interface.

* `getChainAwareTokenFilterFactory()` allows a filter factory to rewrite itself against its preceding
  filter chain, or to resolve references to other filters. It replaces `ReferringFilterFactory` and
  `CustomAnalyzerProvider.checkAndApplySynonymFilter`, and by default returns `this`.
* `getSynonymFilter()` defines whether or not a filter should be applied when building a synonym
  list `Analyzer`. By default it returns `true`.

Fixes #33609
2018-09-19 15:52:14 +01:00
Benjamin Trent 4190a9f1e9
Delete custom index if the only contained job is deleted (#33788)
* Delete custom index if the only contained job is deleted
2018-09-19 07:42:26 -07:00
markharwood c118581617 Test fix - Graph connections could appear in different orders
Graph connections could appear in different orders based on insertion sequence

Closes #33686
2018-09-19 15:16:14 +01:00
Shaunak Kashyap d78966c4c7
Fixing assertions in integration test (#33833) 2018-09-19 04:49:54 -07:00
Martijn van Groningen d9947c631a
[CCR] Rename idle_shard_retry_delay to poll_timout in auto follow patterns (#33821) 2018-09-19 13:13:20 +02:00
Simon Willnauer 251489d59a
Cut over to unwrap segment reader (#33843)
The fix in #33757 introduces some workaround since FilterCodecReader didn't
support unwrapping. This cuts over to a more elegant fix to access the readers
segment infos.
2018-09-19 10:18:03 +02:00
Marios Trivyzas d22b383b9c
SQL: Fix issue with options for QUERY() and MATCH(). (#33828)
Previously multiple comma separated lists of options where not
recognized correctly which resulted in only the last of them
to be taked into account, e.g.:

For the following query:
  SELECT * FROM test WHERE QUERY('search', 'default_field=foo', 'default_operator=and')"
only the `default_operator=and` was finally passed to the ES query.

Fixes: #32602
2018-09-19 10:16:24 +02:00
Martijn van Groningen 013b64a07c
[CCR] Change FollowIndexAction.Request class to be more user friendly (#33810)
Instead of having one constructor that accepts all arguments, all parameters
should be provided via setters. Only leader and follower index are required
arguments. This makes using this class in tests and transport client easier.
2018-09-19 07:18:24 +02:00
Andrei Stefan c6e3231ef3
SQL: day and month name functions tests locale providers enforcement (#33653)
DAYNAME and MONTHNAME functions tests will be skipped if the right JVM parameter (-Djava.locale.providers=COMPAT) is not used in unit testing environment
2018-09-19 07:33:15 +03:00
Martijn van Groningen 805a12361f
[CCR] Fail with a descriptive error if leader index does not exist (#33797)
Closes #33737
2018-09-18 21:47:02 +02:00
Jim Ferenczi 241c74efb2
upgrade to a new snapshot of Lucene 8 (7d0a7782fa) (#33812) 2018-09-18 18:16:40 +02:00
Costin Leau 91e45ca21b
SQL: Better handling of number parsing exceptions (#33776)
Add proper exceptions in case the parsing of numbers (too large, invalid
format) fails.

Close #33622
2018-09-18 18:51:48 +03:00
Martijn van Groningen 9fe5a273aa
[TEST] handle failed search requests differently 2018-09-18 15:55:27 +02:00
Costin Leau ab9c28a2b1
SQL: Grammar tweak for number declarations (#33767)
Consider plus and minus as part of a number declaration (to avoid the
minus be treated as a negation).

Close #33765
2018-09-18 11:44:52 +03:00