Commit Graph

41166 Commits

Author SHA1 Message Date
David Kyle b04faa059b
HLRC: ML PUT Calendar (#33362) 2018-09-14 15:00:18 +01:00
Christoph Büscher bcbbbdf660
[Tests] Fix randomization in StringTermsIT (#33678)
It looks like the COLLECT_SEGMENT_ORDS flag should be randomized.
2018-09-14 15:52:47 +02:00
Ioannis Kakavas d9f5e4fd2e Pin TLS1.2 in SSLConfigurationReloaderTests
Ensure that the SSLConfigurationReloaderTests can run with JDK 11
by pinning the HttpClient to TLS version to TLS1.2. This is necessary
becase even if the MockWebServer is set to user TLS1.2, we don't
set its enabled protocols, so if it receives a TLS1.3 request (which
is the default behavior for HttpClient in JDK11), it will use TLS1.3
and the original issue will manifest again.

Relates  #33127
Resolves #32124
2018-09-14 16:39:20 +03:00
Jason Tedor 39191331d1
Only notify ready global checkpoint listeners (#33690)
When we add a global checkpoint listener, it is also carries along with
it a value that it thinks is the current global checkpoint. This value
can be above the actual global checkpoint on a shard if the listener
knows the global checkpoint from another shard copy (e.g., the primary),
and the current shard copy is lagging behind. Today we notify the
listener whenever the global checkpoint advances, regardless if it goes
above the current global checkpoint known to the listener. This commit
reworks this implementation. Rather than thinking of the value
associated with the listener as the current global checkpoint known to
the listener, we think of it as the value that the listener is waiting
for the global checkpoint to advance to (inclusive). Now instead of
notifying all waiting listeners when the global checkpoint advances, we
only notify those that are waiting for a value not larger than the
actual global checkpoint that we advanced to.
2018-09-14 09:32:03 -04:00
Adrien Grand 4f68104865
Don't count hits via the collector if the hit count can be computed from index stats. (#33701)
This is something that we were already doing when sorting by field, which is
now also done when sorting by score. As-is this change will speed up top-k
`term` queries. This could work for `match_all` queries as well when we
implement the `setMinCompetitiveScore` API on their Scorer.
2018-09-14 14:59:16 +02:00
Jason Tedor 2282150f34
Expose retries for CCR fetch failures (#33694)
This commit exposes the number of times that a fetch has been tried to
the CCR stats endpoint, and to CCR monitoring.
2018-09-14 08:52:46 -04:00
markharwood 140c3bb61c
Test fix - Graph vertices could appear in different orders based on map insertion sequence (#33709)
Solved by sorting the vertices arrays before comparison
Closes #33686
2018-09-14 13:46:36 +01:00
Albert Zaharovits c86e2d5211
Structured audit logging (#31931)
Changes the format of log events in the audit logfile.
It also changes the filename suffix from `_access` to `_audit`.
The new entry format is consistent with Elastic Common Schema.
Entries are formatted as JSON with no nested objects and field
names have a dotted syntax. Moreover, log entries themselves
are not spaced by commas and there is exactly one entry per line.
In addition, entry fields are ordered, unlike a typical JSON doc,
such that a human would not strain his eyes over jumbled 
fields from one line to the other; the order is defined in the log4j2
properties file.
The implementation utilizes the log4j2's `StringMapMessage`.
This means that the application builds the log event as a map
and the log4j logic (the appender's layout) handle the format
internally. The layout, such as the set of printed fields and their
order, can be changed at runtime without restarting the node.
2018-09-14 15:25:53 +03:00
Alexander Reelsen faa3c16241
Core: Add DateFormatter interface for java time parsing (#33467)
The existing approach used date formatters when a format based string
like `date_time||epoch_millis` was used, instead of the custom code.

In order to properly solve this, a new interface called
`DateFormatter` has been added, which now can be implemented for custom
formatters. Currently there are two implementations, one using java time
and one doing the epoch_millis formatter, which simply parses a number
and then converts it to a date in UTC timezone.

The DateFormatter interface now also has a method to retrieve the name
of the formatter pattern, which is needed for mapping changes anyway.

The existing `CompoundDateTimeFormatter` class has been removed, the
name was not really nice anyway.

One more minor change is the fact, that the new java time using
FormatDateFormatter does not try to parse the date with its printer
implementation first (which might be a strict one and fail), but a
printer can now be specified in addition. This saves one potential
failure/exception when parsing less strict dates.

If only a printer is specified, the printer will also be used as a
parser.
2018-09-14 13:55:16 +02:00
Martijn van Groningen 222f42274e
[CCR] Check whether the rejected execution exception has the shutdown flag set (#33703)
and if so debug log it and otherwise rethrow.

This should fix a couple of test failures where during test teardown tests
failed due to uncaught exceptions being detected.
2018-09-14 13:28:11 +02:00
Igor Motov b8fb83d7a4 Mute ClusterDisruptionIT#testSendingShardFailure
Tracked by #33704
2018-09-14 14:24:06 +04:00
Jason Tedor f7c131f118
Revert "Mute FullClusterRestartSettingsUpgradeIT"
This reverts commit e9826164bd.
2018-09-14 05:55:46 -04:00
Jason Tedor d65ff17b7e
Adjust BWC version on settings upgrade test (#33650)
The skip_unavailable setting did not exist until 6.1.0. This means that
we need to skip this test on versions prior to 6.1.0. We need to use
this setting because otherwise we will fail startup without it (since we
are not setting up a real remote cluster connection). This commit adds a
skip for all versions prior to 6.1.0.
2018-09-14 05:55:01 -04:00
David Roberts 568ac10ca6
[ML] Allow overrides for some file structure detection decisions (#33630)
This change modifies the file structure detection functionality
such that some of the decisions can be overridden with user
supplied values.

The fields that can be overridden are:

- charset
- format
- has_header_row
- column_names
- delimiter
- quote
- should_trim_fields
- grok_pattern
- timestamp_field
- timestamp_format

If an override makes finding the file structure impossible then
the endpoint will return an exception.
2018-09-14 09:29:11 +01:00
Jim Ferenczi 5f495c18df Adapt skip version for doc_values format deprecation
This commit fixes bwc rest tests for the doc_values format deprecation
in search. The message of the deprecation changed in 6.4.1 so the bwc test
should not check against 6.4.0.
2018-09-14 10:22:11 +02:00
Martijn van Groningen 4bcad95fe7
[TEST] wait for no initializing shards 2018-09-14 09:59:24 +02:00
Christoph Büscher c9131983f5
[Docs] Minor fix in `has_child` javadoc comment (#33674)
The min and max constants are accidentaly the wrong way around.
2018-09-14 09:41:20 +02:00
Igor Motov e9826164bd Mute FullClusterRestartSettingsUpgradeIT
Tracked by #33697
2018-09-14 11:21:44 +04:00
Yogesh Gaikwad d810f1b094
[Kerberos] Add realm name & UPN to user metadata (#33338)
We have a Kerberos setting to remove realm part from the user
principal name (remove_realm_name). If this is true then
the realm name is removed to form username but in the process,
the realm name is lost. For scenarios like Kerberos cross-realm
authentication, one could make use of the realm name to determine
role mapping for users coming from different realms.
This commit adds user metadata for kerberos_realm and
kerberos_user_principal_name.
2018-09-14 17:17:53 +10:00
Ioannis Kakavas 8ae1eeb303
[TESTS] Disable specific locales for RestrictedTrustManagerTest (#33299)
Disable specific Thai and Japanese locales as Certificate expiration
validation fails due to the date parsing of BouncyCastle (that manifests
in a FIPS 140 JVM as this is the only place we use BouncyCastle).
Added the locale switching logic here instead of subclassing
ESTestCase as these are the only tests that fail for these locales and
JVM combination.

Resolves #33081
2018-09-14 09:42:03 +03:00
Costin Leau 736053c658
SQL: Return functions in JDBC driver metadata (#33672)
Update JDBC database metadata to advertised the supported functions
Add aliases to some date/time functions to match the ODBC spec

Fix #33671
2018-09-14 09:05:34 +03:00
Armin Braun 0b4960ff6b
SCRIPTING: Move terms_set Context to its Own Class (#33602)
* SCRIPTING: Move terms_set Context to its Own Class

* Extracted TermsSetQueryScript
* Kept mechanics close to what they were with SearchScript
2018-09-14 06:21:18 +02:00
Nhat Nguyen 189aaceecf AwaitsFix testRestoreMinmal
Tracked at #33689
2018-09-13 22:15:21 -04:00
Yogesh Gaikwad d3e27ff2f6
[Kerberos] Move tests based on SimpleKdc to evil-tests (#33492)
We have a test dependency on Apache Mina when using SimpleKdcServer
for testing Kerberos. When checking for LDAP backend connectivity,
the code checks for deadlocks which require additional security
permissions accessClassInPackage.sun.reflect. As this is only for
test and we do not want to add security permissions to production,
this commit moves these tests and related classes to
x-pack evil-tests where they can run with security manager disabled.
The plan is to handle the security manager exception in the upstream issue
DIRMINA-1093
and then once the release is available to run these tests with security
manager enabled.

Closes #32739
2018-09-14 10:07:19 +10:00
Jay Modi 3914a980f7
Security: remove wrapping in put user response (#33512)
This change removes the wrapping of the created field in the put user
response. The created field was added as a top level field in #32332,
while also still being wrapped within the `user` object of the
response. Since the value is available in both formats in 6.x, we can
remove the wrapped version for 7.0.
2018-09-13 14:40:36 -06:00
Michael Basnight 7dd22f09dc Mute failing JdbcSqlSpec functions
Relates #33687
2018-09-13 15:17:37 -05:00
Costin Leau 32a22ca00e DOC: improved wording in SQL client app section 2018-09-13 22:07:23 +03:00
Costin Leau 60ab4f97ab
SQL: Return correct catalog separator in JDBC (#33670)
JDBC DatabaseMetadata returns correct separator (:) for catalog/cluster names.

Fix #33654
2018-09-13 21:55:44 +03:00
Martijn van Groningen 53ba253aa4
[CCR] Add validation for max_retry_delay (#33648) 2018-09-13 20:52:00 +02:00
Martijn van Groningen b9d0c8f25c
[CCR] Add monitoring mapping verification test (#33662)
Added test that verifies that all fields in ShardFollowNodeTaskStatus are
mapped in monitoring-es.json
2018-09-13 20:47:24 +02:00
Armin Braun 040695b64e
CORE: Disable Setting Type Validation (#33660) (#33669)
* Reverts setting type validation introduced in #33503
2018-09-13 20:45:48 +02:00
Jason Tedor e4eb631b8e
Revert "Use serializable exception in GCP listeners (#33657)"
This reverts commit 6dfe54c838.
2018-09-13 13:55:19 -04:00
Benjamin Trent 7e51b960fb
Adding index refresh (#33647) 2018-09-13 10:44:33 -07:00
Lisa Cawley c3a817957d
[DOCS] Moves securing-communications to docs (#33640) 2018-09-13 10:42:26 -07:00
Dimitris Athanasiou 9600819cef
[HLRC][ML] Add ML delete datafeed API to HLRC (#33667)
Relates #29827
2018-09-13 17:13:36 +01:00
Nhat Nguyen dcbbaad296 Mute testRecoveryWithConcurrentIndexing
Relates #33473
Relates #33616
2018-09-13 11:53:43 -04:00
Nhat Nguyen b3071133d4 TEST: decrease logging level in the flush test
Relates #31629
2018-09-13 11:18:03 -04:00
Costin Leau a192785fc8 DOC: Add SQL section on client applications
Add setup instructions for a number of GUI SQL applications
2018-09-13 15:44:52 +03:00
Jason Tedor d806a0e59d
Fix race in global checkpoint listeners test
This race can occur if the latch from the listener notifies the test
thread and the test thread races ahead before the scheduler thread has a
chance to emit the log message. This commit fixes this test by not
counting down the latch until after the log message we are going to
assert on has been emitted.
2018-09-13 07:00:40 -04:00
Jason Tedor 6dfe54c838
Use serializable exception in GCP listeners (#33657)
We used TimeoutException here but that's not serializable. This commit
switches to a serializable exception so that we can test for the
exception type on the remote side.
2018-09-13 06:35:36 -04:00
Martijn van Groningen a69ae6b89f
[CCR] Add metadata to keep track of the index uuid of the leader index in the follow index (#33367)
The follow index api checks if the recorded uuid in the follow index matches
with uuid of the leader index and fails otherwise. This validation will
prevent a follow index from following an incompatible leader index.

The create_and_follow api will automatically add this custom index metadata
when it creates the follow index.

Closes #31505
2018-09-13 11:36:52 +02:00
Marios Trivyzas 7d3b99a9b7
SQL: Fix result column names for CAST (#33604)
Previously, when an non-pruned cast (casting as a different
data type) got applied on a table column in the `SELECT` clause,
the name of the result column didn't contain the target data type
of the cast, e.g.:

  SELECT CAST(MAX(salary) AS DOUBLE) FROM "test_emp"

returned as column name:

  CAST(MAX(salary))

instead of:

  CAST(MAX(salary) AS DOUBLE)

Closes #33571

* Added more tests for trivial casts that are pruned
2018-09-13 10:28:05 +02:00
Jim Ferenczi 6ca36bba15
Fix field mapping updates with similarity (#33634)
This change fixes a bug introduced in 6.3 that prevents fields with an explicit
similarity to be updated. It also adds a test that checks this case for similarities
but also for analyzers since they could suffer from the same problem.

Closes #33611
2018-09-13 09:21:27 +02:00
David Turner 5a3fd8e4e7
Use file-based discovery not MockUncasedHostsProvider (#33554)
Today we use a special unicast hosts provider, the `MockUncasedHostsProvider`,
in many integration tests, to deal with the dynamic nature of the allocation of
ports to nodes. However #33241 allows us to use file-based discovery to achieve
the same goal, so the special test-only `MockUncasedHostsProvider` is no longer
required.

This change removes `MockUncasedHostProvider` and replaces it with file-based
discovery in tests based on `EsIntegTestCase`.
2018-09-13 07:37:15 +02:00
David Turner 94f6d4560d Revert "Mark NamedDateTimeProcessorTests as @AwaitsFix"
This reverts commit 44a80d61b2.
2018-09-13 07:32:33 +02:00
David Turner 44a80d61b2 Mark NamedDateTimeProcessorTests as @AwaitsFix 2018-09-13 07:24:51 +02:00
Nhat Nguyen b097eff342
Resync fails to notify on unavaiable exceptions (#33615)
We fail to notify the resync listener if the resync replication hits a
shard unavailable exception. Moreover, we no longer need to swallow
these unavailable exceptions.

Relates #28571
Closes #33613
2018-09-12 21:27:59 -04:00
Shaunak Kashyap b2f78e5a47
[CCR Monitoring] Only collect stats for specified indices (#33646)
Follow up to #33617. Relates to #30086.

As with all other per-index Monitoring collectors, the `CcrStatsCollector` should only collect stats for the indices the user wants to monitor. This list is controlled by the `xpack.monitoring.collection.indices` setting and defaults to all indices.
2018-09-12 15:56:13 -07:00
Jason Tedor eb715d5290
Add follower index to CCR monitoring and status (#33645)
This commit adds the follower index to CCR shard follow task status, and
to monitoring.
2018-09-12 17:35:06 -04:00
Martijn van Groningen b5d8495789
[CCR] Add auto follow pattern APIs to transport client. (#33629) 2018-09-12 21:50:22 +02:00