Commit Graph

45253 Commits

Author SHA1 Message Date
Lisa Cawley 52b0b9cb55 [DOCS] Add reindex steps for internal indices (#40019) 2019-03-14 10:30:13 -07:00
Ioannis Kakavas 8dc8fc507d Handle UTF-8 values in the keystore (#39496)
* Handle UTF8 values in the keystore

Our current implementation uses CharBuffer#array to get the chars
that were decoded from the UTF-8 bytes. The backing array of
CharBuffer is created in CharsetDecoder#decode and gets an initial
length that is the same as the length of the ByteBuffer it decodes,
hence the number of UTF-8 bytes.
This works fine for the first 128 characters where each one needs
one bytes, but for the next UTF-8 characters (other latin alphabets
Greek, Cyrillic etc.) where we need 2 to 4 bytes per character, this
backing char array has a larger size than the number of the actual
chars this CharBuffer contains. Calling `array()` on it will return
a char array that can potentially have extra null chars so the
SecureString we get from the KeystoreWrapper, is not the same as the
one we entered.

This commit changes the behavior to use Arrays#copyOfRange to get
the necessary chars from the CharBuffer and adds a test with
random ( maybe not printable ) UTF-8 strings
2019-03-14 18:03:50 +02:00
Jake Landis e9fa7767ec
Fix test which still uses default type (#39997)
org.elasticsearch.xpack.monitoring.action.MonitoringBulkRequestTests#testAddRequestContent
can still randomly use a defaultType for monitoring. The defaultType
support has been removed as of PR #39888. Prior to its's removal it
would default the type if one is not specified. The _type on the monitoring
bulk end point is currently required, though it is not used as the final index type
(which defaultType would have).

Closes #39980
2019-03-14 10:37:51 -05:00
David Kyle c02f49e9d3
[ML-Dataframe] Add Data Frame client to the Java HLRC (#40040)
Adds DataFrameClient to the Java HLRC and implements PUT and 
DELETE data frame transform.
2019-03-14 14:57:12 +00:00
Jason Tedor 24973cf464
Adjust BWC version on cluster state size response
This work has been backported all the way now, so this commit adjusts
the BWC version.

Relates #40016
2019-03-14 09:42:12 -04:00
Jason Tedor 9181668edf
Stop returning cluster state size by default (#40016)
Computing the compressed size of the cluster state on every invocation
of cluster:monitor/state action is expensive, and the value of this
field is dubious anyway. Therefore we want to remove computing this
field. As a first step, we stop computing and return this field by
default. To avoid breaking users, we will give them a system property to
use to tide them over until the next major release when we will actually
remove this field. This comes with a deprecation warning too, and the
backport to the appropriate minor will also include a note in the
migration guide. There will be a follow-up to remove this field in the
next major version.
2019-03-14 08:57:55 -04:00
Jason Tedor d02bca1314
Upgrade the bouncycastle dependency to 1.61 (#40017)
This commit upgrades the bouncycastle dependency from 1.59 to 1.61.
2019-03-14 08:54:47 -04:00
Marios Trivyzas 4e9657f93f SQL: Fix bug with JDBC timezone setting and DATE type (#39978)
Previously, JDBC's REST call to the server was always sending UTC
instead of the timezone passed through connection string/properties.

Moreover the conversion to java.sql.Date was problematic as a
calculation on the epoch millis was used to set the time to 00:00:00.000
and the timezone info was lost. This caused the resulting java.sql.Date
object which is always using the JVM's timezone (no matter what timezone
setting is used in the connection string/properties) to be wrongly created.

Fixes: #39915
2019-03-14 13:41:53 +01:00
Ioannis Kakavas 2512cf3ec8 Mute testGetSslCertificates in FIPS (#40042) 2019-03-14 14:23:46 +02:00
Yogesh Gaikwad 59201915db Mute DataFrameFeatureSetTests#testUsage test (#40023) 2019-03-14 10:39:14 +00:00
Ioannis Kakavas 2361947731 Log response entity when submitting IDP form
This will allow to troubleshoot the intermittent errors in
SamlAuthneticationIT

Relates to https://github.com/elastic/elasticsearch/issues/40025
2019-03-14 12:33:36 +02:00
Andrei Stefan 4d1305b6df SQL: Extend the multi dot field notation extraction to lists of values (#39823)
(cherry picked from commit 300ae485dd08373727ca111a4d21276dd47d9a27)
2019-03-14 11:21:53 +02:00
Yogesh Gaikwad 20e5994179
Mute failing tests in NodeConnectionsServiceTests (#40034) (#40035) 2019-03-14 19:40:15 +11:00
Benjamin Trent 2016e23285
[ML] Refactor common utils out of ML plugin to XPack.Core (#39976) (#40009)
* [ML] Refactor common utils out of ML plugin to XPack.Core

* implementing GET filters with abstract transport

* removing added rest param

* adjusting how defaults can be supplied
2019-03-13 17:08:43 -05:00
Benjamin Trent 8c6ff5de31
[Data Frame] Refactor PUT transform to not create a task (#39934) (#40010)
* [Data Frame] Refactor PUT transform such that:

 * POST _start creates the task and starts it
 * GET transforms queries docs instead of tasks
 * POST _stop verifies the stored config exists before trying to stop
the task

* Addressing PR comments

* Refactoring DataFrameFeatureSet#usage, decreasing size returned getTransformConfigurations

* fixing failing usage test
2019-03-13 17:08:15 -05:00
debadair a3ef565a25 [DOCS] First pass at upgrade updates for 7.0. (#39944)
* [DOCS] First pass at upgrade updates for 7.0.

* [DOCS] Updates X-Pack terminology

* [DOCS] Incorporated feedback from lcawl.
2019-03-13 14:43:32 -07:00
Igor Motov 4a42e408c5 GEO: Add support for z values to libs/geo classes (#38921)
Adds support for z-values to all Geometry objects in the
libs/geo library.
2019-03-13 15:36:03 -04:00
Przemyslaw Gomulka 8a314a36db
Change zone formatting for all printers backport(#39568) #39952
After the joda-java time migration we were formatting zone ids with zoneOrOffsetId method. This when a date was provided with a ZoneRegion for instance America/Edmonton it was appending this zone identifier instead of zone formatted as +HH:MM.
This fix is changing the format of zone suffix for all printers and also always wrapping a Temporal into a ZonedDateTime when formatting.

closes #38471
backport #39568
2019-03-13 18:27:37 +01:00
Alpar Torok 9b8b47114e Configure TMP for test nodes on Windows (#39959)
This breaks on windows where TMP dir default to C:\Windows and startup
fails with a permission error.
I tried to create a tmp dir and pass in `TMP` env, but it lead to a
class not found error, and since testclusers is already independent of
the calling environment I stopped there.
2019-03-13 19:21:13 +02:00
Jim Ferenczi 7a7658707a
Upgrade to Lucene release 8.0.0 (#39998)
This commit upgrades to the GA release of Lucene 8

Closes #39640
2019-03-13 18:11:50 +01:00
Tim Brooks 352f9f1f39
Remove sizing from `Recycler#obtain` (#39975)
Currently there is a method `Recycler#obtain(size)` that allows a size
parameter to be passed. However all implementations ignore this
parameter and just allocate a page size based on other settings. This
commit removes this method.
2019-03-13 09:32:31 -06:00
Ioannis Kakavas 87ec511684 Mute locale dependent mapping tests (#39996) 2019-03-13 17:07:20 +02:00
Adrien Grand 9731ba4338
Make the `type` parameter optional when percolating existing documents. (#39987) (#39989)
`document_type` is the type to use for parsing the document to percolate, which
is already optional and deprecated. However `percotale` queries also have the
ability to percolate existing documents, identified by an index, an id and a
type. This change makes the latter optional and deprecated.

Closes #39963
2019-03-13 15:04:41 +01:00
Jason Tedor 16e4499b97
Fix CCR forget follower docs example
This example was missing sample values in the forget follower API
call. This commit addresses this.
2019-03-13 10:03:22 -04:00
Andrey Ershov 9300826d8a Do not log unsuccessful join attempt each time (#39756)
When performing the test with 57 master-eligible nodes and one node
crash, we saw messy elections, when multiple nodes were attempting to
become master.
JoinHelper has logged 105 long log messages with lengthy stack
traces during one such election.
To address this, we decided to log these messages every time only on
debug level.
We will log last unsuccessful join attempt (along with a timestamp)
if any with WARN level if the cluster is failing to form.

(cherry picked from commit 17a148cc27b5ac6c2e04ef5ae344da05a8a90902)
2019-03-13 13:30:31 +01:00
Martijn van Groningen 8f7ab84c5c
Add minimal docs around upgrading clusters with ccr enabled (#38037) 2019-03-13 10:11:14 +01:00
Dimitris Athanasiou 79e414df86
[ML] Fix datafeed skipping first bucket after lookback when aggs are … (#39859) (#39958)
The problem here was that `DatafeedJob` was updating the last end time searched
based on the `now` even though when there are aggregations, the extactor will
only search up to the floor of `now` against the histogram interval.
This commit fixes the issue by using the end time as calculated by the extractor.

It also adds an integration test that uses aggregations. This test would fail
before this fix. Unfortunately the test is slow as we need to wait for the
datafeed to work in real time.

Closes #39842
2019-03-13 09:09:07 +02:00
Ioannis Kakavas b8733eab00 Replace Vagrant with Docker for idp-fixture (#39948)
The change replaces the Vagrant box based fixture with a fixture
based on docker compose and 2 docker images, one for an openldap
server and one for a Shibboleth SAML Identity Provider.

The configuration of both openldap and shibboleth is identical to
the previous one, in order to minimize required changes in the
tests
2019-03-13 08:30:03 +02:00
Christoph Büscher b10dd3769c Add analysis modes to restrict token filter use contexts (#36103)
Currently token filter settings are treated as fixed once they are declared and
used in an analyzer. This is done to prevent changes in analyzers that are already
used actively to index documents, since changes to the analysis chain could
corrupt the index. However, it would be safe to allow updates to token
filters at search time ("search_analyzer"). This change introduces a new
property of token filters that allows to mark them as only being usable at search
or at index time. Any analyzer that uses these tokenfilters inherits that property
and can be rejected if they are used in other contexts. This is a first step towards
making specific token filters (e.g. synonym filter) updateable.

Relates to #29051
2019-03-12 23:48:55 +01:00
Yogesh Gaikwad db04288d14
Add pre-upgrade check to test cluster routing allocation is enabled (#39340) (#39815)
When following the steps mentioned in upgrade guide
https://www.elastic.co/guide/en/elastic-stack/6.6/upgrading-elastic-stack.html
if we disable the cluster shard allocation but fail to enable it after
upgrading the nodes and plugins, the next step of upgrading internal
indices fails. As we did not check the bulk request response for reindexing,
we delete the old index assuming it has been created. This is fatal
as we cannot recover from this state.

This commit adds a pre-upgrade check to test the cluster shard
allocation setting and fail upgrade if it is disabled. In case there
are search or bulk failures then we remove the read-only block and
fail the upgrade index request.

Closes #39339
2019-03-13 09:23:32 +11:00
Lisa Cawley 26983d1fdf [DOCS] Replaces CCR terms with attributes (#39516) 2019-03-12 14:35:22 -07:00
Lisa Cawley efd7003ea9 [DOCS] Corrected API path for invalidate token and SSL certificate examples (#39530) 2019-03-12 13:28:37 -07:00
Andy Bristol e2b88bc706 add version 6.6.3 2019-03-12 13:21:36 -07:00
Lisa Cawley 77e46b4f53 [DOCS] Added upgrade step for realm settings (#39672) 2019-03-12 13:04:10 -07:00
David Turner 049970af3e Only connect to new nodes on new cluster state (#39629)
Today, when applying new cluster state we attempt to connect to all of its
nodes as a blocking part of the application process. This is the right thing to
do with new nodes, and is a no-op on any already-connected nodes, but is
questionable on known nodes from which we are currently disconnected: there is
a risk that we are partitioned from these nodes so that any attempt to connect
to them will hang until it times out. This can dramatically slow down the
application of new cluster states which hinders the recovery of the cluster
during certain kinds of partition.

If nodes are disconnected from the master then it is likely that they are to be
removed as part of a subsequent cluster state update, so there's no need to try
and reconnect to them like this. Moreover there is no need to attempt to
reconnect to disconnected nodes as part of the cluster state application
process, because we periodically try and reconnect to any disconnected nodes,
and handle their disconnectedness reasonably gracefully in the meantime.

This commit alters this behaviour to avoid reconnecting to known nodes during
cluster state application.

Resolves #29025.
2019-03-12 19:26:13 +00:00
Jack Conradson aeb0116355 Add Painless cast tests for int and Integer (#39813) 2019-03-12 12:03:36 -07:00
Tal Levy 9ab2410436
Adding an example in the Set processor documentation to address #30604 (#39941) (#39969)
* Added an example of using set to copy values from one field to another.

* Modified the document type to match the test.
2019-03-12 11:14:41 -07:00
Michael Basnight 8c78fc096d More lenient socket binding in LDAP tests (#39864)
The LDAP tests attempt to bind all interfaces,
but if for some reason an interface can't be bound
the tests will stall until the suite times out.

This modifies the tests to be a bit more lenient and allow
some binding to fail so long as at least one succeeds.
This allows the test to continue even in more antagonistic
environments.
2019-03-12 12:00:49 -04:00
Jay Modi ba40230c7f
Add client jar for transport-nio (#39860)
This change marks the transport-nio plugin as having a client jar. The
nio transport can be used from a transport client and the
x-pack-transport artifact depends on the transport-nio jar but this jar
is not published. This change marks the transport-nio project as having
a client jar so that the jar may be published in the same way that we
publish the netty4 transport artifact.

The change to actually publish the jar will be handled separately as an
update to the release manager.
2019-03-12 09:27:08 -06:00
Gordon Brown da67c2f7f8
Deprecation check for indices with very large numbers of fields (#39869)
Indices with very large numbers of fields (>1024 by default) that do not
have index.query.default_field set will experience query failures in 7.0
for Simple Query String and Multi-Match queries. This deprecation check
issues a warning for indices of that size that do not have
index.query.default_field set.

This also adds a deprecation check for index templates with field counts
that would trigger these query failures as well.
2019-03-12 09:06:31 -06:00
Andrey Ershov aa967edd73 Add elasticsearch-node tool docs (#37812)
This commit, mostly authored by @DaveCTurner,
adds documentation for elasticsearch-node tool #37696.

(cherry picked from commit 09425d5a5158c2d3fdad794411b3bbc4bba47b15)
2019-03-12 12:47:10 +01:00
Przemyslaw Gomulka a29bba4ede
Migrate Streamable to writeable for index package backport(#37381) #39949
Migrate streamable classes from index package to Writeable and clean up access modifiers

Related to #34389
backport#37381
2019-03-12 12:10:36 +01:00
lzh3636 ad55e5b80d Log missing file exception when failing to read metadata snapshot (#32920)
Adds the exception to the logged output, which contains info about the file that's missing.
2019-03-12 10:41:44 +01:00
Adrien Grand 66b3a3a546
Warn about the fact that the terms index is moving off-heap. (#39918)
Lucene 8.0 includes a [change](https://issues.apache.org/jira/browse/LUCENE-8635)
that moves the terms index off-heap for all fields but ID fields. I'm
including this in the migration notes so that users who have queries that match
lots of terms won't be surprised in case of slowdown.
2019-03-12 10:16:41 +01:00
Przemyslaw Gomulka 675489d54c
Date_math_expression doc to inform about format backport#39840 #39903
The format since 7 should be compatible with java-time. Users should be informed about this on this page
backport#39840
2019-03-12 09:19:44 +01:00
Adrien Grand 466864710a Update the how-to section of the docs for 7.0: (#37717)
- new `rank_feature`/`script_score` queries
 - new `index_phrases`/`index_prefixes` options
 - disabling `_field_names` doesn't help anymore
 - adaptive replica selection is on by default
2019-03-12 08:24:39 +01:00
Adrien Grand 5528554aac
Generate release notes for 7.0.0-alpha1. (#39899)
We were missing release notes for 7.0.0-alpha1. I generated them by running
the release-notes script with a quick hack that filtered out pull requests
that had been closed on or after 2018-11-15.

Some breaking changes had been documented in the release notes rather than
the migration guide so I moved them.
2019-03-12 08:10:18 +01:00
Daniel Mitterdorfer 34006105ae
Allow IntegTestClusters to run with FIPS 140 JVMs (#39917) (#39923)
The changes in #39732 mean that nodes in the IntegTest clusters will
now run with whichever java version is defined as `runtime.java` and
not JAVA_HOME anymore.
This means that these nodes will also run in JVM with fips approved
mode enabled and as such, need to have access to the password for the
BCFKS keystore that is used as the default keystore/truststore.

This change sets the two necessary system properties.

Resolves #39855
2019-03-12 06:46:51 +01:00
Nhat Nguyen ce5f09ab04 Enforce retention leases require soft deletes (#39922)
If a primary on 6.7 and a replica on 5.6 are running more than 5 minutes
(retention leases background sync interval), the retention leases
background sync will be triggered, and it will trip 6.7 node due to the
illegal checkpoint value. We can fix the problem by making the returned
checkpoint depends on the node version. This PR, however, chooses to
enforce retention leases require soft deletes, and make retention leases
sync noop if soft deletes is disabled instead.

Closes #39914
2019-03-11 22:37:47 -04:00
Nhat Nguyen bf814357ad Enable soft deletes in RetentionLeaseIT
Relates #39922
2019-03-11 22:37:42 -04:00