Commit Graph

48124 Commits

Author SHA1 Message Date
James Rodewig 5990532cb7
[DOCS] Reformat flush API docs (#46875) (#47230) 2019-09-30 08:42:52 -04:00
David Roberts 0807d409bf [ML] Reinstate ML daily maintenance actions (#47103)
A refactoring in 6.6 meant that the ML daily
maintenance actions have not been run at all
since then. This change installs the local
master listener that schedules the ML daily
maintenance, and also defends against some
subtle race conditions that could occur in the
future if a node flipped very quickly between
master and non-master.

Fixes #47003
2019-09-30 13:12:32 +01:00
Yannick Welsch 467596871a Omit writing index metadata for non-replicated closed indices on data-only node (#47285)
Fixes a bug related to how "closed replicated indices" (introduced in 7.2) interact with the index
metadata storage mechanism, which has special handling for closed indices (but incorrectly
handles replicated closed indices). On non-master-eligible data nodes, it's possible for the
node's manifest file (which tracks the relevant metadata state that the node should persist) to
become out of sync with what's actually stored on disk, leading to an inconsistency that is then
detected at startup, refusing for the node to start up.

Closes #47276
2019-09-30 13:56:52 +02:00
Przemyslaw Gomulka d9a7bcef21
Support optional parsers in any order with DateMathParser Backport(46654) (#47217)
Currently DateMathParser with roundUp = true is relying on the DateFormatter build with
combined optional sub parsers with defaulted fields (depending on the formatter).
That means that for yyyy-MM-dd'T'HH:mm:ss||yyyy-MM-dd'T'HH:mm:ss.SSS
Java.time implementation expects optional parsers in order from most specific to
least specific (reverse in the example above).
It is causing a problem because the first parsing succeeds but does not consume the full input.
The second parser should be used.
We can work around this with keeping a list of RoundUpParsers and iterate over them choosing
the one that parsed full input. The same approach we used for regular (non date math) in
relates #40100
The jdk is not considering this to be a bug https://bugs.openjdk.java.net/browse/JDK-8188771

Those below will expect this change first
relates #46242
relates #45284
backport #46654
2019-09-30 13:54:52 +02:00
Jason Tedor 7eed3e5585
Remove use of Setting#getRaw in deprecation tests (#47262)
This commit removes a use of Setting#getRaw from the deprecation header
tests. The use of Setting#getRaw is not needed here, the x-content
infrastructure will take care of emitting the appropriate values here,
and so the caller does not need to convert these to string
representations of the settings values.
2019-09-30 06:35:59 -04:00
Jason Tedor 2cba323b4e
Remove use of get raw in token/API key settings (#47260)
These settings were using get raw to fallback to whether or not SSL is
enabled. Yet, we have a formal mechanism for falling back to a
setting. This commit cuts over to that formal mechanism.
2019-09-30 06:35:58 -04:00
David Roberts a1d3711b52 [TEST] Mute MlConfigIndexMappingsFullClusterRestartIT.testMlConfigIndexMappingsAfterMigratio
Due to https://github.com/elastic/elasticsearch/issues/47286
2019-09-30 11:24:34 +01:00
Yannick Welsch 9dc90e41fc Remove "force" version type (#47228)
It's been deprecated long ago and can be removed.

Relates to #20377

Closes #19769
2019-09-30 11:58:34 +02:00
Ryan Ernst b270f70be1 Wait for elasticsearch exit in packaging tests (#47242)
The archives stopElasticsearch utility method sends SIGTERM to the
elasticsearch process, but does not wait for it to exit. That can cause
subsequent tests to sometimes file. This commit adds wait logic to both
linux and windows for the stopElasticsearch method.

closes #44501
2019-09-30 12:09:26 +03:00
Ryan Ernst 33ebf5f5dc Fix archive tests to not create elasticsearch home (#47239)
The test for java home with special characters on linux would create a
temporary java home under /home/elasticsearch. But our packaging
assertions expect that to not exist. Unfortunately this would fail much
later when the checks were actually done in bats tests. This commit
fixes the linux test to match the behavior of windows, which links the
entire java directory, and now does it into a /tmp dir.

closes #45903
2019-09-30 12:09:26 +03:00
Alpar Torok 3648481eb2 Replace heavy weight build-tools integ tests with a simple unit test. (#44056)
Remove heavy build-tool integ test.
Add a  unit test for the plugin builder plugin.

Closes #41256 #41256
2019-09-30 10:37:52 +03:00
Rory Hunter 1c7dc3a5bf Format Java source files automatically (#46745)
This commit adds a Java source formatter and checker into the build process.
This is not yet enabled for any sub-projects - to format and check a
sub-project, add its Gradle path into `build.gradle` and run:

    ./gradlew spotlessApply

to format, and:

    ./gradlew spotlessJavaCheck
    # or:
    ./gradlew precommit

to verify formatting.
2019-09-30 10:35:47 +03:00
Alpar Torok d229e15b8d Add workaround for building docker on debian 8 (#47106)
Looks like there's a workaround with aufs used in debian 8.
Adding `tsflags=nodocs` works around this issue and results in smaller
image files also.

Closes #47097 and elastic/infra#14780
2019-09-30 09:35:45 +03:00
Yogesh Gaikwad 2be351c5d0
Use 'should' clause instead of 'filter' when querying native privileges (#47019) (#47271)
When we added support for wildcard application names, we started to build
the prefix query along with the term query but we used 'filter' clause
instead of 'should', so this would not fetch the correct application
privilege descriptor thereby failing the _has_privilege checks.
This commit changes the clause to use should and with minimum_should_match
as 1.
2019-09-30 14:14:52 +10:00
Yogesh Gaikwad cec2ff5ef4 Enhance docs for create api keys created when role descriptor not specified (#46897)
This commit adds the documentation to point the user that when one
creates API keys with no role descriptor specified then that API
key will have a point in time snapshot of user permissions.

Closes#46876
2019-09-30 12:15:29 +10:00
Rory Hunter 53a4d2176f
Convert most awaitBusy calls to assertBusy (#45794) (#47112)
Backport of #45794 to 7.x. Convert most `awaitBusy` calls to
`assertBusy`, and use asserts where possible. Follows on from #28548 by
@liketic.

There were a small number of places where it didn't make sense to me to
call `assertBusy`, so I kept the existing calls but renamed the method to
`waitUntil`. This was partly to better reflect its usage, and partly so
that anyone trying to add a new call to awaitBusy wouldn't be able to find
it.

I also didn't change the usage in `TransportStopRollupAction` as the
comments state that the local awaitBusy method is a temporary
copy-and-paste.

Other changes:

  * Rework `waitForDocs` to scale its timeout. Instead of calling
    `assertBusy` in a loop, work out a reasonable overall timeout and await
    just once.
  * Some tests failed after switching to `assertBusy` and had to be fixed.
  * Correct the expect templates in AbstractUpgradeTestCase.  The ES
    Security team confirmed that they don't use templates any more, so
    remove this from the expected templates. Also rewrite how the setup
    code checks for templates, in order to give more information.
  * Remove an expected ML template from XPackRestTestConstants The ML team
    advised that the ML tests shouldn't be waiting for any
    `.ml-notifications*` templates, since such checks should happen in the
    production code instead.
  * Also rework the template checking code in `XPackRestTestHelper` to give
    more helpful failure messages.
  * Fix issue in `DataFrameSurvivesUpgradeIT` when upgrading from < 7.4
2019-09-29 12:21:46 +01:00
Jason Tedor 98989f7b37
Use fallback settings in throttling decider (#47261)
This commit replaces some uses of Setting#getRaw in the throttling
allocation decider settings. Instead, these settings should be using
fallback settings.
2019-09-28 08:06:24 -04:00
Jason Tedor bd603b0a7b
Remove dead leniency in allow rebalance setting use (#47259)
This commit removes some leniency that exists in getting the allow
rebalance setting. Fortunately, that leniency is dead code, this can
never happen. The reason this can never happen is because the settings
infrastructure will not allow setting an invalid value for this
setting. If you try to set this in the elasticsearch.yml, then the node
will fail to start, since parsing the setting will fail. If you try to
set this via an update settings API call, then parsing the setting will
fail and the settings update will be rejected. Therefore, this leniency
can never be activated, so we remove it.

This commit is the first of a few in an attempt to remove the public
uses of Setting#getRaw.
2019-09-28 08:05:37 -04:00
lcawl ab871add36 [DOCS] Skip code snippet testing in monitoring example 2019-09-27 17:11:24 -07:00
Lisa Cawley 62baa85632 [DOCS] Adds production monitoring page (#47184) 2019-09-27 15:01:54 -07:00
James Rodewig b98c57ab21
[DOCS] Document missing query parms for cat recovery API (#47181) (#47235)
Documents the following query parameters for
the /_cat/recovery API:

* `active_only`
* `detailed`
* `index`
2019-09-27 16:31:10 -04:00
Tim Brooks e11c56760d
Fix bind failure logging for mock transport (#47150)
Currently the MockNioTransport uses a custom exception handler for
server channel exceptions. This means that bind failures are logged at
the warn level. This commit modifies the transport to use the common
TcpTransport exception handler which will log exceptions at the correct
level.
2019-09-27 13:53:48 -06:00
Jack Conradson d09965a6dd Add ClassWriter to Painless writing pass (#47140)
This the first part of a series to allow nodes to write all of their appropriate
pieces to the class. Currently, nodes must add their bindings, constants, and
functions to main SClass node for delayed writing. This instead adds a
Painless version of ClassWriter to the write pass. The Painless ClassWriter
contains an appropriate ClassVisitor that can be accessed in any node
during the process along with access to the clinit method, and finally a
shortcut for creating new MethodWriter. The next step will be removing the
delayed writing in SClass, and instead, delegate all writing responsibilities to
the nodes.
2019-09-27 11:04:15 -07:00
Nhat Nguyen 444b47ce88 Relax maxSeqNoOfUpdates assertion in FollowingEngine (#47188)
We disable MSU optimization if the local checkpoint is smaller than
max_seq_no_of_updates. Hence, we need to relax the MSU assertion in
FollowingEngine for that scenario. Suppose the leader has three
operations: index-0, delete-1, and index-2 for the same doc Id. MSU on
the leader is 1 as index-2 is an append. If the follower applies index-0
then index-2, then the assertion is violated.

Closes #47137
2019-09-27 14:00:20 -04:00
James Rodewig e01465eb88
[DOCS] Correct typo in ICU Analysis plugin description (#47175) (#47219) 2019-09-27 13:04:14 -04:00
James Rodewig b159305274
[DOCS] Add redirect for SLM API docs (#46838) (#46865) 2019-09-27 11:05:55 -04:00
David Roberts e943e27954 Spawn controller processes from a different directory on macOS (#47013)
This is the Java side of https://github.com/elastic/ml-cpp/pull/593
with a fallback so that ml-cpp bundles with either the
new or old directory structure work for the time being.
A few days after merging the C++ changes a followup to
this change will be made that removes the fallback.
2019-09-27 14:02:40 +01:00
Marios Trivyzas 01623f9f1c
SQL: Add alias DATETRUNC to DATE_TRUNC function (#47173)
To be on the safe side in terms of use cases also add the alias
DATETRUNC to the DATE_TRUNC function.

Follows: #46473

(cherry picked from commit 9ac223cb1fc66486f86e218fa785a32b61e9bacc)
2019-09-27 15:38:51 +03:00
Andrei Dan 4c909438dd
Fix OriginationDate parsing tests. (#47170) (#47200)
Drop the usage of `SimpleDateFormat` and use the `DateFormatter` instead

(cherry picked from commit 7cf509a7a11ecf6c40c44c18e8f03b8e81fcd1c2)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
2019-09-27 13:16:45 +01:00
Yannick Welsch 6fd3b4723f Remove write lock for Translog.getGeneration (#47036)
No need for the write lock, and currentFileGeneration is already protected by the read lock. 
Also removes the unused method "isCurrent".
2019-09-27 13:58:07 +02:00
Przemysław Witek 3fbd58d156
[7.x] Allow evaluation to consist of multiple steps. (#46653) (#47194) 2019-09-27 13:01:51 +02:00
Henning Andersen a1e2e208ce Mute Snapshot/Restore with repository-azure (#47204)
Relates #47201
2019-09-27 12:13:01 +02:00
István Zoltán Szabó 0ab7132c47 [DOCS] Reformats Profile API (#47168)
* [DOCS] Reformats Profile API.

* [DOCS] Fixes failing docs test.
2019-09-27 11:14:14 +02:00
Colin Goodheart-Smithe 2cd6bed9c8
Updates 7.4.0 release notes 2019-09-27 09:20:08 +01:00
Tanguy Leroux 42ae76ab7c Injected response errors in Azure repository tests should have a body (#47176)
The Azure SDK client expects server errors to have a body, 
something that looks like:

<?xml version="1.0" encoding="utf-8"?>  
<Error>  
  <Code>string-value</Code>  
  <Message>string-value</Message>  
</Error> 

I've forgot to add such errors in Azure tests and that triggers 
some NPE in the client like the one reported in #47120.

Closes #47120
2019-09-27 09:43:29 +02:00
Alpar Torok 813b130e08 Exclude the demo folder form the JDK (#47161)
The folder contains jars with source code that fail the lintian test on
debian (based) distributions.
2019-09-27 10:35:34 +03:00
Alpar Torok 944421627d Work around error building deb on Windows (#47011)
Relates to  #47007 . the `gradle-ospackage-plugin` plugin doesn't
properly support symlink on windows.

This PR changes the way we configure tasks to prevent building these
packages as part of a windows check.
2019-09-27 09:04:29 +03:00
Pius 9c60e15ebd [DOCS] Add 'remote clusters' requirement link to CCR docs (#47185) 2019-09-26 16:15:21 -04:00
Nicole Albee 4714f10650 [DOCS] Remove coming tag from 7.2.0 release highlights (#47183) 2019-09-26 15:48:54 -04:00
Costin Leau b29a2cb360 SQL: Check case where the pivot limit is reached (#47121)
In some cases, the fetch size affects the way the groups are returned
causing the last page to go beyond the limit. Add dedicated check to
prevent extra data from being returned.

Fix #47002

(cherry picked from commit f4c29646f097bbd29855300342823ef4cef61c05)
2019-09-26 22:32:42 +03:00
Jack Conradson 9b4f377474 Change Painless function node to use a block instead of raw statements (#46884)
This change improves the node structure of SFunction. SFunction now uses 
an SBlock instead of a List of AStatments reducing code duplication and 
gives a future target for symbol table scoping.
2019-09-26 10:33:35 -07:00
Igor Motov ae202fda21 SQL: Add support for shape type (#46464)
Enables support for Cartesian geometries shape type. We still need to
decide how to handle the distance function since it is currently using
the haversine distance formula and returns results in meters, which
doesn't make any sense for Cartesian geometries.

Closes #46412
Relates to #43644
2019-09-26 09:47:42 -04:00
James Rodewig 0c575dc1e8 [DOCS] Correct link to `index.store.preload` setting (#47145) 2019-09-26 08:57:09 -04:00
James Rodewig 3b626c2d56
[DOCS] Reformat get pipeline API (#47131) (#47163) 2019-09-26 08:51:12 -04:00
David Roberts 77cc6d5bad [TEST] Work around _cat/indices bug with security enabled (#47160)
When the ML native multi-node tests use _cat/indices/_all
and the request goes to a non-master node, _all is
translated to a list of concrete indices by the authz layer
on the coordinating node before the request is forwarded
to the master node. Then it is possible for the master
node to return an index_not_found_exception if one of
the concrete indices that was expanded on the
coordinating node has been deleted in the meantime.
(#47159 has been opened to track the underlying problem.)

It has been observed that the index that gets deleted when
the problem affects the ML native multi-node tests is
always the ML notifications index. The tests that fail are
only interested in the presence or absense of ML results
indices. Therefore the workaround is to only _cat indices
that match the ML results index pattern.

Fixes #45652
2019-09-26 13:29:40 +01:00
Dimitris Athanasiou 0765bd4bf7
[7.x][ML] Ensure data frame analytics task is only marked completed once (#47119) (#47157)
Closes #46907
2019-09-26 15:26:06 +03:00
Jim Ferenczi 73a09b34b8
Replace SearchContextException with SearchException (#47046)
This commit removes the SearchContextException in favor of a simpler
SearchException that doesn't leak the SearchContext.

Relates #46523
2019-09-26 14:21:23 +02:00
Tanguy Leroux 95e2ca741e
Remove unused private methods and fields (#47154)
This commit removes a bunch of unused private fields and unused
private methods from the code base.

Backport of (#47115)
2019-09-26 12:49:21 +02:00
Yogesh Gaikwad 9a64b7a888
[Backport] Validate `query` field when creating roles (#46275) (#47094)
In the current implementation, the validation of the role query
occurs at runtime when the query is being executed.

This commit adds validation for the role query when creating a role
but not for the template query as we do not have the runtime
information required for evaluating the template query (eg. authenticated user's
information). This is similar to the scripts that we
store but do not evaluate or parse if they are valid queries or not.

For validation, the query is evaluated (if not a template), parsed to build the
QueryBuilder and verify if the query type is allowed.

Closes #34252
2019-09-26 17:57:36 +10:00
jimczi 97d977f381 #47046 Fix serialization version check after backport 2019-09-26 09:56:24 +02:00