Commit Graph

47229 Commits

Author SHA1 Message Date
Armin Braun 90803a5caf
Reenable Integ Tests in native-multi-node-tests (#45482) (#45496)
* Reenable Integ Tests in native-multi-node-tests

* The tests broken here were likely fixed by #45463 => let's reenable them and see if things run fine again
* Relates #45405, #45455
2019-08-13 15:55:54 +02:00
Mayya Sharipova 22ab389531 Clarify that FLS/DLS disable shard request cache (#45462) 2019-08-13 09:05:57 -04:00
James Rodewig 109358ca3f [DOCS] Reformat cat API titles (#45407) 2019-08-13 08:37:10 -04:00
James Rodewig df0df9402d [DOCS] Correct anchor for cat segments API examples 2019-08-13 08:30:20 -04:00
James Rodewig 2543e9eca5 [DOCS] Reformats cat shards API (#45392) 2019-08-13 08:30:14 -04:00
István Zoltán Szabó 0ff723852f [DOCS] Reformats cluster update settings API (#45337) 2019-08-13 14:24:18 +02:00
István Zoltán Szabó f4b001dc4b [DOCS] Adds heading between examples to better separate them. (#45499) 2019-08-13 13:55:15 +02:00
István Zoltán Szabó 356a632b95 [DOCS] Reformats cluster node info API (#45446)
Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2019-08-13 13:33:15 +02:00
István Zoltán Szabó 4ee7ac25ae [DOCS] Reformats cluster node stats API (#45441)
Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2019-08-13 12:48:07 +02:00
Alexander Reelsen dd527b4e91 Fix watcher HttpClient URL creation (#45207)
The http client could end up creating URLs, that did not resemble the
original one, when encoding. This fixes a couple of corner cases, where
too much or too few slashes were added to an URI.

Closes #44970
2019-08-13 12:15:54 +02:00
Armin Braun 00e4fba2fb
Simplify and Optimize RestController Slightly (#45419) (#45485)
* Simplify the path iterator to generate less garbage
* `dispatchRequest` always terminates, adjust code accordingly
2019-08-13 10:43:30 +02:00
Ryan Ernst 488673f636
Guard distro tests for pre oss in distribution filesnames (#45478)
Before #45064, the bats tests skipped the upgrade tests when the random
upgrade version is before 6.3.0. This commit restores that behavior.

closes #45476
2019-08-13 00:13:20 -07:00
Przemysław Witek 1aed388a24
Add view_index_metadata to roles.yml and remove as many df analytics test cases from build.gradle blacklist as possible. (#45451) (#45465) 2019-08-13 08:31:58 +02:00
Yogesh Gaikwad 471d940c44
Refactor cluster privileges and cluster permission (#45265) (#45442)
The current implementations make it difficult for
adding new privileges (example: a cluster privilege which is
more than cluster action-based and not exposed to the security
administrator). On the high level, we would like our cluster privilege
either:
- a named cluster privilege
  This corresponds to `cluster` field from the role descriptor
- or a configurable cluster privilege
  This corresponds to the `global` field from the role-descriptor and
allows a security administrator to configure them.

Some of the responsibilities like the merging of action based cluster privileges
are now pushed at cluster permission level. How to implement the predicate
(using Automaton) is being now enforced by cluster permission.

`ClusterPermission` helps in enforcing the cluster level access either by
performing checks against cluster action and optionally against a request.
It is a collection of one or more permission checks where if any of the checks
allow access then the permission allows access to a cluster action.

Implementations of cluster privilege must be able to provide information
regarding the predicates to the cluster permission so that can be enforced.
This is enforced by making implementations of cluster privilege aware of
cluster permission builder and provide a way to specify how the permission is
to be built for a given privilege.

This commit renames `ConditionalClusterPrivilege` to `ConfigurableClusterPrivilege`.
`ConfigurableClusterPrivilege` is a renderable cluster privilege exposed
as a `global` field in role descriptor.

Other than this there is a requirement where we would want to know if a cluster
permission is implied by another cluster-permission (`has-privileges`).
This is helpful in addressing queries related to privileges for a user.
This is not just simply checking of cluster permissions since we do not
have access to runtime information (like request object).
This refactoring does not try to address those scenarios.

Relates #44048
2019-08-13 09:06:18 +10:00
Ryan Ernst 97efb6a403
Convert vagrant tests to per platform projects (#45064)
The vagrant based tests currently reside in a single project, creating
dozens of tasks to manage starting and stopping the vagrant VM along
with running java and bats tests within each image. This all-in-one
pattern makes parallelizing packaging tests difficult.

This commit rewrites the vagrant testing infrastructure to be
independent of the actual test runners, thus allowing each platform to
be handled in a separate subproject. Additionally, the java and bats
tests are changed to be run through a "destructive" gradle task, which
is run inside the VM. The combination of these will allow
parallelization both locally (through running several VMs at once) as
well as running the destructive tasks in CI machines dedicated to each
platform (thus removing the need for vagrant in CI).
2019-08-12 16:01:53 -07:00
Tim Brooks ae06a9399a
Fix bug in copying bytes for socket write (#45463)
Currently we take the array of nio buffers from the netty channel
outbound buffer and copy their bytes to a direct buffer. In the process
we mutate the nio buffer positions. It seems like netty will continue to
reuse these buffers. This means than any data that is not flushed in a
call is lost. This commit fixes this by incrementing the positions after
the flush has completed. This is similar to the behavior that
SocketChannel would have provided and netty relied upon.

Fixes #45444.
2019-08-12 15:59:26 -06:00
Julie Tibshirani dc1856ca53 Make sure to validate the type before attempting to merge a new mapping. (#45157)
Currently, when adding a new mapping, we attempt to parse + merge it before
checking whether its top-level document type matches the existing type. So when
a user attempts to introduce a new mapping type, we may give a confusing error
message around merging instead of complaining that it's not possible to add
more than one type ("Rejecting mapping update to [my-index] as the final
mapping would have more than 1 type...").

This PR moves the type validation to the start of
`MetaDataMappingService#applyRequest` so that we make sure the type matches
before performing any mapper merging.

We already partially addressed this issue in #29316, but the tests there
focused on `MapperService` and did not catch this problem with end-to-end
mapping updates.

Addresses #43012.
2019-08-12 14:28:03 -07:00
Zachary Tong 4d97d2c50f Revert "Only execute one final reduction in InternalAutoDateHistogram (#45359)"
This reverts commit c0ea8a867e.
2019-08-12 17:17:17 -04:00
Mark Vieira 2a1f0c7b4a
Fix location of reaper jar on Windows (#45464)
(cherry picked from commit 5021c5d8bf815ed11a2fe9cf4d3a2c2ed2e2e0b3)
2019-08-12 13:25:37 -07:00
Chris Dean 82d48cfcc9
[DOCS] Added cross-link to snapshot lifecycle management. Closes #44588. (#45408) (#45468)
merging #44588 changes into 7.x
2019-08-12 15:13:11 -05:00
Julie Tibshirani 8c4394d5d7 Fix a bug where mappings are dropped from rollover requests. (#45411)
We accidentally introduced this bug when adding a typeless version of the
rollover request. The bug is not present if include_type_name is set to true.
2019-08-12 12:46:27 -07:00
Michael Basnight a521e4c86f Retrieve processors instead of checking existence (#45354)
The previous hasProcessors method would validate if a processor was
present within a pipeline, but would not return the contents of the
processors. This does not allow a consumer to inspect the processor for
specific metadata. The method now returns the list of processors based
on the class of the processor passed in.
2019-08-12 13:48:17 -05:00
Zachary Tong 472f6ef41a Mute InternalAutoDateHistogramTests#testReduceRandom() 2019-08-12 14:45:08 -04:00
Zachary Tong c0ea8a867e Only execute one final reduction in InternalAutoDateHistogram (#45359)
Because auto-date-histo can perform multiple reductions while
merging buckets, we need to ensure that the intermediate reductions
are done with a `finalReduce` set to false to prevent Pipeline aggs
from generating their output.

Once all the buckets have been merged and the output is stable,
a mostly-noop reduction can be performed which will allow pipelines
to generate their output.
2019-08-12 14:07:38 -04:00
Ryan Ernst 620cd4cb20 Fix old name of script context in docs (#45410)
The docs contain an example how to limit the scripting contexts allowed,
but the example used an outdated name for the scoring context.

closes #44232
2019-08-12 09:29:13 -07:00
Mark Vieira 7e3379444b
Fix build failure due to unknown task and disable test conventions
(cherry picked from commit 8ed84bc5cef9bcfae6c817059f764d97e4451a4a)
2019-08-12 09:18:39 -07:00
Mark Vieira 250544c4eb
Upgrade to build scan plugin 2.4 (#45426) 2019-08-12 09:10:57 -07:00
Przemyslaw Gomulka 421e9b8e8b
Mute integ tests in native-multi-node-tests (#45457)
Tracked at #45405
2019-08-12 17:42:24 +02:00
James Rodewig e3f618e1d3 [DOCS] Reformat query string query (#45296) 2019-08-12 11:18:00 -04:00
James Rodewig bff4ef2d95 [DOCS] Reformats cat templates API (#45403) 2019-08-12 08:45:45 -04:00
James Rodewig c1c3bd011a [DOCS] Reformats cat snapshots API (#45402) 2019-08-12 08:41:57 -04:00
James Rodewig 7556d37925 [DOCS] Reformats cat thread pool API (#45385) 2019-08-12 08:36:57 -04:00
James Rodewig e8acf6b34f [DOCS] Reformats cat repositories API (#45382) 2019-08-12 08:30:46 -04:00
James Rodewig fb81346055 [DOCS] Reformats cat segments API (#45397) 2019-08-12 08:25:32 -04:00
James Rodewig 4e79e04c0e [DOCS] Reformats cat recovery API (#45345) 2019-08-12 08:17:52 -04:00
Bogdan Pintea 499be8398f SQL: ODBC: document newest connection string parameters (#44185)
* SQL: ODBC: document newest conn string parameters

This commit adds the documentation for two newly added connection string
parameters: AutoEscapePVA and IndexIncludeFrozen.

It also removes the recommended OSes from the prerequisites list and
places the recommendation distinctively: the unmet prerequisites will
fail the installation, while the driver would install on other OSes than
those recommended.

* address review suggestions.

- adjust phrasing for clearer message.

(cherry picked from commit e18ac10c6e163a04f5b7cf7fa72f262882ffb711)
2019-08-12 13:49:11 +02:00
Przemyslaw Gomulka d11ae08467
Muting ForecastIT.testOverflowToDisk (#45435) (#45438)
awaits #45405
2019-08-12 11:01:32 +02:00
Dimitris Athanasiou d02d6e40c2 [ML] Mute regression integ test
Relates #45425
2019-08-12 10:59:24 +03:00
Albert Zaharovits 2cb172f079
CreateIndex and PutIndexTemplate with typeless mapping (#45120)
This commit makes sure that mapping parameters to `CreateIndex` and
`PutIndexTemplate` are keyed by the type name. 

`IndexCreationTask` expects mappings to be keyed by the type name.
It asserts this for template mappings but not for the mappings in the request.
The `CreateIndexRequest` and `RestCreateIndexAction` mostly make it sure
that the mapping is keyed by a type name, but not always.
When building the create-index request outside of the REST handler, there are
a few methods to set the mapping for the request. Some of them add the type
name some of them do not.
For example, `CreateIndexRequest#mapping(String type, Map<String, ?> source)`
adds the type name, but
`CreateIndexRequest#mapping(String type, XContentBuilder source)` does not.
This PR asserts the type name in the request mapping inside `IndexCreationTask`
and makes all `CreateIndexRequest#mapping` methods add the type name.
2019-08-12 08:05:07 +03:00
Armin Braun a9e1402189
Remove Settings from BaseRestRequest Constructor (#45418) (#45429)
* Resolving the todo, cleaning up the unused `settings` parameter
* Cleaning up some other minor dead code in affected classes
2019-08-12 05:14:45 +02:00
Emmanuel DEMEY f8c08c537b Add snippet for the search_type query parameter (#43540) 2019-08-11 18:36:52 -04:00
Nhat Nguyen cf9a73b5ac Call afterWriteOperation after trim translog in peer recovery (#45182)
testShouldFlushAfterPeerRecovery was added #28350 to make sure the
flushing loop triggered by afterWriteOperation eventually terminates.
This test relies on the fact that we call afterWriteOperation after
making changes in translog. In #44756, we roll a new generation in
RecoveryTarget#finalizeRecovery but do not call afterWriteOperation.

Relates #28350
Relates #45073
2019-08-10 22:59:02 -04:00
Nhat Nguyen 25c6102101 Trim local translog in peer recovery (#44756)
Today, if an operation-based peer recovery occurs, we won't trim
translog but leave it as is. Some unacknowledged operations existing in
translog of that replica might suddenly reappear when it gets promoted.
With this change, we ensure trimming translog above the starting
sequence number of phase 2. This change can allow us to read translog
forward.
2019-08-10 22:59:02 -04:00
Armin Braun 1cd464d675
Isolate Request in Call-Chain for REST Request Handling (#45130) (#45417)
* Follow up to #44949
* Stop using a special code path for multi-line JSON and instead handle its detection like that of other XContent types when creating the request
* Only leave a single path that holds a reference to the full REST request
   * In the next step we can move the copying of request content to happen before the actual request handling and make it conditional on the handler in question to stop copying bulk requests as suggested in #44564
2019-08-10 10:21:01 +02:00
Mark Vieira 491880edde
Fix build cache misses caused by embedded reaper jar (#45404)
(cherry picked from commit 788feced760bc2a5f453ebb07f1dbb288c6232b2)
2019-08-09 21:10:48 -07:00
Mark Vieira b5c3c9767c
Publish CI build scans to Gradle Enterprise (#45249)
(cherry picked from commit 957a232afc3ecd43326509043ab73cc6c2a26411)
2019-08-09 15:53:50 -07:00
Benjamin Trent fac1a6f8e8
[ML][Data Frame] have DataFrameTransformConfigUpdate#apply set Version (#45391) (#45400) 2019-08-09 14:32:49 -05:00
Hendrik Muhs bf4da6c6ad
[ML-DataFrame] fix starting a batch data frame after stopping at runtime (#45340) (#45381)
fix loading of next checkpoint after data frame transform has been stopped/started within one run

closes #45339
2019-08-09 20:30:11 +02:00
Dimitris Athanasiou 27497ff75f
[7.x][ML] Add regression analysis to DF analytics (#45292) (#45388)
This commit adds a first draft of a regression analysis
to data frame analytics. There is high probability that
the exact syntax might change.

This commit adds the new analysis type and its parameters as
well as appropriate validation. It also modifies the extractor
and the fields detector to be able to handle categorical fields
as regression analysis supports them.
2019-08-09 19:31:13 +03:00
Armin Braun d1ed9bdbfd
Use StepListener to Simplify SnapshotResiliencyTests (#45233) (#45386)
* Reduces complicated callback relations in `testSuccessfulSnapshotAndRestore` to flat steps of sequential actions
* Will refactor the other tests in this suit as a follow up
   * This format certainly makes it easier to create more complicated tests that involve multiple subsequent snapshots as it would allow adding loops
2019-08-09 18:19:48 +02:00