Commit Graph

48113 Commits

Author SHA1 Message Date
Yogesh Gaikwad b6d1d2e6ec
Add 'create_doc' index privilege (#45806) (#47645)
Use case:
User with `create_doc` index privilege will be allowed to only index new documents
either via Index API or Bulk API.

There are two cases that we need to think:
- **User indexing a new document without specifying an Id.**
   For this ES auto generates an Id and now ES version 7.5.0 onwards defaults to `op_type` `create` we just need to authorize on the `op_type`.
- **User indexing a new document with an Id.**
   This is problematic as we do not know whether a document with Id exists or not.
   If the `op_type` is `create` then we can assume the user is trying to add a document, if it exists it is going to throw an error from the index engine.

Given these both cases, we can safely authorize based on the `op_type` value. If the value is `create` then the user with `create_doc` privilege is authorized to index new documents.

In the `AuthorizationService` when authorizing a bulk request, we check the implied action.
This code changes that to append the `:op_type/index` or `:op_type/create`
to indicate the implied index action.
2019-10-07 23:58:44 +11:00
Yogesh Gaikwad 7c862fe71f
Add support to retrieve all API keys if user has privilege (#47274) (#47641)
This commit adds support to retrieve all API keys if the authenticated
user is authorized to do so.
This removes the restriction of specifying one of the
parameters (like id, name, username and/or realm name)
when the `owner` is set to `false`.

Closes #46887
2019-10-07 23:58:21 +11:00
James Rodewig f93bb9dac5
[DOCS] Reformat type exists API docs (#47601) 2019-10-07 08:49:42 -04:00
Przemyslaw Gomulka 63bb4c91eb
Update deprecation logging doc with logger configuration (#47649) Backport#47508
Explicitly adds a configuration snippet to change logging level
2019-10-07 14:41:22 +02:00
Armin Braun b669b8f046
Simplify Snapshot Delete Further (#47626) (#47644)
This change removes the special path for deleting
the index metadata blobs and moves deleting them to
the bulk delete of unreferenced blobs at the end of
the snapshot delete process.
This saves N RPC calls for a snapshot containing N
indices and simplifies the code.
Also, this change moves the unreferenced data cleanup up
the stack to make it more obvious that any exceptions during
this pahse will be ignored and not fail the delete request.
Lastly, this change removes the needless chaining of
first deleting unreferenced data from the snapshot delete
and then running the stale data cleanup (that would also run
from the cleanup endpoint) and simply fires off the cleanup
right after updating the repository data (index-N) in parallel
to the other delete operations to speed up the delete some more.
2019-10-07 14:18:41 +02:00
Tanguy Leroux b5ac0204d2
Fail earlier Put Follow requests for closed leader indices (#47637)
Backport of (#47582)

Today when following a new leader index, we fetch the remote cluster state, 
check the remote cluster license, check the user privileges, retrieve the 
index shard stats before initiating a CCR restore session.

But if the leader index to follow is closed, we're executing a bunch of 
operations that would inevitability fail at some point (on retrieving the
 index shard stats, because this type of request forbid closed indices 
when resolving indices). We could fail a Put Follow request at the first 
step by checking the leader index state directly from the remote cluster 
state.

This also helps the Resume Follow API to fail a bit earlier.
2019-10-07 13:59:04 +02:00
Alpar Torok bc85b22c1f
Complete testclusters backport (#47623)
* Use versions specific distribution folders so we don't need to clean up (#46539)

* Retry deleting distro dir on windows

When retarting the cluster we clean up old distribution files that might
still be in use by the OS.
Windows closes resources of ded processes async, so we do a couple of
retries to get arround it.

Closes #46014

* Avoid having to delete the distro folder.

* Remove the use of ClusterFormationTasks form RestTestTask (#47022)

This PR removes a use-case of the ClusterFormationTasks and converts a
project that flew under the radar so far.
There's probably more clean-up possible here, but for now the goal is
to be able to remove that code after `RunTask` is also updated.

* Migrate some 7.x only projects
2019-10-07 11:43:57 +03:00
Armin Braun 1359ef73a3
Add IT for Snapshot Issue in 47552 (#47627) (#47634)
* Add IT for Snapshot Issue in 47552 (#47627)

Adding a specific integration test that reproduces the problem
fixed in #47552. The issue fixed only reproduces in the snapshot
resiliency otherwise which are not available in 6.8 where the
fix is being backported to as well.
2019-10-07 10:38:19 +02:00
Armin Braun 6bd033931b
Add Consistency Assertion to SnapshotsInProgress (#47598) (#47633)
Assert given input shards and indices are consistent.
Also, fixed the equality check for SnapshotsInProgress.
Before this change the tests never had more than a single waiting
shard per index so they never failed as a result of the
waiting shards list not being ordered.

Follow up to #47552
2019-10-07 10:37:56 +02:00
Luca Cavanna 736fceb18b
Fold InitialSearchPhase into AbstractSearchAsyncAction (#47182)
Historically, we have two base classes for search actions that generally need to fan out to multiple shards and then move on to the following phase: InitialSearchPhase and AbstractSearchAsyncAction that extends it. Practically, every search action extends the latter, and there are no direct subclasses of InitialSearchPhase in our codebase.

This commit folds InitialSearchPhase into AbstractSearchAsyncAction in the attempt of simplifying things and making the search code running on the coordinating node easier to reason about.
2019-10-07 10:10:04 +02:00
Andrei Dan 4506b37ed5
ILM: Skip rolling indexes that are already rolled (#47324) (#47592)
An index with an ILM policy that has a rollover action in one of the
phases was rolled over when the ILM conditions dictated regardless if
it was already rolled over (eg. manually after modifying an index
template in order to force the creation of a new index that uses the new
mappings).
This changes this behaviour and has ILM check if the index it's about to
roll has not been rolled over in the meantime.

(cherry picked from commit 37d6106feeb9f9369519117c88a9e7e30f3ac797)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
2019-10-07 07:47:47 +01:00
Ioannis Kakavas 36cabbae80
NameID mapping and Single Logout (#47288) (#47561)
Clarify in the documentation that for SAML Single Logout to be
functional, the Identity Provider needs to release a NameID.
2019-10-07 09:19:32 +03:00
debadair 41c04ef39c
[DOCS] Backporting API ref reformatting for document APIs (#47631)
* [DOCS] Reformats bulk API. (#47479)

* Reformats bulk API.

* Update docs/reference/docs/bulk.asciidoc

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Reformats mget API (#47477)

* Reformats mget API

* Update docs/reference/docs/get.asciidoc

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Incorporated feedback.

* Reformats reindex API (#47483)

* Reformats reindex API

* Incorporated review feedback.

* Reformats term vectors APIs (#47484)

* Reformat termvectors APIs

* Reformats mtermvectors

* Apply suggestions from code review

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Incorporated review feedback.
2019-10-06 22:25:21 -07:00
Dimitris Athanasiou ffacfc642c
[7.x][ML] Mute RegressionIT.testStopAndRestart (#47624) (#47625)
Relates #47612
2019-10-05 23:58:32 +03:00
debadair f765ded650 [DOCS] Comment out tag in Task Managment API Docs so it isn't rendered. (#47618)
The tag for the shared content is being rendered in the output.
2019-10-05 12:48:51 -04:00
Armin Braun 22679c7932
Fix Snapshot Corruption in Edge Case (#47552) (#47620)
This fixes missing to marking shard snapshots as failures when
multiple data-nodes are lost during the snapshot process or
shard snapshot failures have occured before a node left the cluster.

The problem was that we were simply not adding any shard entries for completed
shards on node-left events. This has no effect for a successful shard, but
for a failed shard would lead to that shard not being marked as failed during
snapshot finalization. Fixed by corectly keeping track of all previous completed
shard states as well in this case.
Also, added an assertion that without this fix would trip on almost every run of the
resiliency tests and adjusted the serialization of SnapshotsInProgress.Entry so
we have a proper assertion message.

Closes #47550
2019-10-05 15:01:06 +02:00
Armin Braun f2d2ca21e2
Cleaner Handling of Store Refcount in BlobStoreRepository (#47560) (#47594)
If a shard gets closed we properly abort its snapshot
before closing it. We should in thise case make sure to
not throw a confusing exception about trying to increment
the reference on an already closed shard in the async tasks
if the snapshot is already aborted.
Also, added an assertion to make sure that aborts are in
fact the only situation in which we run into a concurrently
closed store.
2019-10-05 09:45:10 +02:00
Gordon Brown e47bdf760e
Fix Rollover error when alias has closed indices (#47148) (#47539)
Rollover previously requested index stats for all indices in the
provided alias, which causes an exception when there is a closed index
with that alias.

This commit adjusts the IndicesOptions used on the index stats
request so that closed indices are ignored, rather than throwing
an exception.
2019-10-04 17:40:05 -06:00
Jason Tedor 43f588a29e
Fix compilation in JVM options parser code
Compilation was accidentally broken here when a backport used code from
JDK 9, which is not supported in 7.x. This commit addresses this by
using JDK 8 compatiable APIs.
2019-10-04 19:29:20 -04:00
Jason Tedor 8a7e5b0847
Move ES_TMPDIR substitution into jvm options parser (#47189)
This commit moves the ES_TMPDIR substitution that we do for JVM options
into the JVM options parser itself. This solves a problem where the fact
that the we do not make the substitution before ergonomics parsing can
lead to the JVM that we start for computing the ergonomic values failing
to start. Additionally, moving this substitution here enables us to
simplify the shell scripts since we do not need to implement this there,
and twice for Bash and Windows.
2019-10-04 19:12:28 -04:00
Mark Vieira d966e5a9b9
Eliminate Gradle task input warnings (#47538)
(cherry picked from commit e86d40ff4576fb20c64fe88f01f13e201f3b948f)
2019-10-04 15:11:41 -07:00
Jason Tedor 35ca3d68d7
Validating monitoring hosts setting while parsing (#47571)
This commit lifts the validation of the monitoring hosts setting into
the setting itself, rather than when the setting is used. This prevents
a scenario where an invalid value for the setting is accepted, but then
later fails while applying a cluster state with the invalid setting.
2019-10-04 17:32:49 -04:00
Mark Tozzi e404f7ea80
DocValueFormat implementation for date range fields (#47472) (#47605) 2019-10-04 17:21:17 -04:00
Lee Hinman 79376b7219 Set default SLM retention invocation time (#47604)
This adds a default for the `slm.retention_schedule` setting, setting it
to `0 30 1 * * ?` which is 1:30am every day.

Having retention unset meant that it would never be invoked and clean up
snapshots. We determined it would be better to have a default than never
to be run. When coming to a decision, we weighed the option of an
absolute time (such as 1:30am) versus a periodic invocation (like every
12 hours). In the end we decided on the absolute time because it has
better predictability and consistency than a periodic invocation, which
would rely on when the master node were elected or restarted.

Relates to #43663
2019-10-04 15:00:20 -06:00
Lisa Cawley f35fcf7204
[DOCS] Adds security content in the Elasticsearch Reference (#47596) 2019-10-04 13:11:05 -07:00
James Rodewig 45f12d18fb
[DOCS] Reformat shrink index API docs (#46711) (#47586) 2019-10-04 14:00:18 -04:00
Martijn van Groningen 63b169b600
Upgrade joni from 2.1.6 to 2.1.29 (#47570)
Backport of #47374

Changed the Grok class to use searchInterruptible(...) instead of search(...)
otherwise we can't interrupt long running matching via the thread watch
dog.

Joni now also provides another way to interrupt long running matches.
By invoking the interrupt() method on the Matcher. We need then to refactor
the watch thread dog to keep track of Matchers instead of Threads, but
it is a better way of doing this, since interrupting would be more direct
(not every 30k iterations) and efficient (checking a volatile field).
This work needs to be done in a follow up.
2019-10-04 12:54:49 -05:00
James Rodewig 0b4fb05540
[DOCS] Reformat refresh API docs (#46667) (#47589) 2019-10-04 13:50:09 -04:00
Przemysław Witek ee952da2e2
[7.x] Implement evaluation API for multiclass classification problem (#47126) (#47343) 2019-10-04 17:54:51 +02:00
Jack Conradson e3aab1295e Add a ScriptRoot to consolidate global data necessary for multiple passes (#47532)
This PR is to get plumbing in for a ScriptRoot class that will consolidate 
several pieces of state required by potentially multiple passes including 
PainlessLookup, CompilerSettings, FunctionTable, the root class node, and a 
synthetic counter. It's possible more may be added to this as we move 
forward and slowly make the the nodes have less mutable state.
2019-10-04 08:37:19 -07:00
Lisa Cawley 9b3e5409c1
[7.x][DOCS] Copies security source files from stack-docs (#47534) 2019-10-04 08:19:10 -07:00
James Rodewig 3cc8081274 [DOCS] Correct headings for split index API docs 2019-10-04 11:07:41 -04:00
James Rodewig bb82addf35
[DOCS] Reformat split index API docs (#46713) (#47578) 2019-10-04 10:41:31 -04:00
Andrei Stefan a46f312ded SQL: fix multi full-text functions usage with aggregate functions (#47444)
* Skip functions involving full-text predicates when replacing multiple
aggregate functions with "stats" or "matrix_stats" aggregations.

(cherry picked from commit bb14ba83128dfb7a70f825ea08b1524072fb9ad0)
2019-10-04 16:27:22 +03:00
Alpar Torok 2b16d7bcf8
Backport testclusters all (#47565)
* Bwc testclusters all (#46265)

Convert all bwc projects to testclusters

* Fix bwc versions config

* WIP fix rolling upgrade

* Fix bwc tests on old versions

* Fix rolling upgrade
2019-10-04 16:12:53 +03:00
Przemysław Witek 8c180a77f0
[7.x] Fix serialization of evaluation response. (#47557) (#47566) 2019-10-04 15:12:18 +02:00
Armin Braun c1be7a802c
Simplify Snapshot Delete Process (#47439) (#47533)
We don't need to read the SnapshotInfo for a
snapshot to determine the indices that need to
be updated when it is deleted as the `RepositoryData`
contains that information already.
This PR makes it so the `RepositoryData` is used to
determine which indices to update and also removes
the special handling for deleting snapshot metadata
and the CS snapshot blob and has those simply be
deleted as part of the deleting of other unreferenced
blobs in the last step of the delete.

This makes the snapshot delete a little faster and
more resilient by removing two RPC calls
(the separate delete and the get).

Also, this shortens the diff with #46250 as a
side-effect.
2019-10-04 13:55:16 +02:00
Przemysław Witek ec9b77deaa
[7.x] Implement new analysis type: classification (#46537) (#47559) 2019-10-04 13:47:19 +02:00
Alpar Torok 65c473bd4b Fix windows packaging tests (#47554)
On windows, it happens that the process we called terminates but some
other process it creates still has the same output strems and thus the
files open, so we can't clean it up.

This PR makes the cleanup a best effort.
2019-10-04 14:02:57 +03:00
David Roberts 31a5e1c7ee [ML] More accurate job memory overhead (#47516)
When an ML job runs the memory required can be
broken down into:

1. Memory required to load the executable code
2. Instrumented model memory
3. Other memory used by the job's main process or
   ancilliary processes that is not instrumented

Previously we added a simple fixed overhead to
account for 1 and 3. This was 100MB for anomaly
detection jobs (large because of the completely
uninstrumented categorization function and
normalize process), and 20MB for data frame
analytics jobs.

However, this was an oversimplification because
the executable code only needs to be loaded once
per machine.  Also the 100MB overhead for anomaly
detection jobs was probably too high in most cases
because categorization and normalization don't use
_that_ much memory.

This PR therefore changes the calculation of memory
requirements as follows:

1. A per-node overhead of 30MB for _only_ the first
   job of any type to be run on a given node - this
   is to account for loading the executable code
2. The established model memory (if applicable) or
   model memory limit of the job
3. A per-job overhead of 10MB for anomaly detection
   jobs and 5MB for data frame analytics jobs, to
   account for the uninstrumented memory usage

This change will enable more jobs to be run on the
same node.  It will be particularly beneficial when
there are a large number of small jobs.  It will
have less of an effect when there are a small number
of large jobs.
2019-10-04 09:57:31 +01:00
David Roberts defc97a300 Remove fallback for controller location (#47104)
This change removes the temporary controller
location fallback introduced in #47013.

Relates elastic/ml-cpp#593
2019-10-04 09:50:26 +01:00
István Zoltán Szabó a57cb5843f [DOCS] Fixes an attribute in the update datafeed API docs. (#47551) 2019-10-04 08:46:16 +02:00
Alpar Torok 67bf3a4f51 Fix default distro bwc tests 2019-10-04 09:44:17 +03:00
Alpar Torok 97a0b7dcbc Make All OS tests run on GCP instances (#46924)
This PR makes the necesary adaptations to the tests and adds a power shell script to
invoke the OS tests on GCP instances connected as CI workers.

Also noticed that logs were not being produced by the tests and that theses were not using log4j so fixed that too.

One of the difficulties in working on theses tests was that the tests just stalled with no indication where the problem is.
To ease with the debugging, after process explorer suggested that the tests are running some commands, we now have multiple timeouts: one for the tests ( which will generate a thread dump ) and one for individual commands ( that bails with the command being ran and output and error so far ) to make it easier to see what went wrong.

The tests were blocking because apparently the pipes to the sub-process were not closing, thus the threads were blocking on them and we were blocking indefinitely on the join. I'm not sure why this doesn't happen in vagrant, but we now properly deal with it.
2019-10-04 08:46:52 +03:00
Ryan Ernst f32692208e
Add explanations to script score queries (#46693) (#47548)
While function scores using scripts do allow explanations, they are only
creatable with an expert plugin. This commit improves the situation for
the newer script score query by adding the ability to set the
explanation from the script itself.

To set the explanation, a user would check for `explanation != null` to
indicate an explanation is needed, and then call
`explanation.set("some description")`.
2019-10-03 21:05:05 -07:00
Yogesh Gaikwad d371f9d44d
Fix for ApiKeyIntegTests related to Expired API keys remover (#43477) (#47546)
When API key is invalidated we do two things first it tries to trigger `ExpiredApiKeysRemover` task
and second, we do index the invalidation for the API key. The index invalidation may happen
before the `ExpiredApiKeysRemover` task is run and in that case, the API key
invalidated will also get deleted. If the `ExpiredApiKeysRemover` runs before the
API key invalidation is indexed then the API key is not deleted and will be
deleted in the future run.
This behavior was not captured in the tests related to `ExpiredApiKeysRemover`
causing intermittent failures.
This commit fixes those tests by checking if the API key invalidated is reported
back when we get API keys after invalidation and perform the checks based on that.

Closes #41747
2019-10-04 13:17:52 +10:00
Nhat Nguyen 5e4732f2bb Limit number of retaining translog files for peer recovery (#47414)
Today we control the extra translog (when soft-deletes is disabled) for
peer recoveries by size and age. If users manually (force) flush many
times within a short period, we can keep many small (or empty) translog
files as neither the size or age condition is reached. We can protect
the cluster from running out of the file descriptors in such a situation
by limiting the number of retaining translog files.
2019-10-03 20:45:29 -04:00
Lisa Cawley f81d9a4aa6
Update link to ls monitoring settings (#47529) (#47542) 2019-10-03 17:06:26 -07:00
James Rodewig b8e7f5069d
[DOCS] Reformat index recovery API docs (#46493) (#47527) 2019-10-03 16:43:26 -04:00
James Rodewig 56e7668c78 [DOCS] Reformat index shard stores API docs (#46504) 2019-10-03 15:57:35 -04:00