Commit Graph

126 Commits

Author SHA1 Message Date
William Brafford 49e30b15a2
Deprecate disabling basic-license features () ()
We believe there's no longer a need to be able to disable basic-license
features completely using the "xpack.*.enabled" settings. If users don't
want to use those features, they simply don't need to use them. Having
such features always available lets us build more complex features that
assume basic-license features are present.

This commit deprecates settings of the form "xpack.*.enabled" for
basic-license features, excluding "security", which is a special case.
It also removes deprecated settings from integration tests and unit
tests where they're not directly relevant; e.g. monitoring and ILM are
no longer disabled in many integration tests.
2020-04-17 15:04:17 -04:00
David Turner 7941f4a47e Add RepositoriesService to createComponents() args ()
Today we pass the `RepositoriesService` to the searchable snapshots plugin
during the initialization of the `RepositoryModule`, forcing the plugin to be a
`RepositoryPlugin` even though it does not implement any repositories.

After discussion we decided it best for now to pass this in via
`Plugin#createComponents` instead, pending some future work in which plugins
can depend on services more dynamically.
2020-04-16 16:27:36 +01:00
Ryan Ernst ae14d1661e
Replace license check isAuthAllowed with isSecurityEnabled () ()
The isAuthAllowed() method for license checking is used by code that
wants to ensure security is both enabled and available. The enabled
state is dynamic and provided by isSecurityEnabled(). But since security
is available with all license types, an check on the license level is
not necessary. Thus, this change replaces isAuthAllowed() with calling
isSecurityEnabled().
2020-04-13 12:26:39 -07:00
Jay Modi 3600c9862f
Reintroduce system index APIs for Kibana ()
This change reintroduces the system index APIs for Kibana without the
changes made for marking what system indices could be accessed using
these APIs. In essence, this is a partial revert of . The changes
for marking what system indices should be allowed access will be
handled in a separate change.

The APIs introduced here are wrapped versions of the existing REST
endpoints. A new setting is also introduced since the Kibana system
indices' names are allowed to be changed by a user in case multiple
instances of Kibana use the same instance of Elasticsearch.

Relates 
Backport of 
2020-04-08 09:08:49 -06:00
Tal Levy 254d1e3543
[7.x] Create new `geo` module and migrate geo_shape registration () ()
This commit introduces a new `geo` module that is intended
to be contain all the geo-spatial-specific features in server.

As a first step, the responsibility of registering the geo_shape
field mapper is moved to this module.

Co-authored-by: Nicholas Knize <nknize@gmail.com>
2020-04-07 16:30:58 -07:00
Mayya Sharipova bf4857d9e0
Search hit refactoring () ()
Refactor SearchHit to have separate document and meta fields.
This is a part of bigger refactoring of issue  to remove
dependency on MapperService to check if a field is metafield.

Relates to PR: 
Relates to issue 

Co-authored-by: sandmannn <bohdanpukalskyi@gmail.com>
2020-04-01 15:19:00 -04:00
Jason Tedor 5fcda57b37
Rename MetaData to Metadata in all of the places ()
This is a simple naming change PR, to fix the fact that "metadata" is a
single English word, and for too long we have not followed general
naming conventions for it. We are also not consistent about it, for
example, METADATA instead of META_DATA if we were trying to be
consistent with MetaData (although METADATA is correct when considered
in the context of "metadata"). This was a simple find and replace across
the code base, only taking a few minutes to fix this naming issue
forever.
2020-03-31 17:24:38 -04:00
Martijn van Groningen 4b4fbc160d
Refactor AliasOrIndex abstraction. ()
Backport of 

In order to prepare the `AliasOrIndex` abstraction for the introduction of data streams,
the abstraction needs to be made more flexible, because currently it really can be only
an alias or an index.

* Renamed `AliasOrIndex` to `IndexAbstraction`.
* Introduced a `IndexAbstraction.Type` enum to indicate what a `IndexAbstraction` instance is.
* Replaced the `isAlias()` method that returns a boolean with the `getType()` method that returns the new Type enum.
* Moved `getWriteIndex()` up from the `IndexAbstraction.Alias` to the `IndexAbstraction` interface.
* Moved `getAliasName()` up from the `IndexAbstraction.Alias` to the `IndexAbstraction` interface and renamed it to `getName()`.
* Removed unnecessary casting to `IndexAbstraction.Alias` by just checking the `getType()` method.

Relates to 
2020-03-30 10:12:16 +02:00
Gordon Brown 0d30b48613
Disallow negative TimeValues ()
This commit causes negative TimeValues, other than -1 which is sometimes used as
a sentinel value, to be rejected during parsing.

Also introduces a hack to allow ILM to load policies which were written to the
cluster state with a negative min_age, treating those values as 0, which should
match the behavior of prior versions.
2020-03-26 13:30:35 -06:00
Ryan Ernst 960d1fb578
Revert "Introduce system index APIs for Kibana ()" ()
This reverts commit c610e0893d.

backport of 
2020-03-23 10:29:35 -07:00
Jake Landis db3420d757
[7.x] Optimize which Rest resources are used by the Rest tests… ()
This should help with Gradle's incremental compile such that projects
only depend upon the resources they use.

related 
2020-03-19 12:28:59 -05:00
Jay Modi c610e0893d
Introduce system index APIs for Kibana ()
This commit introduces a module for Kibana that exposes REST APIs that
will be used by Kibana for access to its system indices. These APIs are wrapped
versions of the existing REST endpoints. A new setting is also introduced since
the Kibana system indices' names are allowed to be changed by a user in case
multiple instances of Kibana use the same instance of Elasticsearch.

Additionally, the ThreadContext has been extended to indicate that the use of
system indices may be allowed in a request. This will be built upon in the future
for the protection of system indices.

Backport of 
2020-03-03 14:11:36 -07:00
Jake Landis 8d311297ca
[7.x] Smarter copying of the rest specs and tests () ()
* Smarter copying of the rest specs and tests ()

This PR addresses the unnecessary copying of the rest specs and allows
for better semantics for which specs and tests are copied. By default
the rest specs will get copied if the project applies
`elasticsearch.standalone-rest-test` or `esplugin` and the project
has rest tests or you configure the custom extension `restResources`.

This PR also removes the need for dozens of places where the x-pack
specs were copied by supporting copying of the x-pack rest specs too.

The plugin/task introduced here can also copy the rest tests to the
local project through a similar configuration.

The new plugin/task allows a user to minimize the surface area of
which rest specs are copied. Per project can be configured to include
only a subset of the specs (or tests). Configuring a project to only
copy the specs when actually needed should help with build cache hit
rates since we can better define what is actually in use.
However, project level optimizations for build cache hit rates are
not included with this PR.

Also, with this PR you can no longer use the includePackaged flag on
integTest task.

The following items are included in this PR:
* new plugin: `elasticsearch.rest-resources`
* new tasks: CopyRestApiTask and CopyRestTestsTask - performs the copy
* new extension 'restResources'
```
restResources {
  restApi {
    includeCore 'foo' , 'bar' //will include the core specs that start with foo and bar
    includeXpack 'baz' //will include x-pack specs that start with baz
  }
  restTests {
    includeCore 'foo', 'bar' //will include the core tests that start with foo and bar
    includeXpack 'baz' //will include the x-pack tests that start with baz
  }
}

```
2020-02-26 08:13:41 -06:00
Jay Modi f3f6ff97ee
Single instance of the IndexNameExpressionResolver ()
This commit modifies the codebase so that our production code uses a
single instance of the IndexNameExpressionResolver class. This change
is being made in preparation for allowing name expression resolution
to be augmented by a plugin.

In order to remove some instances of IndexNameExpressionResolver, the
single instance is added as a parameter of Plugin#createComponents and
PersistentTaskPlugin#getPersistentTasksExecutor.

Backport of 
2020-02-21 07:50:02 -07:00
Martijn van Groningen d17ecb5936
Change the delete policy api to not pass wildcard expressions to the delete index api ()
Backport from 

Don't rely on the delete index api to resolve all the enrich indices for a particular enrich policy using a '[policy_name]-*' wildcard expression. With this change, the delete policy api will resolve the indices to remove and pass that directly to the delete index api.

This resolves a bug, that if `action.destructive_requires_name` setting has been set to true then the delete policy api is unable to remove the enrich indices related to the policy being deleted.

Closes  

Co-authored-by: bellengao <gbl_long@163.com>
2020-02-18 10:53:39 +01:00
Jay Modi 3edadfefd0 RestHandlers declare handled routes ()
This commit changes how RestHandlers are registered with the
RestController so that a RestHandler no longer needs to register itself
with the RestController. Instead the RestHandler interface has new
methods which when called provide information about the routes
(method and path combinations) that are handled by the handler
including any deprecated and/or replaced combinations.

This change also makes the publication of RestHandlers safe since they
no longer publish a reference to themselves within their constructors.

Closes 

Co-authored-by: Jason Tedor <jason@tedor.me>

Backport of 
2020-02-09 22:48:32 -07:00
Rory Hunter d8bd736f8a
Formatting: keep simple if / else on the same line ()
Backport of .

Previous the formatter was breaking simple if/else statements (i.e.
without braces) onto separate lines, which could be fragile because the
formatter cannot also introduce braces. Instead, keep such expressions
on the same line.
2020-01-29 10:42:04 +00:00
Gordon Brown 89c2834b24
Deprecate creation of dot-prefixed index names except for hidden and system indices ()
This commit deprecates the creation of dot-prefixed index names (e.g.
.watches) unless they are either 1) a hidden index, or 2) registered by
a plugin that extends SystemIndexPlugin. This is the first step
towards more thorough protections for system indices.

This commit also modifies several plugins which use dot-prefixed indices
to register indices they own as system indices, and adds a plugin to
register .tasks as a system index.
2020-01-28 10:01:16 -07:00
Rory Hunter 80d925e225
Auto-format buildSrc ()
Backport / reimplementation of  on 7.x.

Opt-in `buildSrc` for automatic formatting. This required a config tweak
in order to pick up all the Java sources, and as a result more files are
now found in the Enrich plugin, that were previously missed.

I also moved the 2 Java files in `buildSrc/src/main/groovy` into the Java
directory, which required some follow-up changes.
2020-01-16 10:26:27 +00:00
Rory Hunter 2f069d8f3f
Tweak formatter config for long generic lines ()
Backport of . The current formatting config allows some long
generic declarations to break the 140 character limit. Tweak the config
to wrap such lines.
2020-01-15 13:17:37 +00:00
Martijn van Groningen e76c3d4d32
Tidy up enrich processors: ()
* Fix generics usages.
* Sealed match processor class.
2020-01-15 08:51:22 +01:00
Martijn van Groningen 2079f1cbeb
Backport: Fix ingest simulate response document order if processor executes async ()
Backport  to 7.x branch.

If a processor executes asynchronously and the ingest simulate api simulates with
multiple documents then the order of the documents in the response may not match
the order of the documents in the request.

Alexander Reelsen discovered this issue with the enrich processor with the following reproduction:

```
PUT cities/_doc/munich
{"zip":"80331","city":"Munich"}

PUT cities/_doc/berlin
{"zip":"10965","city":"Berlin"}

PUT /_enrich/policy/zip-policy
{
  "match": {
    "indices": "cities",
    "match_field": "zip",
    "enrich_fields": [ "city" ]
  }
}

POST /_enrich/policy/zip-policy/_execute

GET _cat/indices/.enrich-*

POST /_ingest/pipeline/_simulate
{
  "pipeline": {
  "processors" : [
    {
      "enrich" : {
        "policy_name": "zip-policy",
        "field" : "zip",
        "target_field": "city",
        "max_matches": "1"
      }
    }
  ]
  },
  "docs": [
    { "_id": "first", "_source" : { "zip" : "80331" } } ,
    { "_id": "second", "_source" : { "zip" : "50667" } }
  ]
}
```

* fixed test compile error
2019-12-17 12:27:07 +01:00
Martijn van Groningen 09c4269097
Add templating support to enrich processor ()
Adds support for templating to `field` and `target_field` options.
2019-11-27 08:53:11 +01:00
James Baiera 6bb6adb8d3
Reuse collected cluster state in EnrichPolicyRunner () ()
The cluster state is obtained twice in the EnrichPolicyRunner when updating 
the final alias. There is a possibility for the state to be slightly different 
between those two calls. This PR just has the function get the cluster state 
once and reuse it for the life of the function call.
2019-11-14 14:14:39 -05:00
Rory Hunter c46a0e8708
Apply 2-space indent to all gradle scripts ()
Backport of . Update `.editorconfig` to make the Java settings the
default for all files, and then apply a 2-space indent to all `*.gradle`
files. Then reformat all the files.
2019-11-14 11:01:23 +00:00
Martijn van Groningen 18d5d73305
Enable spotless for enrich gradle project in 7 dot x branch. ()
Backport of 

The enrich project doesn't have much history as all the other gradle projects,
so it makes sense to enable spotless for this gradle project.
2019-11-12 13:22:34 +01:00
Martijn van Groningen a1dd830cb5
Re-enabled test with longer timeout waiting for monitoring.
See 
2019-11-11 16:07:50 +01:00
Martijn van Groningen c358ecb5fb
Don't preserve indices between enrich qa tests.
This was added because it was suspected to cause the monitoring
enrich verification to fail, but that is not the case.

See 
2019-10-31 14:23:56 +01:00
Martijn van Groningen 05324b7f03
Muted verifying monitoring integration in enrich integration test.
Relates to 
2019-10-24 08:39:53 +02:00
Martijn van Groningen bbe50eca72
Fail with a better error when if there are no ingest nodes ()
when executing enrich execute policy api.
2019-10-22 07:42:04 +02:00
Martijn van Groningen 0ec0ab64c9
Fix executing enrich policies stats ()
The enrich stats api picked the wrong task to be displayed
in the executing stats section.

In case `wait_for_completion` was set to `false` then no task
was being displayed and if that param was set to `true` then
the wrong task was being displayed (transport action task instead
of enrich policy executor task).

Testing executing policies in enrich stats api is tricky.
I have verified locally that this commit fixes the bug.
2019-10-22 07:41:56 +02:00
Martijn van Groningen c09b62d5bf
Backport: also validate source index at put enrich policy time ()
Backport of: 

This changes tests to create a valid
source index prior to creating the enrich policy.
2019-10-22 07:38:16 +02:00
James Baiera 0d12ef8958
Add Enrich Origin () ()
This PR adds an origin for the Enrich feature, and modifies the background 
maintenance task to use the origin when executing client operations. 
Without this fix, the maintenance task fails to execute when security is 
enabled.
2019-10-21 16:40:49 -04:00
Martijn van Groningen 844825a13f
Validate policy type when storing an enrich policy () 2019-10-18 16:26:48 +02:00
Martijn van Groningen a5fe69c344
Include enrich into the info api as feature ()
This commit also fixes a bug, the enrich enabled setting
was not included in the list of settings.

Backport of 
2019-10-17 09:51:32 +02:00
Martijn van Groningen 77164e9017
adjusted minimal supported version 2019-10-15 07:45:00 +02:00
Martijn van Groningen 51c33f3edf
remove eclipse conditional 2019-10-15 07:18:32 +02:00
Martijn van Groningen c4b1a3045a
Fixed test, take into account that Map can be the result if max_matches is 1. 2019-10-15 07:03:01 +02:00
James Baiera 18d7e32b7d Add wait for completion for Enrich policy execution ()
This PR adds the ability to run the enrich policy execution task in the background,
returning a task id instead of waiting for the completed operation.
2019-10-14 16:05:28 -04:00
Martijn van Groningen 7fc9198d46
Change how `max_matches` affects `target_field` option. ()
Prior to this change the `target_field` would always be a json array
field in the document being ingested. This to take into account that
multiple enrich documents could be inserted into the `target_field`.

However the default `max_matches` is `1`. Meaning that by default
only a single enrich document would be added to `target_field` json
array field.

This commit changes this; if `max_matches` is set to `1` then the single
document would be added as a json object to the `target_field` and
if it is configured to a higher value then the enrich documents will be
added as a json array (even if a single enrich document happens to be
enriched).
2019-10-14 21:09:48 +02:00
James Baiera 73263c654a Add basic task support for executing enrich policies ()
Changes the execution logic to create a new task using the execute request,
and attaches the new task to the policy runner to be updated. Also, a new
response is now returned from the execute api, which contains either the task
id of the execution, or the completed status of the run. The fields are mutually
exclusive to make it easier to discern what type of response it is.
2019-10-11 13:32:06 -04:00
Martijn van Groningen aace42d38d
Add HLRC support for enrich stats API ()
This PR also includes HLRC docs for the enrich stats api.

Relates to 
2019-10-10 09:08:29 +02:00
Martijn van Groningen 19393fc5a7
match processor should handler values other than string properly ()
Currently if the document being ingested contains another field value
than a string then the processor fails with an error.

This commit changes the match processor to handle number values
and array values correctly.

If a json array is detected then the `terms` query is used instead
of the `term` query.
2019-10-10 08:49:17 +02:00
Martijn van Groningen f8ebb75fcf
Reuse OperationRouting#searchShards(...) to select local enrich shard ()
The currently logic shard selecting logic selects a random shard copy
instead of selecting the local shard copy and if local copy is not
available then selecting a random shard copy. The latter is desired
behaviour for enrich.

By reusing `OperationRouting#searchShards(...)` we get the desired
behaviour and reuse the same logic that the search api is using.
2019-10-09 17:31:43 +02:00
Martijn van Groningen be0e17770c
required change after merging in 7 dot x branch 2019-10-09 09:16:23 +02:00
James Baiera b9fb354618 Add retry to force merge operation in EnrichPolicyRunner ()
Adds a check when running an Enrich policy to make sure that an Enrich index
is force merged down to one segment, and if it was not fully merged, attempts
the merge again, up to a configurable number of times.
2019-10-08 11:23:02 -04:00
Martijn van Groningen 8b7100eb1f
Don't remove indices to avoid monitoring from intermittently failing
to index monitoring docs.
2019-10-08 17:10:42 +02:00
Tal Levy a17f394e27
Geo-Match Enrich Processor () ()
this commit introduces a geo-match enrich processor that looks up a specific
`geo_point` field in the enrich-index for all entries that have a geo_shape match field
that meets some specific relation criteria with the input field.

For example, the enrich index may contain documents with zipcodes and their respective
geo_shape. Ingesting documents with a geo_point field can be enriched with which zipcode
they associate according to which shape they are contained within.

this commit also refactors some of the MatchProcessor by moving a lot of the shared code to
AbstractEnrichProcessor.

Closes .
2019-10-07 15:03:46 -07:00
James Baiera a66c0dcd95 Add pipeline to ensure unique Enrich index documents ()
Adds a pipeline that removes ids and routing from documents before indexing
them into enrich indices. Enrich documents may come from multiple indices,
and thus have id collisions on them. This pipeline ensures that documents
with colliding id fields do not clobber one another during the reindex operation
while executing an enrich policy.
2019-10-04 12:20:52 -04:00
Michael Basnight 0e1b77568a Add enable checks to missing enrich plugin methods ()
Some of the server side objects that do not need to be created unless
enrich is enabled were still being created. This commit fixes that.
2019-10-01 12:04:46 -05:00