Commit Graph

47301 Commits

Author SHA1 Message Date
James Rodewig e72bda1703 [DOCS] Reformats cat nodes API (#45285) 2019-08-08 08:38:02 -04:00
James Rodewig b806e6edde [DOCS] Reformats cat pending tasks API (#45287) 2019-08-08 08:32:04 -04:00
István Zoltán Szabó 276e9c6697 [DOCS] Adds supported time units ref to the ML and DF API params. (#45322) 2019-08-08 14:26:19 +02:00
Martijn van Groningen e066133016
Change the ingest simulate api to not include dropped documents (#44161)
If documents are dropped by the `drop` processor then
these documents are returned as a `null` value in the response.

=== Example

Create pipeline:

```
PUT _ingest/pipeline/droppipeline
{
    "processors": [
        {
            "set": {
                "field": "bla",
                "value": "val"
            }
        },
        {
            "drop": {}
        }
    ]
}
```

Simulate request:

POST _ingest/pipeline/droppipeline/_simulate
{
    "docs": [
        {
            "_source": {
                "message": "text"
            }
        }
    ]
}

Response:

```
{
    "docs": [
        null
    ]
}
```

Response if verbose is enabled:

```
{
    "docs": [
        {
            "processor_results": [
                {
                    "doc": {
                        "_index": "_index",
                        "_type": "_doc",
                        "_id": "_id",
                        "_source": {
                            "message": "text",
                            "bla": "val"
                        },
                        "_ingest": {
                            "timestamp": "2019-07-10T11:07:10.758315Z"
                        }
                    }
                },
                null
            ]
        }
    ]
}
```

Closes #36150

* Abort pipeline simulation in verbose mode when document has been dropped
by drop processor
2019-08-08 13:04:33 +02:00
Ioannis Kakavas 99ddb8b3d8 Allow empty token endpoint for implicit flow (#45038)
When using the implicit flow in OpenID Connect, the
op.token_endpoint_url should not be mandatory as there is no need
to contact the token endpoint of the OP.
2019-08-08 12:50:53 +03:00
David Turner ddcc38cf1c
More read-only-allow-delete docs (#45320)
Adds to the `index.blocks.read_only_allow_delete` docs the information that
this block may be added or removed automatically, and rewords the
breaking-changes docs to mention the blocks explicitly and to recommend using a
different block.

Relates #42559
2019-08-08 09:58:23 +01:00
István Zoltán Szabó 9f62c04637 [DOCS] Reformats cluster health and cluster state APIs (#45206)
Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2019-08-08 10:25:05 +02:00
Martijn van Groningen fb959d188c
Backport: Add description to force-merge tasks (#41365) (#45191)
* Add description to force-merge tasks (#41365)

This is static information that is part of the force merge request.

Relates to #15975
2019-08-08 08:15:09 +02:00
Mark Vieira fca458f1c8
Use system properties for build cache configuration (#45295) 2019-08-07 13:14:54 -07:00
Gordon Brown e3599fded7
Add warning about versions to Deprecation API docs (#36624)
Add a note that the Deprecation API may not be up to date with all
breaking changes until the last minor version in a major version series.
2019-08-07 14:11:05 -06:00
Michael Basnight 89861d0884 Add ingest processor existence helper method (#45156)
This commit adds a helper method to the ingest service allowing it to
inspect a pipeline by id and verify the existence of a processor in the
pipeline. This work exposed a potential bug in that some processors
contain inner processors that are passed in at instantiation. These
processors needed a common way to expose their inner processors, so the
WrappingProcessor was created in order to expose the inner processor.
2019-08-07 11:19:04 -05:00
Mark Vieira 341ab48ec0
Improve SCM info in build scans (#45264) 2019-08-07 09:06:11 -07:00
Benjamin Trent 5db9982f71
[7.x] [ML][Data Frame] Add update transform api endpoint (#45154) (#45279)
* [ML][Data Frame] Add update transform api endpoint (#45154)

This adds the ability to `_update` stored data frame transforms. All mutable fields are applied when the next checkpoint starts. The exception being `description`.

This PR contains all that is necessary for this addition:
* HLRC
* Docs
* Server side
2019-08-07 10:37:35 -05:00
Benjamin Trent 3a71b91dca
[ML][Data Frame] add support for geo_bounds aggregation (#44441) (#45281)
This adds support for `geo_bounds` aggregation inside the `pivot.aggregations` configuration. 

The two points returned from the `geo_bounds` aggregation are transformed into `geo_shape` whose types are dynamic given the point's similarity.

* `point` if the two points are identical
* `linestring` if the two points share either a latitude or longitude 
* `polygon` if the two points are completely different

The automatically deduced mapping for the resulting field is a `geo_shape`.
2019-08-07 10:37:09 -05:00
István Zoltán Szabó 95d3a8e8ad [DOCS] Reformats cluster stats API and expands common params (#45270)
Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2019-08-07 16:49:58 +02:00
Lee Hinman c7ec0b8431 Include in-progress snapshot for a policy with get SLM policy… (#45245)
This commit adds the "in_progress" key to the SLM get policy API,
returning a policy that looks like:

```json
{
  "daily-snapshots" : {
    "version" : 1,
    "modified_date" : "2019-08-05T18:41:48.778Z",
    "modified_date_millis" : 1565030508778,
    "policy" : {
      "name" : "<production-snap-{now/d}>",
      "schedule" : "0 30 1 * * ?",
      "repository" : "repo",
      "config" : {
        "indices" : [
          "foo-*",
          "important"
        ],
        "ignore_unavailable" : true,
        "include_global_state" : false
      },
      "retention" : {
        "expire_after" : "10m"
      }
    },
    "last_success" : {
      "snapshot_name" : "production-snap-2019.08.05-oxctmnobqye3luim4uejhg",
      "time_string" : "2019-08-05T18:42:23.257Z",
      "time" : 1565030543257
    },
    "next_execution" : "2019-08-06T01:30:00.000Z",
    "next_execution_millis" : 1565055000000,
    "in_progress" : {
      "name" : "production-snap-2019.08.05-oxctmnobqye3luim4uejhg",
      "uuid" : "t8Idqt6JQxiZrzp0Vt7z6g",
      "state" : "STARTED",
      "start_time" : "2019-08-05T18:42:22.998Z",
      "start_time_millis" : 1565030542998
    }
  }
}
```

These are only visible while the snapshot is being taken (or failed),
since it reads from the cluster state rather than from the repository
itself.
2019-08-07 08:29:49 -06:00
István Zoltán Szabó 9384774b4c [DOCS] Adds supported time units ref to the frequency and delay params. (#45283) 2019-08-07 16:19:59 +02:00
Alpar Torok 0ea00e4861 Change how we pick bwc versions to check out (#45189)
Prior to this PR we always checked out the latest bwc branches and had
an external mechanism to store the bwc versions used for every CI run so
we could both reproduce those builds and run additional tests using the
same combination.

This adds complexities in setting up and maintaining CI and makes it
difficult to set up multi jobs.

This change replaces that mechanism with a time based approach
that looks at the commit date of the current revision and picks the
newest on the bwc branch that's still older than that.
It also makes sure there are no merge commits in this interval.

This new behavior will is ment to be enabled in CI only, for everything
except PR checks that will still use last available bwc revision.
2019-08-07 16:44:38 +03:00
James Rodewig 46fc989ca2 [DOCS] Reformats cat nodeattrs API (#45255) 2019-08-07 09:31:37 -04:00
James Rodewig 5ade756275 [DOCS] Reformats cat indices API (#45239) 2019-08-07 09:08:35 -04:00
James Rodewig d8423f0a80 [DOCS] Reformat cat master API (#45240) 2019-08-07 09:04:04 -04:00
Bas ten Berge 15d41a7508 [DOCS] Remove outdated 5.x community plugins (#40306)
Our docs previously included several community plugins that are only supported for versions 5.x and earlier. This removes those plugins for our 6.6+ docs.
2019-08-07 08:45:05 -04:00
Benjamin Trent be911e6a53
[ML][Data Frames] Fix null aggregation handling in indexer (#45061) (#45257)
* [ML][Data Frames] Fix null aggregation handling in indexer

* addressing PR comments

* adjusting error messages
2019-08-07 07:01:13 -05:00
Bukhtawar cd304c4def Auto-release flood-stage write block (#42559)
If a node exceeds the flood-stage disk watermark then we add a block to all of
its indices to prevent further writes as a last-ditch attempt to prevent the
node completely exhausting its disk space. However today this block remains in
place until manually removed, and this block is a source of confusion for users
who current have ample disk space and did not even realise they nearly ran out
at some point in the past.

This commit changes our behaviour to automatically remove this block when a
node drops below the high watermark again. The expectation is that the high
watermark is some distance below the flood-stage watermark and therefore the
disk space problem is truly resolved.

Fixes #39334
2019-08-07 11:03:53 +01:00
Tanguy Leroux a869342910 Restore DefaultShardOperationFailedException's reason after deserialization (#45203)
The reason field of DefaultShardOperationFailedException is lost during serialization. 
This is sad because this field is checked for nullity during xcontent generation and it 
means that the cause won't be included in the generated xcontent and won't be 
printed in two REST API responses (Close Index API and Indices Shard Stores API).

This commit simply restores the reason from the cause during deserialization.
2019-08-07 10:37:15 +02:00
Jason Tedor bd59ee6c72
Fix clock used in update requests (#45262)
We accidentally switched to using the relative time provider here. This
commit fixes this by switching to the appropriate absolute clock.
2019-08-06 21:15:21 -04:00
Julie Tibshirani 9318192578 Correct a code snippet in removal_of_types. (#45118)
Previously, the reindex examples did not include `_doc` as the destination type.
This would result in the reindex failing with the error "Rejecting mapping
update to [users] as the final mapping would have more than 1 type: [_doc,
user]".

Relates to #43100.
2019-08-06 14:09:21 -07:00
Tom Callahan a7a419bee8 Change Ldap SDK License to LGPL-2.1 (#45116)
We currently use the unboundid ldap SDK, which is triply licensed under
GPL-2.0, LGPL-2.1, and the "UnboundID LDAP SDK Free Use License". We currently
identify the license as the latter, but LGPL-2.1 is the one we should be using
per our policy.
2019-08-06 16:48:09 -04:00
Lisa Cawley 7c9c9a9cc4 [DOCS] Reformats ML update APIs (#45253) 2019-08-06 11:16:29 -07:00
James Rodewig 51c1abc112 [DOCS] Reformat match phrase prefix query (#45209) 2019-08-06 14:02:07 -04:00
Jack Conradson fc8a6fc9d0 Decouple Painless AST Lambda generation from the grammar (#45111)
This is the first step in decoupling the Painless AST from the grammar. The
Painless AST should be able to generate classes independently of how the
AST is generated from a grammar. (If I were to build a Painless AST by hand
in code this should be all that's necessary.) This change removes Lambda
name generation from the ANTLR grammar tree walker. It also removes
unnecessary node generation of new array function references from the
tree walker as well.
2019-08-06 10:08:19 -07:00
Jason Tedor 9a142ff25c
Introduce formal node ML role (#45174)
This commit builds on the ability for plugins to introduce new roles to
add a formal node ML role.
2019-08-06 13:00:05 -04:00
Lisa Cawley bb22b2e45f [DOCS] Reformats get rollup jobs API (#45114) 2019-08-06 09:40:18 -07:00
David Turner f5d1381e01 Remove always-true param from IndicesService#stats (#45231)
Parameter `includePrevious` is always true, so this commit inlines it.
2019-08-06 17:22:11 +01:00
David Turner 355713b9ca
Improve slow logging in MasterService (#45241)
Adds a tighter threshold for logging a warning about slowness in the
`MasterService` instead of relying on the cluster service's 30-second warning
threshold. This new threshold applies to the computation of the cluster state
update in isolation, so we get a warning if computing a new cluster state
update takes longer than 10 seconds even if it is subsequently applied quickly.
It also applies independently to the length of time it takes to notify the
cluster state tasks on completion of publication, in case any of these
notifications holds up the master thread for too long.

Relates #45007
Backport of #45086
2019-08-06 17:01:49 +01:00
Mark Vieira bb7f46da62
Avoid building docker images when running precommit task (#45211) 2019-08-06 09:01:06 -07:00
Zachary Tong 422aca9a5d Fix Rollup job creation to work with templates (#43943)
The PutJob API accidentally used an "expert" API of CreateIndexRequest.
That API is semi-lenient to syntax; a type could be omitted and the
request would work as expected.  But if a type was omitted it would
not merge with templates correctly, leading to index creation that
only has the template and not the requested mappings in the request.

This commit refactors the PutJob API to:

- Include the type name
- Use a less "expert" API in an attempt to future proof against errors
- Uses an XContentBuilder instead of string replacing, removes json template
2019-08-06 10:53:44 -04:00
James Rodewig b09b874801 [DOCS] Reformat cat fielddata API (#45202) 2019-08-06 10:33:51 -04:00
Armin Braun 5d7fafec14
Add Assertion to Ensure Retries in S3BlobContainer (#45224) (#45230)
* We need a `markSupported` input stream to retry uploads
* Relates #45153
2019-08-06 16:11:19 +02:00
Tanguy Leroux 772ce1f599
Add deprecation warning for Force Merge API (#44903)
This commit adds a deprecation warning in 7.x for the Force Merge API 
when both only_expunge_deletes and max_num_segments are set in a request.

Relates #44761
2019-08-06 16:04:24 +02:00
Alexander Reelsen 210593d8e5 Add back lowercase processor in docs (#45090)
This got lost in a refactoring in 9137d92ca6
2019-08-06 09:23:13 -04:00
James Rodewig 1c96e8a6af [DOCS] Reformat cat health API (#45218) 2019-08-06 08:41:18 -04:00
James Rodewig 2489508e1f [DOCS] Reformat cat count API (#45160) 2019-08-06 08:37:09 -04:00
Alpar Torok bc60eebfc8 Add retries to vault access in init scripts 2019-08-06 15:39:36 +03:00
Yannick Welsch 17846212bd Fix tests after backport of #44055 2019-08-06 14:19:20 +02:00
Alpar Torok b3076adae6 Fix reaper build failures on Windows (#45205)
We configure the service ID as the node's toString but this containes
characters that Windows doesn't like.
This PR fixes it by allowing only alphanumeric characters
2019-08-06 14:36:36 +03:00
Yannick Welsch a453cd489e Run testExtendedSocketOptions only on JDK11+ (#44055)
This functionality only works on JDK 11 or higher
2019-08-06 13:15:17 +02:00
Jason Tedor 5b1b146099
Normalize environment paths (#45179)
This commit applies a normalization process to environment paths, both
in how they are stored internally, also their settings values. This
normalization is done via two means:
 - we make the paths absolute
 - we remove redundant name elements from the path (what Java calls
   "normalization")

This change ensures that when we compare and refer to these paths within
the system, we are using a common ground. For example, prior to the
change if the data path was relative, we would not compare it correctly
to paths from disk usage. This is because the paths in disk usage were
being made absolute.
2019-08-06 06:04:30 -04:00
Yannick Welsch 7aeb2fe73c Add per-socket keepalive options (#44055)
Uses JDK 11's per-socket configuration of TCP keepalive (supported on Linux and Mac), see
https://bugs.openjdk.java.net/browse/JDK-8194298, and exposes these as transport settings.
By default, these options are disabled for now (i.e. fall-back to OS behavior), but we would like
to explore whether we can enable them by default, in particular to force keepalive configurations
that are better tuned for running ES.
2019-08-06 10:45:44 +02:00
Alpar Torok 81ea08b8ca Update ci bwc versions 2019-08-06 11:45:28 +03:00