Commit Graph

42773 Commits

Author SHA1 Message Date
Ioannis Kakavas 580b5baf21
Add realm information for Authenticate API (#35648)
- Add the authentication realm and lookup realm name and type in the response for the _authenticate API
- The authentication realm is set as the lookup realm too (instead of setting the lookup realm to null or empty ) when no lookup realm is used.
2018-11-27 23:35:42 +02:00
Tal Levy fe603e9163
[ILM] add HLRC docs to remove-policy-from-index (#35759)
This primarily introduces documentation for the HLRC
remove-policy-from-index action.
2018-11-27 13:28:58 -08:00
Zachary Tong 1a1092d08f [Rollup] Update serialization version after backport 2018-11-27 16:02:48 -05:00
Zachary Tong 48fa251812
[Rollup] Add more diagnostic stats to job (#35471)
* [Rollup] Add more diagnostic stats to job

To help debug future performance issues, this adds the
 min/max/avg/count/total latencies (in milliseconds) for search
and bulk phase.  This latency is the total service time including
transfer between nodes, not just the `took` time.

It also adds the count of search/bulk failures encountered during
runtime.  This information is also in the log, but a runtime counter
will help expose problems faster

* review cleanup

* Remove dead ParseFields
2018-11-27 15:46:10 -05:00
Marios Trivyzas af9233b067
Build: Fix gradle build for Mac OS (#35968)
Remove trailing line feed from the output of the command to get
the number of CPUs.

Follow up to: #35789
2018-11-27 21:31:43 +01:00
Jeff Hajewski 49087f16f5 Adds deprecation logging to ScriptDocValues#getValues. (#34279)
`ScriptDocValues#getValues` was added for backwards compatibility but no
longer needed. Scripts using the syntax `doc['foo'].values` when
`doc['foo']` is a list should be using `doc['foo']` instead.

Closes #22919
2018-11-27 14:30:13 -05:00
Chris Earle 3337fa7351
[Monitoring] Make Exporters Async (#35765)
This changes the exporter code -- most notably the `http` exporter --
to use async operations throughout the resource management and bulk
initialization code (the bulk indexing of monitoring documents was
already async).

As part of this change, this does change one semi-core aspect of the
`HttpResource` class in that it will no longer block all concurrent calls
until the first call completes with
`HttpResource::checkAndPublishIfDirty`.
Now, any parallel attempts to check the resources will be skipped until
the first call completes (success or failure). While this is a technical
change, it has very little practical impact because the existing behavior
was either quick success (then every blocked request processed) or
each request timed out and failed anyway, thus being effectively
skipped (and a burden on the system).
2018-11-27 14:28:14 -05:00
Tal Levy c2329cdf1d
[ILM] reduce time restriction on IndexLifecycleExplainResponse (#35954)
step times were set. The assumption was that these are always set.
Tests passed, which led me to believe this was true. There is a time
when shrunk indices have their step phase/action/step details set,
but with no time information (in the CopyExecutionStateStep).
Explain API fails for these
2018-11-27 11:03:47 -08:00
Jay Modi 2061eeb122
Remove use of AbstractComponent in xpack (#35394)
This commit removes the use of AbstractComponent in xpack where it was
still being extended. It has been replaced with explicit logger
declarations.

See #34488
2018-11-27 11:28:26 -07:00
Julie Tibshirani 25c416b12d
Deprecate types in search and multi search templates. (#35669)
This PR adds deprecation warnings to the relevant `Rest*Action` classes, plus tests in `Rest*ActionTests`. No updates to REST tests, the Java HLRC, or documentation were necessary, since they didn't make use of types.
2018-11-27 10:19:19 -08:00
Luca Cavanna 7e772e311d
Remove fromXContent from IndexUpgradeInfoResponse (#35934)
Such method is there only for testing purposes, it is not needed.
2018-11-27 19:04:44 +01:00
Simon Willnauer ad1f0dccd4
Validate metdata on `_msearch` (#35938)
MultiSearchRequests issues through `_msearch` now validate all keys
in the metadata section. Previously unknown keys were ignored
while now an exception is thrown.

Closes #35869
2018-11-27 17:08:24 +01:00
Tim Brooks cc1fa799c8
Remove `TcpChannel#setSoLinger` method (#35924)
This commit removes the dedicated `setSoLinger` method. This simplifies
the `TcpChannel` interface. This method has very little effect as the
SO_LINGER is not set prior to the channels being closed in the abstract
transport test case. We still will set SO_LINGER on the
`MockNioTransport`. However we can do this manually.
2018-11-27 09:08:14 -07:00
Tim Brooks b6ed6ef189
Add sni name to SSLEngine in nio transport (#35920)
This commit is related to #32517. It allows an "sni_server_name"
attribute on a DiscoveryNode to be propagated to the server using
the TLS SNI extentsion. Prior to this commit, this functionality
was only support for the netty transport. This commit adds this
functionality to the security nio transport.
2018-11-27 09:06:52 -07:00
Christophe Bismuth adc0b560c0 Raise a 404 exception when document source is not found (#33384) (#34083)
This pull request makes the `RestGetSourceAction` return a `ResourceNotFoundException` with a proper JSON response when source or document itself is missing (see issue #33384).

Here is below a sample JSON output:

```
{
  "error": {
    "root_cause": [
      {
        "type": "resource_not_found_exception",
        "reason": "Source not found [index1]/[_doc]/[1]"
      }
    ],
    "type": "resource_not_found_exception",
    "reason": "Source not found [index1]/[_doc]/[1]"
  },
  "status": 404
}
```
2018-11-27 10:35:45 -05:00
Tal Levy df341aba4a
[ILM] make IndexLifecycleExplainResponse more resilient to null values (#35800)
added validation for complete information of step details.

also changed the rendering of explain responses so null strings are not rendered

Another thing that I changed is the format of the client-side response. I found it difficult to maintain the two subtly-different objects, so I migrated the usage of long for the fields, to Long (just as it is on the server-side).
2018-11-27 07:29:59 -08:00
Alexander Reelsen a615ab454a
Watcher: Only trigger a watch if new or schedule/changed (#35908)
The trigger engine did always create a new schedule data structure, when
the watcher indexing listener called an add. However the indexing
listener also called add, when the watch status was updated. This means,
that upon a watch status update the watch got retriggered, potentially
waiting a defined interval from the watch status update onwards, instead
of waiting from the last run.

This commit only updates the schedule in the trigger engine, if it
actually has changed, otherwise the existing schedule will not be
touched. This has two results

1. If a watch is updated by an execution, the existing interval will not
be touched (meaning the scheduled time will not move forward).
2. If a watch is updated by a user, but the schedule is not changed, it
will not be reset from the update (for example starting to count from 5
minutes again, if the interval was set to 5 minutes).

Furthermore some minor cleanups were applied, making variables final in
the ctor, preventing double creation of variables.
2018-11-27 16:15:37 +01:00
Nik Everett 38725bd3a1
Build: Pick default test jvms for macOS (#35789)
In #35259 we switched the default number of VMs to fork for unit tests to
the number of physical CPU cores. But because we could only get an accurate
count on machines with a normal `/proc` filesystem, macOS machine did not
pick up the new default. Given that macOS is a huge portion of developer
machines, we'd like to get the right default there. This does that.

It also moves the default-finding process from happening once per testing
task to happening once at startup. This seems like a good choice in general,
but a very good choice for macOS because we have to run a command to list
the count.
2018-11-27 09:59:44 -05:00
Alpar Torok 541869a96a
Add missing entries to conffiles (#35810) 2018-11-27 16:38:56 +02:00
Marios Trivyzas 1da9c6faa0
SQL: Fix translation of math functions to painless (#35910)
`SIGN` and `RADIANS` where wrongly overriding `mathFunction()`.
Converted `mathFunction()` to private in `MathFunction` since it
shouldn't be overriden, as it uses the assigned `MathOperation`
to get the funtion name for painless scripts.

Fixes: #35654
2018-11-27 15:33:43 +01:00
Tim Vernum 8d9e21ffaa Fix mutator in GetRolesRequestTests (#35932)
There was no guarantee that the mutated object would not be the same
as the original object (and was for seed 6A8C4CBF63B5AA63)

Resolves #35947
2018-11-27 16:10:14 +02:00
Nik Everett cb8646ac0e
Tests: If dpkg fails try and log failure (#35551)
If `dpkg` fails, try and look for who has `/var/lib/dpkg/lock` open. If
it exists and is open then return a failure with information about who
has file open. This should help us debug #33762.

Closes #34309
2018-11-27 09:06:10 -05:00
Marios Trivyzas c91ef1105d
SQL: Implement data type verification for conditionals (#35916)
Add special verifier rule to check that the arguments of conditional
functions are of the same or compatible types. This way the user gets
a descriptive error message with line number and column indicating
where is the offending argument.

Closes: #35907
2018-11-27 14:46:11 +01:00
David Roberts 110c4fdd65
[ML] Adjust file structure finder parser config (#35935) 2018-11-27 12:52:52 +00:00
Tim Vernum b6eb73a835
Update BWC checks in HasPrivilegesResponse (#35929)
This serialization has been backported to 6.x
2018-11-27 21:05:12 +11:00
Martijn van Groningen 4a3d66c668
[TEST] wrong bugUrl...
Relates to #35937
2018-11-27 10:53:04 +01:00
Martijn van Groningen fabffb89b8
Muted test. This test expose an issue inside the auto follower coordinator,
it has nothing to do with the hlrc support for put auto follow pattern api,
this test was added for.

Relates to #35480
2018-11-27 10:49:03 +01:00
Yogesh Gaikwad 31fdb76973
[DOCS] Document authorization_realms for Kerberos realm (#35927)
This commit adds documentation for authorization_realms
setting for the Kerberos realm and also corrects a typo in
existing documentation.

Co-authored-by: @A-Hall
2018-11-27 19:09:17 +11:00
Martijn van Groningen df1e02d0d5
[HLRC] Added support for CCR Put Auto Follow Pattern API (#35780)
This change also adds documentation for the Put Auto Follow Pattern API.

Relates to #33824
2018-11-27 08:53:22 +01:00
Martijn van Groningen 447e5d212a
Changed versions in serialization code after backporting #35535 2018-11-27 08:00:06 +01:00
Ioannis Kakavas 998c663f0d
Add Tests for findSamlRealm (#35905)
This commit adds a test for handling correctly all they possible 
`SamlPrepareAuthenticationRequest` parameter combinations that 
we might get from Kibana or a custom web application talking to the
SAML APIs. 
We can match the correct SAML realm based either on the realm name
or the ACS URL. If both are included in the request then both need to 
match the realm configuration.
2018-11-27 08:18:18 +02:00
Tim Vernum 5b427d415e
Add "request.id" to file audit logs (#35536)
This generates a synthesized "id" for each incoming request that is
included in the audit logs (file only).
This id can be used to correlate events for the same request (e.g.
authentication success with access granted).

This request.id is specific to the audit logs and is not used for any
other purpose

The request.id is consistent across nodes if a single request requires
execution on multiple nodes (e.g. search acros multiple shards).
2018-11-27 15:19:47 +11:00
Tim Vernum a18b219f79
Allow noop PutUser updates (#35843)
When assertions are enabled, a Put User action that have no effect (a
noop update) would trigger an assertion failure and shutdown the node.

This change accepts "noop" as an update result, and adds more
diagnostics to the assertion failure message.
2018-11-27 15:08:53 +11:00
Tim Vernum 3435fc4613
HLRC: Add ability to put user with a password hash (#35844)
Update PutUserRequest to support password_hash (see: #35242)

This also updates the documentation to bring it in line with our more
recent approach to HLRC docs.
2018-11-27 15:07:24 +11:00
Ryan Ernst 72d7f236a5
Scripting: Add back joda to whitelist (#35915)
Watcher still exposes some dates as joda DateTime objects. This commit
adds back joda to the painless whitelist so they can still be accessed.

closes #35913
2018-11-26 16:33:05 -08:00
Ryan Ernst 10447dd962
Build: Fix jdbc jar to include deps (#35602)
This commit adds back bundling of all deps of the sql jdbc jar. This was
lost in a refactoring of how the shadow plugin is handled for the entire
elasticsearch project.
2018-11-26 16:18:35 -08:00
Gordon Brown 119835decd
Always enforce cluster-wide shard limit (#34892)
This removes the option to run a cluster without enforcing the
cluster-wide shard limit, making strict enforcement the default and only
behavior.  The limit can still be adjusted as desired using the cluster
settings API.
2018-11-26 17:05:12 -07:00
Christoph Büscher 4d525e3e33
[Docs] Align callouts in search.asciidoc (#35897) 2018-11-27 00:05:15 +01:00
Gordon Brown 5c11b8612a
Add HLRC docs for Explain Lifecycle (#35803)
Adds HLRC documentation for the Explain Lifecycle API.
2018-11-26 15:25:56 -07:00
Sachin Araballi 5f656e4eab Fields are declared at the top of the class. (#35891)
Fields should be declared at the top of the class, before any method declarations, constructors, initializers or inner classes.
2018-11-26 22:05:28 +01:00
Tal Levy 8fe5f66637
[ILM] fix rest-api-spec doc links (#35478) 2018-11-26 12:26:36 -08:00
Igor Motov 663563f64b
Geo: better handling of malformed geo_points (#35554)
Improves handling of malformed geo_points when `ignore_malformed` is
set to true

Closes #35419
2018-11-26 09:44:42 -10:00
Jim Ferenczi 900caa20ef
Handles exists query in composite aggs (#35758)
This commit adds the support for exists query in the sorted execution mode
of the `composite` aggregation. We'll execute the aggregation from the sorted
points and use early termination if the main query is an `exists` query over the
first source of the `composite` aggregation.
2018-11-26 19:08:14 +01:00
Marios Trivyzas 3f7cae3f0d
SQL: Implement GREATEST and LEAST functions (#35879)
Add GREATEST(expr1, expr2, ... exprN) and LEAST(expr1, expr2, exprN)
functions which are in the family of CONDITIONAL functions.

Implementation follows PostgreSQL behaviour, so the functions return
`NULL` when all of their arguments evaluate to `NULL`.

Renamed `CoalescePipe` and `CoalesceProcessor` to `ConditionalPipe` and
`ConditionalProcessor` respectively, to be able to reuse them for
`Greatest` and `Least` evaluations. To achieve that `ConditionalOperation`
has been added to differentiate between the functionalities at execution
time.

Closes: #35878
2018-11-26 18:21:36 +01:00
Christophe Bismuth b95a4db6e6 Throw a parsing exception when boost is set in span_or query (#28390) (#34112) 2018-11-26 12:15:59 -05:00
John 0baffda390 ingest: grok remove duplicated patterns (#35886)
This commit removes the redundant (and incorrect) JAVACLASS
and JAVAFILE grok patterns. This helps to keep parity with 
Logstash's patterns. 

See also: https://github.com/logstash-plugins/logstash-patterns-core/pull/237
 
closes #35699
2018-11-26 11:13:46 -06:00
Jake Landis 4b99a663c1
ingest: fix broken doc link 2018-11-26 10:34:42 -06:00
Ed Savage 13e11966ca
[HLRC][ML] Add delete expired data API (#35906)
Relates to #29827
2018-11-26 16:15:54 +00:00
Jim Ferenczi 3ed3aab127
Fix rest test when numShards is greater than 1 (#35909) 2018-11-26 16:45:56 +01:00
Jake Landis 7f7b31723e
ingest: extended `if` documentation (#35044)
part of #33188
2018-11-26 09:35:45 -06:00