Commit Graph

40065 Commits

Author SHA1 Message Date
Colin Goodheart-Smithe cd89ca2685
Merge branch 'master' into index-lifecycle
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackSetti
ngs.java
/Users/colings86/dev/work/git/elasticsearch/.git/worktrees/elasticsearch
-ilm/MERGE_HEAD

buildSrc/src/main/groovy/org/elasticsearch/gradle/vagrant/VagrantTestPlu
gin.groovy
client/rest-high-level/src/main/java/org/elasticsearch/client/RestHighLe
velClient.java
client/rest-high-level/src/main/java/org/elasticsearch/client/XPackClien
t.java
client/rest-high-level/src/test/java/org/elasticsearch/client/PingAndInf
oIT.java
client/rest-high-level/src/test/java/org/elasticsearch/client/documentat
ion/MiscellaneousDocumentationIT.java
docs/reference/analysis/tokenfilters/synonym-graph-tokenfilter.asciidoc
docs/reference/analysis/tokenfilters/synonym-tokenfilter.asciidoc
modules/ingest-common/src/test/java/org/elasticsearch/ingest/common/Grok
ProcessorTests.java
modules/ingest-common/src/test/java/org/elasticsearch/ingest/common/Json
ProcessorTests.java
qa/vagrant/src/main/java/org/elasticsearch/packaging/test/ArchiveTestCas
e.java
qa/vagrant/src/main/java/org/elasticsearch/packaging/util/Archives.java
qa/vagrant/src/main/java/org/elasticsearch/packaging/util/Cleanup.java
qa/vagrant/src/main/java/org/elasticsearch/packaging/util/FileUtils.java
qa/vagrant/src/main/java/org/elasticsearch/packaging/util/Installation.j
ava
qa/vagrant/src/main/java/org/elasticsearch/packaging/util/Platforms.java
qa/vagrant/src/main/java/org/elasticsearch/packaging/util/ServerUtils.ja
va
qa/vagrant/src/main/java/org/elasticsearch/packaging/util/Shell.java
qa/vagrant/src/test/resources/packaging/tests/20_tar_package.bats
server/src/main/java/org/elasticsearch/index/analysis/ESSolrSynonymParse
r.java
server/src/main/java/org/elasticsearch/index/analysis/ESWordnetSynonymPa
rser.java
server/src/main/java/org/elasticsearch/index/analysis/SynonymGraphTokenF
ilterFactory.java
server/src/main/java/org/elasticsearch/index/analysis/SynonymTokenFilter
Factory.java
server/src/test/java/org/elasticsearch/index/analysis/ESSolrSynonymParse
rTests.java
server/src/test/java/org/elasticsearch/index/analysis/ESWordnetSynonymPa
rserTests.java
server/src/test/java/org/elasticsearch/index/shard/IndexShardIT.java
test/framework/src/main/java/org/elasticsearch/ingest/IngestDocumentMatc
her.java
test/framework/src/test/java/org/elasticsearch/ingest/IngestDocumentMatc
herTests.java
x-pack/plugin/core/src/main/java/org/elasticsearch/license/License.java
x-pack/plugin/core/src/main/java/org/elasticsearch/license/XPackLicenseS
tate.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackClien
tPlugin.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackField
.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackSetti
ngs.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/beats/Beat
sFeatureSetUsage.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/a
uthz/permission/Role.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/a
uthz/store/ReservedRolesStore.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/action/Tra
nsportXPackInfoActionTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/security/a
uthz/store/ReservedRolesStoreTests.java
x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/datafeed/MlRem
oteLicenseChecker.java
x-pack/plugin/ml/src/test/java/org/elasticsearch/xpack/ml/datafeed/MlRem
oteLicenseCheckerTests.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/au
thz/store/CompositeRolesStore.java
x-pack/protocol/src/main/java/org/elasticsearch/protocol/xpack/XPackInfo
Response.java
x-pack/protocol/src/main/java/org/elasticsearch/protocol/license/License
Status.java ->
x-pack/protocol/src/main/java/org/elasticsearch/protocol/xpack/license/L
icenseStatus.java
x-pack/protocol/src/main/java/org/elasticsearch/protocol/license/package
-info.java ->
x-pack/protocol/src/main/java/org/elasticsearch/protocol/xpack/license/p
ackage-info.java
x-pack/protocol/src/main/java/org/elasticsearch/protocol/security/packag
e-info.java ->
x-pack/protocol/src/main/java/org/elasticsearch/protocol/xpack/security/
package-info.java
x-pack/protocol/src/main/java/org/elasticsearch/protocol/watcher/package
-info.java ->
x-pack/protocol/src/main/java/org/elasticsearch/protocol/xpack/watcher/p
ackage-info.java
x-pack/protocol/src/test/java/org/elasticsearch/protocol/xpack/XPackInfo
ResponseTests.java
x-pack/protocol/src/test/java/org/elasticsearch/protocol/license/License
StatusTests.java ->
x-pack/protocol/src/test/java/org/elasticsearch/protocol/xpack/license/L
icenseStatusTests.java
x-pack/qa/rolling-upgrade/src/test/resources/rest-api-spec/test/mixed_cl
uster/10_basic.yml
2018-07-11 10:26:44 +01:00
Armin Braun b4087d69d2
Fix assertIngestDocument wrongfully passing (#31913)
* Fix assertIngestDocument wrongfully passing

* Previously docA being subset of docB passed because iteration was over docA's keys only
* Scalars in nested fields were not compared in all cases
* Assertion errors were hard to interpret (message wasn't correct since it only mentioned the class type)
* In cases where two paths contained different types a ClassCastException was thrown instead of an AssertionError
* Fixes #28492
2018-07-11 10:24:21 +02:00
Christoph Büscher 4b8b831517
Remove unused reference to filePermissionsCache (#31923)
Currently Role.Builder keeps a reference to the FieldPermissionsCache that is
passed into its constructors. This seems to be unused except for passing it on
to convertFromIndicesPrivileges() in the second ctor itself, but we don't need
to keep the internal reference in that case, so it can be removed.

Relates to #31876
2018-07-11 09:56:21 +02:00
Boaz Leskes c6666fc6cb rolling upgrade should use a replica to prevent relocations while running a scroll
A scroll holds a reference to the shard store. If the cluster is moving shards
around that reference can prevent a shard from relocating back to node it used
to be on, causing test failures.

Closes #31827
2018-07-11 08:13:38 +02:00
Nik Everett 4d83a0dd5a
HLREST: Bundle the x-pack protocol project (#31904)
The `:x-pack:protocol` project is an implementation detail shared by the
xpack projects and the high level rest client and really doesn't deserve
its own maven coordinants and published javadoc. This change bundles
`:x-pack:protocol` into the high level rest client.

Relates to #29827
2018-07-10 17:59:15 -04:00
Nhat Nguyen 25cd835010 Increase logging level for testStressMaybeFlush
Relates #31629
2018-07-10 17:44:48 -04:00
Sohaib Iftikhar 88c270d844 Added lenient flag for synonym token filter (#31484)
* Added lenient flag for synonym-tokenfilter.

Relates to #30968

* added docs for synonym-graph-tokenfilter

-- Also made lenient final
-- changed from !lenient to lenient == false

* Changes after review (1)

-- Renamed to ElasticsearchSynonymParser
-- Added explanation for ElasticsearchSynonymParser::add method
-- Changed ElasticsearchSynonymParser::logger instance to static

* Added lenient option for WordnetSynonymParser

-- also added more documentation

* Added additional documentation

* Improved documentation
2018-07-10 17:11:50 -04:00
Shaunak Kashyap 3189ef49a5
[X-Pack] Beats centralized management: security role + licensing (#30520)
* Adding Beats x-pack plugin + index templates

* Adding built-in roles for Beats central management

* Fixing typo

* Refactoring: extract common code into method

* More refactoring for more code reuse

* Use a single index for Beats management

* Rename "fragment" to "block"

* Adding configuration block type

* Expand kibana_system role to include Beats management index privileges

* Fixing syntax

* Adding test

* Adding asserting for reserved role

* Fixing privileges

* Updating template

* Removing beats plugin

* Fixing tests

* Fixing role variable name

* Fixing assertions

* Switching to preferred syntax for boolean false checks

* Making class final

* Making variables final

* Updating Basic license message to be more accurate
2018-07-10 10:06:41 -07:00
Nik Everett dcbb1154bf
HLRest: Move xPackInfo() to xPack().info() (#31905)
Originally I put the X-Pack info object into the top level rest client
object. I did that because we thought we'd like to squash `xpack` from
the name of the X-Pack APIs now that it is part of the default
distribution. We still kind of want to do that, but at least for now we
feel like it is better to keep the high level rest client aligned with
the other language clients like C# and Python. This shifts the X-Pack
info API to align with its json spec file.

Relates to #31870
2018-07-10 13:01:28 -04:00
Michael Basnight e85bb734cf
Docs: add security delete role to api call table (#31907) 2018-07-10 11:17:21 -05:00
Andy Bristol 006c2c9ab0
[test] port archive distribution packaging tests (#31314)
Recreates the rest of the bats packaging tests for the tar distribution
in the java packaging test project, with support for both tar and zip
packaging, both oss and default flavors, and on Linux and Windows. Most
tests are followed fairly closely, some have either been dropped if
unnecessary or folded into others if convenient.
2018-07-10 08:21:20 -07:00
Tal Levy 952859d87d Merge branch 'master' into index-lifecycle 2018-07-10 10:55:50 -04:00
albendz 8ec33b742d Watcher: Slack message empty text (#31596)
Slack accepts an empty text or attachments, but not both. This commit
ensures that both are not empty when creating a watch.

Closes #30071

Replacing old pull request: #31288
2018-07-10 09:47:00 -05:00
David Kyle 1f0421aa69 [ML] Mute failing DetectionRulesIT.testCondition() test 2018-07-10 11:47:47 +01:00
Christoph Büscher 2ac7e49924
Fix broken NaN check in MovingFunctions#stdDev() (#31888)
The initial check will never be true, because of the special semantics of NaN,
where no value is equal to Nan, including NaN. Thus, x == Double.NaN always
evaluates to false. The method still works correct because later computations
will also return NaN if the avg argument is NaN, but the intended shortcut
doesn't work.
2018-07-10 09:34:17 +02:00
Alexander Reelsen 1c32497c44
Date: Add DateFormatters class that uses java.time (#31856)
A newly added class called DateFormatters now contains java.time based
builders for dates, which also intends to be fully backwards compatible,
when the name based date formatters are picked. Also a new class named 
CompoundDateTimeFormatter for being able to parse multiple different 
formats has been added.

A duelling test class has been added that ensures the same dates when
parsing java or joda time formatted dates for the name based dates.

Note, that java.time and joda time are not fully backwards compatible,
which also means that old formats will currently not work with this
setup.
2018-07-10 09:28:28 +02:00
David Kyle 991d17c8c4
[ML] Switch native QA tests to a 3 node cluster (#31757) 2018-07-10 08:26:48 +01:00
Christoph Büscher e60987b95f
Change trappy float comparison (#31889)
Comparing primitive floats with '==' can be trappy because e.g. special handling
of NaN values. It can be avoided by using Float.compare().
2018-07-10 09:19:11 +02:00
Tim Vernum a5d5234eff
Fix building AD URL from domain name (#31849)
The steps to read the settings and build URLs happen in a non-obvious
order, which meant that we would build the default URL (from the
domain name, and port) before we'd actually read the port settings.
This would cause the URL to always have a port of `0`.

Relates: bccf988
2018-07-10 11:54:24 +10:00
Tal Levy e9cd0556c1 Merge branch 'master' into index-lifecycle 2018-07-09 19:44:12 -04:00
Igor Motov dd21ad0d9d
Add opaque_id to audit logging (#31878)
Logs opaque_id if it is available with all audit log messages.

Relates #31521
2018-07-09 13:40:23 -07:00
Tal Levy 60c1ea2c00 re-enable backcompat tests 2018-07-09 16:06:57 -04:00
Tal Levy 68a8d13828
add support for is_write_index in put-alias body parsing (#31674)
* add support for is_write_index in put-alias body parsing

The Rest Put-Alias Action does separate parsing of the alias body
to construct the IndicesAliasesRequest. This extra parsing
was missed in #30703.

* test flag was not just ignored by the parser

* disable backcompat tests
2018-07-09 16:03:17 -04:00
Tom Callahan 3428dc20a4
Improve release notes script (#31833)
Improve release notes script

The current release notes script does not handle area labels that
are not two parts, such as ":ml".  As these area labels are rare,
I have simply hard-coded a title for these area labels.  In
addition, the script will not explicitly call out instances where
multiple area labels are present on an issue.
2018-07-09 15:32:10 -04:00
lcawl de273651ae [DOCS] Fix broken link in painless example 2018-07-09 10:47:42 -07:00
Mayya Sharipova 5481fbc249
Handle missing values in painless (#30975)
* Handle missing values in painless

Throw an exception for `doc['field'].value`
if this document is missing a value for the `field`.

For 7.0:
This is the default behaviour from 7.0

For 6.x:
To enable this behavior from 6.x, a user can set a jvm.option:
 `-Des.script.exception_for_missing_value=true` on a node.
If a user does not enable this behavior, a deprecation warning is logged on start up.

Closes #29286
2018-07-09 11:59:49 -04:00
Jim Ferenczi 584fa261cc
Remove the ability to index or query context suggestions without context (#31007)
This is a follow up of #30712 that removes the ability to index or query
and context enabled completion field without context.

Relates #30712
2018-07-09 16:01:01 +02:00
Armin Braun 5f5157a2dc
Ingest: Enable Templated Fieldnames in Rename (#31690)
* Ingest: Enable Templated Fieldnames in Rename
2018-07-09 13:50:21 +02:00
Damien Alexandre 5dcaac32b0 [Docs] Fix typo in the Rollup API Quick Reference (#31855)
The Stop endpoint was not correct in the Quick Reference.
2018-07-09 12:51:57 +02:00
Armin Braun e46ed73379
Ingest: Add ignore_missing option to RemoveProc (#31693)
Added `ignore_missing` setting to the RemoveProcessor to fix #23086
2018-07-09 10:24:34 +02:00
Nicolas Ruflin c5c860dadb
Add template config for Beat state to X-Pack Monitoring (#31809)
With https://github.com/elastic/beats/pull/7075 Beats introduces state reporting for X-Pack Monitoring. The data sent up to Elasticsearch ends up stored in the following format.

```
"beats_state": {
  "timestamp": "2018-07-05T07:21:03.581Z",
  "state": {
    "module": {
      "count": 1,
      "names": [
        "http"
      ]
    }
  },
  "beat": {
    "uuid": "594039b5-6353-4d78-9bad-778ecc0fe83f",
    "type": "metricbeat",
    "version": "7.0.0-alpha1",
    "name": "ruflin",
    "host": "ruflin"
  }
}
```

This PR adds the new fields to the template.
2018-07-09 10:04:23 +02:00
Alexander Reelsen 3aa0de9e10
Watcher: Add ssl.trust email account setting (#31684)
In order to allow users to specify hosts, where SSL is always trusted,
this setting is exposed. Otherwise the system keystore needs to be
configured properly.

For more info see https://javaee.github.io/javamail/docs/api/com/sun/mail/smtp/package-summary.html
2018-07-09 09:44:06 +02:00
Russ Cam 0dac73c4fb Remove link to oss-MSI (#31844)
This commit removes the link to an oss-MSI; there is only one version of the MSI, which includes X-Pack.

(cherry picked from commit d2e5db8a806ec8a25162f79db5209aceed4f30f7)
2018-07-09 11:31:38 +10:00
Jack Conradson d9a92011bc
Painless: Restructure Definition/Whitelist (#31879)
Create lookup package
rename Definition to PainlessLookup and move to lookup package
rename Definition.Method to PainlessMethod
rename Definition.MethodKey to PainlessMethod
rename Definition.Field to PainlessField
rename Definition.Struct to PainlessClass
rename Definition.Cast to PainlessCast
rename Whitelist.Struct to WhitelistClass
rename Whitelist.Constructor to WhitelistConstructor
rename Whitelist.Method to WhitelistMethod
rename Whitelist.Field to WhitelistField
2018-07-08 12:00:23 -07:00
Nik Everett fb27f3e7f0
HLREST: Add x-pack-info API (#31870)
This is the first x-pack API we're adding to the high level REST client
so there is a lot to talk about here!

= Open source

The *client* for these APIs is open source. We're taking the previously
Elastic licensed files used for the `Request` and `Response` objects and
relicensing them under the Apache 2 license.

The implementation of these features is staying under the Elastic
license. This lines up with how the rest of the Elasticsearch language
clients work.

= Location of the new files

We're moving all of the `Request` and `Response` objects that we're
relicensing to the `x-pack/protocol` directory. We're adding a copy of
the Apache 2 license to the root fo the `x-pack/protocol` directory to
line up with the language in the root `LICENSE.txt` file. All files in
this directory will have the Apache 2 license header as well. We don't
want there to be any confusion. Even though the files are under the
`x-pack` directory, they are Apache 2 licensed.

We chose this particular directory layout because it keeps the X-Pack
stuff together and easier to think about.

= Location of the API in the REST client

We've been following the layout of the rest-api-spec files for other
APIs and we plan to do this for the X-Pack APIs with one exception:
we're dropping the `xpack` from the name of most of the APIs. So
`xpack.graph.explore` will become `graph().explore()` and
`xpack.license.get` will become `license().get()`.

`xpack.info` and `xpack.usage` are special here though because they
don't belong to any proper category. For now I'm just calling
`xpack.info` `xPackInfo()` and intend to call usage `xPackUsage` though
I'm not convinced that this is the final name for them. But it does get
us started.

= Jars, jars everywhere!

This change makes the `xpack:protocol` project a `compile` scoped
dependency of the `x-pack:plugin:core` and `client:rest-high-level`
projects. I intend to keep it a compile scoped dependency of
`x-pack:plugin:core` but I intend to bundle the contents of the protocol
jar into the `client:rest-high-level` jar in a follow up. This change
has grown large enough at this point.

In that followup I'll address javadoc issues as well.

= Breaking-Java

This breaks that transport client by a few classes around. We've
traditionally been ok with doing this to the transport client.
2018-07-08 11:03:56 -04:00
Dimitris Athanasiou 49ba271bd8
[ML] Fix master node deadlock during ML daily maintenance (#31836)
This is the implementation for master and 6.x of #31691.
Native tests are changed to use multi-node clusters in #31757.

Relates #31683
2018-07-07 09:43:28 +01:00
Nik Everett 20cafd3159
Build: Switch integ-test-zip to OSS-only (#31866)
We mistakenly enabled bundling of the default distribution's bin scripts
into the `integ-test-zip` artifact used by plugin authors to test plugins.
These didn't change the version of Elasticsearch used for testing but as
a side effect changed the LICENSE.txt from the Apache 2 license to the
Elastic license. We really didn't mean for that to happen. The bin script
and the elasticsearch-sql-cli jar file bundled into the distribution are
indeed governed by the Elastic license but we didn't intend for them to be
in the testing artifact in the first place. This removes them and fixes
the license of the `integ-test-zip` artifact.
2018-07-06 14:41:19 -04:00
Costin Leau 9ffb26ab02
SQL: Remove restriction for single column grouping (#31818)
For historical reasons SQL restricts GROUP BY to only one field.
This commit removes the restriction and improves the test suite with
multi group by tests.

Close #31793
2018-07-06 20:55:27 +03:00
Nik Everett 994a251075
Build: Fix detection of Eclipse Compiler Server (#31838)
It looks like Eclipse's compiler server recently changed something so
our "eclipse detector" stopped working for it. I've updated the detector
so it ought to work now.
2018-07-06 12:48:00 -04:00
Piotr Prądzyński 99030e7af5 Docs: Inconsistency between description and example (#31858) 2018-07-06 12:44:20 -04:00
Martijn van Groningen c6e7df3ac2
Re-enable bwc tests now that #29538 has been backported and 6.x intake
build succeeded.
2018-07-06 18:03:25 +02:00
Costin Leau 6f5d8caeb7
QA: build improvements related to SQL projects (#31862)
Register SQL as an xpackModule
Specify group for SQL QA to disambiguate projects (otherwise due to an
old Gradle bug (https://github.com/gradle/gradle/issues/847) any
subprojects under SQL QA will not be able to refer to SQL xpackModule

Co-authored-by: Alpar Torok <torokalpar@gmail.com>
2018-07-06 18:24:12 +03:00
Christoph Büscher 3c11c7c261
[Docs] Add clarification to analysis example (#31826)
There have been at least two PRs trying to fix the spelling of "lazi" because it
isn't very clear from the example that the english analyzer will stem each token
in the example. This adds a short description of the analysis process to make
this clearer.

Relates to #31797
2018-07-06 14:36:58 +02:00
hanbj 03adbf2a39 Check timeZone() argument in AbstractSqlQueryRequest (#31822)
A value of `null` will throw an IAE.
2018-07-06 14:35:03 +02:00
Costin Leau 89cb0872cf
SQL: Fix incorrect HAVING equality (#31820)
Fix bug that causes `HAVING a = b` to be translated ad-litteram in
Painless which uses `==` for equality checks not `=`.

Close #31796
2018-07-06 12:46:57 +03:00
Yannick Welsch cce7dc20ad
Smaller aesthetic fixes to InternalTestCluster (#31831)
Allows cluster to auto-reconfigure faster by starting up nodes in parallel.
2018-07-06 11:42:09 +02:00
Christoph Büscher 450a450b2c
[Docs] Clarify accepted sort case (#31605)
Rescore only works with an explicite "sort" element if it is on descending
"_score". Even using "order" : "asc" will throw an error.
2018-07-06 10:11:36 +02:00
Martijn van Groningen cae304406e
Temporarily disable bwc test in order to backport #29538 2018-07-06 10:08:11 +02:00
Alan Woodward d3a8094d35
Remove obsolete parameters from analyze rest spec (#31795)
This commit also fixes a typo in the analyze high-level client
documentation.
2018-07-06 09:05:34 +01:00
Jim Ferenczi 573613e7ca
[Docs] Fix wrong link in Korean analyzer docs (#31815) 2018-07-06 09:30:48 +02:00