Commit Graph

38852 Commits

Author SHA1 Message Date
Adrien Grand d223bcf7ab
Add the `include_type_name` option to the search and document APIs. (#29506)
This commit add the `include_type_name` option to the `index`, `update`,
`delete`, `get`, `bulk` and `search` APIs. When set to `false`, the response
will omit the `_type` in the response. This option doesn't work if the endpoint
contains a type. For instance, the following call would succeed:

```
GET index/_doc/1?include_type_name=false
```

But the following one would fail:

```
GET index/some_type/1?include_type_name=false
```

Relates #15613
2018-04-17 11:29:08 +02:00
Tim Vernum eccf3899a2 Tie zip output to command line options in certutil (elastic/x-pack-elasticsearch#4354)
Previously "certutil" would generate a zip file if there were multiple
certificates.
However, this means that if the user specified "-multiple" or "-in"
then the output format will vary based on whether they entered
multiple instance names (-multiple) or whether the input file
contained multiple instance records (-in).
It is better if the output format is predictable based on the command
line options, so this change forces zip output whenever any of the
following command line options are supplied:
    -pem
    -keep-ca-key
    -multiple
    -in

Original commit: elastic/x-pack-elasticsearch@344baa5f17
2018-04-17 11:05:30 +10:00
Tim Vernum 5eac9fd1a4 [DOCS] Improve docs for "--pass" option in certutil (elastic/x-pack-elasticsearch#4352)
The handling of the "--pass" option has complex rules due to the
different requirements of different output formats, and the way
joptsimple handles specs with optional arguments.

A standard unix shell will parse
    certutil --pass=""
as being identical to
    certutil --pass=
and joptsimple treats that as being "no argument", i.e. the same as
    certutil --pass

The only way to pass a blank password is
    certutil --pass ""
(though this is shell dependent)

Original commit: elastic/x-pack-elasticsearch@bc09d0f298
2018-04-17 11:05:09 +10:00
Tim Vernum 7cd99d2f96 Trigger log4j loading prior to context classloader (elastic/x-pack-elasticsearch#4355)
OpenSAML requires that the context classloader be set to the
classloader that contains the opensaml jar files in order to
initialise itself. However, sometimesrthis seems to cause the
classloading for slf4j/log4j to take place while that context
classloader is set, which fails.

This change forces slf4j/log4j to be initialised prior to setting
the context classloader.

Original commit: elastic/x-pack-elasticsearch@85ed8eb488
2018-04-17 11:04:31 +10:00
Nhat Nguyen fd161d2659 TEST: Mute testEnsureWeReconnect
Relates #29547
2018-04-16 18:31:34 -04:00
Lisa Cawley fa44406cea [DOCS] Add X-Pack monitoring details for Elasticsearch (elastic/x-pack-elasticsearch#4328)
Original commit: elastic/x-pack-elasticsearch@0d160df9b6
2018-04-16 14:57:42 -07:00
Jason Tedor 21e73403d6 Mute full cluster restart test recovery test
This test is failing because of an addition of a call to GET
/_cluster/health with the parameter wait_for_no_initializing_shards set
to true. As older versions of Elasticsearch do not understand this
parameter, this request fails and the test fails. This commit marks this
test as awaiting a fix.
2018-04-16 17:26:35 -04:00
olcbean b3e3b80f1b REST high-level client: add support for Indices Update Settings API [take 2] (#29327)
Relates to #27205
2018-04-16 21:39:11 +02:00
Ryan Ernst e3d954c6a5
Plugins: Fix native controller confirmation for non-meta plugin (#29434)
This commit fixes plugin warning confirmation to include native
controller confirmation when no security policy exists. The case was
already covered for meta plugins, but not for normal plugins. Tests are
also added for all cases.
2018-04-16 12:27:34 -07:00
Ryan Ernst 0ac8b78986 Remove legacy help files (elastic/x-pack-elasticsearch#4343)
This commit removes help files that existed from the cli infrastructure
long ago.

Original commit: elastic/x-pack-elasticsearch@df11f59763
2018-04-16 12:26:42 -07:00
Jason Tedor a8d4ee1620
Remove PipelineExecutionService#executeIndexRequest (#29537)
With the move long ago to execute all single-document indexing requests
as bulk indexing request, the method
PipelineExecutionService#executeIndexRequest is unused and will never be
used in production code. This commit removes this method and cuts over
all tests to use PipelineExecutionService#executeBulkRequest.
2018-04-16 14:55:26 -04:00
Chris Earle 64a3339178 [Monitoring] Ignore data when no Cluster UUID exists (elastic/x-pack-elasticsearch#4344)
This ignores data collection when the cluster is not ready, in addition to
the existing check that ignores when the cluster state's version is
unknown.

Original commit: elastic/x-pack-elasticsearch@54257d7e6f
2018-04-16 13:58:20 -04:00
Zachary Tong b73c16287b [Rollup] Document type should be _doc (elastic/x-pack-elasticsearch#4363)
Rollup was using a historical `rollup` doc type, when we should be using
`_doc` so that it is forward-compatible with the removal in 8.0

Original commit: elastic/x-pack-elasticsearch@4b3188e6c8
2018-04-16 10:52:13 -07:00
Shaunak Kashyap 31b118552f Give the logstash_admin role cluster:monitor/main privilege (elastic/x-pack-elasticsearch#4318)
This is required so the Logstash Centralized Configuration Management UI in Kibana may make the GET / request to Elasticsearch and retrieve the cluster UUID. It then uses this cluster UUID to make a call to a Kibana Monitoring API to retrieve a list of pipelines from Monitoring. In order for the Kibana Monitoring API request to succeed, the logged-in user needs to have the built-in monitoring_user role anyway, so we give this role the cluster:monitor/main privilege.

Original commit: elastic/x-pack-elasticsearch@bf6ad5c1df
2018-04-16 10:47:01 -07:00
Lisa Cawley 5742ec92b2 [DOCS] Adds links to Getting Started with Security (elastic/x-pack-elasticsearch#4349)
Original commit: elastic/x-pack-elasticsearch@28148bd72e
2018-04-16 10:37:45 -07:00
Costin Leau f7bed219f3 SQL: improve conversion of Date types (elastic/x-pack-elasticsearch#4382)
When dealing with dates, the conversion now returns a proper DateTime
instance instead of a long

Relates elastic/x-pack-elasticsearch#4331

Original commit: elastic/x-pack-elasticsearch@bba9f2c79f
2018-04-16 19:58:32 +03:00
Alexander Reelsen 25895e0a3c Tests: Check watcher state before starting/stopping (elastic/x-pack-elasticsearch#4362)
This changes the behaviour of AbstractWatcherIntegrationTestCase and its
startWatcher/stopWatcher methods. Instead of checking for the target
state and just starting or stopping if it does not match, the methods
now wait for certain states to be reached before starting or stopping.

This will fix test failures where a failure is started instead of
stopped or vice versa.

Original commit: elastic/x-pack-elasticsearch@f0b0954803
2018-04-16 18:53:04 +02:00
Costin Leau fd7b3e4d0b SQL: add conversion to same and NULL data type (elastic/x-pack-elasticsearch#4378)
Add trivial data type conversions to same (identity) and NULL type

Original commit: elastic/x-pack-elasticsearch@dc825443d5
2018-04-16 19:48:22 +03:00
Costin Leau 7266042546 SQL: Add folding to Math functions (elastic/x-pack-elasticsearch#4377)
Math functions now support folding

Original commit: elastic/x-pack-elasticsearch@4af7a22b32
2018-04-16 19:48:10 +03:00
Costin Leau e69f820423 SQL: Expand SYS TABLES behavior to support ODBC spec (elastic/x-pack-elasticsearch#4375)
Add behavior for enumerating catalogs, table types, etc when the main
param is SQL_ALL_* while all the rest are empty strings

Fix elastic/x-pack-elasticsearch#4334

Original commit: elastic/x-pack-elasticsearch@28e7b15904
2018-04-16 19:47:38 +03:00
Costin Leau bf5f7e1847 SQL: Enable some date agg tests (elastic/x-pack-elasticsearch#4374)
Now that composite aggs return dates as long (instead of Strings) enable
relevant tests

Relates https://github.com/elastic/elasticsearch/pull/29370

Original commit: elastic/x-pack-elasticsearch@6653db6d0f
2018-04-16 19:47:08 +03:00
Costin Leau e113d5dda1 SQL: Fix bug in transforming Expression properties (elastic/x-pack-elasticsearch#4373)
Fix bug in checking whether the expression properties has been  
modified or not.

Change test to move from SIN to SQRT (as the former seem to return
different values (extra digit) across JDK versions - 8 vs 10)

Fix elastic/x-pack-elasticsearch#4335

Original commit: elastic/x-pack-elasticsearch@8f672c455d
2018-04-16 19:46:33 +03:00
Igor Motov e334baf6fc
Fix overflow error in parsing of long geohashes (#29418)
Fixes a possible overflow error that geohashes longer than 12 characters
can cause during parsing.

Fixes #24616
2018-04-16 12:37:38 -04:00
jaymode 13d08f9c42 Test: don't reset node in ClusterPrivilegeTests
In the ClusterPrivilegeTests class, the code was resetting the node
after each test and failures were seen in CI that were HTTP 401 when
a 403 was expected. This commit removes the resetting of the node
between tests as this was not necessary.

Additionally, there is an issue in the SecuritySingleNodeTestCase where
the rest client was not torn down afterstopping a node and starting a
new node. This means the client used in other tests would not be
connected to the right cluster. This change resolves this by tearing
down the rest client after the old node is torn down.

relates elastic/x-pack-elasticsearch#4383

Original commit: elastic/x-pack-elasticsearch@2f81a4b2e2
2018-04-16 10:31:13 -06:00
Lisa Cawley da6ff8fddd [DOCS] Clarified xpack.monitoring.enabled setting (elastic/x-pack-elasticsearch#4384)
Original commit: elastic/x-pack-elasticsearch@e713aee367
2018-04-16 09:16:29 -07:00
David Turner 34ec403a2e
Remove unused index.ttl.disable_purge setting (#29527)
This setting does nothing, and is deprecated in the 6.x series by #29526. This 
change removes it entirely in 7.0.
2018-04-16 17:10:55 +01:00
Boaz Leskes 8f91743768 FullClusterRestartIT.testRecovery should wait for all initializing shards
Shards that are not fully bake may not execute the flush call, causing
subsequent unexpected translog recoveries which fails the test.

Closes #27817
2018-04-16 17:51:08 +02:00
Nik Everett 69aabb7e40
Build: Fail if any libs depend on non-core libs (#29336)
Fails the build if any subprojects of `:libs` have dependencies in `:libs`
except for `:libs:elasticsearch-core`.

Since we now have three places where we resolve project substitutions
I've added `dependencyToProject` to `project.ext` in all projects. It
resolves both `project` style dependencies and "external" style (like
"org.elasticsearch:elasticsearch-core:${version}") dependencies to
`Project`s using the `projectSubstitutions`. I use this new function all
three places where resovle project substitutions.

Finally this pulls `apply plugin: 'elasticsearch.build'` out of
`libs/*/build.gradle` and into a subprojects clause in
`libs/build.gradle`. I do this entirely so that I can call
`tasks.precommit.dependsOn checkDependencies` without waiting for the
subprojects to be evaluated or worrying about whether or not they have
`precommit` set up in a normal way.
2018-04-16 11:49:27 -04:00
javanna 719bad0cc3 Mute ClusterPrivilegeTests#testThatSnapshotAndRestore
See elastic/x-pack-elasticsearch#4383

Original commit: elastic/x-pack-elasticsearch@95d57374bc
2018-04-16 17:38:44 +02:00
Luca Cavanna 62e33eeef3
[TEST] REST client request without leading '/' (#29471)
The following is the current behaviour, tested now through a specific
test.

The low-level REST client doesn't add a leading wildcard when not
provided, unless a `pathPrefix` is configured in which case a trailing
slash will be automatically added when concatenating the prefix and the
provided uri.

Also when configuring a pathPrefix, if it doesn't start with a '/' it
will be modified by adding the missing leading '/'.
2018-04-16 15:06:57 +02:00
Ke Li 0bfb59dcf2 Using ObjectParser in UpdateRequest (#29293)
CRUD: Parsing changes for UpdateRequest (#29293)

Use `ObjectParser` to parse `UpdateRequest` so we reject unknown fields
and drop support for the `_fields` parameter because it was deprecated
in 5.x.
2018-04-16 08:39:35 -04:00
Dimitris Athanasiou 5bd467eec8 [ML] Respect max_result_window setting in result APIs (elastic/x-pack-elasticsearch#4380)
Currently there is a hardcoded check against 10000, which
is the default value of the max_result_window setting. This
is a relic of the past. Removing this hardcoded validation
means we respect the setting so that a user may alter it
when appropriate.

relates elastic/x-pack-elasticsearch#3672

Original commit: elastic/x-pack-elasticsearch@9c9c5bab89
2018-04-16 13:12:24 +01:00
Christoph Büscher a004a33803
Prevent accidental changes of default values (#29528)
The default percentiles values and the default highlighter per- and
post-tags are currently publicly accessible and can be altered any time.
This change prevents this by restricting field access.
2018-04-16 13:41:42 +02:00
Bolarinwa Saheed Olayemi 7931cf87f0 [Docs] Add definitions to glossary (#29127)
Definitions for "filter" and "query" are added to the glossary of terms.

Closes #29127
2018-04-16 10:49:54 +02:00
Yogesh Gaikwad 1701934dd4 SAML: Support multiple decryption keys for SP (elastic/x-pack-elasticsearch#4289)
- Changes in build SAML SP metadata to support multiple
  encryption keys.
- Changes in Saml metadata command to support the use of
   protected keystores.
- Changes to export and set proper usage type in key
   descriptors of SP saml metadata XML.
- Changes in SAML realm to create chaining key info
   credential resolver backed by Collection of encryption
   keys as per SP configuration.
- Unit tests and test enhancements

relates elastic/x-pack-elasticsearch#3980,elastic/x-pack-elasticsearch#4293

Original commit: elastic/x-pack-elasticsearch@e02ebcc9e6
2018-04-16 17:17:39 +10:00
Jason Tedor 00fd73acc4
Avoid self-deadlock in the translog (#29520)
Today when reading an operation from the current generation fails
tragically we attempt to close the translog. However, by invoking close
before releasing the read lock we end up in self-deadlock because
closing tries to acquire the write lock and the read lock can not be
upgraded to a write lock. To avoid this, we move the close invocation
outside of the try-with-resources that acquired the read lock. As an
extra guard against this, we document the problem and add an assertion
that we are not trying to invoke close while holding the read lock.
2018-04-15 16:26:09 -04:00
Jason Tedor 9125684d86 Minor cleanup in NodeInfo.groovy
This commit is a minor cleanup of a code block in NodeInfo.groovy. We
remove an unused variable, make the formatting of the code consistent,
and cast a property that is typed as an Object to a String to avoid an
annoying IDE warning.
2018-04-14 20:48:48 -04:00
Jason Tedor 0e89f07c3f Lazy configure build tasks that require older JDKs (elastic/x-pack-elasticsearch#4376)
Some build tasks require older JDKs. For example, the BWC build tasks
for older versions of Elasticsearch require older JDKs. It is onerous to
require these be configured when merely compiling Elasticsearch, the
requirement that they be strictly set to appropriate values should only
be enforced if these tasks are going to be executed. To address this, we
lazy configure these tasks.

Original commit: elastic/x-pack-elasticsearch@804a11c243
2018-04-14 15:45:01 -04:00
Jason Tedor b883e1217f
Lazy configure build tasks that require older JDKs (#29519)
Some build tasks require older JDKs. For example, the BWC build tasks
for older versions of Elasticsearch require older JDKs. It is onerous to
require these be configured when merely compiling Elasticsearch, the
requirement that they be strictly set to appropriate values should only
be enforced if these tasks are going to be executed. To address this, we
lazy configure these tasks.
2018-04-14 15:44:43 -04:00
Jason Tedor efa823bd79 Simplify snapshot check in root build file
Rather than checking a substring match, now that
VersionProperties#elasticsearch is a strongly-typed instance of Version,
we can use the Version#isSnapshot convenience method. This commit
switches the root build file to do this.
2018-04-13 17:11:37 -04:00
Jason Tedor 85ac541ab3
Make NodeInfo#nodeVersion strongly-typed as Version (#29515)
Today we have a nodeVersion property on the NodeInfo class that we use
to carry around information about a standalone node that we will start
during tests. This property is a String which we usually end up parsing
to a Version anyway to do various checks on it. This can end up
happening a lot during configuration so it would be more efficient and
safer to have this already be strongly-typed as a Version and parsed
from a String only once for each instance of NodeInfo. Therefore, this
commit makes NodeInfo#nodeVersion strongly-typed as a Version.
2018-04-13 16:57:59 -04:00
Jason Tedor 82a753dcc7
Enable license header exclusions (#29379)
There are some scenarios where the license on a source file is one that
is compatible with our projects yet we do not want to add the license to
the list of approved license headers (to keep the number of files with
that compatible license contained). This commit adds the ability to
exclude a file from the license check.
2018-04-13 15:59:33 -04:00
lcawley 614d28cee7 [DOCS] Fixed xpack.monitoring.elasticsearch.ssl.ca setting
Original commit: elastic/x-pack-elasticsearch@ce453e1a0c
2018-04-13 11:27:31 -07:00
Jason Tedor 2b201d8f4a Use proper Java version for BWC builds (elastic/x-pack-elasticsearch#4365)
This commit sets the BWC builds to use the version of the JDK that is
appropriate for the indvidual version of Elasticsearch under test.

Original commit: elastic/x-pack-elasticsearch@967a497a20
2018-04-13 12:41:32 -04:00
Jason Tedor 27fafa24f5
Use proper Java version for BWC builds (#29493)
Today we have JAVA_HOME for the compiler Java home and RUNTIME_JAVA_HOME
for the test Java home. However, when we compile BWC nodes and run them,
neither of these Java homes might be the version that was suitable for
that BWC node (e.g., 5.6 requires JDK 8 to compile and to run). This
commit adds support for the environment variables JAVA\d+_HOME and uses
the appropriate Java home based on the version of the node being
started. We even do this for reindex-from-old which requires JDK 7 for
these very old nodes. Note that these environment variables are not
required if not running BWC tests, and they are strictly required if
running BWC tests.
2018-04-13 12:41:18 -04:00
Costin Leau 177121fa05 SQL: fix bug in mutate method
Make sure mutateInstance() returns a different instance

Original commit: elastic/x-pack-elasticsearch@4d403473d0
2018-04-13 19:11:18 +03:00
Costin Leau 4252b3094c SQL: Fix incorrect field type returned by SYS TYPE (elastic/x-pack-elasticsearch#4358)
SYS TYPE returns an integer instead of a boolean (the bug was caused
by reading the ODBC spec which refers to the wire representation instead
of the JDBC one which uses primitives)

Original commit: elastic/x-pack-elasticsearch@f9fe64ab0d
2018-04-13 18:22:53 +03:00
Costin Leau fc5e1631f1 SQL: generalize the use of ? for STRING (elastic/x-pack-elasticsearch#4359)
Improve grammar to allow use of ? as an alternative to STRING
through-out all commands
Add various parsing tests checking the ? usage for SYS commands

Original commit: elastic/x-pack-elasticsearch@d0d1feeb4c
2018-04-13 18:16:47 +03:00
javanna 485d5d19bc Mute TranslogTests#testFatalIOExceptionsWhileWritingConcurrently
This test has been failing quite a few times with a suite timeout,
opened #29509 for it.
2018-04-13 17:03:09 +02:00
Jason Tedor 03ce3dd4a4 Enable skipping fetching latest for BWC builds (#29497)
The BWC builds always fetch the latest from the elastic/elasticsearch
repository for the BWC branches. Yet, there are use-cases for using the
local checkout without fetching the latest. This commit enables these
use-cases by adding a tests.bwc.git.fetch.latest property to skip the
fetches.
2018-04-13 09:31:06 -04:00