Commit Graph

39580 Commits

Author SHA1 Message Date
Jay Modi 2b8d3e8520
Security: fix dynamic mapping updates with aliases ()
This commit fixes an issue with dynamic mapping updates when an index
operation is performed against an alias and when the user only has
permissions to the alias. Dynamic mapping updates resolve the concrete
index early to prevent issues so the information about the alias that
the triggering operation was being executed against is lost. When
security is enabled and a user only has privileges to the alias, this
dynamic mapping update would be rejected as it is executing against the
concrete index and not the alias. In order to handle this situation,
the security code needs to look at the concrete index and the
authorized indices of the user; if the concrete index is not authorized
the code will attempt to find an alias that the user has permissions to
update the mappings of.

Closes 
2018-05-24 10:28:46 -06:00
Julie Tibshirani 638a719370
Ensure that ip_range aggregations always return bucket keys. () 2018-05-24 08:55:14 -07:00
Simon Willnauer 8bbfdf1f45
Use remote client in TransportFieldCapsAction ()
We now have a remote cluster client exposed which can
talk to a given remote cluster and manages reconnects etc.
This makes code more readable than using the transport layer directly.
2018-05-24 17:02:47 +02:00
Yannick Welsch 39c4f89f9b
Move Watcher versioning setting to meta field ()
The .watcher-history-* template is currently using a plugin-custom index setting xpack.watcher.template.version,
which prevents this template from being installed in a mixed OSS / X-Pack cluster, ultimately
leading to the situation where an X-Pack node is constantly spamming an OSS master with (failed)
template updates. Other X-Pack templates (e.g. security-index-template or security_audit_log)
achieve the same versioning functionality by using a custom _meta field in the mapping instead.
This commit switches the .watcher-history-* template to use the _meta field instead.
2018-05-24 15:16:20 +02:00
Christoph Büscher 3f78b3f5e1
[Docs] Explain incomplete dates in range queries ()
The current documentation isn't very clear about how incomplete dates are
treated when specifying custom formats in a `range` query. This change adds a
note explaining how missing month or year coordinates translate to dates that
have the missings slots filled with unix time start date (1970-01-01)

Closes 
2018-05-24 11:20:00 +02:00
David Roberts aafcd85f50
Move persistent task registrations to core ()
Persistent tasks was moved from X-Pack to core in .
However, registration of the named writables and named
X-content was left in X-Pack.

This change moves the registration of the named writables
and named X-content into core.  Additionally, the persistent
task actions are no longer registered in the X-Pack client
plugin, as they are already registered in ActionModule.
2018-05-24 09:17:17 +01:00
David Turner ff0b6c795a
Decouple ClusterStateTaskListener & ClusterApplier ()
Today, the `ClusterApplier` and `MasterService` both use the
`ClusterStateTaskListener` interface to notify their callers when asynchronous
activities have completed. However, this is not wholly appropriate: none of the
callers into the `ClusterApplier` care about the `ClusterState` arguments that
they receive.  This change introduces a dedicated ClusterApplyListener
interface for callers into the `ClusterApplier`, to distinguish these listeners
from the real `ClusterStateTaskListener`s that are waiting for responses from
the `MasterService`.
2018-05-24 09:05:09 +01:00
Simon Willnauer 0bdfb5c5b5
Send client headers from TransportClient ()
This change adds a simple header to the transport client
that is present on the servers thread context that ensures
we can detect if a transport client talks to the server in a
specific request. This change also adds a header for xpack
to detect if the client has xpack installed.
2018-05-24 09:46:48 +02:00
Ryan Ernst 2c7559c575
Packaging: Ensure upgrade_is_oss flag file is always deleted ()
This commit ensures the delete of the upgrade_is_oss indicator for
the packaging tests is always deleted before each run. It works by
moving the check on version which skips the task into the doFirst block,
replacing the onlyIf.

closes 
2018-05-23 23:20:13 -07:00
Jason Tedor e8b543b8cd
Force stable file modes for built packages ()
If you have an unusual umask (e.g., 0002) and clone the GitHub
repository then files that we stick into our packages like the
README.textile and the license will have a file mode of 0664 on disk yet
we expect them to be 0644. Additionally, the same thing happens with
compiled artifacts like JARs. We try to set a default file mode yet it
does not seem to take everywhere. This commit adds explicit file modes
in some places that we were relying on the defaults to ensure that the
built artifacts have a consistent file mode regardless of the underlying
build host.
2018-05-23 23:15:19 -04:00
lcawl e76c09f642 [DOCS] Fixes typos in security settings 2018-05-23 16:41:04 -07:00
Igor Motov 699153edc7 Fix GeoShapeQueryBuilder serialization after backport
Aligns the routing value serialization version after backport of
2018-05-23 18:45:19 -04:00
lcawl 94ba78e09a [DOCS] Splits auditing.asciidoc into smaller files 2018-05-23 15:43:43 -07:00
Tim Brooks d7040ad7b4
Reintroduce mandatory http pipelining support ()
This commit reintroduces 31251c9 and 63a5799. These commits introduced a
memory leak and were reverted. This commit brings those commits back
and fixes the memory leak by removing unnecessary retain method calls.
2018-05-23 14:38:52 -06:00
Jack Conradson a96a45c6ae
Painless: Types Section Clean Up ()
Clean up of types section, casting section, and a large number of examples.
2018-05-23 13:36:58 -07:00
Igor Motov 4b6915976c
Add support for indexed shape routing in geo_shape query ()
Adds ability to specify the routing value for the indexed shape in the
geo_shape query.

Closes 
2018-05-23 15:15:19 -04:00
Andy Bristol a1b538122c
[test] java tests for archive packaging ()
Ports the first couple tests for archive distributions from the old bats
project to the new java project that includes windows platforms,
consolidating them into one test method that tests that the
distributions can be extracted and their contents verified. Includes the
zip distributions which were not tested in the bats project.
2018-05-23 10:37:57 -07:00
Colin Goodheart-Smithe 4fd0a3e492 Revert "Make http pipelining support mandatory ()" ()
This reverts commit 31251c9 introduced in .

We suspect this commit is causing the OOME's reported in  and we will use this PR to test this assertion.
2018-05-23 10:54:46 -06:00
Lisa Cawley ab047ca28d
[DOCS] Fix more edit URLs in Stack Overview () 2018-05-23 09:37:55 -07:00
Yannick Welsch cfa2b069f3
Use correct cluster state version for node fault detection ()
Since its introduction in ES 1.4, node fault detection has been using the wrong cluster state version to send
as part of the ping request, by using always the constant -1 (ClusterState.UNKNOWN_VERSION). This can, in an
unfortunate series of events, lead to a situation where a previous stale master can regain its authority and
revert the cluster to an older state.

This commit makes NodesFaultDetection use the correct current cluster state for sending ping requests, avoiding
the situation where a stale master possibly forces a newer master to step down and rejoin the stale one.
2018-05-23 18:35:25 +02:00
Adrien Grand 405eb7a751 Change serialization version of doc-value fields.
Relates 
2018-05-23 18:34:05 +02:00
lcawl fcd54cdb7c [DOCS] Fixes broken link for native realm 2018-05-23 09:32:07 -07:00
Lisa Cawley b3619bdd3e
[DOCS] Clarified audit.index.client.hosts () 2018-05-23 08:41:06 -07:00
Yannick Welsch 28d690d1cf [TEST] Don't expect acks when isolating nodes
With , acking expects *all* nodes to successfully apply the cluster state.
The testElectMasterWithLatestVersion test was checking for an ack while isolating
one node in the test.

Relates to 
2018-05-23 16:35:22 +02:00
Adrien Grand a19df4ab3b
Add a `format` option to `docvalue_fields`. ()
This commit adds the ability to configure how a docvalue field should be
formatted, so that it would be possible eg. to return a date field
formatted as the number of milliseconds since Epoch.

Closes 
2018-05-23 14:39:04 +02:00
Colin Goodheart-Smithe 4aa345e6dc
Fixes UpdateSettingsRequestStreamableTests mutate bug
The mutate function in UpdateSettingsRequestStreamableTests did not
guarantee that the masterNodeTimeout and timeout values are definitely
changed and occassionally the randomTimeValue() method would select the
sime time value as the original request which caused a failure.
2018-05-23 13:31:43 +01:00
Colin Goodheart-Smithe 483b25330b
Mustes {p0=snapshot.get_repository/10_basic/*} YAML test
This is awaiting a fix for
https://github.com/elastic/elasticsearch/issues/30807
2018-05-23 11:32:58 +01:00
Yannick Welsch 03607f646b Revert "Mutes MachineLearningTests.testNoAttributes_givenSameAndMlEnabled"
This reverts commit ca999ad569.
2018-05-23 11:49:52 +02:00
Yannick Welsch 8145a820c2
Only allow x-pack metadata if all nodes are ready ()
Enables a rolling restart from the OSS distribution to the x-pack based distribution by preventing
x-pack code from installing custom metadata into the cluster state until all nodes are capable of
deserializing this metadata.
2018-05-23 11:41:23 +02:00
Colin Goodheart-Smithe ca999ad569
Mutes MachineLearningTests.testNoAttributes_givenSameAndMlEnabled
This is awaiting fix on
https://github.com/elastic/elasticsearch/issues/30804
2018-05-23 10:39:00 +01:00
Yannick Welsch 30b004f582
Use original settings on full-cluster restart ()
When doing a node restart using the test framework, the restarted node does not only use the
settings provided to the original node, but also additional settings provided by plugin extensions,
which does not correspond to the settings that a node would have on a true restart.
2018-05-23 09:02:01 +02:00
Yannick Welsch cceaa9a0f1
Only ack cluster state updates successfully applied on all nodes ()
The cluster state acking mechanism currently incorrectly acks cluster state updates that have not
successfully been applied on all nodes. In a situation, for example, where some of the nodes
disconnect during publishing, and don't acknowledge receiving the new cluster state, the user-facing
action (e.g. create index request) will still consider this as an ack.
2018-05-23 08:56:32 +02:00
Adrien Grand 886db84ad2
Expose Lucene's FeatureField. ()
Lucene has a new `FeatureField` which gives the ability to record numeric
features as term frequencies. Its main benefit is that it allows to boost
queries with the values of these features and efficiently skip non-competitive
documents at the same time using block-max WAND and indexed impacts.
2018-05-23 08:55:21 +02:00
Fernando Medina Corey 739bb4f0ec Fix a grammatical error in the 'search types' documentation.
Simple grammatical fix.
2018-05-22 22:09:04 -07:00
Tim Brooks 63a5799526
Remove http pipelining from integration test case ()
This is related to . We are removing the ability to disable http
pipelining. This PR removes the references to disabling pipelining in
the integration test case.
2018-05-22 17:18:05 -06:00
Nik Everett a5d90e919f
QA: Add xpack tests to rolling upgrade ()
A rolling upgrade from oss Elasticsearch to the default distribution of
Elasticsearch is significantly different than a full cluster restart to
install a plugin and is again different from starting a new cluster with
xpack installed. So this adds some basic tests to make sure that the
rolling upgrade that enables xpack works at all.

This also removes some unused imports from the tests that I modified in
PR . I didn't mean to leave them.
2018-05-22 17:17:34 -04:00
Michael Basnight a8cea90e10
Modify state of VerifyRepositoryResponse for bwc ()
The VerifyRepositoryResponse class holds a DiscoveryNode[], but the
nodes themselves are not serialized to a REST API consumer. Since we do
not want to put all of a DiscoveryNode over the wire, be it REST or
Transport since its unused, this change introduces a BWC compatible
change in ser/deser of the Response. Anything 6.4 and above will
read/write a NodeView, and anything prior will read/write a
DiscoveryNode. Further changes to 7.0 will be introduced to remove the
BWC shim and only read/write NodeView, and hold a List<NodeView> as the
VerifyRepositoryResponse internal state.
2018-05-22 14:55:20 -05:00
Jason Tedor 0fc22de336
Reduce CLI scripts to one-liners on Windows ()
This commit reduces the Windows CLI scripts to one-liners by moving all
of the redundant logic to an elasticsearch-cli script. This commit is
only the Windows side, a previous commit covered the Linux side.
2018-05-22 14:34:29 -04:00
Jason Tedor 2984734197
Simplify number of shards setting ()
This is code that was leftover from the move to one shard by
default. Here in index metadata we were preserving the default number of
shards settings independently of the area of code where we set this
value on an index that does not explicitly have an number of shards
setting. This took into consideration the es.index.max_number_of_shards
system property, and was used in search requests to set the default
maximum number of concurrent shard requests. We set the default there
based on the default number of shards so that in a one-node case a
search request could concurrently hit all shards on an index with the
defaults. Now that we default to one shard, we expect fewer shards in
clusters and this adjustment of the node count as the max number of
concurrent shard requests is no longer needed. This commit then changes
the default number of shards settings to be consistent with the value
used when an index is created, and removes the now unneeded adjustment
in search requests.
2018-05-22 14:33:16 -04:00
Luca Cavanna a17d6cab98
Replace Request#setHeaders with addHeader ()
Adding headers rather than setting them all at once seems more
user-friendly and we already do it in a similar way for parameters
(see Request#addParameter).
2018-05-22 20:32:30 +02:00
Luca Cavanna 0d37ac4e8c
[TEST] remove endless wait in RestClientTests ()
This commit adds a max wait timeout of one second to all the latch.await
calls made in RestClientTests. It also makes clearer that the `onSuccess`
listener method will never be called given that the underlying http
client is mocked and makes sure that `latch.countDown` is always called
2018-05-22 20:31:36 +02:00
Christoph Büscher f7b5986682
[Docs] Fix script-fields snippet execution ()
Currently the first snippet in the documentation test in script-fields.asciidoc
isn't executed, although it has the CONSOLE annotation. Adding a test setup
annotation to it seems to fix the problem.
2018-05-22 20:22:42 +02:00
Nhat Nguyen 1918a30237
Upgrade to Lucene-7.4.0-snapshot-cc2ee23050 ()
The new snapshot includes LUCENE-8324 which fixes missing checkpoint
after a fully deletes segment is dropped on flush. This snapshot should
resolves failed tests in the CorruptedFileIT suite.

Closes 
Closes 
2018-05-22 13:11:48 -04:00
Lisa Cawley 9ffeb171e0
[DOCS] Add SAML configuration information () 2018-05-22 08:50:35 -07:00
Lisa Cawley 1094ec0917
[DOCS] Remove X-Pack references from SQL CLI () 2018-05-22 08:33:14 -07:00
Tim Brooks 31251c9a6d
Make http pipelining support mandatory ()
This is related to  and . This commit removes the abilitiy
to disable http pipelining. After this commit, any elasticsearch node
will support pipelined requests from a client. Additionally, it extracts
some of the http pipelining work to the server module. This extracted
work is used to implement pipelining for the nio plugin.
2018-05-22 09:29:31 -06:00
Lee Jones 37f67d9e21 [Docs] Fix typo in circuit breaker docs ()
The previous description had a part that didn't fit and was probably
from a copy/paste of the in flight requests description above.
2018-05-22 16:43:45 +02:00
Itamar Syn-Hershko 5f172b6795 [Feature] Adding a char_group tokenizer ()
=== Char Group Tokenizer

The `char_group` tokenizer breaks text into terms whenever it encounters
a
character which is in a defined set. It is mostly useful for cases where
a simple
custom tokenization is desired, and the overhead of use of the
<<analysis-pattern-tokenizer, `pattern` tokenizer>>
is not acceptable.

=== Configuration

The `char_group` tokenizer accepts one parameter:

`tokenize_on_chars`::
    A string containing a list of characters to tokenize the string on.
Whenever a character
    from this list is encountered, a new token is started. Also supports
escaped values like `\\n` and `\\f`,
    and in addition `\\s` to represent whitespace, `\\d` to represent
digits and `\\w` to represent letters.
    Defaults to an empty list.

=== Example output

```The 2 QUICK Brown-Foxes jumped over the lazy dog's bone for $2```

When the configuration `\\s-:<>` is used for `tokenize_on_chars`, the
above sentence would produce the following terms:

```[ The, 2, QUICK, Brown, Foxes, jumped, over, the, lazy, dog's, bone,
for, $2 ]```
2018-05-22 16:26:31 +02:00
Tanguy Leroux 74474e99d6 [Docs] Fix broken cross link in documentation 2018-05-22 16:03:33 +02:00
Jay Modi 0a3b9e2138
Test: wait for netty threads in a JUnit ClassRule ()
This commit changes the wait for a few netty threads to wait for these
threads to complete after the cluster has stopped. Previously, we were
waiting for these threads before the cluster was actually stopped; the
cluster is stopped in an AfterClass method of ESIntegTestCase, while
the wait was performed in the AfterClass of a class that extended
ESIntegTestCase, which is always executed before the AfterClass of
ESIntegTestCase.

Now, the wait is contained in an ExternalResource ClassRule that
implements the waiting for the threads to terminate in the after
method. This rule is executed after the AfterClass method in
ESIntegTestCase. The same fix has also been applied in
SecuritySingleNodeTestCase.

Closes 
2018-05-22 07:35:16 -06:00