Commit Graph

7975 Commits

Author SHA1 Message Date
Tim Vernum 7cd99d2f96 Trigger log4j loading prior to context classloader (elastic/x-pack-elasticsearch#4355)
OpenSAML requires that the context classloader be set to the
classloader that contains the opensaml jar files in order to
initialise itself. However, sometimesrthis seems to cause the
classloading for slf4j/log4j to take place while that context
classloader is set, which fails.

This change forces slf4j/log4j to be initialised prior to setting
the context classloader.

Original commit: elastic/x-pack-elasticsearch@85ed8eb488
2018-04-17 11:04:31 +10:00
Lisa Cawley fa44406cea [DOCS] Add X-Pack monitoring details for Elasticsearch (elastic/x-pack-elasticsearch#4328)
Original commit: elastic/x-pack-elasticsearch@0d160df9b6
2018-04-16 14:57:42 -07:00
Ryan Ernst 0ac8b78986 Remove legacy help files (elastic/x-pack-elasticsearch#4343)
This commit removes help files that existed from the cli infrastructure
long ago.

Original commit: elastic/x-pack-elasticsearch@df11f59763
2018-04-16 12:26:42 -07:00
Chris Earle 64a3339178 [Monitoring] Ignore data when no Cluster UUID exists (elastic/x-pack-elasticsearch#4344)
This ignores data collection when the cluster is not ready, in addition to
the existing check that ignores when the cluster state's version is
unknown.

Original commit: elastic/x-pack-elasticsearch@54257d7e6f
2018-04-16 13:58:20 -04:00
Zachary Tong b73c16287b [Rollup] Document type should be _doc (elastic/x-pack-elasticsearch#4363)
Rollup was using a historical `rollup` doc type, when we should be using
`_doc` so that it is forward-compatible with the removal in 8.0

Original commit: elastic/x-pack-elasticsearch@4b3188e6c8
2018-04-16 10:52:13 -07:00
Shaunak Kashyap 31b118552f Give the logstash_admin role cluster:monitor/main privilege (elastic/x-pack-elasticsearch#4318)
This is required so the Logstash Centralized Configuration Management UI in Kibana may make the GET / request to Elasticsearch and retrieve the cluster UUID. It then uses this cluster UUID to make a call to a Kibana Monitoring API to retrieve a list of pipelines from Monitoring. In order for the Kibana Monitoring API request to succeed, the logged-in user needs to have the built-in monitoring_user role anyway, so we give this role the cluster:monitor/main privilege.

Original commit: elastic/x-pack-elasticsearch@bf6ad5c1df
2018-04-16 10:47:01 -07:00
Lisa Cawley 5742ec92b2 [DOCS] Adds links to Getting Started with Security (elastic/x-pack-elasticsearch#4349)
Original commit: elastic/x-pack-elasticsearch@28148bd72e
2018-04-16 10:37:45 -07:00
Costin Leau f7bed219f3 SQL: improve conversion of Date types (elastic/x-pack-elasticsearch#4382)
When dealing with dates, the conversion now returns a proper DateTime
instance instead of a long

Relates elastic/x-pack-elasticsearch#4331

Original commit: elastic/x-pack-elasticsearch@bba9f2c79f
2018-04-16 19:58:32 +03:00
Alexander Reelsen 25895e0a3c Tests: Check watcher state before starting/stopping (elastic/x-pack-elasticsearch#4362)
This changes the behaviour of AbstractWatcherIntegrationTestCase and its
startWatcher/stopWatcher methods. Instead of checking for the target
state and just starting or stopping if it does not match, the methods
now wait for certain states to be reached before starting or stopping.

This will fix test failures where a failure is started instead of
stopped or vice versa.

Original commit: elastic/x-pack-elasticsearch@f0b0954803
2018-04-16 18:53:04 +02:00
Costin Leau fd7b3e4d0b SQL: add conversion to same and NULL data type (elastic/x-pack-elasticsearch#4378)
Add trivial data type conversions to same (identity) and NULL type

Original commit: elastic/x-pack-elasticsearch@dc825443d5
2018-04-16 19:48:22 +03:00
Costin Leau 7266042546 SQL: Add folding to Math functions (elastic/x-pack-elasticsearch#4377)
Math functions now support folding

Original commit: elastic/x-pack-elasticsearch@4af7a22b32
2018-04-16 19:48:10 +03:00
Costin Leau e69f820423 SQL: Expand SYS TABLES behavior to support ODBC spec (elastic/x-pack-elasticsearch#4375)
Add behavior for enumerating catalogs, table types, etc when the main
param is SQL_ALL_* while all the rest are empty strings

Fix elastic/x-pack-elasticsearch#4334

Original commit: elastic/x-pack-elasticsearch@28e7b15904
2018-04-16 19:47:38 +03:00
Costin Leau bf5f7e1847 SQL: Enable some date agg tests (elastic/x-pack-elasticsearch#4374)
Now that composite aggs return dates as long (instead of Strings) enable
relevant tests

Relates https://github.com/elastic/elasticsearch/pull/29370

Original commit: elastic/x-pack-elasticsearch@6653db6d0f
2018-04-16 19:47:08 +03:00
Costin Leau e113d5dda1 SQL: Fix bug in transforming Expression properties (elastic/x-pack-elasticsearch#4373)
Fix bug in checking whether the expression properties has been  
modified or not.

Change test to move from SIN to SQRT (as the former seem to return
different values (extra digit) across JDK versions - 8 vs 10)

Fix elastic/x-pack-elasticsearch#4335

Original commit: elastic/x-pack-elasticsearch@8f672c455d
2018-04-16 19:46:33 +03:00
jaymode 13d08f9c42 Test: don't reset node in ClusterPrivilegeTests
In the ClusterPrivilegeTests class, the code was resetting the node
after each test and failures were seen in CI that were HTTP 401 when
a 403 was expected. This commit removes the resetting of the node
between tests as this was not necessary.

Additionally, there is an issue in the SecuritySingleNodeTestCase where
the rest client was not torn down afterstopping a node and starting a
new node. This means the client used in other tests would not be
connected to the right cluster. This change resolves this by tearing
down the rest client after the old node is torn down.

relates elastic/x-pack-elasticsearch#4383

Original commit: elastic/x-pack-elasticsearch@2f81a4b2e2
2018-04-16 10:31:13 -06:00
Lisa Cawley da6ff8fddd [DOCS] Clarified xpack.monitoring.enabled setting (elastic/x-pack-elasticsearch#4384)
Original commit: elastic/x-pack-elasticsearch@e713aee367
2018-04-16 09:16:29 -07:00
javanna 719bad0cc3 Mute ClusterPrivilegeTests#testThatSnapshotAndRestore
See elastic/x-pack-elasticsearch#4383

Original commit: elastic/x-pack-elasticsearch@95d57374bc
2018-04-16 17:38:44 +02:00
Dimitris Athanasiou 5bd467eec8 [ML] Respect max_result_window setting in result APIs (elastic/x-pack-elasticsearch#4380)
Currently there is a hardcoded check against 10000, which
is the default value of the max_result_window setting. This
is a relic of the past. Removing this hardcoded validation
means we respect the setting so that a user may alter it
when appropriate.

relates elastic/x-pack-elasticsearch#3672

Original commit: elastic/x-pack-elasticsearch@9c9c5bab89
2018-04-16 13:12:24 +01:00
Yogesh Gaikwad 1701934dd4 SAML: Support multiple decryption keys for SP (elastic/x-pack-elasticsearch#4289)
- Changes in build SAML SP metadata to support multiple
  encryption keys.
- Changes in Saml metadata command to support the use of
   protected keystores.
- Changes to export and set proper usage type in key
   descriptors of SP saml metadata XML.
- Changes in SAML realm to create chaining key info
   credential resolver backed by Collection of encryption
   keys as per SP configuration.
- Unit tests and test enhancements

relates elastic/x-pack-elasticsearch#3980,elastic/x-pack-elasticsearch#4293

Original commit: elastic/x-pack-elasticsearch@e02ebcc9e6
2018-04-16 17:17:39 +10:00
Jason Tedor 0e89f07c3f Lazy configure build tasks that require older JDKs (elastic/x-pack-elasticsearch#4376)
Some build tasks require older JDKs. For example, the BWC build tasks
for older versions of Elasticsearch require older JDKs. It is onerous to
require these be configured when merely compiling Elasticsearch, the
requirement that they be strictly set to appropriate values should only
be enforced if these tasks are going to be executed. To address this, we
lazy configure these tasks.

Original commit: elastic/x-pack-elasticsearch@804a11c243
2018-04-14 15:45:01 -04:00
lcawley 614d28cee7 [DOCS] Fixed xpack.monitoring.elasticsearch.ssl.ca setting
Original commit: elastic/x-pack-elasticsearch@ce453e1a0c
2018-04-13 11:27:31 -07:00
Jason Tedor 2b201d8f4a Use proper Java version for BWC builds (elastic/x-pack-elasticsearch#4365)
This commit sets the BWC builds to use the version of the JDK that is
appropriate for the indvidual version of Elasticsearch under test.

Original commit: elastic/x-pack-elasticsearch@967a497a20
2018-04-13 12:41:32 -04:00
Costin Leau 177121fa05 SQL: fix bug in mutate method
Make sure mutateInstance() returns a different instance

Original commit: elastic/x-pack-elasticsearch@4d403473d0
2018-04-13 19:11:18 +03:00
Costin Leau 4252b3094c SQL: Fix incorrect field type returned by SYS TYPE (elastic/x-pack-elasticsearch#4358)
SYS TYPE returns an integer instead of a boolean (the bug was caused
by reading the ODBC spec which refers to the wire representation instead
of the JDBC one which uses primitives)

Original commit: elastic/x-pack-elasticsearch@f9fe64ab0d
2018-04-13 18:22:53 +03:00
Costin Leau fc5e1631f1 SQL: generalize the use of ? for STRING (elastic/x-pack-elasticsearch#4359)
Improve grammar to allow use of ? as an alternative to STRING
through-out all commands
Add various parsing tests checking the ? usage for SYS commands

Original commit: elastic/x-pack-elasticsearch@d0d1feeb4c
2018-04-13 18:16:47 +03:00
Jay Modi 0d83edbca5 Test: speed up IndexPrivilegeTests (elastic/x-pack-elasticsearch#4348)
The IndexPrivilegeTests have been notoriously slow for years.
@polyfractal identified the primary issue, which is that these tests
were running against an internal cluster with 1 or 2 data nodes and had
the number of replicas set to 1 for indices by default and the methods
in the test would perform a wait for green. This wait for green would
take the full thirty seconds when there was a single data node as the
index could never reach green health due to an unassigned replica. This
could have been caught earlier by asserting the request did not timeout
but this assertion was not present.

This change does a few things to address the issues above. The first is
that these tests now extend SecuritySingleNodeTestCase, which is a new
class that extends ESSingleNodeTestCase and contains the necessary
logic for the setup and teardown of security; much of which is based
off of SecurityIntegTestCase. This means that these tests always run
against a single node cluster and have a much simpler setup. The
default index template for these tests applies settings so that indices
are created with a single shard and no replicas.

Assertions have been added to ensure the health checks with a wait for
green status have not timed out. A handcoded wait for snapshots to
finish has been replaced with an assertBusy call. Finally, the BadApple
annotation has been removed from the test.

Relates elastic/x-pack-elasticsearch#324

Original commit: elastic/x-pack-elasticsearch@572919273d
2018-04-13 06:30:40 -06:00
lcawley c09c9e13d7 [DOCS] Removes redundant role examples
Original commit: elastic/x-pack-elasticsearch@ac17cbbdfd
2018-04-12 16:47:45 -07:00
Nik Everett 76c764b379 SQL: Prepare JDBC for release (elastic/x-pack-elasticsearch#4341)
This changes JDBC so it can be released. It bundles the
`sql-shared-client` and `sql-proto` jars into the jar for the jdbc client.
It also Generates a pom for the jdbc driver when you run `gradle assemble`
on it. This will allow us to release the jdbc driver.

It also adds a zip distribution of the jdbc driver with all of its
dependencies bundled in the zip. It'd be nice to bundle all of the jdbc
driver's dependencies in the jar but we can't quite do that yet. So, for
now, to help folks using BI tools use the JDBC driver, we build a zip.

Original commit: elastic/x-pack-elasticsearch@9c668231d4
2018-04-12 15:43:35 -04:00
Lisa Cawley e521bb55ff [DOCS] Removed X-Pack installation terminology (elastic/x-pack-elasticsearch#4339)
Original commit: elastic/x-pack-elasticsearch@4987e15148
2018-04-12 10:23:07 -07:00
Costin Leau e5d18ad45c SQL: GROUP BY use composite instead of terms aggregation (elastic/x-pack-elasticsearch#4282)
Rewrote the GROUP BY to use composite aggregation instead of terms 
(and everything that comes with it) but instead rely on composite aggregation

This not only works better but simplifies the code complexity since
composite is a straight, two-level tree:

1. root/group-by/composite-keys
2. (metric) aggregations

This removes a lot of complexity from all stages that involve creating,
assembling and especially parsing the results.
By moving to composite agg, the aggregation/GROUP BY are now pageable
so the consumer/listener had to be extended to include a dedicated
cursor and specific (bucket) extractors inline with the scroll requests.

While at it, also improved the support for implicit GROUP BY by
formalizing it (previously it supported only counts and no other
agg).

In addition:
Fixed a JDBC bug that caused incorrect timeout to be passed
Improved the returned RowSet a bit and add better naming
Pick up @Nullable move from core
Make sure to specify the TimeZone for DateTimeHistogram extraction
Add missing javadoc
To avoid delegating NamedWriteableRegistry (NWR) and to keep the scope
clean, SQL writeables now handle their own serialization, keeping the
boundary between the Elasticsearch's NWR in place.
Pass NamedWriteableRegistry only when looking at the next page
To keep in line with the existing patter and simplify the code
bureaucracy, the deserialization happens directly.
Since the SearchSourceBuilder deserialization happens explicitly (and
it's otherwise opaque), the declarative invocation isn't necessary
anymore.
Add a bit more randomization in tests

Original commit: elastic/x-pack-elasticsearch@f5af046386
2018-04-12 20:20:10 +03:00
lcawley df61103b05 Merge remote-tracking branch 'upstream/master'
Original commit: elastic/x-pack-elasticsearch@8a529e7890
2018-04-12 10:00:06 -07:00
lcawley a925768de2 [DOCS] Updated terms re wildcard support
Original commit: elastic/x-pack-elasticsearch@76510b0c8c
2018-04-12 09:59:17 -07:00
Jay Modi 6ff75f7aa3 Build: use built-in support for keystore files (elastic/x-pack-elasticsearch#4353)
This commit switches the manual creation and addition of files to the
keystore to use the built-in support available in the integTestCluster
configuration closure.

This change removes the need to worry about the creation of the
keystore and possibly dealing with a prompt from the creation command.

Original commit: elastic/x-pack-elasticsearch@8a4026a096
2018-04-12 07:29:27 -06:00
Adrien Grand 4d1314906e Stop filtering on `_type`.
Relates elastic/elasticsearch#29468

Original commit: elastic/x-pack-elasticsearch@b96bcd937f
2018-04-12 14:18:18 +02:00
Adrien Grand 98815655c1 Cache number of live documents with document-level security. (elastic/x-pack-elasticsearch#4255)
Currently numDocs() is computed lazily, but this doesn't help since
BaseCompositeReader calls numDocs() on its sub readers eagerly. This may cause
performance issues since every time we wrap a reader with DocumentSubSetReader
(which means for every query when DLS is enabled) we need to recompute the
number of live documents, which runs in linear time with the number of matches
of the role query.

Not computing numDocs() eagerly in DocumentSubSetReader might help, but it
would also be fragile since callers of this method still usually assume that
it runs in constant time. So I am proposing that we add a cache of the number
of live docs in order to decrease the performance hit of document-level
security. I would expect this cache to be efficient as it will not only reuse
entries in-between refreshes, but also across refreshes for segments that
haven't received any new updates.

Original commit: elastic/x-pack-elasticsearch@5a3af1b174
2018-04-12 09:12:16 +02:00
Hendrik Muhs 81a3f367f8 Improve watcher smoke test stability
port test re-factorings from elastic/x-pack-elasticsearch#4240 to improve smoke test stability

relates elastic/x-pack-elasticsearch#4311 elastic/x-pack-elasticsearch#3812

Original commit: elastic/x-pack-elasticsearch@d5fb16eef4
2018-04-12 07:50:05 +02:00
Lee Hinman 9eaec0c808 Adjust to decoupling TimeValue from Writeable (elastic/x-pack-elasticsearch#4338)
This is the x-pack side of https://github.com/elastic/elasticsearch/pull/29454

Original commit: elastic/x-pack-elasticsearch@fab4d511ca
2018-04-11 14:58:19 -06:00
Adrien Grand bd834b7357 Remove legacy mapping code. (elastic/x-pack-elasticsearch#4258)
This is a sibling of elastic/elasticsearch#29224.

Original commit: elastic/x-pack-elasticsearch@1c16d86f78
2018-04-11 08:43:52 +02:00
Zachary Tong 20dbd75623 [Rollup] Rename job config `size` to `page_size` (elastic/x-pack-elasticsearch#4309)
Renaming should hopefully make it more clear that this is the size
of pages to process during rolling up, nothing to do with the size
of the various groups, metrics, etc.

Original commit: elastic/x-pack-elasticsearch@8a0a44f04b
2018-04-10 13:34:40 -07:00
Zachary Tong cf965dca27 [Docs] Mark Rollup as Experimental instead of Beta
Original commit: elastic/x-pack-elasticsearch@bce120daae
2018-04-10 15:51:09 +00:00
Lee Hinman 9e3b03531c Remove custom PeriodType for watcher PeriodThrottler (elastic/x-pack-elasticsearch#4327)
This constructor was actually never used, other than in tests, and even then,
there is no need for a custom period type as the human-readable toString value
will suffice.

Original commit: elastic/x-pack-elasticsearch@fc666a04b9
2018-04-10 08:01:32 -06:00
Dimitris Athanasiou 4c7bd71bdf [ML] Improve comment on why task is marked completed on kill process
Original commit: elastic/x-pack-elasticsearch@5c16cf8e26
2018-04-10 13:36:13 +01:00
David Kyle 7e4e1dabcf [ML] Add categorical exclude condition (elastic/x-pack-elasticsearch#4326)
Original commit: elastic/x-pack-elasticsearch@6c80988e08
2018-04-10 13:19:00 +01:00
Tim Vernum 411f683521 Document ssl.verification_mode (elastic/x-pack-elasticsearch#4253)
It is common for users to wish to adjust the verification_mode in SSL
settings, usually with the intention of skipping hostname
verification. This has been supported for a long time, but the
relevant configuration setting was not clearly documented, which would
sometimes lead users to set `verification_mode` to `none`, and disable
more checks than they intended.

This commit adds clearer documentation regarding the options available
for `verification_mode` and actively discourages the use of `none`.

Original commit: elastic/x-pack-elasticsearch@2fdf53b42f
2018-04-10 20:27:23 +10:00
Igor Motov 2ba7258603 SQL: Extract CSV spec parser into Utils class (elastic/x-pack-elasticsearch#4317)
For Geo SQL we need to have a separate set of CSV tests. This commit
extracts CSV spec parsing logic into a separate file for a more
straight forward reuse.

Relates elastic/x-pack-elasticsearch#4080

Original commit: elastic/x-pack-elasticsearch@29034ef051
2018-04-09 17:01:24 -04:00
Lisa Cawley 17543dbc00 [DOCS] Augmented token API description (elastic/x-pack-elasticsearch#4321)
Original commit: elastic/x-pack-elasticsearch@c24713ee43
2018-04-09 11:39:21 -07:00
Lee Hinman 218e9a57bb Correct documentation for number of salt generation rounds (elastic/x-pack-elasticsearch#4322)
For the user cache, the crypt option rounds are actually the log2 of the number
of rounds. This commits updates the documentation to reflect this.

Original commit: elastic/x-pack-elasticsearch@d3cc2b7f29
2018-04-09 10:59:06 -06:00
David Kyle 11a6cd18ac [ML] Create calendars with job groups (elastic/x-pack-elasticsearch#4308)
Original commit: elastic/x-pack-elasticsearch@dc42dccb1f
2018-04-09 16:18:01 +01:00
Alexander Reelsen 6bd5e9ef91 Watcher: Reenable email property settings (elastic/x-pack-elasticsearch#4319)
With the change of requiring to configure account settings properly by
using affix settings, we forgot another special snowflake, namely the
configuration of mail properties, which can be arbitrary in the
configuration. Those properties are used when an email is sent.

This commit adds a few (but not all of those) options back and removes
the link in the documentation to refer to all of those settings.

Some settings are useless, as they only change the execution
expectations when a mail is sent, which the watch has control over.

The following settings are supported

* smtp.{host,port,user,password}
* smtp.auth
* smtp.starttls.{enable,required}
* smtp.{timeout,connection_timeout,write_timeout}
* smtp.{local_address,local_port}
* smtp.send_partial
* smtp.wait_on_quit

relates elastic/x-pack-elasticsearch#4048

Original commit: elastic/x-pack-elasticsearch@39d5624710
2018-04-09 15:42:16 +02:00
Lisa Cawley b6d901f530 [DOCS] Augmented security configuration steps (elastic/x-pack-elasticsearch#4316)
Original commit: elastic/x-pack-elasticsearch@48b47b19ae
2018-04-06 11:48:03 -07:00