Commit Graph

7495 Commits

Author SHA1 Message Date
Jason Tedor 04b48324aa Drop native controller from descriptors (except ML)
These were copied wholesale from the pre-split X-Pack
descriptor. However, only ML has a native controller. This removes the
plugin installation asking multiple times to approve the existence of a
native controlled for every bundled plugin.

Relates elastic/x-pack-elasticsearch#3650

Original commit: elastic/x-pack-elasticsearch@4fca606243
2018-01-21 11:00:51 -05:00
Jason Tedor a582acddf8 Add API JAR task to x-pack-core
This commit adds back the task for assembling the API JAR from the
x-pack-core JAR.

Original commit: elastic/x-pack-elasticsearch@10385ecf17
2018-01-21 10:00:47 -05:00
Jason Tedor 377bb53cca Fix format of third party audit in saml-idp-tests
The formatting here was off since it contains tabs and tabs are evil.

Original commit: elastic/x-pack-elasticsearch@fadbb02157
2018-01-20 19:53:26 -05:00
Tim Vernum 47213f5675 [Security] Add SAML authentication support (elastic/x-pack-elasticsearch#3646)
Introduces:

- SAML Realm
- REST & Transport actions to support SAML single signon / signout
- Tests for above
- More XML than you ever wanted to see.

Original commit: elastic/x-pack-elasticsearch@b0fe7bb652
2018-01-21 08:43:00 +10:00
Jason Tedor 50864eabce Introduce plugin-specific env scripts
With plugins of a meta-plugin now loaded in separate classloaders, we
should not be loading all classes in all plugins of a meta-plugin when
executing scripts. This is particularly problematic in the case of
security extensions where the install plugin extension command would be
running with the classpath of all plugins. However, if there is JAR hell
in this classpath, installation would fail. This is not realistic though
since the plugins are run in separate classloaders. To fix this, for the
scripts of a plugin, we only set the classpath to include the JARs for
that plugin and the JARs of core. This leads us to the introduction of
plugin-specific env scripts.

Relates elastic/x-pack-elasticsearch#3649

Original commit: elastic/x-pack-elasticsearch@543df37eed
2018-01-20 15:56:00 -05:00
Jason Tedor 51c53710d7 Keep imports consistent with 6.x
Some imports were changed in 6.x to address line-length issues
there. This commit pulls the same changes to master to keep the branches
consistent to simplify backports.

Original commit: elastic/x-pack-elasticsearch@190f9d41f5
2018-01-20 14:52:07 -05:00
David Roberts 64cfa017f0 [TEST] Update ML categorization expected result following C++ change
Original commit: elastic/x-pack-elasticsearch@cc78772aa2
2018-01-20 14:50:24 +00:00
Jason Tedor 60e577a9c4 Fix test base class that disables ML autodetect
This is no longer needed for general X-Pack since the tests will not be
depending on ML. We move this class to an ML specific directory and
remove the dependency from other tests.

Original commit: elastic/x-pack-elasticsearch@9b287f7460
2018-01-20 01:27:33 -05:00
Ryan Ernst 0bc496ae04 add pattern excludes for binary files
Original commit: elastic/x-pack-elasticsearch@f1077e35f8
2018-01-19 22:05:01 -08:00
Michael Basnight f3ec4a5208 Split up xpack plugins into their own modules (elastic/x-pack-elasticsearch#3643)
Thanks to some great work by a bunch of amazing people, the chuck norris xpack split is a go!

Original commit: elastic/x-pack-elasticsearch@dad98e28f4
2018-01-19 23:30:17 -06:00
Nik Everett 51a6285ba1 SQL: Allow CSV tests to look more like the CLI (elastic/x-pack-elasticsearch#3640)
This allows CSV tests to include a line between the header and the
values that is ignored by the test framework. This optional line can be
added to the tests to make them a little easier to read which is useful
when they are included in the documentation. As a side effect they also
closely mimick the output of the CLI. To the point where you can copy
directly from the CLI and paste into the CSV tests.

Example:
```
constantYear
// tag::year
SELECT YEAR(CAST('2018-01-19T10:23:27Z' AS TIMESTAMP)) as year;
     year
---------------
2018
// end::year
;
```

This can be extracted with a construct like this in the docs:
```
["source","sql",subs="attributes,callouts,macros"]
--------------------------------------------------
include-tagged::{sql-specs}/datetime.csv-spec[year]
--------------------------------------------------
```

Which makes documentation that looks like this:
```
SELECT YEAR(CAST('2018-01-19T10:23:27Z' AS TIMESTAMP)) as year;
     year
---------------
2018
```

Which is fairly nice.

Original commit: elastic/x-pack-elasticsearch@8c10b5cb10
2018-01-19 16:24:42 -05:00
Lisa Cawley 744d9325e9 [DOCS] Add forecasting to ML tutorial (elastic/x-pack-elasticsearch#3489)
Original commit: elastic/x-pack-elasticsearch@ffb681ae96
2018-01-19 12:43:58 -08:00
Nik Everett b165f1c71e SQL: Fix constant folding of datetime functions (elastic/x-pack-elasticsearch#3637)
I went to write some docs for datetime functions that look like:
```
SELECT YEAR(CAST('2018-01-19T10:23:27Z' AS TIMESTAMP)) as year;

  year
2018

```
because I figured they'd be pretty easy to read because they didn't
require any knowledge of a data set. But it turns out that constant
folding doesn't work properly for date time functions because they don't
actually apply the extraction.

Original commit: elastic/x-pack-elasticsearch@aa9c66b2c7
2018-01-19 15:11:37 -05:00
Jason Tedor 0602d38bd5 Split transport implementations into client/server
This commit splits the transport implementations into components that
can be used client-side (in the transport client) and server-side (in
the server). This enables removing security as a dependency for the
transport client.

Relates elastic/x-pack-elasticsearch#3635

Original commit: elastic/x-pack-elasticsearch@e480eb7eb2
2018-01-19 12:09:34 -05:00
Dimitrios Athanasiou 9e834cbf8a [ML] Fix javadoc in MlStrings#hasValidLengthForId
Original commit: elastic/x-pack-elasticsearch@ef5ecad5e2
2018-01-19 11:45:41 +00:00
Dimitris Athanasiou 21f692c02b [ML] Further validate calendar_id and add calendar description (elastic/x-pack-elasticsearch#3624)
relates elastic/x-pack-elasticsearch#3595

Original commit: elastic/x-pack-elasticsearch@fade977361
2018-01-19 10:44:39 +00:00
Ryan Ernst 19874e35ee More license file corrections
See elastic/x-pack-elasticsearch@0af9ffc

Original commit: elastic/x-pack-elasticsearch@360a88381f
2018-01-18 16:34:25 -08:00
Ryan Ernst 43486d6a70 Remove unnecessary license files
These are no longer necessary after https://github.com/elastic/elasticsearch/pull/28304

Original commit: elastic/x-pack-elasticsearch@0af9ffc4d1
2018-01-18 16:19:10 -08:00
Lee Hinman f69a09ce83 SQL: Remove unneeded licenses, notices, and shas
Original commit: elastic/x-pack-elasticsearch@2407eeabd3
2018-01-18 17:16:25 -07:00
Lee Hinman 5818986775 Merge remote-tracking branch 'origin/master' into feature/sql_2
Original commit: elastic/x-pack-elasticsearch@8475419a00
2018-01-18 16:51:56 -07:00
Nik Everett d7944722cf SQL: Allow IN from empty lists
It won't find anything, but it is valid.

This fixes a rare test failure.

Original commit: elastic/x-pack-elasticsearch@95ee6be507
2018-01-18 17:02:56 -05:00
Lee Hinman ed65987500 SQL: Remove getting named writeables from SqlPlugin (elastic/x-pack-elasticsearch#3627)
Since we don't do any serialization of Cursor classes from the Transport
side (it is treated entirely as a string), we don't actually need to tell anyone
about SQL's writeables

Original commit: elastic/x-pack-elasticsearch@ad2c10e327
2018-01-18 14:16:37 -07:00
Nik Everett b9dbb6d204 SQL: Run fewer tests against multiple nodes (elastic/x-pack-elasticsearch#3625)
Rather than run every SQL test against multiple nodes we instead run a
subset of the tests simply for speed. The subset is both small but
exercises the CLI, JDBC, and REST interface and a few special "these
might be different in against multiple nodes" cases.

This drops the run time from of `gradle clean check` of these tests from
```
BUILD SUCCESSFUL in 4m 49s
```

To:
```
BUILD SUCCESSFUL in 2m 49s
```

Original commit: elastic/x-pack-elasticsearch@dcd99bcd82
2018-01-18 15:59:40 -05:00
Costin Leau e48589aa65 Fix CLI test
Original commit: elastic/x-pack-elasticsearch@8a31108b09
2018-01-18 22:33:47 +02:00
Tim Brooks 685b75da3a Support changes in nio channel contexts (elastic/x-pack-elasticsearch#3609)
This is related to elastic/elasticsearch#elastic/x-pack-elasticsearch#28275. It modifies x-pack to
support the changes in channel contexts. Additionally, it simplifies
the SSLChannelContext by relying on some common work between it and
BytesChannelContext.

Original commit: elastic/x-pack-elasticsearch@8a8fcce050
2018-01-18 13:06:42 -07:00
jaymode e775e84a7e Test: add additional logging to SslMultiPortTests
This commit adds some additional logging to the SslMultiPortTests to help with debugging an
intermittent failure.

Relates elastic/x-pack-elasticsearch#1809

Original commit: elastic/x-pack-elasticsearch@ae9c967249
2018-01-18 12:46:01 -07:00
jaymode 40d315b244 Test: only use a single shard in RemoteIndexAuditTrailStartingTests
Relates elastic/x-pack-elasticsearch#435

Original commit: elastic/x-pack-elasticsearch@05db61297b
2018-01-18 12:07:30 -07:00
Tim Brooks fb12a0e383 Support TLS/SSL renegotiation (elastic/x-pack-elasticsearch#3600)
This commit is related to elastic/x-pack-elasticsearch#3246. It adds support for receiving TLS/SSL
renegotiation requests for peers.

Original commit: elastic/x-pack-elasticsearch@c22c16b3bc
2018-01-18 10:59:44 -07:00
Lisa Cawley 0ea43c1aa1 [DOCS] Move auditing settings to Elasticsearch Reference (elastic/x-pack-elasticsearch#3608)
Original commit: elastic/x-pack-elasticsearch@a108afd26b
2018-01-18 09:18:24 -08:00
Nik Everett fbb7654c58 SQL: Fix spelling in ci script
s/pluin/plugin/

Original commit: elastic/x-pack-elasticsearch@92771bd932
2018-01-18 11:27:45 -05:00
Nik Everett 375ec896ce SQL: Move top level sql directory into plugin/sql (elastic/x-pack-elasticsearch#3618)
This will conform to how we want the x-pack repo structured.

relates elastic/x-pack-elasticsearch#3363

Original commit: elastic/x-pack-elasticsearch@f67e597bc8
2018-01-18 11:15:02 -05:00
Dimitris Athanasiou eefd8e7940 [ML] Allow categorical conditions to use all analysis fields (elastic/x-pack-elasticsearch#3615)
Categorical conditions should be allowed to use any of the analysis
fields as a field_name.

Original commit: elastic/x-pack-elasticsearch@a0d25e7445
2018-01-18 15:48:56 +00:00
Nik Everett 9b6f417df6 SQL: Move the jdbc directory into plugin/sql (elastic/x-pack-elasticsearch#3617)
This is the next step in removing the top level `sql` directory.

Related to elastic/x-pack-elasticsearch#3363

Original commit: elastic/x-pack-elasticsearch@948c3e3583
2018-01-18 09:33:16 -05:00
Costin Leau 1dd98eab83 SQL: Fix show tables as admin (elastic/x-pack-elasticsearch#3597)
Fix show tables as JDBC with security enabled
This commit fixes the test SqlSecurityTestCase.testShowTablesWorksAsAdmin to
skip over any indices/aliases that exist starting with `.security`.
Use value comparison instead of the result sets
Fix an offset bug while at it (columns start at 1 not 0)
Resolves elastic/x-pack-elasticsearch#3423

Original commit: elastic/x-pack-elasticsearch@6fffda6070
2018-01-18 16:13:01 +02:00
Nik Everett 753d21f3c6 SQL: Move CLI into plugin/sql directory (elastic/x-pack-elasticsearch#3606)
This is the next step in removing the top level sql directory.

I named the directory `sql-cli` instead of `cli` because that puts it at
the maven coordinates `org.elasticsearch.plugin:sql-cli` instead of
`org.elasticsearch.plugin:cli`.

Relates to elastic/x-pack-elasticsearch#3363

Original commit: elastic/x-pack-elasticsearch@d41a57a136
2018-01-18 08:39:02 -05:00
Ryan Ernst b785f9c61b Build: Use md5 to determine whether ml snapshot needs downloading (elastic/x-pack-elasticsearch#3612)
This commit makes ML snapshot downloading happen less often. It does
that by first moving the download location to a directory outside the
destructive power of gradle clean, and then also uses the md5 of the zip
to compare to that found in s3. This allows us to do a cheap HEAD
request to find if the file has changed.

Original commit: elastic/x-pack-elasticsearch@cd8b00fd31
2018-01-17 22:12:23 -08:00
Lisa Cawley 9f6064f9ac [DOCS] Edited documentation for ML categorization_analyzer (elastic/x-pack-elasticsearch#3587)
Original commit: elastic/x-pack-elasticsearch@6dd179107a
2018-01-17 13:11:36 -08:00
Nik Everett 310d1d2302 SQL: Move `sql:server` to `plugin:sql` (elastic/x-pack-elasticsearch#3604)
This moves SQL's server project into `plugin:sql` without modifying how the integration is performed. I know that it is not correct with regards to the x-pack modularization but I think it is a good first step.

Original commit: elastic/x-pack-elasticsearch@2f40d02e4d
2018-01-17 15:48:58 -05:00
Costin Leau c93247e5db SQL: Make NodeSubClass work on windows (elastic/x-pack-elasticsearch#3603)
Original commit: elastic/x-pack-elasticsearch@99a6f3a99b
2018-01-17 22:36:04 +02:00
Costin Leau 41c1c5fdd1 SQL: close ResultSet after asserting (elastic/x-pack-elasticsearch#3601)
The on-closing Connection is closed so there's no leak however closing
the ResultSet is good practice and tests another piece of code as well

Original commit: elastic/x-pack-elasticsearch@3a9cee70a3
2018-01-17 22:14:07 +02:00
Nik Everett d6e292087b SQL: Drop embedded qa mode (elastic/x-pack-elasticsearch#3602)
It will conflict with moving the `sql:server` project to `plugin:sql`
and we're fairly sure we need to rework or remove it anyway.

relates elastic/x-pack-elasticsearch#3557

Original commit: elastic/x-pack-elasticsearch@763072c182
2018-01-17 15:04:14 -05:00
Jay Modi 60d4b7e53e Add the ability to refresh tokens obtained via the API (elastic/x-pack-elasticsearch#3468)
This commit adds the ability to refresh tokens that have been obtained by the API using a refresh
token. Refresh tokens are one time use tokens that are valid for 24 hours. The tokens may be used
to get a new access and refresh token if the refresh token has not been invalidated or
already refreshed.

relates elastic/x-pack-elasticsearch#2595

Original commit: elastic/x-pack-elasticsearch@23435eb815
2018-01-17 12:18:44 -07:00
Dimitris Athanasiou f91631664a [ML] Rename Condition member valueFilter -> value (elastic/x-pack-elasticsearch#3599)
Original commit: elastic/x-pack-elasticsearch@5ef8415ca5
2018-01-17 18:39:47 +00:00
Nik Everett 74ae8e3373 SQL: Replace Node's reflection with explicit code (elastic/x-pack-elasticsearch#3490)
This isn't pretty but it removes our need to compile with parameter
names in the debug symbols and the use of reflection during tree
transforms. `instanceof` is still used. It does so by forcing all
subclasses of `Node` to implement two methods like this:

```
    @Override
    protected NodeInfo<PercentileRank, Expression> info() {
        return info(this, PercentileRank::new, field(), value());
    }

    @Override
    protected Expression replaceChildren(List<Expression> newChildren) {
        if (newChildren.size() != 2) {
            throw new IllegalArgumentException("Expected [2] children but got [" + newChildren.size() + "]");
        }
        return new PercentileRank(location(), newChildren.get(0), newChildren.get(1));
    }
```

Every. Single. One.

This is tedious and painful and you have to do each one perfectly,
but it *is* checked by the compiler so it is less scary then the reflection
based approach it is replacing. Marginally. It is still pretty terrifying because
it requires so many tiny changes. While the compiler *does* check that
you've made all the right methods it doesn't check that you've implemented
them correctly.

Technically relates elastic/x-pack-elasticsearch#2871 but doesn't really close the "OO all the things" spirit
of elastic/x-pack-elasticsearch#2871.

A change like this deserves a million tests. Instead, I've created a hacky
reflection based test that attempts to verify that all subclasses of `Node`
implement these method correctly for some test verifiable definition of
"correct".

Original commit: elastic/x-pack-elasticsearch@a69ab634f4
2018-01-17 12:49:47 -05:00
Igor Motov c3b82e5ee1 SQL: Remove test-utils project (elastic/x-pack-elasticsearch#3583)
Makes Sql Translate Action consistent with other SQL Actions and removes test-utils project

Follow up for elastic/x-pack-elasticsearch#3543

Original commit: elastic/x-pack-elasticsearch@8ff2148d67
2018-01-17 12:18:25 -05:00
Tim Brooks dda3a8dee0 Add TLS/SSL enabled SecurityNioTransport (elastic/x-pack-elasticsearch#3519)
This is related to elastic/x-pack-elasticsearch#3246. This commit adds a SSL/TLS layer to the nio
work implemented in the SSLChannelContext and SSLDriver classes.
This work is used to build up a SecurityNioTransport implementation.
This transport does yet offer feature parity with our normal security
transport. It mainly offers SSL/TLS security.

Original commit: elastic/x-pack-elasticsearch@d0e0484418
2018-01-17 09:44:31 -07:00
Lisa Cawley a4fad02d9a [DOCS] Added SSL certificates API (elastic/x-pack-elasticsearch#3136)
Original commit: elastic/x-pack-elasticsearch@62cb574fcf
2018-01-17 08:14:02 -08:00
Costin Leau 78b0cec4ad Add missing SHAs
Original commit: elastic/x-pack-elasticsearch@7f11371304
2018-01-17 16:55:12 +02:00
Costin Leau 9ef4089f89 SQL: Upgrade to Joda 2.9.9 (sync with master)
Original commit: elastic/x-pack-elasticsearch@d8e641cb71
2018-01-17 16:42:42 +02:00
Alexander Reelsen ef2d2764a5 Watcher: Improve cluster state listener behaviour (elastic/x-pack-elasticsearch#3538)
The cluster state listener used by watch now have two additional checks.
First, when no master node exists in the cluster state, watcher will
stop and the indexing listener will not try to trigger any new watch.
Second, when there is a global cluster write level block, it would not
be possible to update the watches index or write into the watcher
history, so the listener can bail at that case as well.

In addition this also changes the log level from debug to info when
watcher is stopped. It turned out that there are zero insights when or
if watcher is stopped when normal logging is activated. This makes it
super hard for support to know when watcher is stopped or started at all
due to shards being moved around.

Original commit: elastic/x-pack-elasticsearch@5e9ce24380
2018-01-17 14:18:17 +01:00