Commit Graph

2016 Commits

Author SHA1 Message Date
Jason Tedor fbb752c273 Use Elasticsearch IOUtils (elastic/x-pack-elasticsearch#4105)
This commit replaces the usage of Lucene IOUtils with Elasticsearch
IOUtils, the former of which is now forbidden.

Original commit: elastic/x-pack-elasticsearch@8e0554001f
2018-03-13 12:49:52 -04:00
Nik Everett 5c3b69fe57 SQL: Clean up catch
Use a mutli-catch because it is a little cleaner.

Original commit: elastic/x-pack-elasticsearch@8555e82bca
2018-03-13 10:05:58 -04:00
Nik Everett 47a2f63d5e SQL: Be more careful with break and eof (elastic/x-pack-elasticsearch#4092)
The SQL CLI was being a bit cavalier about `null`, `ctrl-c`, and
`ctrl-d` while reading passwords to the point where it'd halt with
an exception if the user hit `ctrl-d` while typing a password. This
changes it so that the CLI will instead shut down if the user
`ctrl-c`s or `ctrl-d`s while on the password prompt with an
ENOPERM error code.

This also fixes a packaging test failure I caused by a copy and paste
error where the CLI was always enforcing things as though it was reading
a password all the time. This error was causing packaging test failures.

Original commit: elastic/x-pack-elasticsearch@a882c50fc7
2018-03-13 09:00:53 -04:00
Tim Brooks 7f7ac08447 Add api to start basic license (elastic/x-pack-elasticsearch#4083)
This is related to elastic/x-pack-elasticsearch#3877. This commit adds a route /start_basic that
will self generate a basic license. The only validation that is
performed is to check that you do not already have a basic license
installed. Additionally, if you lose features from switching to a basic
license, you must acknowledge the changes.

Original commit: elastic/x-pack-elasticsearch@7b8eeb50b1
2018-03-12 14:39:58 -06:00
Tim Brooks de10e61765 Disable ClusterStatsMonitoringDocTests test
ClusterStatsMonitoringDocTests.testToXContent is currently failing on
master. An issue (elastic/x-pack-elasticsearch#4100) has been created for this test. This commit
disables it in the meantime.

Original commit: elastic/x-pack-elasticsearch@76557313e6
2018-03-12 12:36:23 -06:00
Jason Tedor 839a776dad Adapt tests to package keystore creation (elastic/x-pack-elasticsearch#4068)
This commit adapts the X-Pack packaging tests to the change in
Elasticsearch to create the keystore on package installation.

Original commit: elastic/x-pack-elasticsearch@e86c98fa83
2018-03-12 12:49:50 -04:00
Yannick Welsch 4bce53a1ad Disallow logger methods with Object parameter
Relates to elastic/elasticsearch#28969

Original commit: elastic/x-pack-elasticsearch@1eff5eecd9
2018-03-12 11:03:23 +01:00
Tanguy Leroux 15ab4af157 [Monitoring] Align indices/index stats with local cluster state (elastic/x-pack-elasticsearch#4079)
A small bug in the `IndexStatsCollector` can potentially returns
statistics for newly created indices that does not exist yet in the
collector's `ClusterState` local instance.

It happens because an instance of the current `ClusterState` is
captured and passed to all the collectors before they are executed (so
that they all share the same view of the state of the cluster). On
some clusters, if an index is created after the `ClusterState` is
captured but before the `IndicesStatsRequest` is executed then it can
appears in the index stats but have no corresponding entry in the
local cluster state.

This commit changes the IndexStatsCollector so that it only return
statistics for indices that already exist in the cluster state. This
way a consistent view is possible between indices/index/shard stats.

Original commit: elastic/x-pack-elasticsearch@da173ae0b0
2018-03-12 10:32:54 +01:00
Tim Vernum 41af46688a Make PKI BootstrapCheck work with SecureSettings (elastic/x-pack-elasticsearch#3993)
SslConfiguration can depend on SecureSettings, so it must be
constructed during the correct lifecycle phase.
For PkiRealmBootstrapCheck, moved the construction of SslConfiguration
objets into the constructor rather than the check method

Original commit: elastic/x-pack-elasticsearch@1a4d147216
2018-03-12 10:48:35 +10:00
Albert Zaharovits d31d90d378 Auditing requests with null indices (elastic/x-pack-elasticsearch#4016)
Adds null check.

relates elastic/x-pack-elasticsearch#3988

Original commit: elastic/x-pack-elasticsearch@64bab62ca6
2018-03-11 13:13:14 +02:00
Tim Brooks 8d68b03cb6 Stop using basic license in put license test
This is related to elastic/x-pack-elasticsearch#4095. That test uses the a basic license in a test
of the route put license. Occasionally, that license is extended due to
recent work related to indefinite basic licenses before the test
assertions can be performed. This commit changes the test to use a gold
license.

Original commit: elastic/x-pack-elasticsearch@bf2550f044
2018-03-10 09:53:20 -07:00
David Roberts 32bc247789 [ML] Adjust the name of the ML C++ repo (elastic/x-pack-elasticsearch#4020)
Relates elastic/machine-learning-cpp#544

Companion to elastic/release-manager#296

Original commit: elastic/x-pack-elasticsearch@d65ea9add5
2018-03-09 22:53:38 +00:00
Tim Brooks ae383462a7 Modify self-generated basic licenses to not expire (elastic/x-pack-elasticsearch#3952)
This is related to elastic/x-pack-elasticsearch#3877. It modifies self-generated basic licenses to
(practically) never expire. Specifically, self-generated basic licenses
will be set with an expiration date 1 year before Long.MAX_VALUE 
Additionally, basic licenses with a different expiration date will be
replaced with a new self-generated basic licenses at startup.

Original commit: elastic/x-pack-elasticsearch@de8b343089
2018-03-09 09:54:30 -07:00
Ioannis Kakavas 558679f997 Disregard comments in XML documents (elastic/x-pack-elasticsearch#4047)
* Disregard comments in XML documents
* Add tests to verify comments in XML are ignored

Original commit: elastic/x-pack-elasticsearch@8b2d8d32ef
2018-03-08 22:32:33 -08:00
Tim Vernum 3a4fa16f03 [SAML] Handle ACS URL with existing query params (elastic/x-pack-elasticsearch#4060)
If the Assertion Consumer Service URL already contained query
parameters, we would incorrectly append an addtional '?' rather than
adding the SAML parameters to the end with '&'

Original commit: elastic/x-pack-elasticsearch@60b6a977d8
2018-03-09 17:15:55 +11:00
Chris Earle c658238f33 [Logstash][Monitoring] Fix Registered Usage and Add Tests (elastic/x-pack-elasticsearch#4075)
This properly registers the `XPackFeatureSetUsage` for Logstash and
it tests it by invoking the Usage API in a Monitoring QA test.

Without those being properly registered, the test will consistently fail.

Original commit: elastic/x-pack-elasticsearch@2e8f2376fd
2018-03-08 14:53:05 -08:00
Zachary Tong aa877161ff [Rollup] Register FeatureSetUsage with xpack, add tests (elastic/x-pack-elasticsearch#4040)
We had a Usage class before, but weren't registering it with XPack.
Would be nice to add more usage info in the future (like the running
jobs on each node), but unclear the best way to do it since we'd need
to filter through the list of allocated tasks.

Original commit: elastic/x-pack-elasticsearch@5207d2758b
2018-03-08 08:06:42 -08:00
Dimitris Athanasiou 1ed31af2c6 [ML] Allow model_memory_limit to be reduced (elastic/x-pack-elasticsearch#3998)
Up to now a job update that reduces the model memory limit
was not allowed. However, there could definitely be cases
where reducing the limit is necessary and reasonable.

This commit makes it possible to decrease the limit as long
as it does not go below the current memory usage. We obtain
the latter from the model size stats.

The conditions under which updating the model_memory_limit
is not allowed are now:

 - when the job is open
 - latest model_size_stats.model_bytes < new value

relates elastic/x-pack-elasticsearch#2461

Original commit: elastic/x-pack-elasticsearch@5b35923590
2018-03-08 06:14:18 -08:00
Shaunak Kashyap 11cd9097b6 Reverting change made for node group IDs (elastic/x-pack-elasticsearch#4069)
The node group feature merits further discussion so we're going to undo this change for now.

Original commit: elastic/x-pack-elasticsearch@48f82b4cc0
2018-03-07 14:40:19 -08:00
Lee Hinman 2ecce78b13 Decouple XContentType from StreamInput/Output (elastic/x-pack-elasticsearch#4065)
This removes the readFrom and writeTo methods from XContentType, instead using
the more generic `readEnum` and `writeEnum` methods. Luckily they are both
encoded exactly the same way, so there is no compatibility layer needed for
backwards compatibility.

This is the x-pack side of https://github.com/elastic/elasticsearch/pull/28927

Original commit: elastic/x-pack-elasticsearch@f1c0928490
2018-03-07 14:50:33 -07:00
Lee Hinman 5f95e7ce87 Remove BytesRef usage from WatcherXContentParser (elastic/x-pack-elasticsearch#4024)
This conforms WatcherXContentParser for the changes in
https://github.com/elastic/elasticsearch/pull/28792

Original commit: elastic/x-pack-elasticsearch@d2d214ef89
2018-03-07 10:10:03 -07:00
Igor Motov a0800c0b09 SQL: Move parameter processing from the client to the server (elastic/x-pack-elasticsearch#3822)
Moves the `?` parameter substitution in prepared from the client to 
the server side, where it can be reused by the ODBC driver and can 
be implemented in a safer manner.

relates elastic/x-pack-elasticsearch#3699

Original commit: elastic/x-pack-elasticsearch@b876ccd8ae
2018-03-07 09:25:25 -05:00
Jason Tedor ead1c6c315 Fix Javadocs for o.e.x.r.j.RollupIndexer
This commit fixes the Javadocs for the class o.e.x.r.j.RollupIndexer as
these Javadocs were referring to instance methods on the class
incorrectly (using a this prefix).

Original commit: elastic/x-pack-elasticsearch@fdcc7338f9
2018-03-06 14:12:42 -08:00
Lee Hinman 2147d217df Wrap stream passed to createParser in try-with-resources (elastic/x-pack-elasticsearch#4055)
This wraps the stream (`.streamInput()`) that is passed to many of the
`createParser` instances in the enclosing (or a new) try-with-resources block.
This ensures the `BytesReference.streamInput()` is closed.

Relates to elastic/x-pack-elasticsearch#28504

Original commit: elastic/x-pack-elasticsearch@7546e3b4d4
2018-03-04 16:48:15 -07:00
Alexander Reelsen 792fcccf46 REST Tests: Remove human parameter (elastic/x-pack-elasticsearch#4032)
This parameter already exists in the core _common.json, no need to
repeat.

relates elastic/x-pack-elasticsearch#3575

Original commit: elastic/x-pack-elasticsearch@49cd3e0df5
2018-03-04 11:54:09 -08:00
Chris Earle af7ad4f366 [TEST][Monitoring] Move payload = null above call (elastic/x-pack-elasticsearch#4053)
This moves the `payload = null;` statement to above the asynchronous
HTTP call. This helps to avoid a race condition relative to `doClose`
asserting that it is `null`.

This is only a realistic situation during a shutdown situation because
the thread responds immediately before it can be nullified and freeable,
which is not a realistic scenario in any other situation.

Original commit: elastic/x-pack-elasticsearch@eb3a6ff118
2018-03-03 21:08:35 -08:00
Costin Leau 58f43ad4f0 SQL: Re-enable matrix aggregations (elastic/x-pack-elasticsearch#3977)
Remove functions without a backing matrix agg

MatrixAgg works across multiple fields and exposing it directly in SQL
does not work. Instead isolated functions are exposed which get folded
and optimized into one matrix agg per field. Thus not all matrix 
functions can be exposed in SQL, at least at this time.

Instead of depending on the plugin directly, depend on the plugin client
jar (matrix-agg-client)

Remove outdated test

Original commit: elastic/x-pack-elasticsearch@ec9b31bf59
2018-03-04 01:53:56 +02:00
Costin Leau 8fd361ba83 SQL: Fix type and value in SYS TYPES (elastic/x-pack-elasticsearch#3979)
The type of BUFFER_LENGTH needs to be an integer (not NULL) and unsigned
indicate the opposite of signed.
Change isSigned from Object to primitive
Since all the consumer of isSigned expect a primitive, an Object is just causing troubles by being null.
Update description table

Original commit: elastic/x-pack-elasticsearch@8e1960bdb5
2018-03-03 22:12:22 +02:00
polyfractal 933738c264 [Rollup] Don't use lucene's newSearcher() method in tests
Use AggregatorTestCase's `newIndexSearcher()` instead.  Lucene's
version can randomly wrap with IndexReader with things we can't handle
like ParallelCompositeReader

Original commit: elastic/x-pack-elasticsearch@b4c0e9a601
2018-03-02 17:07:57 -08:00
polyfractal faac0d2a52 [Rollup] Don't randomize index name in test
The test job helper randomizes the index name with 1-10 characters,
which can lead to randomized index names to overlap and show fewer
caps than the test expects.

The solution is to just use index names "0"-"24" to ensure none
of the names overlap, and thus the caps don't overlap.

Original commit: elastic/x-pack-elasticsearch@74a6d13213
2018-03-02 16:16:11 -08:00
polyfractal 7fbe289d30 [Rollup] Fix bad await in tests
The arrangement of the final latch meant the latch could countdown,
then the test ends before the await() triggers which caused the
thread to be interrupted and fail.  The whole arrangement was incorrect
anyhow.

We need to await the latch before sending the search response as before,
but move the final atomicBoolean to the second time the persistent
task status is updated which is a signal that we are done
and can end the test

If these tests continues to be flaky, we should probably just remove them.
The headers are tested elsewhere and not required to be tested in this
context.

Original commit: elastic/x-pack-elasticsearch@0cf5603972
2018-03-02 16:05:36 -08:00
Dimitris Athanasiou 79d46d1d17 [ML] Set explicit defaults to AnalysisLimits (elastic/x-pack-elasticsearch#4015)
Analysis limits contain settings that affect the resources
used by ML jobs. Those limits always take place. However,
explictly setting them is not required as they have reasonable
defaults. For a long time those defaults lived on the c++ side.
The job could just not have any explicit limits and that meant
defaults would be used at the c++ side. This has the disadvantage
that it is not obvious to the users what these settings are set to.
Additionally, users might not be aware of the settings existence.

On top of that, since 6.1, the default model_memory_limit was lowered
from 4GB to 1GB. For BWC, this meant that jobs where model_memory_limit
is null, the default of 4GB applies. Jobs that were created from 6.1
onwards, contain an explicit setting for model_memory_limit, which is
1GB unless the user sets it differently. This adds additional confusion.

This commit makes analysis limits an always explicit setting on the job.
Regardless of whether the user sets custom limits or not, the job object
(and response) will contain the full analysis limits values.

The possibilities for interpretation of missing values are:

  - the entire analysis_limits is null: this may only happen for jobs
  created prior to 6.1. Thus we set the model_memory_limit to 4GB.
  - analysis_limits are non-null but model_memory_limit is: this also
  may only happen for jobs prior to 6.1. Again, we set memory limit to
  4GB.
  - model_memory_limit is non-null: this either means the user set an
  explicit value or the job was created from 6.1 onwards and it has
  the explicit default of 1GB. We simply keep the given value.

For categorization_examples_limit the default has always been 4, so
we fill that in when it's missing.

Finally, note that we still need to handle potential null values
for the situation of a mixed cluster.

Original commit: elastic/x-pack-elasticsearch@5b6994ef75
2018-02-27 17:49:05 +00:00
Nik Everett d962d33a2a SQL: Replace the cli fixture with in-process testing (elastic/x-pack-elasticsearch#3889)
I'm really really sad to be removing the cli-fixture but I've had
trouble with it leaking recently it is pretty slow. Beyond that, we'd
prefer that our test fixture only fixture things that are external
depndencies.

So, yeah, I'm removing it. So we get faster tests and no chance of
leaking processes. We lose some "realness" in the tests. Instead of
interacting with the CLI like a real user we embed it in the test
process. That means we don't test the forking, we don't test the
executable jar, and we don't test the jLine console detection stuff. On
the other hand we were kind of forcing the jLine console detection stuff
in a funky way with the fixture anyway. And we test the executable jar
in the packaging tests. And that'll have to do.

I haven't renamed `RemoteCli` because it'd bloat this commit with
mechanical changes that'd make it hard to review. I'll rename it in a
followup commit.

This also updates jLine so we can disable blinking to matching
parentheses during testing. I have no clue why, but this wasn't
happening when we used the fixture. The trouble with the blinking is
that it is based on *time* so it slows things down. Worse, it works
inconsistently! Sometimes it spits out sensible ascii codes and
sometimes it, well, spits out weird garbage. When you use it in person
it works fine though. So we keep it on when not testing.

Cleans up some redundancy in when testing CLI errors. Less copy and
paste good.

I was tempted to disable the xterm emulation entirely while working on
this because upgrading jLine changed a few things and it was a real pain
to update. But If we turned that off then we'd have *nothing* testing
the colors and such. That'd be a shame because we use color in the
output to commicate stuff. I like it so I don't want to break it.

While I was there, I replaces the cli connector's `PrintWriter` with a
`BufferedWriter`. The `PrintWriter` was kind of a trap because `println`
would fail to work properly on windows because we force the terminal
into xterm mode and it doesn't know what to do with windows line
endings. Windows.....

Additionally I fixed a race condition between disabling echo when
reading passwords and fast writers. We were disabling the echo shortly
after sending the prompt. A fast enough writer could send us text before
the echo disable kicked in. Now I delegate to `LineReader#readLine`
with a special echo mask that disables echo. This is both easier to test
and doesn't seem to have the race condition. This race condition was
failing the tests because they are so much faster now. Yay!

Original commit: elastic/x-pack-elasticsearch@d0ec027396
2018-02-27 09:24:16 -08:00
Chris Earle 065f9aa834 [Monitoring] Remove support for setting interval -1 (elastic/x-pack-elasticsearch#4035)
This removes the ability to set `xpack.monitoring.collection.interval` to
`-1`.

Original commit: elastic/x-pack-elasticsearch@60f2db4bd1
2018-02-26 17:53:09 -08:00
David Kyle 9786b38747 [ml] Don’t open failed jobs after restart (elastic/x-pack-elasticsearch#3997)
Original commit: elastic/x-pack-elasticsearch@f4b2ff084b
2018-02-25 13:47:15 +00:00
Costin Leau b8c9c5325c SQL: add catalog filter to SYS COLUMNS command (elastic/x-pack-elasticsearch#3978)
Add basic support for catalog parameters in SYS COLUMN
Pass an empty string instead of a null inside the prepared statement
Don't use pattern for catalog in getColumns

Original commit: elastic/x-pack-elasticsearch@17e9e851a0
2018-02-25 05:08:48 +02:00
Costin Leau b2631f9ac8 SQL: Extend field resolution to deal with unquoted qualified columns (elastic/x-pack-elasticsearch#3894)
* foo.bar can mean either: no table field foo.bar or table foo, field bar
The resolution rule has been extended to include the latter case as
fallback.
* Always check field ambiguity
* Since field with dots can create confusion (when not qualified), the
analyzer always now always checks for both qualified and not-qualified
fields and throws an ambiguity message with the potential candidates.
This forces the use of qualifiers or quotes to better indicate the
desired field.
* Add example of aliasing the table to remove ambiguity

Original commit: elastic/x-pack-elasticsearch@8b70b9c4ee
2018-02-25 04:39:48 +02:00
Zachary Tong 3b474d8868 [Test] Fix slow rollup job task test
Incorrect latch caused this test to run slowly (until the await
finished), and could probably cause failure due to incorrect ordering

Original commit: elastic/x-pack-elasticsearch@ebeb8655da
2018-02-24 20:04:33 +00:00
Zachary Tong eb82e3cf61 [Test] Fix bad latches in rollup state tests
The latches were not placed correctly, allowing the aborts
to be set before we checked the state for Indexing the first time.

This was due to using the DelayingIndexer's built in latch, which
isn't placed quite where we needed it.

Original commit: elastic/x-pack-elasticsearch@590cfa07b0
2018-02-24 18:44:51 +00:00
Zachary Tong ea531c0135 [Rollup] Silence delete job yaml tests
See issue elastic/x-pack-elasticsearch#4041

Original commit: elastic/x-pack-elasticsearch@e5e589593b
2018-02-24 18:15:57 +00:00
Zachary Tong 390e64aabd Add empty policy file to Rollups
Packaging tests seem to require a policy file for the time being

Original commit: elastic/x-pack-elasticsearch@ce34b023db
2018-02-24 03:28:28 +00:00
Lee Hinman c763a14132 Pass InputStream when creating XContent parser (elastic/x-pack-elasticsearch#4001)
* Pass InputStream when creating XContent parser

Rather than passing the raw `BytesReference` in when creating the xcontent
parser, this passes the StreamInput (which is an InputStream), this allows us to
decouple XContent from BytesReference.

This is the x-pack side of https://github.com/elastic/elasticsearch/pull/28754

* Use the streamInput variant, not sourceAsString

Original commit: elastic/x-pack-elasticsearch@dd5d8b1654
2018-02-23 15:22:28 -07:00
Zachary Tong bf1550a0b2 Rollups for Elasticsearch (elastic/x-pack-elasticsearch#4002)
This adds a new Rollup module to XPack, which allows users to configure periodic "rollup jobs" to pre-aggregate data.  That data is then available later for search through a special RollupSearch API, which mimics the DSL and functionality of regular search.

Rollups are used to drastically reduce the on-disk footprint of metric-based data (e.g. timestamped document with numeric and keyword fields).  It can also be used to speed up aggregations over large datasets, since the rolled data will be considerably smaller and fewer documents to search.

The PR adds seven new endpoints to interact with Rollups; create/get/delete job, start/stop job, a capabilities API similar to field-caps, and a Rollup-enabled search.

Original commit: elastic/x-pack-elasticsearch@dcde91aacf
2018-02-23 17:10:37 -05:00
Chris Earle 6e5343192b [TEST] Wait for index to be green before searching (elastic/x-pack-elasticsearch#4034)
Since adding, I have not been able to reproduce shard failure.

Original commit: elastic/x-pack-elasticsearch@9f9fc6fb8f
2018-02-23 13:44:04 -05:00
Chris Earle 71372752dc [TEST] Remove AwaitsFix after elastic/x-pack-elasticsearch#4031
Original commit: elastic/x-pack-elasticsearch@240168baae
2018-02-23 13:13:18 -05:00
Chris Earle 0402ef4853 [TEST] Wait for Exporters independent of results (elastic/x-pack-elasticsearch#4031)
This should force waiting on the exporters so that the data can be awaited
separately.

Original commit: elastic/x-pack-elasticsearch@1e675f2786
2018-02-23 13:09:10 -05:00
Chris Earle cf0fe1bbff [Monitoring] Deprecate setting interval to -1 (elastic/x-pack-elasticsearch#4023)
This deprecates setting `xpack.monitoring.collection.interval` to `-1`.

Original commit: elastic/x-pack-elasticsearch@7c5fa35aad
2018-02-23 10:33:34 -05:00
Alexander Reelsen 7f166e1927 Cleanup: Move tests calling external services into QA projects (elastic/x-pack-elasticsearch#3869)
This also moves the hipchat/pagerduty/slack tests to REST tests which put real watches and also check the history.

Relates elastic/x-pack-elasticsearch#3800


Original commit: elastic/x-pack-elasticsearch@fce08d9cb7
2018-02-23 13:31:28 +01:00
Alexander Reelsen 9624788c17 Monitoring: Mute MonitoringIT tests
Relates elastic/x-pack-elasticsearch#4027, elastic/x-pack-elasticsearch#3962

Original commit: elastic/x-pack-elasticsearch@9ef5cac4ba
2018-02-23 12:12:40 +01:00
Luca Cavanna df8dcb22b8 [TEST] share code between streamable/writeable/xcontent base test classes (elastic/x-pack-elasticsearch#4021)
This is the xpack side of https://github.com/elastic/elasticsearch/pull/28785

Original commit: elastic/x-pack-elasticsearch@f1202e23af
2018-02-23 10:49:04 +01:00
Alexander Reelsen 2ebb21f108 Watcher: Reduce threads on non-data nodes (elastic/x-pack-elasticsearch#3932)
The watcher thread pool is scaled by the number of CPUs and has by
default up to 5x the number of cores. This is needed because we assumme
I/O based waiting workloads for watcher. However if the node is not a
data node, there will not be any execution of watches with the exception
of a user calling the execute watch API on that node.

This means, we can get away with just one thread, so that there is no
need for the JVM to manage more threads on master/client or ingest only
nodes.

Original commit: elastic/x-pack-elasticsearch@b5899401d3
2018-02-23 09:16:32 +01:00
Alexander Reelsen 45fb3db062 Tests: Fix JiraIssueTests to not hide secrets
Original commit: elastic/x-pack-elasticsearch@f4c83c2da3
2018-02-23 09:10:56 +01:00
Lee Hinman 13e47ab315 Decouple XContentGenerator and JsonXContentGenerator from BytesReference (elastic/x-pack-elasticsearch#4009)
This is the x-pack side of https://github.com/elastic/elasticsearch/pull/28772

Original commit: elastic/x-pack-elasticsearch@8a37e38f62
2018-02-22 14:28:44 -07:00
Chris Earle b009dec503 [Monitoring] Clean .watcher-history-* indices by default in 6.3+ (elastic/x-pack-elasticsearch#2720)
This flips the setting default to true for 6.3+.

Original commit: elastic/x-pack-elasticsearch@47a09d8449
2018-02-22 13:20:53 -05:00
Tim Brooks 33ae455e6c Selectors operate on channel contexts (elastic/x-pack-elasticsearch#3803)
This is related to elastic/elasticsearch#28468. It is a compatibility
commit to ensure that x-pack is compatible with those changes.

Original commit: elastic/x-pack-elasticsearch@3ddf719adc
2018-02-22 09:45:07 -07:00
javanna 531d44f446 Adjust serialization versions after backport
Now that elastic/x-pack-elasticsearch#3985 has been backported to 6.x, we can update the version conditions from 7.0.0-alpha1 to 6.3.0

Original commit: elastic/x-pack-elasticsearch@b068ca92a7
2018-02-22 16:00:47 +01:00
Luca Cavanna 79bc6d9a53 Remove AcknowledgedRestListener in favour of RestToXContentListener (elastic/x-pack-elasticsearch#3985)
Adapt to AcknowledgedRestListener removal 

Original commit: elastic/x-pack-elasticsearch@74c08fcf02
2018-02-22 09:13:58 +01:00
Chris Earle 309adaf38e [Monitoring] Disable Monitoring Collection by Default (elastic/x-pack-elasticsearch#3962)
This adds a new setting, `xpack.monitoring.collection.enabled`, and
disables it by default (`false`).

Original commit: elastic/x-pack-elasticsearch@4b3a5a1161
2018-02-21 23:10:20 -05:00
Tim Vernum 828387a25a Add more info logging when loading roles.yml (elastic/x-pack-elasticsearch#3992)
Although not frequently used in production, we make heavy use of the
FileRolesStore within integration tests. This change adds a little bit
more logging at INFO level when the roles.yml file is (re)loaded.

Original commit: elastic/x-pack-elasticsearch@bbacd46e28
2018-02-22 11:10:27 +11:00
Alexander Reelsen 0bf354eb38 Watcher: Fix hipchat message failure serialization (elastic/x-pack-elasticsearch#3939)
The current toXContent serialization of a failed hipchat message writes
the same field called status twice and thus cannot be stored in the
watch history.

This commit ensures the field gets only written once.

relates elastic/x-pack-elasticsearch#3919

Original commit: elastic/x-pack-elasticsearch@fb499e8055
2018-02-21 09:38:53 +01:00
Jay Modi e3e82ccb2d Security: IndexAuditTrail updates mappings when indexing locally (elastic/x-pack-elasticsearch#3944)
This change makes the IndexAuditTrail update mappings for the current
index even when indexing locally. This is needed in a couple of
scenarios; the first being upgrades from 5x to 6x where the document
type was changed and the second is when upgrading minor versions and
a new field has been introduced. A new field was introduced in 6.2.

Additionally, writing tests for the rolling upgrade revealed a bug
where the actual node name was never being set as it was using the name
key instead of the node.name key.

relates elastic/x-pack-elasticsearch#3901
relates elastic/x-pack-elasticsearch#3718

Original commit: elastic/x-pack-elasticsearch@7374a62ff1
2018-02-20 11:09:56 -07:00
Jay Modi 0fc0034509 Test: remove outdated logic for file writing in security tests (elastic/x-pack-elasticsearch#3947)
This commit removes some outdated logic in the SecurityTestUtils class
for writing files and creating directories. A long, long time ago there
was a global cluster for tests that was used across suites and because
of this there were calls to delete files if they already existed. The
global cluster has been removed, so we no longer need the code that
deletes the content of a directory if it already exists.

Additionally, the file writing used in SecurityTestUtils did not use
atomic moves when possible and this commit changes the code such that a
temp file is written and we try to atomically move it to the correct
path; if atomic moves are not supported a regular move is performed.

relates elastic/x-pack-elasticsearch#3912

Original commit: elastic/x-pack-elasticsearch@973fcfe2e1
2018-02-20 09:34:17 -07:00
Lee Hinman 6728912c87 Handle removing log4j dependency from elasticsearch-core jar (elastic/x-pack-elasticsearch#3976)
This handles the removal of the log4j dependency from our elasticsearch-core
project. It was originally necessary only for our jar classpath checking. It is
now replaced by a `Consumer<String>`.

This is the x-pack side of https://github.com/elastic/elasticsearch/pull/28705

Original commit: elastic/x-pack-elasticsearch@1186a7eaf6
2018-02-20 09:15:56 -07:00
David Kyle e365a255c7 [ML] Filter non-ml attributes from data feed stats (elastic/x-pack-elasticsearch#3994)
Original commit: elastic/x-pack-elasticsearch@9cdc78c000
2018-02-20 12:22:28 +00:00
Alexander Reelsen c9d77d20fd Watcher: Never return credentials after watch creation... (elastic/x-pack-elasticsearch#3581)
... yet support updates. This commit introduces a few changes of how
watches are put.

The GET Watch API will never return credentials like basic auth
passwords, but a placeholder instead now. If the watcher is enabled to
encrypt sensitive settings, then the original encrypted value is
returned otherwise a "::es_redacted::" place holder.

There have been several Put Watch API changes.

The API now internally uses the Update API and versioning. This has
several implications. First if no version is supplied, we assume an
initial creation. This will work as before, however if a credential is
marked as redacted we will reject storing the watch, so users do not
accidentally store the wrong watch.

The watch xcontent parser now has an additional methods to tell the
caller if redacted passwords have been found. Based on this information
an error can be thrown.

If the user now wants to store a watch that contains a password marked
as redacted, this password will not be part of the toXContent
representation of the watch and in combinatination with update request
the existing password will be merged in. If the encrypted password is
supplied this one will be stored.

The serialization for GetWatchResponse/PutWatchRequest has changed.
The version checks for this will be put into the 6.x branch.

The Watcher UI now needs specify the version, when it wants to store a
watch. This also prevents last-write-wins scenarios and is the reason
why the put/get watch response now contains the internal version.

relates elastic/x-pack-elasticsearch#3089

Original commit: elastic/x-pack-elasticsearch@bb63be9f79
2018-02-20 10:09:27 +01:00
Tim Vernum 56c761f241 Log roles.yml changes in TokenAuthIntegTests
This test recently failed and it appears that the cause was that
roles.yml was deleted while the cluster was still running.
This turns on logging for the FileRolesStore so that changes to file
based roles can traced more easily.

Original commit: elastic/x-pack-elasticsearch@fbce5a8027
2018-02-20 17:19:39 +11:00
Tim Vernum c4582cdcd0 Additional settings for SAML NameID policy (elastic/x-pack-elasticsearch#3969)
* Additional settings for SAML NameID policy

We should not be populating SPNameQualifier by default as it is
intended to be used to specify an alternate SP EntityID rather than
our own. Some IdPs (ADFS) fail when presented with this value.

This commit
- makes the SPNameQualifier a setting that defaults to blank
- adds a setting for "AllowCreate"
- documents the above

Original commit: elastic/x-pack-elasticsearch@093557e88f
2018-02-20 13:51:42 +11:00
Albert Zaharovits 5833ed296e Fix embarrassing NPE in audit trail.
relates elastic/x-pack-elasticsearch#3988

Original commit: elastic/x-pack-elasticsearch@72c16ea50f
2018-02-19 20:14:37 +02:00
Martijn van Groningen c621be626f test: index documents with the same type and increase root log level
Original commit: elastic/x-pack-elasticsearch@41153e52d4
2018-02-19 16:51:21 +01:00
markharwood dd9962cff2 [TEST] Add deterministic routing of docs to shards to prevent
randomisation of scores produced in results.
Reinstated failing assertions from elastic/x-pack-elasticsearch#918

Original commit: elastic/x-pack-elasticsearch@42318f1d8a
2018-02-19 15:46:51 +00:00
David Kyle 137488c9f9 [ML] Kill process after executor has shutdown (elastic/x-pack-elasticsearch#3958)
Original commit: elastic/x-pack-elasticsearch@4c58aa51bf
2018-02-19 12:11:22 +00:00
Shaunak Kashyap 50be43bcd3 Adding new queue metric fields to mapping (elastic/x-pack-elasticsearch#3949)
This will allow these fields to be used in aggregations by Monitoring UI code

Original commit: elastic/x-pack-elasticsearch@6ce0aecb0c
2018-02-16 08:51:15 -08:00
Tanguy Leroux eb11cb96a1 [Tests] Ensure that Watcher templates are created before creating alias (elastic/x-pack-elasticsearch#3956)
* [Tests] Ensure that Watcher templates are created before creating alias

The createWatcherIndicesOrAliases() method randomly created watcher
indices with or without aliases. On slow machines it is possible that
the indices get created before WatcherIndexTemplateRegistry put the
templates. When it happens, the indices will be created without the
right index.format value, preventing the WatcherService to start (as
it checks the index.format before starting, in case an upgrade is
required)

relates elastic/x-pack-elasticsearch#3965

Original commit: elastic/x-pack-elasticsearch@82946a2be0
2018-02-16 11:48:42 +01:00
Lee Hinman fbd9d6c7f5 Call createParser with deprecation handler (elastic/x-pack-elasticsearch#3943)
There are only two remaining callers from x-pack that don't pass the deprecation
handler in.

Relates to elastic/x-pack-elasticsearch#28504

Original commit: elastic/x-pack-elasticsearch@cb542613fd
2018-02-15 15:38:25 -07:00
Costin Leau 749d0e3880 SQL: Regen grammar
Original commit: elastic/x-pack-elasticsearch@7cac9f5ea0
2018-02-15 22:34:12 +02:00
Costin Leau ae8152b55b SQL: add catalog and type arguments to SYS TABLES (elastic/x-pack-elasticsearch#3920)
Extend the SYS TABLES command to allow filtering by catalog and types
Add creation of EnumSet inside node tests
Handle EnumSet inside test for replaceChildren

Original commit: elastic/x-pack-elasticsearch@407ce7b006
2018-02-15 22:07:51 +02:00
Costin Leau 844c485e0d SQL: introduce SYS CATALOGS and TABLE TYPES commands (elastic/x-pack-elasticsearch#3941)
These are the equivalent of JDBC/ODBC getCatalogs and getTableTypes on
the server side.

Original commit: elastic/x-pack-elasticsearch@a6fa733680
2018-02-15 21:51:23 +02:00
David Roberts 54b0fca88e [TEST] Update strings in ML categorization test (elastic/x-pack-elasticsearch#3959)
This makes the samples match the C++ code

Original commit: elastic/x-pack-elasticsearch@54b71f9299
2018-02-15 16:41:16 +00:00
Tim Vernum bad812d920 [TEST] Silence LocalExporterIntegTests.testExport
See: elastic/x-pack-elasticsearch#3954, elastic/x-pack-elasticsearch#3910

Original commit: elastic/x-pack-elasticsearch@a298c21579
2018-02-15 18:04:20 +11:00
Tim Brooks d45f28339b Support configuring license type for gradle run (elastic/x-pack-elasticsearch#3942)
This commit is related to elastic/x-pack-elasticsearch#3877. It allows the passing of a system
property to gradle run indicating whether a `basic` or `trial` license
should be generated. If the `trial` license is indicated, platinum
features will be enabled.

Original commit: elastic/x-pack-elasticsearch@b2b89a4606
2018-02-14 19:34:55 -07:00
Chris Earle 176411e55e [Monitoring] Ignore _bulk if Collection is Disabled (elastic/x-pack-elasticsearch#3910)
This blocks incoming requests from Kibana, Logstash, and Beats when X-Pack monitoring is effectively disabled by setting `xpack.monitoring.collection.interval: -1`.

Original commit: elastic/x-pack-elasticsearch@016a9472f1
2018-02-14 16:24:03 -05:00
Tanguy Leroux 00dec27d9f SQL: Relax RemoteFailure parsing so that it also parses error metadata (elastic/x-pack-elasticsearch#3938)
The current parsing of errors in the RemoteFailure class is strict and
fails on any field that is not one of [caused_by, reason, root_cause,
stack_trace, type]. Sadly some exceptions adds more headers or
metadata when they are printed as XContent and such fields can't be
easily ignored at parsing time.

This commit changes the RemoteFailure.parseFromResponse() method so that
it parses errors using the same behavior as the high level rest client:
it parses any unknown field as a metadata if it's string value or an
array of string and just ignores and skips everything else without
throwing an exception.

Original commit: elastic/x-pack-elasticsearch@1348706807
2018-02-14 21:05:23 +01:00
Igor Motov 89f15ed9d3 Tests: suppress testIndexUpgradeInfoLicense on release builds (elastic/x-pack-elasticsearch#3930)
The testIndexUpgradeInfoLicense test is generating a license, which
fails on release builds because they are using production public keys
for license verification.

relates elastic/x-pack-elasticsearch#3921

Original commit: elastic/x-pack-elasticsearch@e1c37bfd91
2018-02-14 13:56:40 -05:00
Lee Hinman affc9e3563 Move more XContent.createParser calls to non-deprecated version (elastic/x-pack-elasticsearch#3928)
Part 2

This moves more of the callers to pass in the DeprecationHandler.

Relates to elastic/x-pack-elasticsearch#28504

Original commit: elastic/x-pack-elasticsearch@e0b52bf050
2018-02-14 09:41:26 -07:00
Lee Hinman e8ef20b219 Move more XContent.createParser calls to non-deprecated version (elastic/x-pack-elasticsearch#3927)
* Move more XContent.createParser calls to non-deprecated version

This moves more of the callers to pass in the DeprecationHandler.

Relates to https://github.com/elastic/elasticsearch/issues/28504

* Mockito ಠ_ಠ

Original commit: elastic/x-pack-elasticsearch@81cb99ba52
2018-02-14 09:24:47 -07:00
Tim Vernum e7e3584560 Remove Issuer/Serial from AKI in certutil/certgen (elastic/x-pack-elasticsearch#3931)
The Issuer and Serial number are optional and can cause problems with
chain validation when using OpenSSL based tools

relates elastic/x-pack-elasticsearch#3915

Original commit: elastic/x-pack-elasticsearch@defcdd934f
2018-02-14 23:15:11 +11:00
Costin Leau a5b129c10c SQL: Introduce the catalog concept inside the grammar (elastic/x-pack-elasticsearch#3911)
Indices can now specify the catalog (the cluster) in which they belong.
This is a first step in supporting both CCS and BI tools which tend to
be explicit

Original commit: elastic/x-pack-elasticsearch@0b4633facc
2018-02-14 12:30:32 +02:00
Alexander Reelsen 742c7001c9 Watcher: Ensure watcher service calls are properly caught (elastic/x-pack-elasticsearch#3906)
In order to prevent exceptions to bubble up to the thread pool exception
handler, this properly wraps all the calls for the watcher service
within an executor into an AbstractRunnable to catch and log a possible
exception.

relates elastic/x-pack-elasticsearch#3854

Original commit: elastic/x-pack-elasticsearch@c0b39e6b5b
2018-02-14 10:45:29 +01:00
Adrien Grand 48847720e8 Disable IndexUpgradeIT.testIndexUpgradeInfoOnEmptyCluster too.
Relates elastic/x-pack-elasticsearch#3921

Original commit: elastic/x-pack-elasticsearch@88e66c5564
2018-02-14 09:05:51 +01:00
Adrien Grand 8dbd1fd2cd Disable IndexUpgradeIT.testIndexUpgradeInfo and testIndexUpgradeInfoLicense for now.
Relates elastic/x-pack-elasticsearch#3921

Original commit: elastic/x-pack-elasticsearch@a32e8fd557
2018-02-14 09:02:54 +01:00
Yogesh Gaikwad 20503a997f X-Pack-Security: Improve error messaging during setup password (elastic/x-pack-elasticsearch#3809)
* X-Pack-Security: Improve error messaging during setup password

Changes are done to improve logging and exception messages when
setup passwords. Now it checks whether
- X-Pack plugin is available
- X-Pack Security is available 
- X-Pack Security is enabled
by invoking /_xpack query and throws a specific error message.
1. Modifications to simple http client to return http response.
2. Changes in SetupPasswordTool for throwing appropriate exceptions.
2. Unit tests

relates elastic/x-pack-elasticsearch#3476

Original commit: elastic/x-pack-elasticsearch@eb0c5a19c5
2018-02-14 13:11:00 +11:00
Michael Basnight 495a0d83e5 Relax remote check for bwc project checkouts (elastic/x-pack-elasticsearch#3924)
Relates elastic/elasticsearch#28666

Original commit: elastic/x-pack-elasticsearch@7c67ee305a
2018-02-13 14:54:38 -06:00
Andy Bristol e520453551 Fix location of sql cli jar
The `elasticsearch` directory in the plugin folder structure was
removed, so make sure the CLI jar ends up in $ES_HOME/bin/x-pack as
intended

Follow up from elastic/x-pack-elasticsearch#28589

Original commit: elastic/x-pack-elasticsearch@6580b38fe0
2018-02-13 11:30:56 -08:00
Michael Basnight e10ffb4486 Remove snapshot conditional for bwc snapshots (elastic/x-pack-elasticsearch#3922)
Relates elastic/elasticsearch#28657

Original commit: elastic/x-pack-elasticsearch@0916a77179
2018-02-13 12:59:54 -06:00
Dimitris Athanasiou 2f4dcf36a9 [ML] Add notification for job updates coming from a user action (elastic/x-pack-elasticsearch#3890)
We were missing a notification for when a job is updated. This is
useful so users know that there's been changes which could justify
a change in the job behaviour.

In addition, having those notifications allows our integrations
tests to know when the update was processed which avoids having
to use `sleep()` with its instabilities.



Original commit: elastic/x-pack-elasticsearch@0b4eda2232
2018-02-13 18:46:00 +00:00
David Kyle 938cf239c9 [M] Clean noisy error message in Yaml tests (elastic/x-pack-elasticsearch#3923)
Original commit: elastic/x-pack-elasticsearch@5f0d4040dc
2018-02-13 17:05:27 +00:00
Adrien Grand c633b919ab Watcher's ExecutionService is not thread-safe. (elastic/x-pack-elasticsearch#3861)
If start, stop and/or clearExecutions are called concurrently, you could end up
in an inconsistent state where `currentExecutions.sealAndAwaitEmpty` is called
twice on the same instance.

These `synchronized` keywords should make it a bit better, though I suspect
there are other issues and thread safety of this class should be reviewed.

This might have caused
https://internal-ci.elastic.co/job/elastic+x-pack-elasticsearch+5.6+multijob-unix-compatibility/932/consoleFull

Original commit: elastic/x-pack-elasticsearch@2f7e7a34d9
2018-02-13 11:27:11 +01:00
Ryan Ernst 19e9afdf10 Test: Re-enable upgrade integ tests (elastic/x-pack-elasticsearch#3917)
This commit removes awaitsfixes for upgrade tests. The fix was
previously made on a branch and did not make it to master until now.

relates elastic/x-pack-elasticsearch#3729

Original commit: elastic/x-pack-elasticsearch@3402bbe78e
2018-02-12 20:46:22 -08:00
Igor Motov c82fdad41d SQL: Consolidate more type information into DataType (elastic/x-pack-elasticsearch#3850)
Consolidates type handling into DataType, makes DataType available to 
JDBC by moving to sql-proto and removes support for all parameter types 
that cannot be handled by the server.

Original commit: elastic/x-pack-elasticsearch@b8024f5c46
2018-02-12 16:40:09 -05:00
Tim Brooks 5dbbe8fef8 Default to basic license at startup (elastic/x-pack-elasticsearch#3878)
This is related to elastic/x-pack-elasticsearch#3877. This commit modifies the license settings to
default to self generating a basic license.

Original commit: elastic/x-pack-elasticsearch@cd6ee8e06f
2018-02-12 12:57:04 -07:00
Martijn van Groningen fde2adb1b3 increase watcher logging when delegating to trigger service
Original commit: elastic/x-pack-elasticsearch@448ee71f37
2018-02-12 16:12:50 +01:00
Alexander Reelsen d5281e89b7 Watcher: Register HTML sanitizer settings (elastic/x-pack-elasticsearch#3904)
The HTML sanitizer settings were not registered correctly, when the
grouping functionality was removed in elastic/x-pack-elasticsearch#3685

This registers the settings properly in watcher.

Relates elastic/support-dev-help-elasticsearch#413

Original commit: elastic/x-pack-elasticsearch@787bb3d9d3
2018-02-12 15:34:04 +01:00
Alexander Reelsen 54cb890eb7 Watcher: Allow to defer startup for tests (elastic/x-pack-elasticsearch#3903)
This commit introduces a new watcher setting to defer starting watcher
until it has been called with the API for the first time. This is
primarily useful in testing environments, as this ensures that watcher
does not try to reload itself because of starting first and then
creating watcher indices.

In addition the undocumented and unused option
xpack.watcher.start_immediately has been removed.

Relates elastic/x-pack-elasticsearch#3854

Original commit: elastic/x-pack-elasticsearch@2b55aec4ad
2018-02-12 14:24:05 +01:00
Alexander Reelsen 48f6a752cb Watcher: Fix proxy scheme to default to HTTP (elastic/x-pack-elasticsearch#3844)
This fixes a regression introduced in Elasticsearch 6.0, when switching
from HttpURLConnection to the Apache HTTP Client.

In the old implementation there was no way to specify if you wanted to use HTTP
or HTTPS for your proxy, only HTTP. If people needed to use HTTPs, they
could just use the CONNECT feature of the proxy.

The new implementation used the scheme of the request that was about to
be sent out as the proxy scheme to be used. So if the request was HTTPS
but the proxy server was HTTP this created a problem.

This commit changes the default scheme to be just HTTP, so that then the
standard CONNECT procecure is taken care off.

Without a real proxy server this is super hard to test. I have verified
this with the following test against a tinyproxy running on port 8888,
but I do not have a great idea how to test this in a unit testable way using a real proxy.

Original commit: elastic/x-pack-elasticsearch@f68e72d8f1
2018-02-12 13:12:38 +01:00
Alexander Reelsen f4fa274aa8 Tests: Do not set number of replicas for watcher index (elastic/x-pack-elasticsearch#3882)
The watcher index uses auto expanding of replicas nowadays, so there is
no need to set the number of replicas.

Also ensuring that all watcher indices are green on startup has been moved
to one call, in order to prevent indices being moved around after adding
the triggered watches or watch history index.

Original commit: elastic/x-pack-elasticsearch@dbe04adf64
2018-02-12 09:35:03 +01:00
Chris Earle 63c2d3e6db [Monitoring] Change MonitoringIT into ESIntegTestCase (elastic/x-pack-elasticsearch#3899)
Change MonitoringIT into ESIntegTestCase

Original commit: elastic/x-pack-elasticsearch@d06f6cc0b3
2018-02-10 12:37:17 -05:00
Shaunak Kashyap 7ee5ef333c Add fields to `.logstash`'s mapping in template (elastic/x-pack-elasticsearch#3872)
* Add fields to `.logstash`'s mapping in template

This "makes room" in the index for pipeline settings and node groups. Due to this change, users will be able to specify settings and node groups for a pipeline via the Centralized Config Management UI in Kibana. Logstash will only retrieve pipelines associated with the node group specified via the `xpack.management.group.id` setting in `logstash.yml`. For the retrieved pipelines, Logstash will apply any (optionally) specified pipeline settings before (re)loading the pipelines.

* Making field name more explicit + adding multi field for better search

Original commit: elastic/x-pack-elasticsearch@2df101f0b1
2018-02-09 15:18:37 -08:00
Nik Everett 45ac81127a SQL: Drop requirement to have keystore (elastic/x-pack-elasticsearch#3896)
SQL doesn't need the keystore. We required it by mistake.

Original commit: elastic/x-pack-elasticsearch@6840637feb
2018-02-09 17:17:46 -05:00
Michael Basnight 24c6a21d8f BWC changes for x-pack (elastic/x-pack-elasticsearch#3841)
relates elastic/elasticsearch#28505

Original commit: elastic/x-pack-elasticsearch@eda97ded76
2018-02-09 15:03:08 -06:00
Chris Earle 9c3257e158 [Monitoring] Register affixed Header Setting (elastic/x-pack-elasticsearch#3893)
This adds the `HEADERS_SETTING` as setting reported by the Monitoring plugin, which was changed in 6.2.

It also adds an IT to ensure that it is not missed in the future (existing integration tests were passing in the Settings and not being validated by the cluster!).

Original commit: elastic/x-pack-elasticsearch@d6f0b4b503
2018-02-09 15:53:27 -05:00
Ryan Ernst 4e578fe7cc Build: Replace provided configuration with compileOnly (elastic/x-pack-elasticsearch#3868)
This is the xpack side of elastic/elasticsearch#28564

Original commit: elastic/x-pack-elasticsearch@60033824a4
2018-02-09 11:30:43 -08:00
Costin Leau 3c6f9d4d29 SQL: fix name of Arithmetic functions (elastic/x-pack-elasticsearch#3884)
* SQL: fix name of Arithmetic functions

Remove id from name of Arithmetic function and improve name of Neg

Original commit: elastic/x-pack-elasticsearch@4f3e8d6a2d
2018-02-09 19:48:22 +02:00
Ioannis Kakavas 7504e33da7 Change value of METADATA_REFRESH for the tests (elastic/x-pack-elasticsearch#3879)
Ensure that the next triggered metadata refresh is always in the
future. relates elastic/x-pack-elasticsearch#3815

Original commit: elastic/x-pack-elasticsearch@edd9ba911d
2018-02-09 18:22:26 +01:00
Martijn van Groningen d2dc630e84 Changes required for elastic/elasticsearch#28588
Original commit: elastic/x-pack-elasticsearch@11d11c2d39
2018-02-09 17:14:30 +01:00
Lee Hinman dc833f2448 Remove all instances of the deprecated `ParseField.match` method (elastic/x-pack-elasticsearch#3874)
* Remove all instances of the deprecated `ParseField.match` method

This removes all the server references to the deprecated `ParseField.match`
method in favor of the method that passes in the deprecation logger.

Relates to https://github.com/elastic/elasticsearch/issues/28504

* Fix line-length

Original commit: elastic/x-pack-elasticsearch@e7cb2611f7
2018-02-09 09:01:18 -07:00
Alexander Reelsen 98e9365aa1 Tests: Increase logging for GetWatchTests
In order to get more insights into a specific test failure, that we
suppose to be in core, increase search logging to debug level.

Relates elastic/x-pack-elasticsearch#3854

Original commit: elastic/x-pack-elasticsearch@5c37535e03
2018-02-09 16:37:33 +01:00
Albert Zaharovits fb8adb4a1c [Security][Tests] Azeri(Turkish) locale tripps opensaml dependency
Original commit: elastic/x-pack-elasticsearch@9c4ecabe48
2018-02-09 13:57:09 +02:00
Ryan Ernst 8d6f402ac9 Fix silly monitoring test checking exact output
Original commit: elastic/x-pack-elasticsearch@58c23551f6
2018-02-08 14:39:03 -08:00
Ryan Ernst a2c9c97650 Fix usage of PluginInfo
See https://github.com/elastic/elasticsearch/pull/28556

Original commit: elastic/x-pack-elasticsearch@1e14b14526
2018-02-08 09:51:20 -08:00
David Kyle d6ca53cd26 [ML] Fix exception categorising an empty string (elastic/x-pack-elasticsearch#3870)
Original commit: elastic/x-pack-elasticsearch@1840a74415
2018-02-08 16:18:48 +00:00
David Kyle 8e73085047 [ML] Enable adding multiple jobs to a calendar (elastic/x-pack-elasticsearch#3786)
Original commit: elastic/x-pack-elasticsearch@56a70a4580
2018-02-08 11:44:16 +00:00
Albert Zaharovits a57999e1e1 [Security][Tests] Print LDAP reconnect stacktrace on fail
Original commit: elastic/x-pack-elasticsearch@18f235197c
2018-02-08 13:41:02 +02:00
markharwood bce8d6f9e6 Test: Remove off-by-one test logic that fails every one in several thousand tests. relates elastic/x-pack-elasticsearch#270 and elastic/x-pack-elasticsearch#869 (which is also now re-enabled).
relates elastic/x-pack-elasticsearch#270

Original commit: elastic/x-pack-elasticsearch@2e52efd601
2018-02-08 10:48:59 +00:00
David Roberts f4b32bef8c [ML] Reject headers supplied directly in the put datafeed body (elastic/x-pack-elasticsearch#3853)
Since elastic/x-pack-elasticsearch#3254 security headers have been stored in datafeed cluster state
to allow the datafeed to run searches using the credentials of the user
who created/updated it.  As a result the parser was changed to read the
"headers" field so that cluster state could be reloaded.  However, this
meant that datafeed configs could be submitted with a "headers" field.
No security loophole arose from this, as subsequent code overwrites the
contents of any supplied headers.  But it could be confusing that an
erroneously supplied field did not cause a parse failure as it usually
would.

This change makes the config parser for datafeeds reject a "headers"
field.  Now only the metadata parser used for reloading cluster state
will read a "headers" field.

Original commit: elastic/x-pack-elasticsearch@afa503275f
2018-02-08 10:43:33 +00:00
Alexander Reelsen 064a0819d9 Cleanup: Remove HaltedClock (elastic/x-pack-elasticsearch#3731)
java.time features it's own halted clock, called a fixed clock, we can
use that one.

On top of that the watcher xcontent parser does not need a clock at all,
just a timestamp when parsing happened.

Original commit: elastic/x-pack-elasticsearch@2061aeffe1
2018-02-08 09:24:08 +01:00
Tim Vernum 3102b94946 [TEST] Don't fail IndexAuditTrailTests if the test runs very fast
Relates: elastic/x-pack-elasticsearch#3472

Original commit: elastic/x-pack-elasticsearch@00fde8973f
2018-02-08 16:44:58 +11:00
Ryan Ernst 7d71c3f1cc Build: Remove refs to x-pack-elasticsearch (elastic/x-pack-elasticsearch#3863)
These were removed before, but sql added a bunch of new hard references
to the project name.

Original commit: elastic/x-pack-elasticsearch@2eb156fd67
2018-02-07 20:50:12 -08:00
Ryan Ernst 8ee3de122b Build: Remove x-pack-api jar (elastic/x-pack-elasticsearch#3864)
The api jar was added for xpack extensions. However, extensions have
been removed in favor of using SPI, and the individual xpack jars like
core and security are published to enable this. This commit removes the
api jar, and switches the transport client to use the core jar (which
the api jar was just a rename of).

Original commit: elastic/x-pack-elasticsearch@58e069e66c
2018-02-07 19:21:43 -08:00
Ryan Ernst f360d2d8d7 Build: Cleanup gradle files (elastic/x-pack-elasticsearch#3867)
There were a number of leftover unnecessary elements in the module
build.gradle files that were holdovers from copying the original plugin
build.gradle. This commit removes these elements.

Original commit: elastic/x-pack-elasticsearch@08babbd520
2018-02-07 16:58:09 -08:00
Martijn van Groningen eb2fce9d54 Changes required for elastic/elasticsearch#28531
Original commit: elastic/x-pack-elasticsearch@e77398e93f
2018-02-07 17:45:33 +01:00
Nik Everett b95745956e Merge branch 'master' into feature/sql_2
Original commit: elastic/x-pack-elasticsearch@5f6bbb588a
2018-02-07 09:38:13 -05:00
Tim Vernum 7bb2cba14f [Security] Reset IndexAuditTrail to INITIALISED before start (elastic/x-pack-elasticsearch#3807)
Calling start() when already in the STARTING state doesn't do anything, so the component
gets stuck in STARTING state forever.

Also: wait on the required index name not just the cluster.

Also: added more logging to help diagnose such issues (either in RemoteIndexAuditTrailStartingTests or production)

Original commit: elastic/x-pack-elasticsearch@fb81214fe7
2018-02-07 12:30:23 +11:00
Nik Everett 10efd82694 SQL: Shift sql-cli into its proper home
The old home made sense before x-pack was split. The new home lines up
with where security keeps its scripts.

The jar file wasn't being included any more so this re-adds that.

Original commit: elastic/x-pack-elasticsearch@d3ec941397
2018-02-06 16:17:18 -05:00
Nik Everett a2f1ed3f62 SQL: Fix javadoc in JdbcConfiguration
It wasn't building....

Original commit: elastic/x-pack-elasticsearch@059c6397a2
2018-02-06 14:41:06 -05:00
Nik Everett 5b46899cc3 Merge branch 'master' into feature/sql_2
Original commit: elastic/x-pack-elasticsearch@9a49ecbbd5
2018-02-06 14:40:55 -05:00
Jay Modi 18c04c714d [Security] handle null values when attempting to parse as DN when mapping roles (elastic/x-pack-elasticsearch#3791)
This commit adds special handling for null values when building a DistinguishedNamePredicate for
role mapping. Previously this would have resulted in an exception from the unboundid ldapsdk as
the DN is invalid.

relates elastic/x-pack-elasticsearch#3787

Original commit: elastic/x-pack-elasticsearch@9386dae03e
2018-02-06 10:19:49 -07:00
Lee Hinman b2b95cb25d [TEST] Fix compilation for XContent parser
Original commit: elastic/x-pack-elasticsearch@bf03cc34fe
2018-02-05 16:56:36 -07:00
Lee Hinman 5a3004300c SQL: Add some Javadocs (elastic/x-pack-elasticsearch#3816)
* SQL: Add some Javadocs

* Reflow javadoc for fewer columns

Original commit: elastic/x-pack-elasticsearch@d23239b2e5
2018-02-05 09:03:06 -07:00
Nik Everett 876aebf7e0 SQL: Make extract work for any datetime function (elastic/x-pack-elasticsearch#3756)
This allows any datetime function to be present in `EXTRACT` which feels
more consistent. `EXTRACT(FOO FROM bar)` is now just sugar for
`FOO(bar)`. This is *much* simpler to explain in the documentation then
"these 10 fields are supported by extract and they are the same as this
subset of the datetime functions."

The implementation of this is a little simpler then the old way. Instead
of resolving the function in the parser we create an
`UnresolvedFunction` that looks *almost* just like what we'd create for
a single argument function and resolve the function in the `Analyzer`.
This feels like a net positive as it allows us to group `EXTRACT`
resolution failures with other function resolution failures.

This also creates `UnresolvedFunctionTests` and
`UnresolvedAttributeTests`. I had to create `UnresolvedFunctionTests`
because `UnreolvedFunction` now has three boolean parameters which is
incompatible with the generic `NodeSubclassTests`'s requirement that all
ctor parameters be unique. I created `UnresolvedAttributeTests` because
I didn't want `UnresolvedFunctionTests` to call `NodeSubclassTests` and
figured that we'd want `UnresolvedAttributeTest` eventually and now felt
like as good a time as any.

Added a 

Original commit: elastic/x-pack-elasticsearch@358aada308
2018-02-03 16:10:09 -05:00
Costin Leau 34fe0beb30 SQL: add comments on JDBC thread-safety (or rather lack of)
Original commit: elastic/x-pack-elasticsearch@6ea5adc6a8
2018-02-02 18:11:24 +02:00
Costin Leau bda97c6e59 SQL: More polish
Original commit: elastic/x-pack-elasticsearch@cedbad466b
2018-02-02 18:01:51 +02:00
Costin Leau b5dd4c649a SQL: Minor fixes
Original commit: elastic/x-pack-elasticsearch@14ea747e20
2018-02-02 17:50:33 +02:00
Costin Leau 817c4759c6 Remove unused classes
Original commit: elastic/x-pack-elasticsearch@f7aee26821
2018-02-02 17:14:54 +02:00
Nik Everett f66e01369a Merge branch 'master' into feature/sql_2
Original commit: elastic/x-pack-elasticsearch@6ba2651f46
2018-02-01 17:21:37 -05:00
David Turner a1cb0911d3 Allow use of LocalDateTime#atZone() in DateTimeFormatterTimestampConverter (elastic/x-pack-elasticsearch#3811)
Adds a `@SuppressForbidden` annotation to permit use of a
trappy method from `java.time`, pending resolution of elastic/x-pack-elasticsearch#3810.

X-pack component of elasticsearch#28476.

Original commit: elastic/x-pack-elasticsearch@c5e1ae0970
2018-02-01 14:33:23 +00:00
Martijn van Groningen ee834026d1 Merge changes related to moving persistent tasks to elasticsearch repo into master branch.
Original commit: elastic/x-pack-elasticsearch@fb61e4853c
2018-02-01 15:31:57 +01:00
Dimitrios Athanasiou 05515b370a [ML][TEST] Increase timeout in MlDailyManagementServiceTests
relates elastic/x-pack-elasticsearch#2916

Original commit: elastic/x-pack-elasticsearch@2925dc45ea
2018-02-01 14:09:33 +00:00
Tim Vernum 415bb7f039 Clear Realm Caches on role mapping health change (elastic/x-pack-elasticsearch#3782)
If any of the follow take place on security index, then any cached role mappings
are potentially invalid and the associated realms need to clear any cached users.
- Index recovers from red
- Index is deleted
- Index becomes out-of-date / not-out-of-date

Original commit: elastic/x-pack-elasticsearch@1bcd86fcd4
2018-02-01 10:01:57 +11:00
Lee Hinman 1b36133988 Merge remote-tracking branch 'origin/master' into feature/sql_2
Original commit: elastic/x-pack-elasticsearch@4176773659
2018-01-31 10:49:25 -07:00
Tal Levy a627fec53e fix timeout usage in _bulk to `timeout` from `master_timeout` (elastic/x-pack-elasticsearch#3796)
The HTTP Exporter in Monitoring allowed users to set a timeout parameters
for the requests. When set, this was setting the `master_timeout` query parameter
in Bulk Requests. The problem is that Bulk Requests do not support this type of
timeout.

Original commit: elastic/x-pack-elasticsearch@9be194006e
2018-01-31 09:13:46 -08:00