Commit Graph

1310 Commits

Author SHA1 Message Date
Alexander Reelsen 7557168a0a Smoke tester: Adapt to new download structure (elastic/elasticsearch#3575)
This makes the smoke tester in x-pack work again after the work
on the unified release.

Original commit: elastic/x-pack-elasticsearch@4d4f1ec26c
2016-09-23 09:53:17 +02:00
Simon Willnauer 3c650e483b Followup from elastic/elasticsearchelastic/elasticsearch#20627 - Removal of AnalysisService
Original commit: elastic/x-pack-elasticsearch@75c14534a7
2016-09-23 08:54:44 +02:00
Tanguy Leroux 375bf95fb1 Remove duplicate methods in ByteSizeValue (elastic/elasticsearch#20560)
Some methods have been renamed in elastic/elasticsearchelastic/elasticsearch#20560. This commit change a .bytes() call to a .getBytes() call.

Original commit: elastic/x-pack-elasticsearch@4a0ff77361
2016-09-20 14:07:02 +02:00
Tanguy Leroux 1a7fbf9679 [Tests] Fix cat.templates to work when other templates exist
This commit fixes the cat.templates REST tests so that it works when other templates exist (like monitoring)

Original commit: elastic/x-pack-elasticsearch@2e27ad88b4
2016-09-20 12:05:21 +02:00
Ryan Ernst 803f20e267 Merge pull request elastic/elasticsearch#3534 from rjernst/license_prod_key
Build: Use licensing prod key when building release

Original commit: elastic/x-pack-elasticsearch@682c4eace5
2016-09-19 15:48:43 -07:00
Ryan Ernst 39160b5b22 Build: Use licensing prod key when building release
This change switches the build to use the licensing prod key when
building the xpack jar for release.

Original commit: elastic/x-pack-elasticsearch@54a21dae5b
2016-09-19 15:46:04 -07:00
Chris Earle a94c27d3de [Monitoring] Future-proof Monitoring Bulk API with "interval" param
This adds an "interval" placeholder parameter that is required to the Monitoring Bulk API, and adds it to the Kibana side of the plumbing.

Having this will allow us to add it to all incoming documents and start to report against it with the Insights, as well as to detect the _lack_ of incoming documents.

By adding it now, we can avoid having a non-BWC API change for Kibana in 5.1. We'll just pickup new data in our documents.

Original commit: elastic/x-pack-elasticsearch@5ba8aafe03
2016-09-19 18:21:09 -04:00
Simon Willnauer 2dde85ab33 Unguice Transport and friends (elastic/elasticsearch#3510)
This is a followup for elastic/elasticsearchelastic/elasticsearch#20526 removes the pluggability of
transport / http server transport via guice.

Original commit: elastic/x-pack-elasticsearch@5fb84949aa
2016-09-19 22:11:17 +02:00
Ryan Ernst 2a03af9849 Build: Remove x-pack client jar
The x-pack client jar and api jar are exactly the same: the entirety of
x-pack. Since we added the x-pack-transport jar, we no longer really
want the client jar as it is confusing. Additionally, it causes jar hell
when a test for an extension uses the transport client. This change
removes the client jar, and makes the x-pack transport client use the
api jar instead. This sounds odd at first, but since transport client is
going away eventually, it is a stopgap, and works.

closes elastic/elasticsearch#3309

Original commit: elastic/x-pack-elasticsearch@ee7a2c12c0
2016-09-19 11:42:27 -07:00
Alexander Reelsen 8c51b3b21a Docs/Release notes: Removed `force` parameter in delete watch API
Also updated the rest test descriptions

Original commit: elastic/x-pack-elasticsearch@890ba7703e
2016-09-19 11:49:10 +02:00
Alexander Reelsen 273a9fb46f Watcher: Fix possible chained input NPE for execution failures (elastic/elasticsearch#3490)
Due to untested code there was an NPE happening in production,
when a chained input execution failed, but the chained input
tried to access the resulting payload (which is never set on
failures). This payload now defaults to being empty.

This commit also drive-by fixes a broken logging statement, that
on the one side returned not the watch id, but a useless watch
toString() representation, and on the other hand only logs an error
message, but not a stack trace into the log, as this is what the
history is for.

Original commit: elastic/x-pack-elasticsearch@7dbe1afd90
2016-09-19 10:06:02 +02:00
Alexander Reelsen 5b265ea569 Watcher: Remove locking of watches for write operations elastic/elasticsearch#3481 (elastic/elasticsearch#3481)
Whenever a watch is updated (put, delete, set state), until now we
happened to reject those operations when a watch was executed at the
same time. However with long running reporting this might mean, that a
watch can never be changed, because it always gets executed.

* Removes the ability of write requests to obtain a lock at all (executing watches is still protected by a lock)
* Replaced the FairKeyedLock in watcher with the KeyedLock in Elasticsearch, which also takes a fair option, removed the FairKeyedLock
* Removed all the timeout parameters that are no longer needed, because there is no lock anymore
* Removed also the force parameter for watch deletion. Just do it[tm]
* Added a test that deleting a watch while it is being executed does not leave any leftovers

In case of a deletion of a watch during an execution, so that updating the status of the watch fails,
a warning is logged.

Closes elastic/elasticsearch#3417

Original commit: elastic/x-pack-elasticsearch@22fad1b797
2016-09-19 09:44:32 +02:00
jaymode e069c1f090 test: stop IndexAuditTrail before ESIntegTestCase#after runs
As part of the review of elastic/elasticsearch#3287, the stopping of the IndexAuditTrail was moved to the tearDown
method. This works sometimes but other times it fails because tearDown is run after
ESIntegTestCase#after, so the IndexAuditTrail is still running during the after checks which will
cause the test to fail since the shard lock cannot be obtained.

Closes elastic/elasticsearch#3520

Original commit: elastic/x-pack-elasticsearch@4cb52b15a2
2016-09-17 07:38:31 -04:00
Simon Willnauer 7be765d2a0 Ensure we have a consistent view on OperationMode in Watcher (elastic/elasticsearch#3507)
Today the operation mode can be set to default for a short amout of
time until it's reset to the actual mode this can cause weird sideeffects
for users if it's read concurrently. Also the test relies on a certain
happens before relationship that is not guaranteed since the operation
mode is set before the listerner is run. This change also rewrites the test
to not use busy waiting but wait for the actual listern to be executed.

Original commit: elastic/x-pack-elasticsearch@a2a42b89e5
2016-09-16 22:10:51 +02:00
Nik Everett c21a922778 Add backwards compatibility support to monitoring
1. We only support indexes created by Marvel 2.3+. All other indexes
are just ignored.
2. The tests don't assert a ton of interesting stuff because there
isn't a java API for Monitoring that we can just use. Instead we assert
that a few objects are there and look sane.
3. We don't migrate the contents of the data index. Instead we just
rely on Monitoring recreating it.

Original commit: elastic/x-pack-elasticsearch@86216c2d61
2016-09-16 13:20:02 -04:00
Jay Modi a6d55f26c6 security: simplify index audit trail stopping
The IndexAuditTrail had both a stop and close method that needed to be called in order
to stop the service. There was a race where we called either flush or close in a non
blocking fashion and then immediately closed the underlying client. This change makes
the stop method wait for up to 10 seconds when closing the bulk processor.

Closes elastic/elasticsearch#3279

Original commit: elastic/x-pack-elasticsearch@0d776bc91a
2016-09-16 10:31:27 -04:00
Simon Willnauer efeb9cefce Cut over SecurityServerTransportService to use the new Interceptor infrastructure (elastic/elasticsearch#3491)
TransportService is not pluggable anymore in core. Instead we now have a interceptor
infrastructure that allows to intercept send and receive calls on the transport layer.

Relates to elastic/elasticsearchelastic/elasticsearch#20505

Original commit: elastic/x-pack-elasticsearch@04194ecb09
2016-09-16 09:48:25 +02:00
Boaz Leskes 635b5a6800 fix import for ElectMasterService
it changed with https://github.com/elastic/elasticsearch/pull/20384

Original commit: elastic/x-pack-elasticsearch@c1e51de6a8
2016-09-15 23:45:24 +02:00
Alexander Reelsen efa2678691 Watcher: Add proxy support to hipchat action (elastic/elasticsearch#3475)
This adds proxy support to the hipchat action. Right now
neither hipchat nor slack nor pagerduty allow for this,
but if you dont need a proxy for internal http connections,
but you do for external, then this configuration cannot be done
without setting a proxy for those actions.

You can set it like this in the JSON

```
"actions" : {
  "notify-hipchat" : {
    "hipchat" : {
      "account" : "integration-account",
      "proxy" : {
        "host" : "localhost",
        "port" : 8080
      },
      "message" : {
        ...
      }
    }
  }
}

```

Relates elastic/elasticsearch#3372

Original commit: elastic/x-pack-elasticsearch@4e8447ce37
2016-09-15 11:24:04 +02:00
Yannick Welsch d35131ad52 Fix wrong logger usages (elastic/elasticsearch#3485)
Relates to elastic/elasticsearchelastic/elasticsearch#20490

Original commit: elastic/x-pack-elasticsearch@1ae7217b59
2016-09-15 10:48:06 +02:00
Jason Tedor b534bfec5a Complete Elasticsearch logger names
This commit modifies the loggers used within x-pack to adapt to the
change in core Elasticsearch where loggers are now named by the
fully-qualified class name.

Relates elastic/elasticsearch#3465

Original commit: elastic/x-pack-elasticsearch@4cac9ac62f
2016-09-13 22:47:10 -04:00
Chris Earle 6085c5aba9 [Monitoring UI] Cleanup Cluster Overview and Node Summary
This cleans up some of the reported stats to be a little clearer, including making the JVM Heap chart behave like Kibana's memory chart. This solves two problems: you can now determine the max heap size and you know what "x%" actually means relative to it.

Original commit: elastic/x-pack-elasticsearch@450f6fd546
2016-09-13 18:58:01 -04:00
Jason Tedor 6d4e4f5131 Fix failing logging audit tests
This commit fixes the logging audit tests which were broken due to an
upstream change in core Elasticsearch relating to the fact that prefixes
are no longer considered part of the log message, but are instead
implemented via markers.

Original commit: elastic/x-pack-elasticsearch@abd7ec23d8
2016-09-13 17:43:19 -04:00
Britta Weber 2c6d0b0cd2 Add option to deny access to fields (elastic/elasticsearch#2879)
To deny access to a fields users can name exceptions to field permissions with the following syntax:

"fields": {
     "grant": [list of field names patterns],
     "except": [list of patterns that are forbidden]
}

See doc for the rules for this.

This commit also reverts elastic/elasticsearch#2720

closes elastic/elasticsearch#2681



Original commit: elastic/x-pack-elasticsearch@d6537028ec
2016-09-13 16:38:58 +02:00
Martijn van Groningen 3b97936587 test: Remove WatcherBackwardsCompatibilityTests as it was specifically build for testing upgrade from 2.x to 5.x and to verify the scripts and template work/serialize as expected. On the master this is test is no longer relevant.
The OldWatcherIndicesBackwardsCompatibilityIT covers a major upgrade too.

Original commit: elastic/x-pack-elasticsearch@657881916b
2016-09-13 09:49:53 +00:00
Alexander Reelsen 67f7da18da Licensing: Parse start date in milliseconds as well
In order to behave like expiration date and and the issue date,
license parsing should be able to parse the start date in milliseconds
as well.

Relates elastic/elasticsearch#3385

Original commit: elastic/x-pack-elasticsearch@54c821192e
2016-09-13 09:57:29 +02:00
Simon Willnauer 3c619e8824 Follow up for elastic/elasticsearchelastic/elasticsearch#20423
Original commit: elastic/x-pack-elasticsearch@53c72d913a
2016-09-12 22:51:30 +02:00
jaymode 2358309f72 security: allow enabled and username fields in put user request body
The enabled and username fields are both now allowed in the request body for the put user
request. This makes it easier to perform a get and update a user without needing to edit more
of the request body than necessary.

Closes elastic/elasticsearch#3391

Original commit: elastic/x-pack-elasticsearch@ab763e843b
2016-09-12 16:14:57 -04:00
jaymode 5f4e6164e5 security: add a built-in role for reporting
This commit adds a built-in role that grants read and write privileges to the reporting
indices.

See elastic/elasticsearch#2374
Closes elastic/elasticsearch#3196

Original commit: elastic/x-pack-elasticsearch@c8c1b465f8
2016-09-12 16:04:08 -04:00
jaymode 7a321534ea security: allow the reserved realm to be disabled
This change allows the reserved realm to be disabled via a setting that is undocumented.

Closes elastic/elasticsearch#3399

Original commit: elastic/x-pack-elasticsearch@3c6c93d7eb
2016-09-12 14:13:47 -04:00
jaymode a119f7ccf1 update the HttpClient to allow for disabling hostname verification
The HttpClient used in xpack supports the new SSL configuration but did not properly obey
the hostname verification disabling that can be specified with these settings. This change
adds the functionality with a test.

Relates elastic/elasticsearch#3240
Relates elastic/elasticsearch#3164

Original commit: elastic/x-pack-elasticsearch@df8e12a5aa
2016-09-12 13:47:08 -04:00
jaymode 6d2fcbe688 security: fix typo is syskeygen script
Original commit: elastic/x-pack-elasticsearch@36abb4284a
2016-09-12 12:26:14 -04:00
Alexander Reelsen 0c3466180f Licensing: Add start date to licenses (elastic/elasticsearch#3385)
Start dates are a required feature for cloud. This functionality adds support
for specifying and enforcing a start date on licenses.

Behaviour: If the start date is > than now, the license will be rejected.

Due to another field in the license class, the version of the License class as well
as its serialization methods are adapted to this.

Closes elastic/elasticsearch#3370

Original commit: elastic/x-pack-elasticsearch@eb2a6f5be3
2016-09-12 17:53:33 +02:00
Nik Everett caf4bd2c82 Be careful when old index tests start nodes
We were starting nodes at weird times and then shutting them down again,
slowing down the tests and causing the watcher tests to fail because
watcher wasn't being shut down with its traditional kid gloves.

Original commit: elastic/x-pack-elasticsearch@2fd81b3eaf
2016-09-12 11:14:51 -04:00
Alexander Reelsen 2a6a9a10f7 Watcher: Improve http attachment history information (elastic/elasticsearch#3436)
When the HTTP attachment was not able to successfully retrieve the
data from and endpoint, there was no indication in the watch history
of what went wrong. Instead a logger was used, which is not useful
for the person running the watches.

This commit removes the logger statement and throws an exception,
so that the exception message can be stored in the watch history.

Source of this issue was a forum post:
https://discuss.elastic.co/t/sending-e-mail-with-generated-report-fails/60263/6

Original commit: elastic/x-pack-elasticsearch@acdaf7abef
2016-09-12 17:04:22 +02:00
Martijn van Groningen 5c8ece8583 test: mute test
Original commit: elastic/x-pack-elasticsearch@112f853bac
2016-09-10 15:10:41 +02:00
javanna 69c4c693f6 Merge branch 'enhancement/cleanup_parse_elements'
Original commit: elastic/x-pack-elasticsearch@0f86204352
2016-09-09 22:47:29 +02:00
Nik Everett 2aa0781971 Explicitly shut down watcher after tests
In our tests you have to explicitly shut down watcher rather than shut
down the node it is running on because of thread leak detection. Just
shutting down the node that it is running on will cause it to start up
on another node if there is another one running and then not properly
shut down. This is probably something that should be fixed in watcher
somehow but for now lets just be more careful with the tests.

Closes elastic/elasticsearch#2365
Closes elastic/elasticsearch#2588

Original commit: elastic/x-pack-elasticsearch@fb8a172972
2016-09-09 14:43:46 -04:00
javanna 5829d112d1 Move search ext section parsing to the coordinating node
Original commit: elastic/x-pack-elasticsearch@1ce38fcff9
2016-09-09 18:58:10 +02:00
Jason Tedor 592ca5ab7b Adjust to logger API changes in Elasticsearch
Previously core Elasticsearch had methods in a test class for removing
and adding appenders. However, these methods were moved to production
code. This commit adjusts x-plugins for this change.

Original commit: elastic/x-pack-elasticsearch@83e37ef65a
2016-09-09 09:15:53 -04:00
Martijn van Groningen b74f1e6cb2 watcher: Add limited capability to upgrade the source of a watcher upon startup.
This particular change focuses on upgrading the source of a watch when it comes to scripts that have no language specified explicitly.
The default language in version 5 changed to painless from whatever is specified in `script.default_lang` setting (this defaulted to groovy). In order to make sure that scripts in watcher remain to work we should rewrite the search source upon startup and set the legacy default language explicitly. The legacy script language is now controlled by `script.legacy.default_lang` setting and that defaults to groovy.

Changing the source upon startup should do the trick and only change the source of watches with scripts that don't have an explicit language set. For new watches the default language used in scripts is painless and because we now always serialize the language explicitly in scripts these watches won't be changed on startup.

 The upgrade logic added here tries to upgrade scripts in the following places in a watch:
 * script condition
 * script transform
 * any script defined inside of a search input

Original commit: elastic/x-pack-elasticsearch@4d578819eb
2016-09-09 14:26:24 +02:00
Britta Weber 4bf685cd31 [TEST] wait for pending tasks finished before testing cluster state collector
Original commit: elastic/x-pack-elasticsearch@d42c8c4bbe
2016-09-09 12:07:52 +02:00
Tanguy Leroux 6ca086b997 Fix line length in LoggingAuditTrailTests.java
Original commit: elastic/x-pack-elasticsearch@3f4185dce7
2016-09-09 10:50:43 +02:00
Colin Goodheart-Smithe 8b6d988cfb Security: Audit all HTTP requests
Adds a new audit event (authentication_success) which logs each request made to
the REST API along with the body of the request

Closes elastic/elasticsearch#912

Original commit: elastic/x-pack-elasticsearch@650b9d70c0
2016-09-09 09:28:27 +01:00
Simon Willnauer 923949ec67 Downgrade hard bwc break to a TODO for now
Original commit: elastic/x-pack-elasticsearch@73c2a6ee3e
2016-09-08 22:05:36 +02:00
Simon Willnauer 5c407cc526 Revert "Remove bwc layer for TimeUnit parsing"
This reverts commit elastic/x-pack@c6b2bd2c12.

Original commit: elastic/x-pack-elasticsearch@b4fbfb8afe
2016-09-08 22:03:46 +02:00
Jason Tedor 9baf88dfaa Handle node validation exception when starting
Node#start can now throw a checked NodeValidationException; this commit
adapts x-plugins for this new API.

Relates elastic/elasticsearch#3381

Original commit: elastic/x-pack-elasticsearch@660c5ab8c1
2016-09-08 10:56:54 -04:00
Simon Willnauer 20fce3280e Remove bwc layer for TimeUnit parsing
Original commit: elastic/x-pack-elasticsearch@79145f279b
2016-09-08 16:43:46 +02:00
Tanguy Leroux 97182fefb9 Clean up XContentBuilder in X-Pack
This commit cleans most of the methods of XContentBuilder so that:
    - Jackson's convenience methods are used instead of our custom ones (ie field(String,long) now uses Jackson's writeNumberField(String, long) instead of calling writeField(String) then writeNumber(long))
    - null checks are added for all field names and values
    - methods are grouped by type in the class source
    - methods have the same parameters names
    - duplicated methods like field(String, String...) and array(String, String...) are removed
    - varargs methods now have the "array" name to reflect that it builds arrays
    - unused methods like field(String,BigDecimal) are removed
    - all methods now follow the execution path: field(String,?) -> field(String) then value(?), and value(?) -> writeSomething() method. Methods to build arrays also follow the same execution path.

Original commit: elastic/x-pack-elasticsearch@d83f3aa6e2
2016-09-08 14:17:15 +02:00
jaymode 2257cc77f6 security: add reserved metadata to the ingest admin role
Closes elastic/elasticsearch#3376

Original commit: elastic/x-pack-elasticsearch@1909da2fb0
2016-09-07 19:19:02 -04:00
Alexander Reelsen 0f571685b9 Watcher: Fix proxy xcontent serialization (elastic/elasticsearch#3364)
Calling to xcontent in a HttpRequest, with a proxy enabled, lead to
serialization exceptions, resulting in failing to write the watch
history.

Closes elastic/elasticsearch#3334

Original commit: elastic/x-pack-elasticsearch@a04dff686c
2016-09-07 16:29:15 +02:00
Alexander Reelsen a296e31a7c Watcher: Ensure triggered watch is deleted on thread pool rejection (elastic/elasticsearch#3049)
This fixes a bug I found with a customer when he updated from 1.x to 2.x.
Due to an BWC incompatible change in the watch history mapping and a thread
pool rejection during execution a watch was not removed from the triggered
watches and tried to be executed again.

While trying to fix it it turned out that the execution of the failure
test case was still done in the transport thread and thus required some
offloading to another thread pool.

Original commit: elastic/x-pack-elasticsearch@df04ce31f2
2016-09-07 15:55:33 +02:00
Alexander Reelsen bb033f1e00 Watcher: Clear out WatchStore on watch index deletion (elastic/elasticsearch#2807)
If someone deletes the watch index (i.e. by deleting all indices), the watcher
in memory store still contains all the watches and tries to execute watches -
which results in exceptions as the watch itself cannot be updated anymore.

In order to minimize this problem (it cant be get rid of completely), we should
act accordingly if the watch index goes missing (either deleted or closed) and
clear out the memory representation of watches in the watchstore as well as trying
to finish all the current executions.

Closes elastic/elasticsearch#2794

Original commit: elastic/x-pack-elasticsearch@12d98cd566
2016-09-07 15:06:03 +02:00
javanna ad5606d1f7 [TEST] don't use null script lang in WatcherUtilsTests
same as elastic/x-pack@9e1e0988c1 but for testSerializeSearchRequest this time.

Original commit: elastic/x-pack-elasticsearch@aa7e3814b6
2016-09-06 23:00:56 +02:00
Nik Everett edb8c12f75 Always use mustache language
Using a null language just defaults to painless which is wrong for
watcher.

Original commit: elastic/x-pack-elasticsearch@48bdc31695
2016-09-06 15:45:23 -04:00
Nik Everett 05222b0d3f Deal with ConcreteShardRequest
It exists now and the intercepters have to deal with it.

Original commit: elastic/x-pack-elasticsearch@fdef0578ed
2016-09-06 15:42:06 -04:00
Martijn van Groningen 36486a7993 Changes for PR: https://github.com/elastic/elasticsearch/pull/20310
Original commit: elastic/x-pack-elasticsearch@9db3d05274
2016-09-06 18:44:41 +02:00
jaymode 7965608add security: make control of logfile audit output consistent with index output
This change moves the logfile audit output from determining what to log based on the
logger level to a enum based configuration that is used by the index output.

A few notable changes were made:

* We alway log all the information we have except for the request body
* The request body is no longer logged by default for REST events; the user needs to
explicitly opt in as there could be sensitive data in the body
* Added a `realm_authentication_failed` event that separates overall authentication
failure from that of an individual realm

Original commit: elastic/x-pack-elasticsearch@343a2bcdd9
2016-09-06 10:50:33 -04:00
Nik Everett 89ce4ebb08 Fix remaining fractional TimeValue issues with watcher
Removes the remaining spots where watcher makes fractional TimeValues.

Closes elastic/elasticsearch#3231

Original commit: elastic/x-pack-elasticsearch@22b0d37ed3
2016-09-06 09:53:22 -04:00
jaymode cd9add5350 test: remove unnecessary verify check
Original commit: elastic/x-pack-elasticsearch@f748c0a565
2016-09-06 08:36:04 -04:00
jaymode 1e6a924e74 security: add support for disabling users
This change adds support for disabling users. Users can be disabled by setting the enabled
property to false and the AuthenticationService will check to make sure that the user is enabled.
If the user is not enabled, this will be audited as an authentication failure.

Also as part of this work, the AnonymousUser was cleaned up to remove having a static instance
that caused issues with tests.

Finally, the poller of users was removed to simplify the code in the NativeUsersStore. In our other
realms we rely on the clear cache APIs and the timeout of the user cache. We should have the
same semantics for the native realm.

Closes elastic/elasticsearch#2172

Original commit: elastic/x-pack-elasticsearch@0820e40183
2016-09-06 07:41:48 -04:00
Chris Earle ec008ec4a9 Looong lines
Original commit: elastic/x-pack-elasticsearch@8c7931c179
2016-09-05 19:05:28 -04:00
Chris Earle 5d5a9afb57 [Monitoring] Use Low Level REST Client for HTTP Exporter
This rewrites the HTTP Exporter to use the REST client underneath. Functionality is improved in resource blocking (templates and pipelines existing) and the majority of the code fundamentall simplified by removing direct HTTP calls.

This is blocked by the SSLService pull request. After that is merged, the I will update this PR to reflect those changes and it could possibly allow us to remove the security privileges required for monitoring.

Original commit: elastic/x-pack-elasticsearch@1ad25f17f8
2016-09-05 15:16:03 -04:00
Alexander Reelsen 41334abda0 Watcher: Reenabling HipChatServiceTests
The authentication token stopped working, deleted it and created a new one
on the hipchat web site. Tests are working now again.

Closes elastic/elasticsearch#3162

Original commit: elastic/x-pack-elasticsearch@a0623af799
2016-09-05 12:27:14 +02:00
Martijn van Groningen 09c7f534b3 added jdocs and rename
Original commit: elastic/x-pack-elasticsearch@8a97b420c1
2016-09-03 11:01:47 +02:00
Nik Everett 3c33879018 Mark OldWatchIndices test AwaitsFix
Is freaking out in CI.

Original commit: elastic/x-pack-elasticsearch@3ab43467ed
2016-09-02 18:31:19 -04:00
Martijn van Groningen 565f50dbe5 Remove custom parsing and toXContent logic from TextTemplate and delegate to the Script as much as possible
Original commit: elastic/x-pack-elasticsearch@6d23549dc1
2016-09-02 23:36:07 +02:00
Nik Everett 203faaf4f4 2.x backwards compatibility support for watcher
Basic backwards compatibility support for watcher.

Closes elastic/elasticsearch#3230

Relates to elastic/elasticsearch#3231 - this actually should fix all the failures caused
by fractional time values but it does so by being able to parse them.
Being able to parse them is important for 2.x compatibility but 5.0
watches shouldn't produce fractional time values. This fixes the
particular way of making fractional time values mentioned in elastic/elasticsearch#3231
but I expect there are a half dozen more places to fix. The actual
watcher tests are fairly basic.

Original commit: elastic/x-pack-elasticsearch@328717455c
2016-09-02 09:24:46 -04:00
javanna 119bb67967 adapt to elastic/elasticsearch#20288: More info classes to implement Writeable rather than Streamable
Original commit: elastic/x-pack-elasticsearch@22cadf604d
2016-09-02 10:24:15 +02:00
Jack Conradson de4c776f0f Merge pull request elastic/elasticsearch#3272 from jdconrad/deprecate
Deprecate Groovy, Python, and Javascript

Original commit: elastic/x-pack-elasticsearch@e5c1d5a112
2016-09-01 16:30:25 -07:00
Chris Earle bb5ff7ab36 Fix test on Linux; setting name missing 'total.'
Original commit: elastic/x-pack-elasticsearch@de0183b3ef
2016-09-01 18:12:00 -04:00
Jack Conradson 670a57274a Merge branch 'master' into deprecate
Original commit: elastic/x-pack-elasticsearch@c9636bd3f1
2016-09-01 14:53:42 -07:00
Chris Earle 6d40cb63d0 [Monitoring] Collect IO Stats
This ensures that the data exists in 5.0.

Original commit: elastic/x-pack-elasticsearch@bc6e7931d1
2016-09-01 15:45:51 -04:00
Chris Earle b60e8aebd2 [Monitoring] Publish X-Pack Usage with Cluster Info
This publishes X-Pack usage data to the cluster info from the elected master node. This allows phone home to retrieve this data from the index, rather than fetching it live from the connected cluster (thereby not getting it from any n - 1 clusers that are not connceted).

Original commit: elastic/x-pack-elasticsearch@79bfaaaf0b
2016-09-01 15:38:54 -04:00
jaymode 2e010d52e9 security: do not allow built-in user/role names to be defined in the file realm
This change restricts built-in user/role names from passing validation when we are reading or modifying
the files used by this realm.

Closes elastic/elasticsearch#2078

Original commit: elastic/x-pack-elasticsearch@9f6b34f39d
2016-09-01 14:46:38 -04:00
jaymode 74f55bf46e security: add option to specify the certificate validity period
This commit adds the option to specify the number of days that a certificate is valid for.

Closes elastic/elasticsearch#2877

Original commit: elastic/x-pack-elasticsearch@fcf0a28321
2016-09-01 14:21:30 -04:00
jaymode 5d5bd015c0 add a PreBuiltXPackTransportClient
This change adds a transport client that comes preconfigured with the same plugins as the
PreBuiltTransportClient and also adds x-pack.

Closes elastic/elasticsearch#2970

Original commit: elastic/x-pack-elasticsearch@bb60534bd4
2016-09-01 13:49:10 -04:00
Jack Conradson 1d6d20679d Merge branch 'master' into deprecate
Original commit: elastic/x-pack-elasticsearch@afdfa26dc0
2016-09-01 08:52:34 -07:00
Chris Earle 6f115a532d [Monitoring] Remove "agent" package
This removes the "agent" package from org.elasticsearch.xpack.monitoring.agent.*, so that now everything is simply org.elasticsearch.xpack.monitoring.*.

Follow-on work will be refactoring some of the other code, but this is a first step now that it's always the agent (in effect).

Original commit: elastic/x-pack-elasticsearch@14025cb17c
2016-09-01 10:56:27 -04:00
jaymode ad9a7c9b96 Migrate xpack to use the common ssl configuration
This change migrates xpack (security, watcher, and monitoring) to use the common ssl
configuration for the elastic stack. As part of this work, several aspects of how we deal
with SSL has been modified.

From a functionality perspective, an xpack wide configuration for SSL was added and
all of the code that needs SSL uses the SSLService now. The following is a list of all
of the aspects of xpack that can have their own SSL configuration, which are separate
from the xpack wide configuration:

* Transport
* Transport profiles
* HTTP Transport
* Realms
* Monitoring Exporters
* HTTP Client

In terms of the code, some cleanups were made with these changes. SSLConfiguration is
now a concrete class and SSLConfiguration.Custom and SSLConfiguration.Global have been
removed. The validate method on key and trust configurations has been removed and these
classes will now throw exceptions when they are constructed with bad values. The
OptionalSettings helper class has been removed as it was just a file with one line functions
that made the code harder to understand. The SSL configuration and service classes have
been moved from the security source directories to the main xpack source set. The SSLService
now handles more of the configuration of the SSLEngine it returns to prevent callers from
having to handle those aspects. The settings that get registered for SSL have been moved to
XPackSettings.

Also included in this PR is a update to the docs around SSL. This includes a large simplification to
the documentation in that the certificate authority configuration section has been removed and the
process that is documented for generating certificates only includes the CLI tool that we bundle.

Closes elastic/elasticsearch#3104
Closes elastic/elasticsearch#2971
Closes elastic/elasticsearch#3164

Original commit: elastic/x-pack-elasticsearch@5bd9e5ef38
2016-09-01 10:51:41 -04:00
jaymode 54103127d0 fix line length
Original commit: elastic/x-pack-elasticsearch@caca0fc0b7
2016-09-01 09:37:05 -04:00
jaymode c5cde120b7 test: add 2.4.0 bwc index
Original commit: elastic/x-pack-elasticsearch@58bcf3abaf
2016-09-01 09:09:48 -04:00
jaymode 836e1d3a28 security: add a reserved role for the ingest feature in Kibana
Closes elastic/elasticsearch#1667

Original commit: elastic/x-pack-elasticsearch@da2f6d9c4e
2016-09-01 08:09:36 -04:00
Simon Willnauer 3638c8639f [TEST] Don't set create=true if no ID is specified
Original commit: elastic/x-pack-elasticsearch@874def17e8
2016-09-01 10:35:44 +02:00
Jason Tedor b29cdc0877 Fix additional exception logging call
This commit modifies an exception logging calls to use a parameterized
message from Log4j.

Original commit: elastic/x-pack-elasticsearch@7b0df7d056
2016-08-31 23:16:07 -04:00
Jack Conradson 0ca243dc83 Merge branch 'master' into deprecate
Original commit: elastic/x-pack-elasticsearch@25985e9144
2016-08-31 15:57:44 -07:00
Jack Conradson 31c9d258c7 Made deprecation of Groovy, Javascript, and Python more explicit.
Original commit: elastic/x-pack-elasticsearch@04d725e9c1
2016-08-31 15:57:09 -07:00
Jason Tedor cd0f17c372 Merge branch 'master' into log4j2
* master:
  Changes tests to conform with new cluster health API, calling setWaitForNoRelocatingShards(true) instead of setWaitForRelocatingShards(0)

Original commit: elastic/x-pack-elasticsearch@bde6ad8c8a
2016-08-31 16:17:28 -04:00
Ali Beyad 8757c2f6e1 Changes tests to conform with new cluster health API, calling
setWaitForNoRelocatingShards(true) instead of setWaitForRelocatingShards(0)

Original commit: elastic/x-pack-elasticsearch@c7c12fe64c
2016-08-31 11:56:38 -04:00
Jason Tedor a89f0110f4 Merge branch 'master' into log4j2
* master:
  Use releasable locks in NativeRolesStore
  security: limit the size of the role store cache
  security: remove explicit handshake wait in netty4 transport
  test: smoke-test-plugins-ssl no longer relies on logging to start
  kibana monitoring/uuid config key reference update
  Docs: Updated release date for 2.4 in RNs.
  Update README.md
  Build: Add apijar task to assemble so it gets built with other artifacts
  monitoring ui/license: cluster listing status cell treatment for basic/unsupported cluster
  monitoring ui:fix cluster overview when cluster has no indices/shards
  monitoring ui/license: logic cleanup per feedback
  monitoring ui/license: primary cluster asterisk styling
  monitoring ui/license: allow clicking into primary cluster if all are basic
  monitoring ui: add isPrimary property to cluster listing response
  Security: throw exception if we cannot extract indices from an indices request
  Security: add tests for delete and update by query

Original commit: elastic/x-pack-elasticsearch@3cb41739ee
2016-08-31 10:26:30 -04:00
Jason Tedor 82079185c2 Use releasable locks in NativeRolesStore
This commit replaces the use of try/finally blocks to handle safe
locking/unlocking for role cache read/write lock with releasable locks
in try-with-resources blocks.

Relates elastic/elasticsearch#3278

Original commit: elastic/x-pack-elasticsearch@fbd659cd85
2016-08-31 10:18:09 -04:00
jaymode 7d78911082 security: limit the size of the role store cache
Previously the roles store cache was unbounded as it was a just using a ConcurrentHashMap,
which could lead to excessive memory usage in cases where there are a large number of roles
as we tried to eagerly load the roles into the cache if they were not present. The roles store now
loads roles on demand and caches them for a finite period of time.

Additionally, the background polling of roles has been removed to reduce complexity. A best effort
attempt is made to clear the roles cache upon modification and if necessary the cache can be
cleared manually.

See elastic/elasticsearch#1837

Original commit: elastic/x-pack-elasticsearch@450dd779c8
2016-08-31 09:22:25 -04:00
jaymode 06ff97f63d security: remove explicit handshake wait in netty4 transport
Netty 4's SslHandler does not require the application to wait for the handshake to
be completed before data is written. This change removes the explicit wait on each
handshake future.

Original commit: elastic/x-pack-elasticsearch@c19bcebb83
2016-08-31 08:23:51 -04:00
jaymode 942a70328c test: smoke-test-plugins-ssl no longer relies on logging to start
This change adds a HTTPS check for smoke-test-plugins-ssl so it no longer has to wait for
a debug level log message.

Closes elastic/elasticsearch#2303

Original commit: elastic/x-pack-elasticsearch@f3eaaad5d4
2016-08-31 08:08:41 -04:00
Luca Cavanna 82649355a0 Merge pull request elastic/elasticsearch#3091 from javanna/fix/reindex_repleaceable
Security: add tests for delete and update by query

Original commit: elastic/x-pack-elasticsearch@6f6786a26e
2016-08-31 09:48:34 +02:00
Jason Tedor b1f97ea5ed Remove unnecessary calls to Path#toString
This commit removes some unnecessary calls to Path#toString when logging
a path in OperationFileModeWatcher#onChage. The calls to Path#toString
are not necessary since the logging infrastructure will do this anyway.

Original commit: elastic/x-pack-elasticsearch@da1bc30d60
2016-08-30 21:08:20 -04:00
Jason Tedor 4cd3e02206 Remove empty Javadocs in ExecutableSearchTransform
This commit removes some inadvertent blank Javadocs for
ExecutableSearchTransform.

Original commit: elastic/x-pack-elasticsearch@b73a5860cb
2016-08-30 21:05:18 -04:00
Jason Tedor 71b7818ead Remove unnecessary logging of exception message
This commit removes an unnecessary logging of an exception message from
HttpEmailAttachementParser#toAttachment since the full exception is
logged anyway.

Original commit: elastic/x-pack-elasticsearch@55d4bada00
2016-08-30 21:03:46 -04:00
Jack Conradson 49779a1549 Fixed test case.
Original commit: elastic/x-pack-elasticsearch@6177c03d32
2016-08-30 16:52:00 -07:00
Jason Tedor 0a285a9a0c Avoid allocating log parameterized messages
This commit modifies the call sites that allocate a parameterized
message to use a supplier so that allocations are avoided unless the log
level is fine enough to emit the corresponding log message.

Original commit: elastic/x-pack-elasticsearch@955ab89b8b
2016-08-30 18:20:15 -04:00
Ryan Ernst 1bfef646b8 Build: Add apijar task to assemble so it gets built with other artifacts
Original commit: elastic/x-pack-elasticsearch@ab9ce9c8d6
2016-08-30 11:23:22 -07:00
Jason Tedor 702d55f0be Introduce Log4j 2
This commit introduces Log4j 2 to the stack.

Original commit: elastic/x-pack-elasticsearch@02b2bb1c89
2016-08-30 13:29:28 -04:00
Nik Everett da8540b10b Handle removing more ctors
Original commit: elastic/x-pack-elasticsearch@dd1b62ba71
2016-08-30 10:27:45 -04:00
Nik Everett 2049a45f52 Handle core chaning IndexRequest
Original commit: elastic/x-pack-elasticsearch@8cd311b3e8
2016-08-30 10:27:45 -04:00
javanna 603db388d7 Security: throw exception if we cannot extract indices from an indices request
This used to be an assertion but we move it to an exception to be able to catch this at all times without requiring assertion enabled

Original commit: elastic/x-pack-elasticsearch@fcb5fbe852
2016-08-30 15:23:02 +02:00
javanna fdfc66a8ba Security: add tests for delete and update by query
Original commit: elastic/x-pack-elasticsearch@e85877d03f
2016-08-30 15:22:55 +02:00
jaymode 88abfcfea9 security: fix audit logging.yml to have the correct prefix
When migrating the package and class names from shield to xpack.security, the logging.yml
file was updated incorrectly which resulted in audit entries going to the wrong file. This change
specifies the correct package name so audit logs go to the correct file.

Closes elastic/elasticsearch#3063

Original commit: elastic/x-pack-elasticsearch@5c07108392
2016-08-30 07:39:25 -04:00
jaymode bb4777b7d6 allow extensions to define settings filter
This change allows extensions to define their own filtered settings. This is used when there
may be sensitive credentials stored in a custom realm that should be filtered out of some
API calls.

Closes elastic/elasticsearch#2847

Original commit: elastic/x-pack-elasticsearch@952474daba
2016-08-30 07:20:00 -04:00
jaymode 473728afb4 security: extract indices from CompositeIndicesRequests and exclude duplicates
This change allows us to log the indices from a composite indices request by getting the indices
from all of the sub requests. Additionally, indices are no longer duplicated when auditing.

Closes elastic/elasticsearch#2623
Closes elastic/elasticsearch#2618

Original commit: elastic/x-pack-elasticsearch@f1d3d87bcf
2016-08-30 07:06:04 -04:00
Tanguy Leroux d3dff6659b Returns empty set of filters in MonitoringIndexNameResolver and fix tests
Original commit: elastic/x-pack-elasticsearch@e901347de9
2016-08-30 10:10:08 +02:00
Tanguy Leroux 203ad03d26 Use sets of String in XContentBuilders
With elastic/elasticsearchelastic/elasticsearch#19865 the XContentBuilder has changed to support both inclusive and exclusive filters and now uses Set<String> instead of arrays of Strings. This change updates the various places in x-plugins where string arrays were used.

Original commit: elastic/x-pack-elasticsearch@1f8d4485f4
2016-08-30 09:08:49 +02:00
Nik Everett 8f22eaf1b7 Security: Add tests for indexes created in 2.x (elastic/elasticsearch#3203)
The actual backwards compatibility support is handled by core's ability to downgrade `keyword` and `text` into `string` for indexes created in 2.x.

Original commit: elastic/x-pack-elasticsearch@6b615d9a45
2016-08-29 12:00:19 -04:00
Yannick Welsch cdc41f6082 Add recovery source to ShardRouting
Relates to elastic/elasticsearchelastic/elasticsearch#19516

Original commit: elastic/x-pack-elasticsearch@051fb25bfc
2016-08-27 15:19:49 +02:00
jaymode 40c2672f12 test: remove the version compatibility tests
The x-plugins version is now kept in sync with the elasticsearch version and the
VersionCompatibilityTests just adds noise when updating versions.

Closes elastic/elasticsearch#3079
See elastic/elasticsearch#3212

Original commit: elastic/x-pack-elasticsearch@5998aa2ec0
2016-08-26 13:52:35 -04:00
jaymode e07ae87cf4 security: do not allow security APIs to execute when unlicensed
Although most of the security functionality was disabled when a basic license was applied,
some of the functionality still could be executed such as using the authenticate API or using
a transport client. The issue here is the UI calls the authenticate API and this gives the impression
that security is really in use when it is not.

Original commit: elastic/x-pack-elasticsearch@881453fc4c
2016-08-26 13:46:56 -04:00
Nik Everett 9c2b3d79ad Fix bulk update tests
It was relying on fields extracting fields from the source but it
doesn't do that any more.

Original commit: elastic/x-pack-elasticsearch@23b534c068
2016-08-26 12:52:06 -04:00
Mike McCandless ab074da873 Don't illegally reuse scorer in this test
Closes elastic/elasticsearch#3216

Original commit: elastic/x-pack-elasticsearch@63654c5585
2016-08-25 11:42:48 -04:00
jaymode 098e61fbc1 security: remove the realtime request interceptor
The realtime request interceptor was added to stop realtime requests from bypassing DLS
or FLS as the request could read a document from the translog. After
elastic/elasticsearchelastic/elasticsearch#20102 we no longer read documents from the translog so we can
allow realtime requests even when DLS or FLS is enabled.

Original commit: elastic/x-pack-elasticsearch@069b501500
2016-08-25 10:22:46 -04:00
jaymode dfdf77c536 security: upgrade to bouncy castle 1.55
Bouncy castle 1.55 was released earlier today and this commit updates the version
we use to 1.55.

Original commit: elastic/x-pack-elasticsearch@95e002431b
2016-08-25 09:41:12 -04:00
jaymode 26c1da4230 security: roles.yml is empty and add built in monitoring users
This change removes all default roles from the roles.yml file that is distributed with xpack and
adds built in roles for monitoring users and remote monitoring agents.

Closes elastic/elasticsearch#3122

Original commit: elastic/x-pack-elasticsearch@b04508bd56
2016-08-23 15:11:28 -04:00
jaymode 334aa94946 security: simplify lookup of files inside the config dir
This commit simplifies the code used for resolving the files used for security to always
resolve against the `config` file. Elasticsearch no longer offers a way to disable the security
manager, so the files read by x-pack should not really be configurable and only exist in their
default locations since that is what can be read by the process.

As part of this, the documentation was updated to indicate that these files should always be in
the default location and the settings to change the locations have been removed. Also, a bug
was fixed in a few places where settings were still using `shield.` instead of `xpack.security.`.
Finally, some outdated and unused files were deleted from the repository.

Closes elastic/elasticsearch#305

Original commit: elastic/x-pack-elasticsearch@3884f080a0
2016-08-23 14:54:18 -04:00
Ryan Ernst 1aa72336a0 Merge pull request elastic/elasticsearch#3191 from rjernst/api_jar
Build: Add api jar and client jar for xpack

Original commit: elastic/x-pack-elasticsearch@7743f43e23
2016-08-23 11:51:05 -07:00
jaymode 7536acdc9f security: cleanup logging and other minor enhancements/fixes
This change cleans up some of the log messages and levels that could now be considered misleading.

While performing these cleanups, the following was done:

* remove creation of dummy user for gradle run as we have the `elastic` user
* Request interceptors are not bound if field and document level security is disabled
* FLS/DLS interceptors skip execution if document and field level security is disabled by the license state
* The roles store that loaded the role is logged at the TRACE level
* The TransportXPackUsageAction was using the incorrect action name when registering a handler

Closes elastic/elasticsearch#2096
Closes elastic/elasticsearch#1861
Closes elastic/elasticsearch#2229
See elastic/elasticsearch#1879

Original commit: elastic/x-pack-elasticsearch@ac16b21c0c
2016-08-23 14:35:23 -04:00
jaymode 507196dca5 security: allow superusers access to the security index
This change allows users with the superuser role to access the security index. We previously allowed
the XPackUser to access this with the intent that the XPackUser was also the `elastic` user. When the
`elastic` user was split out into the ElasticUser, we did not update the check to allow this user access
to the security index.

Original commit: elastic/x-pack-elasticsearch@fa556d9845
2016-08-23 14:24:43 -04:00
jaymode 9c76211393 security: do not use hidden filenames when generating certs
This commit changes how we get the file and directory name for certificates in the tool. The
tool now prompts the user for the filename. If the provided instance name will result in a
valid filename, this is provided as a default. Otherwise the user must provide a valid
filename.

Closes elastic/elasticsearch#2854

Original commit: elastic/x-pack-elasticsearch@3c923d736b
2016-08-23 14:11:04 -04:00
Jack Conradson feefd070ef Make Painless the default scripting language.
Closes elastic/elasticsearch#3124

Original commit: elastic/x-pack-elasticsearch@7e458c07a6
2016-08-22 17:41:18 -07:00
Ryan Ernst f28f77f73c Build: Add api jar and client jar for xpack
This adds back (again) building a transport client plugin jar for
x-pack, and also adds producing an "api" jar which extension authors can
build against. For now, both these jars are exactly the same, but
eventually they could differ, and be reduced to less than the real
x-pack jar.

see elastic/stackelastic/elasticsearch#7

Original commit: elastic/x-pack-elasticsearch@0a989de18b
2016-08-22 15:40:46 -07:00
Areek Zillur e28c2a8c94 Merge pull request elastic/elasticsearch#3178 from areek/fix/3177
Fix license expiry logging

Original commit: elastic/x-pack-elasticsearch@29d9f90af2
2016-08-22 13:16:05 -04:00
Areek Zillur 87eb69fdc6 Fix license expiry logging
Original commit: elastic/x-pack-elasticsearch@51580eaba7
2016-08-22 12:20:21 -04:00
Martijn van Groningen f10fbeeb70 watcher: remove WatchScript class
Original commit: elastic/x-pack-elasticsearch@fb2f9a28f1
2016-08-22 09:39:13 +02:00
Ryan Ernst 8ae939fec7 Remove client jar
This reverts building a separate client jar for xpack. It is not
necessary because we already build x-pack as a jar and publish to
elastic maven, since extension authors need that.

Original commit: elastic/x-pack-elasticsearch@2fab06b42c
2016-08-19 16:17:54 -07:00
Ryan Ernst 9586e68470 Merge pull request elastic/elasticsearch#3165 from rjernst/client_jar
Build: Enable x-pack to build a transport client plugin jar

Original commit: elastic/x-pack-elasticsearch@a36305d736
2016-08-19 09:29:47 -07:00
Ryan Ernst aa4fad9009 Build: Enable x-pack to build a transport client plugin jar
Original commit: elastic/x-pack-elasticsearch@f42f7a1cd2
2016-08-19 09:15:54 -07:00
Martijn van Groningen 64eec5afb3 security: Prohibit the use of `terms` query with lookup, `geo_shape` with indexed shapes, `has_child`, `has_parent` and `percolator` query inside DLS role query.
Closes elastic/elasticsearch#3145
Closes elastic/elasticsearch#613

Original commit: elastic/x-pack-elasticsearch@5962089b6c
2016-08-19 16:59:36 +02:00
Tanguy Leroux 30eab329a1 Muted HipChatServiceTests
These tests failed regularly. It seems that the user_account authentification token is not accepted anymore by the HipChat service that respond with a 401 HTTP code.

    See https://github.com/elastic/x-plugins/issues/3162

Original commit: elastic/x-pack-elasticsearch@793ad494d3
2016-08-19 16:48:44 +02:00
Tanguy Leroux 56be936ace Watcher: Use search template in Search Input/Transform REST tests
These tests would have caught the regression (introduced in elastic/x-pack@95a29c6a42 and fixed by elastic/x-pack@9b834b5f50) that cause search template to have "groovy" lang by default instead of "mustache"

Original commit: elastic/x-pack-elasticsearch@e27e5ae821
2016-08-19 09:30:24 +02:00
Ryan Ernst f92b576661 Merge pull request elastic/elasticsearch#3142 from rjernst/deguice20
Remove SecurityTransportModule

Original commit: elastic/x-pack-elasticsearch@e609503c7b
2016-08-18 08:03:31 -07:00
Ryan Ernst 9da4d827c1 Rename transport service method to clarify a user is always used, but
not necessarily the system user

Original commit: elastic/x-pack-elasticsearch@0357d3718d
2016-08-18 08:03:31 -07:00
Tanguy Leroux 5b2c7dd503 Fix watrcher script parsing tests
closes elastic/elasticsearch#3135, elastic/elasticsearch#3134, elastic/elasticsearch#3136

Original commit: elastic/x-pack-elasticsearch@a1b0402be4
2016-08-18 10:06:05 +02:00
Ryan Ernst 49ac29f2e3 Remove SecurityTransportModule
SecurityTransportModule handled binding two things in guice. First, IPFilter,
for which createComponents already had the necessary dependencies. The
second was ClientTransportFilter. For transport clients, this was a
noop and could be removed. For nodes, this just attaches the system
user, which could be done directly from SecurityServerTransportService.

Original commit: elastic/x-pack-elasticsearch@da327de476
2016-08-18 00:09:31 -07:00
Chris Earle 572869087d Change use of generics in test
Original commit: elastic/x-pack-elasticsearch@39473681b2
2016-08-17 15:50:20 -04:00
Chris Earle c16860e901 Fix Watcher REST test
Original commit: elastic/x-pack-elasticsearch@7cb5b08948
2016-08-17 15:35:18 -04:00
Jason Tedor 76319495b6 Mark failing Watcher REST test as awaits fix
This commit marks a failing Watcher REST test with seed 97AAFD9CA37864EE
as awaits fix.

Original commit: elastic/x-pack-elasticsearch@5e19f0169c
2016-08-17 15:13:26 -04:00
jaymode 934b69b0c6 test: use valid names to prevent false test failures
In some cases, the random ascii value could match a forbidden name, which will cause this
test to fail as it expects valid names. This changes ensures the names are valid.

Original commit: elastic/x-pack-elasticsearch@c79f8fc4cc
2016-08-17 14:31:17 -04:00
Jason Tedor de4851329d Mark WUT#testDeserializeSearchRequests awaits fix
This commit marks WatcherUtilsTests#testDeserializeSearchRequest which
is failing with seed 2203D3AD59DB5223 as awaits fix.

Original commit: elastic/x-pack-elasticsearch@4b6cbe544b
2016-08-17 13:43:12 -04:00
Jason Tedor ca896d2dca Mark STT#testParser as awaits fix
This commit marks SearchTransformTests#testParser which is failing with
seed 97BC9E2543410D55 as awaits fix.

Original commit: elastic/x-pack-elasticsearch@3d69e9648c
2016-08-17 13:13:10 -04:00
Jason Tedor bed5cc5882 Mark WUT#testSerializeSearchRequest as awaits fix
This commit marks WatcherUtilsTests#testSerializeSearchRequest which is
failing with seed 97BC9E2543410D55 as awaits fix.

Original commit: elastic/x-pack-elasticsearch@45a174c1f2
2016-08-17 13:10:42 -04:00
Tanguy Leroux c2dbd5ed4a Watcher: Use Mustache as default search template lang
With the latest clean ups and changes in Watcher, the default search template lang has been switched to WatcherScript.DEFAULT_LANG which points to "groovy" but it should be "mustache" instead.

Original commit: elastic/x-pack-elasticsearch@1d9ef1963e
2016-08-17 18:16:30 +02:00
Chris Earle 53d022a20a [Watcher] Add Condition to Action
This adds a "condition" to every action (via the ActionWrapper) that prevents execution of the action if the condition fails. An action-level condition is only useful when there is more than one action, but nothing checks to ensure that it's only used in that scenario.

Original commit: elastic/x-pack-elasticsearch@704cfb1a86
2016-08-17 11:45:43 -04:00
Martijn van Groningen 101d791ec4 move test to the right package
Original commit: elastic/x-pack-elasticsearch@0693b8657d
2016-08-17 10:40:26 +02:00
Martijn van Groningen f291f292bf watcher: Watcher search templates shouldn't serialize SearchSourceBuilder to a string, template that and turn it back into a SearchSourceBuilder
Instead watcher search template should be agnostic of SearchSourceBuilder and just work with BytesReference, so that serializing to a string before templating isn't needed.

Original commit: elastic/x-pack-elasticsearch@36d21ec819
2016-08-17 09:52:07 +02:00
Ryan Ernst 7aa557d8f0 Build: Add task to build zip of license tools
This roughly matches what was done in 2.x for the license tools (except
without any dependency exclusions).

Original commit: elastic/x-pack-elasticsearch@4e1f07b5fa
2016-08-16 14:23:56 -07:00
Ryan Ernst e4e2601b11 Merge branch 'master' into search_parser
Original commit: elastic/x-pack-elasticsearch@c347445a69
2016-08-16 11:29:19 -07:00
Ryan Ernst 4dac802c5d Use consolidated search parsers container
This is the xplugins side of elastic/elasticsearchelastic/elasticsearch#20000, using the new
SearchRequestParsers.

Original commit: elastic/x-pack-elasticsearch@d7cee93a8f
2016-08-16 02:01:56 -07:00
Nik Everett 8ee190d741 Handle moving rest support
Original commit: elastic/x-pack-elasticsearch@99076b2936
2016-08-15 22:41:43 -04:00
Chris Earle af28d05515 Disabling local test while it awaits a fix
Original commit: elastic/x-pack-elasticsearch@3a8afb331f
2016-08-12 23:44:28 -04:00
Chris Earle e7ae3fa1f5 Give a longer timeout for test
Original commit: elastic/x-pack-elasticsearch@461d7eee8f
2016-08-12 17:18:44 -04:00
Chris Earle 3c9749b2b8 [Monitoring] Add Empty Ingest Pipeline to Future Proof Monitoring Documents
This adds an empty _ingest/pipeline named after the _current_ version of the Monitoring API (currently 2) from both exporters.

This allows us to eventually change the API version (to 3, 4, etc.) and go _back_ and modify the pipeline that exists. The modified pipeline would then "fix" the documents as best as possible and rename the index. As a result, different versions (starting with 5.0) will be able to report to the same monitoring cluster regardless of the running API version.

Note: This has no impact on stale data (e.g., the day before the upgrade) _and_ it implies that the monitoring cluster should always be updated first. A simple reindexing script can be supplied for old data, which can be done at the discretion of the user.

Original commit: elastic/x-pack-elasticsearch@45df5ee87b
2016-08-12 11:50:39 -04:00
jaymode 72f580c82d test: fix concurrency bug in SSLConfigurationReloaderTests
The SSLConfigurationReloaderTests rarely failed during some local runs. This turned out to be due to
signaling that the reload happened before we actually reloaded. This led to a race condition where we
attempted to validate the config was reloaded properly and actually reloading. This change fixes the
ordering of operations and uses a CountDownLatch instead of a AtomicInteger and awaitBusy.

Original commit: elastic/x-pack-elasticsearch@9615f225d6
2016-08-12 08:50:53 -04:00
jaymode a3e7536205 fix serialization of XPackFeatureSet.Usage
The XPackFeatureSet.Usage writeTo method failed to write the name to the
StreamOutput whereas the deserialization code expected it to be the first value
present in the StreamInput. This causes xpack usage requests made to nodes
other than the master to fail.

This change fixes the serialization and adds tests to all of the feature sets to ensure
that the usage is serialized properly.

Closes elastic/elasticsearch#3072

Original commit: elastic/x-pack-elasticsearch@eccab616ca
2016-08-12 07:48:55 -04:00
Lee Hinman 52f23afb4b Fix checkstyle violations in HttpExporter*.java
Two files had out-of-order @Nullable clauses and one had an invalid
license header

Original commit: elastic/x-pack-elasticsearch@8057871b18
2016-08-10 16:46:31 -06:00
Chris Earle c41550d36a [Monitoring] Allow HTTP Exporter to be given custom HTTP Headers for every request
This allows X-Pack Monitoring 5.0 to be given custom headers for any request.

Original commit: elastic/x-pack-elasticsearch@04050181bf
2016-08-10 16:33:32 -04:00
Adrien Grand 632871be84 Fix after the `mapper` package was collapsed.
Original commit: elastic/x-pack-elasticsearch@fb708c72c3
2016-08-10 17:52:07 +02:00
jaymode af44ea0440 security: move to a single SSLService
This change refactors the ssl services into a single class and reduces the number of
public methods that are exposed by the class. As part of this change, ssl configurations
must be loaded during construction of the service otherwise an exception will be thrown
when trying to retrieve the configuration. Additionally, the reloading of key material and
trust material is now handled by the SSLService instead of the SSLConfiguration. The
SSLConfiguration class has been made package private as its usage has been contained
to the ssl package entirely.

See elastic/elasticsearch#2971

Original commit: elastic/x-pack-elasticsearch@46e4535bb1
2016-08-10 07:24:47 -04:00
Ryan Ernst 20041446f3 Merge pull request elastic/elasticsearch#3069 from rjernst/deguice18
Consolidate settings for enabling xpack features

Original commit: elastic/x-pack-elasticsearch@bdc505dc6b
2016-08-09 16:31:06 -07:00
jaymode 2b2ad14f65 test: mute version compatibility tests
Original commit: elastic/x-pack-elasticsearch@f3eee3a223
2016-08-09 15:22:32 -04:00
Britta Weber 7b1d7c8e27 use pattern instead of hard coding the version (elastic/elasticsearch#3078)
Original commit: elastic/x-pack-elasticsearch@b86e1cd40c
2016-08-09 20:50:46 +02:00
Britta Weber e8d139eb97 update mappings and template for .security index if they are not up t… (elastic/elasticsearch#3030)
* update mappings and template for .security index if they are not up to date

closes elastic/elasticsearch#2986

* nits


Original commit: elastic/x-pack-elasticsearch@b63aebbed8
2016-08-09 17:52:29 +02:00
Ryan Ernst 0039f9a2b2 Merge pull request elastic/elasticsearch#3032 from rjernst/deguice16
Internal: Deguice notification services

Original commit: elastic/x-pack-elasticsearch@9739742373
2016-08-08 14:10:52 -07:00
Ryan Ernst ce496938cd Consolidate settings for enabling xpack features
Currently there are a number of helper methods, all used at various
stages of initializing xpack, for reading "enabled" settings. Almost all
of these read the setting directly rather than use the new settings
infra. This change adds an XPackSettings class to contain these settings
constants, and adds all the enabled settings to it.

Note: While there is an existing MonitoringSettings, I think we do not
have enough settings (especially for all features) to warrant a separate
class per "feature" (eg graph would have nothing). We should decide
if/when this is necessary on a case by case basis, but I think the
enabled settings should all go here in XPackSettings.

Original commit: elastic/x-pack-elasticsearch@086152d5a5
2016-08-08 14:01:35 -07:00
Jason Tedor b2be99a869 Remove network settings when security is disabled
When security is disabled, we currently set the transport and HTTP type
to Netty 4. However, this is unnecessary as these are the default
settings so this commit removes these explicit settings.

Relates elastic/elasticsearch#3065

Original commit: elastic/x-pack-elasticsearch@afa360ceec
2016-08-08 14:46:23 -04:00
jaymode 129a2e594e test: add test for Kibana user executing a get mappings request
See elastic/elasticsearch#3048

Original commit: elastic/x-pack-elasticsearch@cd0e8ec29f
2016-08-08 10:48:34 -04:00
Tanguy Leroux 9d26ac0378 [TEST] Fix TransformIntegrationTests.testScriptTransform and wrong stored script id
Original commit: elastic/x-pack-elasticsearch@dfbb60d265
2016-08-08 10:45:41 +02:00
Tanguy Leroux b07394090f [Test] Re enable some messy Watcher tests
This commit reenable some Watcher tests that were muted in elastic/elasticsearch#724. It removes the Groovy dependency for tests that don't really use Groovy features and replace scripts with mocked scripts. It converts a GroovyScriptConditionIT test into a REST test and moves it in the smoke-test-watcher-with-groovy project.

closes elastic/elasticsearch#724

Original commit: elastic/x-pack-elasticsearch@f4c8161946
2016-08-08 10:06:30 +02:00
Jason Tedor 7b7411f78d Remove mock Netty plugins
These mock Netty plugins were needed to assist in setting
sun.nio.ch.bugLevel. Yet, settings this property should no longer be
needed on JDK 8 and the missing privileged block when writing this
property can be fixed upstream.

Relates elastic/elasticsearch#3047

Original commit: elastic/x-pack-elasticsearch@e2aa4f6fb5
2016-08-05 14:58:50 -04:00
Britta Weber 656a6bb624 mute test, we have an issue for it
Original commit: elastic/x-pack-elasticsearch@59fc88a9a3
2016-08-05 17:35:03 +02:00
Tanguy Leroux ec950a2ca7 Update to Jackson 2.8.1
This commit updates Jackson to the 2.8.1 version, which is more strict when it comes to build objects.

Original commit: elastic/x-pack-elasticsearch@fb23208e7d
2016-08-05 12:25:42 +02:00
Boaz Leskes a2cca18c49 allow joining nodes, conflicting with existing nodes to elect a master (elastic/elasticsearch#2996)
this is a companion PR to elastic/elasticsearchelastic/elasticsearch#19743

Original commit: elastic/x-pack-elasticsearch@193c92d99a
2016-08-05 09:07:29 +02:00
Ryan Ernst 42076f89ce Internal: Deguice notification services
This change removes guice from creating notification componenents. It
also removes the Notification helper class, as it just makes looking at
what stuff xpack adds more obfuscated.

Original commit: elastic/x-pack-elasticsearch@69b8ea4735
2016-08-03 15:32:56 -07:00
Ryan Ernst 5ebc9c4569 Fix signature of createComponents after addition of script service in core
Original commit: elastic/x-pack-elasticsearch@768e992715
2016-08-03 13:14:13 -07:00
Jason Tedor 46d7baa319 Merge pull request elastic/elasticsearch#3021 from jaymode/netty4_transport
Introduce Netty 4 security

Original commit: elastic/x-pack-elasticsearch@fb5e7fa66e
2016-08-03 14:10:52 -04:00
Alexander Reelsen 07b8bcf219 Watcher: Ensure watch status is always updated on condition check (elastic/elasticsearch#2980)
When a watch is executed, but the condition is not met, it is not automatically marked as dirty.
This results in a divergence of what is displayed when someone is just getting a watch via the get
watch API (due to using the in memory store) compare to when someone is using the GET API to get the
watch via the .watches index call.

This commit sets the watch status always to dirty after a condition check, thus forcing an update.

Closes elastic/elasticsearch#2699

Original commit: elastic/x-pack-elasticsearch@e6a00260f7
2016-08-03 18:21:57 +02:00
jaymode 60cb867d47 security: ignore empty query value in roles
Users are allowed to create query objects with an empty string value as we do
not currently validate the input against a query parser. In this case, we can ignore
the empty value when parsing the role. If we pass an empty BytesReference in
the role then trying to determine the XContentType will fail in the
SecurityIndexSearcherWrapper.

Closes elastic/elasticsearch#2997

Original commit: elastic/x-pack-elasticsearch@fc593943c4
2016-08-03 11:24:49 -04:00
Jason Tedor 33ba52e735 Merge branch 'master' into netty4_transport
* master:
  Restore interruption flag correctly in life cycle service

Original commit: elastic/x-pack-elasticsearch@5da6471c9a
2016-08-03 11:11:16 -04:00
Jason Tedor 832cb4fff2 Simplify test setup in Netty 4 transport tests
This commit collapes some duplicated setup code in every test in the
Netty 4 security transport tests to a common method.

Original commit: elastic/x-pack-elasticsearch@0a088db78d
2016-08-03 11:08:00 -04:00
Daniel Mitterdorfer 196f74984c Merge pull request elastic/elasticsearch#2969 from danielmitterdorfer/fix-interruption
Restore interruption flag correctly in life cycle service

Original commit: elastic/x-pack-elasticsearch@fe73226e47
2016-08-03 17:03:33 +02:00
jaymode b3f8495a03 clarify comments and add assert client auth type
Original commit: elastic/x-pack-elasticsearch@0e3d134bc6
2016-08-03 10:39:37 -04:00
Jason Tedor 94e7d82a20 Kill newline in SecurityNetty4HttpServerTransport
This commit removes an extra newline in
SecurityNetty4HttpServerTransport.java.

Original commit: elastic/x-pack-elasticsearch@218b29c9b6
2016-08-03 09:42:51 -04:00
Jason Tedor 4874d84f82 Merge branch 'master' into netty4_transport
* master: (25 commits)
  docs: remove unused file and add link to invalid role examples
  Remove interfaces for notification services
  Redirect to URL specified by next parameter, if it is set
  Fix fixture and tests so they pass again
  Update error message to be more actionable
  Switch to NamedWriteable pull based extension in xpack
  Fixing issue with infinite redirect loop
  Toggle display of security nav controls more dynamically
  Pass in xpackMainPlugin instead of xpackMainPlugin.info
  Wrap the return in a Promise
  Only unset the cookie if it is currently set
  Clarifying intent of code
  Updating tests fixtures + adding assertion for client cookie deletion
  If security is disabled, do not attempt to call the authenticate ES API
  Disambiguate between resolve function names
  Revert to not using xpackMainPlugin.info until the xpackMainPlugin is ready
  Redirect /login => / if security is disabled in ES
  Register/deregister security management items depending on whether there's an auth'd user
  Show/hide the username + logout button depending on whether there is an auth'd user
  If security is disabled, continue without auth + delete client cookie
  ...

Original commit: elastic/x-pack-elasticsearch@16b92a1a59
2016-08-03 09:18:42 -04:00
jaymode 00e963fef9 ensure transport is set for external cluster clients take 2
Original commit: elastic/x-pack-elasticsearch@b92943303e
2016-08-03 08:12:05 -04:00
jaymode 417bc49230 ensure transport is set for external cluster clients
Original commit: elastic/x-pack-elasticsearch@ee8aa8600a
2016-08-03 08:11:19 -04:00
jaymode 76591e54f6 mock channelhandlercontext instead of a null implementation
Original commit: elastic/x-pack-elasticsearch@e4ba6b96cc
2016-08-03 07:27:37 -04:00
jaymode 55ccd27acf ensure the transport type is a security transport
Original commit: elastic/x-pack-elasticsearch@6c7e46e103
2016-08-03 07:23:55 -04:00
Ryan Ernst d69bb4cc51 Merge pull request elastic/elasticsearch#3017 from rjernst/writeable_registry
Switch to NamedWriteable pull based extension in xpack

Original commit: elastic/x-pack-elasticsearch@e077ba2d6c
2016-08-03 01:36:43 -07:00
Ryan Ernst 549a5d3e73 Remove interfaces for notification services
We have 4 types of notification services, and all of them have an
interface with a single implementation class. They also all
unnecessarily are lifecycle componenets, but the only thing start does
is read the settings.

This change converts all 4 notification services to classes, and makes
them regular components instead of lifecycle services.

Original commit: elastic/x-pack-elasticsearch@897115ae65
2016-08-03 00:07:26 -07:00
Jason Tedor dd181e3e13 Cleanup HTTP and transport security settings
After this commit, we have "security3" and "security4" as possible
transport and HTTP settings, we default to "security4" if it is not set,
and we randomize the selection in some of the integration tests.

Original commit: elastic/x-pack-elasticsearch@e56718911a
2016-08-03 00:23:58 -04:00
Jason Tedor f4ba670b25 Cleanup Netty handshake waiting handlers
This commit cleans up the Netty handshake waiting handlers. We rename
the Netty 3 implementation to include "Netty3" in the name, the Netty 4
implementation is not needed, and we improve the handling of waiting for
the handshakes to complete when connecting.

Original commit: elastic/x-pack-elasticsearch@f736fdc8f0
2016-08-02 23:55:20 -04:00
Ryan Ernst cdae14a5b9 Switch to NamedWriteable pull based extension in xpack
This is the xplugins side of elastic/elasticsearchelastic/elasticsearch#19764. It converts
the one use of registering custom NamedWriteable readers in xpack for
xpack feature sets to register them up front with the new pull based
registration.

Original commit: elastic/x-pack-elasticsearch@48e2020816
2016-08-02 16:00:06 -07:00
Jason Tedor 8579dbf80b Remove nocommit from Security
This commit removes a nocommit from Security so that work can continue.

Original commit: elastic/x-pack-elasticsearch@4d35ef6174
2016-08-02 17:11:32 -04:00
Jason Tedor c88aea19d8 Migrate Netty 4 unit tests
This commit migrates some unit tests to Netty 4.

Original commit: elastic/x-pack-elasticsearch@a38dc82706
2016-08-02 16:43:07 -04:00