Commit Graph

329 Commits

Author SHA1 Message Date
Lisa Cawley 621ec7cb84 [DOCS] Removes out-dated info from Watcher limitations (#44252) 2019-07-17 13:24:02 -07:00
James Rodewig ac07eef86c [DOCS] Remove :edit_url: overrides. (#44445)
These overrides do not work in Asciidoctor and are no longer needed.
2019-07-16 15:04:44 -04:00
Lee Hinman fb0461ac76
[7.x] Add Snapshot Lifecycle Management (#44382)
* Add Snapshot Lifecycle Management (#43934)

* Add SnapshotLifecycleService and related CRUD APIs

This commit adds `SnapshotLifecycleService` as a new service under the ilm
plugin. This service handles snapshot lifecycle policies by scheduling based on
the policies defined schedule.

This also includes the get, put, and delete APIs for these policies

Relates to #38461

* Make scheduledJobIds return an immutable set

* Use Object.equals for SnapshotLifecyclePolicy

* Remove unneeded TODO

* Implement ToXContentFragment on SnapshotLifecyclePolicyItem

* Copy contents of the scheduledJobIds

* Handle snapshot lifecycle policy updates and deletions (#40062)

(Note this is a PR against the `snapshot-lifecycle-management` feature branch)

This adds logic to `SnapshotLifecycleService` to handle updates and deletes for
snapshot policies. Policies with incremented versions have the old policy
cancelled and the new one scheduled. Deleted policies have their schedules
cancelled when they are no longer present in the cluster state metadata.

Relates to #38461

* Take a snapshot for the policy when the SLM policy is triggered (#40383)

(This is a PR for the `snapshot-lifecycle-management` branch)

This commit fills in `SnapshotLifecycleTask` to actually perform the
snapshotting when the policy is triggered. Currently there is no handling of the
results (other than logging) as that will be added in subsequent work.

This also adds unit tests and an integration test that schedules a policy and
ensures that a snapshot is correctly taken.

Relates to #38461

* Record most recent snapshot policy success/failure (#40619)

Keeping a record of the results of the successes and failures will aid
troubleshooting of policies and make users more confident that their
snapshots are being taken as expected.

This is the first step toward writing history in a more permanent
fashion.

* Validate snapshot lifecycle policies (#40654)

(This is a PR against the `snapshot-lifecycle-management` branch)

With the commit, we now validate the content of snapshot lifecycle policies when
the policy is being created or updated. This checks for the validity of the id,
name, schedule, and repository. Additionally, cluster state is checked to ensure
that the repository exists prior to the lifecycle being added to the cluster
state.

Part of #38461

* Hook SLM into ILM's start and stop APIs (#40871)

(This pull request is for the `snapshot-lifecycle-management` branch)

This change allows the existing `/_ilm/stop` and `/_ilm/start` APIs to also
manage snapshot lifecycle scheduling. When ILM is stopped all scheduled jobs are
cancelled.

Relates to #38461

* Add tests for SnapshotLifecyclePolicyItem (#40912)

Adds serialization tests for SnapshotLifecyclePolicyItem.

* Fix improper import in build.gradle after master merge

* Add human readable version of modified date for snapshot lifecycle policy (#41035)

* Add human readable version of modified date for snapshot lifecycle policy

This small change changes it from:

```
...
"modified_date": 1554843903242,
...
```

To

```
...
"modified_date" : "2019-04-09T21:05:03.242Z",
"modified_date_millis" : 1554843903242,
...
```

Including the `"modified_date"` field when the `?human` field is used.

Relates to #38461

* Fix test

* Add API to execute SLM policy on demand (#41038)

This commit adds the ability to perform a snapshot on demand for a policy. This
can be useful to take a snapshot immediately prior to performing some sort of
maintenance.

```json
PUT /_ilm/snapshot/<policy>/_execute
```

And it returns the response with the generated snapshot name:

```json
{
  "snapshot_name" : "production-snap-2019.04.09-rfyv3j9qreixkdbnfuw0ug"
}
```

Note that this does not allow waiting for the snapshot, and the snapshot could
still fail. It *does* record this information into the cluster state similar to
a regularly trigged SLM job.

Relates to #38461

* Add next_execution to SLM policy metadata (#41221)

* Add next_execution to SLM policy metadata

This adds the next time a snapshot lifecycle policy will be executed when
retriving a policy's metadata, for example:

```json
GET /_ilm/snapshot?human
{
  "production" : {
    "version" : 1,
    "modified_date" : "2019-04-15T21:16:21.865Z",
    "modified_date_millis" : 1555362981865,
    "policy" : {
      "name" : "<production-snap-{now/d}>",
      "schedule" : "*/30 * * * * ?",
      "repository" : "repo",
      "config" : {
        "indices" : [
          "foo-*",
          "important"
        ],
        "ignore_unavailable" : true,
        "include_global_state" : false
      }
    },
    "next_execution" : "2019-04-15T21:16:30.000Z",
    "next_execution_millis" : 1555362990000
  },
  "other" : {
    "version" : 1,
    "modified_date" : "2019-04-15T21:12:19.959Z",
    "modified_date_millis" : 1555362739959,
    "policy" : {
      "name" : "<other-snap-{now/d}>",
      "schedule" : "0 30 2 * * ?",
      "repository" : "repo",
      "config" : {
        "indices" : [
          "other"
        ],
        "ignore_unavailable" : false,
        "include_global_state" : true
      }
    },
    "next_execution" : "2019-04-16T02:30:00.000Z",
    "next_execution_millis" : 1555381800000
  }
}
```

Relates to #38461

* Fix and enhance tests

* Figured out how to Cron

* Change SLM endpoint from /_ilm/* to /_slm/* (#41320)

This commit changes the endpoint for snapshot lifecycle management from:

```
GET /_ilm/snapshot/<policy>
```

to:

```
GET /_slm/policy/<policy>
```

It mimics the ILM path only using `slm` instead of `ilm`.

Relates to #38461

* Add initial documentation for SLM (#41510)

* Add initial documentation for SLM

This adds the initial documentation for snapshot lifecycle management.

It also includes the REST spec API json files since they're sort of
documentation.

Relates to #38461

* Add `manage_slm` and `read_slm` roles (#41607)

* Add `manage_slm` and `read_slm` roles

This adds two more built in roles -

`manage_slm` which has permission to perform any of the SLM actions, as well as
stopping, starting, and retrieving the operation status of ILM.

`read_slm` which has permission to retrieve snapshot lifecycle policies as well
as retrieving the operation status of ILM.

Relates to #38461

* Add execute to the test

* Fix ilm -> slm typo in test

* Record SLM history into an index (#41707)

It is useful to have a record of the actions that Snapshot Lifecycle
Management takes, especially for the purposes of alerting when a
snapshot fails or has not been taken successfully for a certain amount of
time.

This adds the infrastructure to record SLM actions into an index that
can be queried at leisure, along with a lifecycle policy so that this
history does not grow without bound.

Additionally,
SLM automatically setting up an index + lifecycle policy leads to
`index_lifecycle` custom metadata in the cluster state, which some of
the ML tests don't know how to deal with due to setting up custom
`NamedXContentRegistry`s.  Watcher would cause the same problem, but it
is already disabled (for the same reason).

* High Level Rest Client support for SLM (#41767)

* High Level Rest Client support for SLM

This commit add HLRC support for SLM.

Relates to #38461

* Fill out documentation tests with tags

* Add more callouts and asciidoc for HLRC

* Update javadoc links to real locations

* Add security test testing SLM cluster privileges (#42678)

* Add security test testing SLM cluster privileges

This adds a test to `PermissionsIT` that uses the `manage_slm` and `read_slm`
cluster privileges.

Relates to #38461

* Don't redefine vars

*  Add Getting Started Guide for SLM  (#42878)

This commit adds a basic Getting Started Guide for SLM.

* Include SLM policy name in Snapshot metadata (#43132)

Keep track of which SLM policy in the metadata field of the Snapshots
taken by SLM. This allows users to more easily understand where the
snapshot came from, and will enable future SLM features such as
retention policies.

* Fix compilation after master merge

* [TEST] Move exception wrapping for devious exception throwing

Fixes an issue where an exception was created from one line and thrown in another.

* Fix SLM for the change to AcknowledgedResponse

* Add Snapshot Lifecycle Management Package Docs (#43535)

* Fix compilation for transport actions now that task is required

* Add a note mentioning the privileges needed for SLM (#43708)

* Add a note mentioning the privileges needed for SLM

This adds a note to the top of the "getting started with SLM"
documentation mentioning that there are two built-in privileges to
assist with creating roles for SLM users and administrators.

Relates to #38461

* Mention that you can create snapshots for indices you can't read

* Fix REST tests for new number of cluster privileges

* Mute testThatNonExistingTemplatesAreAddedImmediately (#43951)

* Fix SnapshotHistoryStoreTests after merge

* Remove overridden newResponse functions that have been removed

* Fix compilation for backport

* Fix get snapshot output parsing in test

* [DOCS] Add redirects for removed autogen anchors (#44380)

* Switch <tt>...</tt> in javadocs for {@code ...}
2019-07-16 07:37:13 -06:00
Lisa Cawley 753da8feac [DOCS] Updates terminology for alerting features (#43945) 2019-07-15 14:47:33 -07:00
Lisa Cawley aa6b544fac [DOCS] Moves Watcher troubleshooting page (#44250) 2019-07-12 08:28:18 -07:00
Ioannis Kakavas 475752be75
Make plugin verification FIPS 140 compliant (#44266)
This change makes the process of verifying the signature of
official plugins FIPS 140 compliant by defaulting to use the
BouncyCastle FIPS provider and adding a dependency to bcpg-fips
that implement parts of openPGP in a FIPS compliant manner.

In already FIPS 140 enabled environments that use the 
BouncyCastle FIPS provider, the bcfips dependency is redundant
but doesn't cause an issue as it will be added only in the classpath
 of the cli-tools

This is a backport of #44224
2019-07-12 14:34:15 +03:00
Mark Vieira 263f76e5ea
Revert "[DOCS] Moves Watcher troubleshooting page (#44144)"
This reverts commit 11375926ec.
2019-07-11 17:13:08 -07:00
Lisa Cawley 11375926ec [DOCS] Moves Watcher troubleshooting page (#44144) 2019-07-11 14:42:32 -07:00
Lisa Cawley 5b71340f99 [DOCS] Moves Watcher limitations (#44141) 2019-07-10 11:17:12 -07:00
Albert Zaharovits 018d946bba [DOC] Backup & Restore Security Configuration (#42970)
This commit documents the backup and restore of a cluster's
security configuration.

It is not possible to only backup (or only restore) security
configuration, independent to the rest of the cluster's conf,
so this describes how a full configuration backup&restore
will include security as well. Moreover, it explains how part
of the security conf data resides on the special .security
index and how to backup that using regular data snapshot API.

Co-Authored-By: Lisa Cawley <lcawley@elastic.co>
Co-Authored-By: Tim Vernum <tim@adjective.org>
2019-07-10 14:53:56 +03:00
marcos ramos 88ee47c9ba Fix OIDC documentation settings (#44115)
Current kibana setting is xpack.security.auth.oidc.realm, 
but the correct  one is xpack.security.authc.oidc.realm
2019-07-09 18:44:35 +03:00
Sachin Frayne 389c923a82 [Docs] Fix json syntax in watcher compare condition (#44032) 2019-07-09 13:43:18 +02:00
Martijn van Groningen adc06ffd89
take builtin role into account in docs tests 2019-07-05 08:06:18 +02:00
Alexander Reelsen 9077c4402f Watcher: Allow to execute actions for each element in array (#41997)
This adds the ability to execute an action for each element that occurs
in an array, for example you could sent a dedicated slack action for
each search hit returned from a search.

There is also a limit for the number of actions executed, which is
hardcoded to 100 right now, to prevent having watches run forever.

The watch history logs each action result and the total number of actions
the were executed.

Relates #34546
2019-07-03 11:28:50 +02:00
Tim Vernum 2a8f30eb9a
Support builtin privileges in get privileges API (#43901)
Adds a new "/_security/privilege/_builtin" endpoint so that builtin
index and cluster privileges can be retrieved via the Rest API

Backport of: #42134
2019-07-03 19:08:28 +10:00
Tim Vernum 14884c871f
Document API-Key APIs require manage_api_key priv (#43869)
Add the "Authorization" section to the API key API docs.
These APIs require The new manage_api_key cluster privilege.

Relates: #43865
Backport of: #43811
2019-07-03 13:51:44 +10:00
Yogesh Gaikwad 480453aa24
Make role descriptors optional when creating API keys (#43481) (#43614)
This commit changes the `role_descriptors` field from required
to optional when creating API key. The default behavior in .NET ES
client is to omit properties with `null` value requiring additional
workarounds. The behavior for the API does not change.
Field names (`id`, `name`) in the invalidate api keys API documentation have been
corrected where they were wrong.

Closes #42053
2019-06-26 14:30:51 +10:00
Yogesh Gaikwad 2f173402ec
Add kerberos grant_type to get token in exchange for Kerberos ticket (#42847) (#43355)
Kibana wants to create access_token/refresh_token pair using Token
management APIs in exchange for kerberos tickets. `client_credentials`
grant_type requires every user to have `cluster:admin/xpack/security/token/create`
cluster privilege.

This commit introduces `_kerberos` grant_type for generating `access_token`
and `refresh_token` in exchange for a valid base64 encoded kerberos ticket.
In addition, `kibana_user` role now has cluster privilege to create tokens.
This allows Kibana to create access_token/refresh_token pair in exchange for
kerberos tickets.

Note:
The lifetime from the kerberos ticket is not used in ES and so even after it expires
the access_token/refresh_token pair will be valid. Care must be taken to invalidate
such tokens using token management APIs if required.

Closes #41943
2019-06-19 18:26:52 +10:00
jalvar08 b77be89c9a Remove Comma in Example (#41873)
The comma is there in error as there are no other parameter after 'value'
2019-06-07 08:39:27 -04:00
Mark Vieira e44b8b1e2e
[Backport] Remove dependency substitutions 7.x (#42866)
* Remove unnecessary usage of Gradle dependency substitution rules (#42773)

(cherry picked from commit 12d583dbf6f7d44f00aa365e34fc7e937c3c61f7)
2019-06-04 13:50:23 -07:00
Chris Cho 1514d1a1be Change shard allocation filter property and api (#42602)
The current example is not working and a bit confused. This change tries
to match it with the sample of the watcher blog.
2019-06-04 12:30:39 -04:00
Ioannis Kakavas 440ec4d9f5
[Backport 7.x] OpenID Connect realm guide (#42836)
This commit adds a configuration guide for the newly introduced
OpenID Connect realm. The guide is similar to the style of the
SAML Guide and shares certain parts where applicable (role mapping)
It also contains a short section on how the realm can be used for
authenticating users without Kibana.

Co-Authored-By: Lisa Cawley <lcawley@elastic.co>

Backport of #41423 and #42555
2019-06-04 14:08:41 +03:00
Alpar Torok eb1639c5fc TestClusters: Convert docs (#42100)
* TestClusters: Convert docs
2019-05-22 14:44:08 +03:00
Lisa Cawley fd2d4d761b [DOCS] Updates TLS configuration info (#41983) 2019-05-20 09:13:37 -04:00
Lisa Cawley 9284a70ec8 [DOCS] Updates security configuration overview (#41982) 2019-05-09 07:50:51 -07:00
Jason Tedor d7fd51a84e
Provide names for all artifact repositories (#41857)
This commit adds a name for each Maven and Ivy repository used in the
build.
2019-05-07 06:35:28 -04:00
James Rodewig 66d8549083 [DOCS] Move block delimiter and anchor in 'Configuring Security' for Asciidoctor migration (#41604) 2019-04-30 16:41:26 -04:00
Christoph Büscher 52495843cc [Docs] Fix common word repetitions (#39703) 2019-04-25 20:47:47 +02:00
RomainGeffraye a02f5544fe Update configuring-ldap-realm.asciidoc (#40427) 2019-04-25 13:29:46 -04:00
Albert Zaharovits fe5789ada1 Fix Has Privilege API check on restricted indices (#41226)
The Has Privileges API allows to tap into the authorization process, to validate
privileges without actually running the operations to be authorized. This commit
fixes a bug, in which the Has Privilege API returned spurious results when checking
for index privileges over restricted indices (currently .security, .security-6,
.security-7). The actual authorization process is not affected by the bug.
2019-04-25 12:03:27 +03:00
James Rodewig 122e727d28 [DOCS] Fix broken link to elasticsearch-php security page 2019-04-23 12:43:40 -04:00
Ioannis Kakavas b8d054e73b OpenID Connect realm settings and rest API docs (#40740)
This commit adds the relevant docs for the OpenID Connect
realm settings and the REST APIs that are exposed.
2019-04-22 15:50:18 +03:00
Lisa Cawley c03394c236 [DOCS] Fixes deprecation notice in pagerduty action (#41362) 2019-04-18 17:33:07 -07:00
Tim Vernum 2c770ba3cb
Support mustache templates in role mappings (#40571)
This adds a new `role_templates` field to role mappings that is an
alternative to the existing roles field.

These templates are evaluated at runtime to determine which roles should be
granted to a user.
For example, it is possible to specify:

    "role_templates": [
      { "template":{ "source": "_user_{{username}}" } }
    ]

which would mean that every user is assigned to their own role based on
their username.

You may not specify both roles and role_templates in the same role
mapping.

This commit adds support for templates to the role mapping API, the role
mapping engine, the Java high level rest client, and Elasticsearch
documentation.

Due to the lack of caching in our role mapping store, it is currently
inefficient to use a large number of templated role mappings. This will be
addressed in a future change.

Backport of: #39984, #40504
2019-04-02 20:55:10 +11:00
James Rodewig 39756fa538
[DOCS] Correct keystore commands for Email and Jira actions in Watcher (#40417) (#40612) 2019-04-01 08:25:26 -04:00
Sébastien Loix 59c98fbb66 [Docs] Fix command to save SMTP password for email account (#40444) 2019-03-27 10:36:53 -04:00
James Rodewig 03aaeb35cc Deprecate elasticsearch.yml as supported Slack config method (#40410) 2019-03-27 10:32:25 -04:00
Yogesh Gaikwad 280567da8d
Correct documentation link for authorization engine example (#40261) (#40292)
This commit fixes the link for authorization engine example.
2019-03-22 12:38:03 +11:00
Lisa Cawley 696cb22e4a [DOCS] Enable testing for API key examples (#39583) 2019-03-19 11:13:09 -07:00
Mark Vovchuk a054a9866e Update managing-roles.asciidoc 2019-03-19 08:21:29 -07:00
Lisa Cawley d093205b6a [DOCS] Fixes edit_url attributes that were externalized as text strings (#40161) 2019-03-18 14:32:02 -07:00
Lisa Cawley e050d50759 [DOCS] Removes X-Pack settings section (#39870) 2019-03-14 14:37:37 -07:00
Lisa Cawley efd7003ea9 [DOCS] Corrected API path for invalidate token and SSL certificate examples (#39530) 2019-03-12 13:28:37 -07:00
lcawl 4dadeba039 [DOCS] Sorts security APIs 2019-03-04 15:06:33 -08:00
Lisa Cawley f1a7166708 [DOCS] Adds link to list of built-in users (#39529) 2019-03-01 10:32:49 -08:00
Shajahan Palayil 8ced21db88
[DOCS] Corrected API path for /_security/api_key (#39521) 2019-02-28 20:08:39 +01:00
Lisa Cawley 8b26f59958 [DOCS] Removes problematic footer from Watcher docs (#39474) 2019-02-27 15:45:56 -08:00
Lisa Cawley 9c8c158f21 [DOCS] Fix inline callout in Watcher documentation (#39423) 2019-02-27 09:45:10 -08:00
Lisa Cawley dedbe60e0a [DOCS] Fixes table and code block separators in Watcher documentation (#39426) 2019-02-27 08:21:19 -08:00
Lee Hinman 7b8178c839
Remove Hipchat support from Watcher (#39374)
* Remove Hipchat support from Watcher (#39199)

Hipchat has been shut down and has previously been deprecated in
Watcher (#39160), therefore we should remove support for these actions.

* Add migrate note
2019-02-25 15:08:46 -07:00
Martijn Laarman 9b4d96534b
Fix #38623 remove xpack namespace REST API (#38625) (#39036)
* Fix #38623 remove xpack namespace REST API

Except for xpack.usage and xpack.info API's, this moves the last remaining API's out of the xpack namespace

* rename xpack api's inside inside the files as well

* updated yaml tests references to xpack namespaces api's

* update callsApi calls in the IT subclasses

* make sure docs testing does not use xpack namespaced api's

* fix leftover xpack namespaced method names in docs/build.gradle

* found another leftover reference

(cherry picked from commit ccb5d934363c37506b76119ac050a254fa80b5e7)
2019-02-18 12:40:07 +01:00
Lisa Cawley 339a15bb09 [DOCS] Edits warning in put watch API (#38582) 2019-02-15 09:40:12 -08:00
Alexander Reelsen 73fcea4d2c Remove ticks in chain input documentation (#38109)
The ticks created a literal string instead of actually accessing the
payload value.
2019-02-11 11:04:32 +01:00
Yogesh Gaikwad 6ff4a8cfd5
Add API key settings documentation (#38490)
This commit adds missing
API key service settings documentation.
2019-02-06 20:58:22 +11:00
Jay Modi 7ca5495d86
Allow custom authorization with an authorization engine (#38358)
For some users, the built in authorization mechanism does not fit their
needs and no feature that we offer would allow them to control the
authorization process to meet their needs. In order to support this,
a concept of an AuthorizationEngine is being introduced, which can be
provided using the security extension mechanism.

An AuthorizationEngine is responsible for making the authorization
decisions about a request. The engine is responsible for knowing how to
authorize and can be backed by whatever mechanism a user wants. The
default mechanism is one backed by roles to provide the authorization
decisions. The AuthorizationEngine will be called by the
AuthorizationService, which handles more of the internal workings that
apply in general to authorization within Elasticsearch.

In order to support external authorization services that would back an
authorization engine, the entire authorization process has become
asynchronous, which also includes all calls to the AuthorizationEngine.

The use of roles also leaked out of the AuthorizationService in our
existing code that is not specifically related to roles so this also
needed to be addressed. RequestInterceptor instances sometimes used a
role to ensure a user was not attempting to escalate their privileges.
Addressing this leakage of roles meant that the RequestInterceptor
execution needed to move within the AuthorizationService and that
AuthorizationEngines needed to support detection of whether a user has
more privileges on a name than another. The second area where roles
leaked to the user is in the handling of a few privilege APIs that
could be used to retrieve the user's privileges or ask if a user has
privileges to perform an action. To remove the leakage of roles from
these actions, the AuthorizationService and AuthorizationEngine gained
methods that enabled an AuthorizationEngine to return the response for
these APIs.

Ultimately this feature is the work included in:
#37785
#37495
#37328
#36245
#38137
#38219

Closes #32435
2019-02-05 13:39:29 -07:00
Julie Tibshirani 3ce7d2c9b6
Make sure to reject mappings with type _doc when include_type_name is false. (#38270)
`CreateIndexRequest#source(Map<String, Object>, ... )`, which is used when
deserializing index creation requests, accidentally accepts mappings that are
nested twice under the type key (as described in the bug report #38266).

This in turn causes us to be too lenient in parsing typeless mappings. In
particular, we accept the following index creation request, even though it
should not contain the type key `_doc`:

```
PUT index?include_type_name=false
{
  "mappings": {
    "_doc": {
      "properties": { ... }
    }
  }
}
```

There is a similar issue for both 'put templates' and 'put mappings' requests
as well.

This PR makes the minimal changes to detect and reject these typed mappings in
requests. It does not address #38266 generally, or attempt a larger refactor
around types in these server-side requests, as I think this should be done at a
later time.
2019-02-05 10:52:32 -08:00
Yogesh Gaikwad fe36861ada
Add support for API keys to access Elasticsearch (#38291)
X-Pack security supports built-in authentication service
`token-service` that allows access tokens to be used to 
access Elasticsearch without using Basic authentication.
The tokens are generated by `token-service` based on
OAuth2 spec. The access token is a short-lived token
(defaults to 20m) and refresh token with a lifetime of 24 hours,
making them unsuitable for long-lived or recurring tasks where
the system might go offline thereby failing refresh of tokens.

This commit introduces a built-in authentication service
`api-key-service` that adds support for long-lived tokens aka API
keys to access Elasticsearch. The `api-key-service` is consulted
after `token-service` in the authentication chain. By default,
if TLS is enabled then `api-key-service` is also enabled.
The service can be disabled using the configuration setting.

The API keys:-
- by default do not have an expiration but expiration can be
  configured where the API keys need to be expired after a
  certain amount of time.
- when generated will keep authentication information of the user that
   generated them.
- can be defined with a role describing the privileges for accessing
   Elasticsearch and will be limited by the role of the user that
   generated them
- can be invalidated via invalidation API
- information can be retrieved via a get API
- that have been expired or invalidated will be retained for 1 week
  before being deleted. The expired API keys remover task handles this.

Following are the API key management APIs:-
1. Create API Key - `PUT/POST /_security/api_key`
2. Get API key(s) - `GET /_security/api_key`
3. Invalidate API Key(s) `DELETE /_security/api_key`

The API keys can be used to access Elasticsearch using `Authorization`
header, where the auth scheme is `ApiKey` and the credentials, is the 
base64 encoding of API key Id and API key separated by a colon.
Example:-
```
curl -H "Authorization: ApiKey YXBpLWtleS1pZDphcGkta2V5" http://localhost:9200/_cluster/health
```

Closes #34383
2019-02-05 14:21:57 +11:00
Boaz Leskes b11732104f
Move watcher to use seq# and primary term for concurrency control (#37977)
* move watcher to seq# occ

* top level set

* fix parsing and missing setters

* share toXContent for PutResponse and rest end point

* fix redacted password

* fix username reference

* fix deactivate-watch.asciidoc have seq no references

* add seq# + term to activate-watch.asciidoc

* more doc fixes
2019-01-30 20:14:59 -05:00
Jake Landis 6a78b6a31c
Remove types from watcher docs (#38002)
Types have been deprecated and this commit removes the documentation
for specifying types in the index action, and search input/transform.

Relates #37594 #35190
2019-01-30 13:12:13 -06:00
Przemyslaw Gomulka 4f4113e964
Rename security audit.log to _audit.json (#37916)
in order to keep json logs consistent the security audit logs are renamed from .log to .json
relates #32850
2019-01-29 14:53:55 +01:00
Albert Zaharovits b6936e3c1e
Remove index audit output type (#37707)
This commit removes the Index Audit Output type, following its deprecation
in 6.7 by 8765a31d4e6770. It also adds the migration notice (settings notice).

In general, the problem with the index audit output is that event indexing
can be slower than the rate with which audit events are generated,
especially during the daily rollovers or the rolling cluster upgrades.
In this situation audit events will be lost which is a terrible failure situation
for an audit system.
Besides of the settings under the `xpack.security.audit.index` namespace, the
`xpack.security.audit.outputs` setting has also been deprecated and will be
removed in 7. Although explicitly configuring the logfile output does not touch
any deprecation bits, this setting is made redundant in 7 so this PR deprecates
it as well.

Relates #29881
2019-01-24 12:36:10 +02:00
Christoph Büscher 34f2d2ec91
Remove remaining occurances of "include_type_name=true" in docs (#37646) 2019-01-22 15:13:52 +01:00
Albert Zaharovits ff0f540255
Permission for restricted indices (#37577)
This grants the capability to grant privileges over certain restricted
indices (.security and .security-6 at the moment).
It also removes the special status of the superuser role.

IndicesPermission.Group is extended by adding the `allow_restricted_indices`
boolean flag. By default the flag is false. When it is toggled, you acknowledge
that the indices under the scope of the permission group can cover the
restricted indices as well. Otherwise, by default, restricted indices are ignored
when granting privileges, thus rendering them hidden for authorization purposes.
This effectively adds a confirmation "check-box" for roles that might grant
privileges to restricted indices.

The "special status" of the superuser role has been removed and coded as
any other role:
```
new RoleDescriptor("superuser",
    new String[] { "all" },
    new RoleDescriptor.IndicesPrivileges[] {
        RoleDescriptor.IndicesPrivileges.builder()
            .indices("*")
            .privileges("all")
            .allowRestrictedIndices(true)
// this ----^
            .build() },
            new RoleDescriptor.ApplicationResourcePrivileges[] {
                RoleDescriptor.ApplicationResourcePrivileges.builder()
                    .application("*")
                    .privileges("*")
                    .resources("*")
                    .build()
            },
            null, new String[] { "*" },
    MetadataUtils.DEFAULT_RESERVED_METADATA,
    Collections.emptyMap());
```
In the context of the Backup .security work, this allows the creation of a
"curator role" that would permit listing (get settings) for all indices
(including the restricted ones). That way the curator role would be able to 
ist and snapshot all indices, but not read or restore any of them.

Supersedes #36765
Relates #34454
2019-01-20 23:19:40 +02:00
Albert Zaharovits a88c050a05
Docs be explicit on how to turn off deprecated auditing (#37316)
Just be explicit about turning off the deprecated audit log appender
because we really want people to turn it off.
2019-01-15 14:29:32 +02:00
Julie Tibshirani 36a3b84fc9
Update the default for include_type_name to false. (#37285)
* Default include_type_name to false for get and put mappings.

* Default include_type_name to false for get field mappings.

* Add a constant for the default include_type_name value.

* Default include_type_name to false for get and put index templates.

* Default include_type_name to false for create index.

* Update create index calls in REST documentation to use include_type_name=true.

* Some minor clean-ups around the get index API.

* In REST tests, use include_type_name=true by default for index creation.

* Make sure to use 'expression == false'.

* Clarify the different IndexTemplateMetaData toXContent methods.

* Fix FullClusterRestartIT#testSnapshotRestore.

* Fix the ml_anomalies_default_mappings test.

* Fix GetFieldMappingsResponseTests and GetIndexTemplateResponseTests.

We make sure to specify include_type_name=true during xContent parsing,
so we continue to test the legacy typed responses. XContent generation
for the typeless responses is currently only covered by REST tests,
but we will be adding unit test coverage for these as we implement
each typeless API in the Java HLRC.

This commit also refactors GetMappingsResponse to follow the same appraoch
as the other mappings-related responses, where we read include_type_name
out of the xContent params, instead of creating a second toXContent method.
This gives better consistency in the response parsing code.

* Fix more REST tests.

* Improve some wording in the create index documentation.

* Add a note about types removal in the create index docs.

* Fix SmokeTestMonitoringWithSecurityIT#testHTTPExporterWithSSL.

* Make sure to mention include_type_name in the REST docs for affected APIs.

* Make sure to use 'expression == false' in FullClusterRestartIT.

* Mention include_type_name in the REST templates docs.
2019-01-14 13:08:01 -08:00
Jay Modi f3edbe2911
Security: remove SSL settings fallback (#36846)
This commit removes the fallback for SSL settings. While this may be
seen as a non user friendly change, the intention behind this change
is to simplify the reasoning needed to understand what is actually
being used for a given SSL configuration. Each configuration now needs
to be explicitly specified as there is no global configuration or
fallback to some other configuration.

Closes #29797
2019-01-14 14:06:22 -07:00
Albert Zaharovits 6fd57d90da
Security Audit includes HTTP method for requests (#37322)
Adds another field, named "request.method", to the structured logfile audit.
This field is present for all events associated with a REST request (not a
transport request) and the value is one of GET, POST, PUT, DELETE, OPTIONS,
HEAD, PATCH, TRACE and CONNECT.
2019-01-13 15:26:23 +02:00
Ioannis Kakavas 80084138dd [DOCS] Fix link to role mapping doc 2019-01-11 09:22:40 +02:00
Josh Soref edb48321ba [DOCS] Various spelling corrections (#37046) 2019-01-07 14:44:12 +01:00
Ioannis Kakavas 3c12c372d0
[DOCS] Adjust role mapping docs for SAML (#37083)
Explicitly mention that file based role mappings cannot be used with
the SAML realm.
2019-01-04 08:26:41 +02:00
lcawl 32bed098bb [DOCS] Synchs titles of X-Pack APIs 2018-12-20 10:27:24 -08:00
lcawl 8b8121279a [DOCS] Fix typos 2018-12-19 15:28:25 -08:00
Ioannis Kakavas c4e12f06d7
Link to the SAML troubleshooting doc from guide (#36754)
Explicitly call out the existence of the troubleshooting guide so
that hopefully users can solve common and easy problems with their
initial configuration
2018-12-20 01:11:55 +02:00
Lisa Cawley 4140b9eede
[DOCS] Update X-Pack terminology in security docs (#36564) 2018-12-19 14:53:37 -08:00
Albert Zaharovits 63aa8756b2
Add X-Forwarded-For to the logfile audit (#36427)
Extracts the value of the X-Forwarded-For HTTP request header and
places it in the audit entries from the logfile output.
2018-12-19 14:56:40 +02:00
lcawl a8387592db [DOCS] Fixes broken links to tcp_tranport_profiles 2018-12-18 14:43:58 -08:00
Tim Brooks 47a9a8de49
Update transport docs and settings for changes (#36786)
This is related to #36652. In 7.0 we plan to deprecate a number of
settings that make reference to the concept of a tcp transport. We
mostly just have a single transport type now (based on tcp). Settings
should only reference tcp if they are referring to socket options. This
commit updates the settings in the docs. And removes string usages of
the old settings. Additionally it adds a missing remote compress setting
to the docs.
2018-12-18 13:09:58 -07:00
Ioannis Kakavas 7b9ca62174
Enhance Invalidate Token API (#35388)
This change:

- Adds functionality to invalidate all (refresh+access) tokens for all users of a realm
- Adds functionality to invalidate all (refresh+access)tokens for a user in all realms
- Adds functionality to invalidate all (refresh+access) tokens for a user in a specific realm
- Changes the response format for the invalidate token API to contain information about the 
   number of the invalidated tokens and possible errors that were encountered.
- Updates the API Documentation

After back-porting to 6.x, the `created` field will be removed from master as a field in the 
response

Resolves: #35115
Relates: #34556
2018-12-18 10:05:50 +02:00
docteurjay c1e3bdd42d docs: Remove issue key from jira documentation (#36136)
This commit removes an incorrect "issues" key from the documentation.
see: https://developer.atlassian.com/server/jira/platform/rest-apis/

Fixes #36128
2018-12-11 15:17:15 -06:00
Ioannis Kakavas d7c5d8049a
Deprecate /_xpack/security/* in favor of /_security/* (#36293)
* This commit is part of our plan to deprecate and ultimately remove the use of _xpack in the REST APIs.

- REST API docs
- HLRC docs and doc tests
- Handle REST actions with deprecation warnings
- Changed endpoints in rest-api-spec and relevant file names
2018-12-11 11:13:10 +02:00
Michael Basnight b5b6e37a60
Deprecate X-Pack centric watcher endpoints (#36218)
This commit is part of our plan to deprecate and ultimately remove the use of
_xpack in the REST APIs.

Relates #35958
2018-12-08 12:57:16 -06:00
Alexander Reelsen 7693d538ca
Docs: Mention that fields with dots do not work (#36295)
The dot is used as a splitting character internally for looking up
values in the array compare condition, thus the user should use the
script condition in such cases.
2018-12-07 09:38:12 +01:00
Alexander Reelsen 97259f0efc
Docs: Add password keystore setting for email account passwords (#33409)
Always refer to secure settings when setting up email account passwords
2018-12-06 14:35:53 +01:00
Jim Ferenczi 18866c4c0b
Make hits.total an object in the search response (#35849)
This commit changes the format of the `hits.total` in the search response to be an object with
a `value` and a `relation`. The `value` indicates the number of hits that match the query and the
`relation` indicates whether the number is accurate (in which case the relation is equals to `eq`)
or a lower bound of the total (in which case it is equals to `gte`).
This change also adds a parameter called `rest_total_hits_as_int` that can be used in the
search APIs to opt out from this change (retrieve the total hits as a number in the rest response).
Note that currently all search responses are accurate (`track_total_hits: true`) or they don't contain
`hits.total` (`track_total_hits: true`). We'll add a way to get a lower bound of the total hits in a
follow up (to allow numbers to be passed to `track_total_hits`).

Relates #33028
2018-12-05 19:49:06 +01:00
Lisa Cawley e1fb1505f9
[DOCS] Moves security config file info (#36232) 2018-12-04 13:18:54 -08:00
Lisa Cawley 46962308aa
[DOCS] Replace deprecated ldap setting (#36022) 2018-11-30 16:58:19 -08:00
Albert Zaharovits 36819f78ef
DOCS Audit event attributes in new format (#35510)
Accounts for the `Structured Audit Entries` in the format
documentation.
2018-11-28 01:24:03 +02:00
Ioannis Kakavas 580b5baf21
Add realm information for Authenticate API (#35648)
- Add the authentication realm and lookup realm name and type in the response for the _authenticate API
- The authentication realm is set as the lookup realm too (instead of setting the lookup realm to null or empty ) when no lookup realm is used.
2018-11-27 23:35:42 +02:00
Ioannis Kakavas e8ec4fad7b
[DOCS] Adjust Invalidate Token REST API docs (#35622)
- Renames API to Invalidate Token
- Explicitly calls out the possibility to invalidate refresh tokens
via this API
2018-11-21 09:32:56 +02:00
Tim Vernum d367894c31
DOCS: Add password_hash & refresh to Put User API (#35574)
PR #35242 formalised support for the password_hash field in the body
of the Put User security API.
Since this field is now validated and tested, it can also be
documented.

The Put User API also supports a "refresh" query parameter that was
not documented. This commit adds it to the docs.
2018-11-20 13:03:22 +11:00
lcawl 3a232d7897 [DOCS] Adds anchor for SAML Kibana basic info 2018-11-19 11:42:45 -08:00
Tanguy Leroux a8433a3164
[Docs] Fix `service_api_key` in PagerDuty documentation (#35589)
closes #35572
2018-11-19 16:30:20 +01:00
Albert Zaharovits 617f91bb0f
DOCS Auditing search queries (#35301)
This documents how to include the search queries in the audit log.

There is a catch, that even if enabling `emit_request_body`, which should
output queries included in request bodies, search queries were not output
because, implicitly, no REST layer audit event type was included.

This folk knowledge is herein imprinted.
2018-11-09 11:38:45 +02:00
Jake Landis d7a4cef483
watcher: Fix integration tests to ensure correct start/stop of Watcher (#35271)
Ensure that Watcher is correctly started and stopped between tests for
SmokeTestWatcherWithSecurityIT,
SmokeTestWatcherWithSecurityClientYamlTestSuiteIT,
SmokeTestWatcherTestSuiteIT, WatcherRestIT,
XDocsClientYamlTestSuiteIT, and XPackRestIT

The change here is to throw an `AssertionError` instead of `break;` to
allow the `assertBusy()` to continue to busy wait until the desired
state is reached.

closes #33291, closes #29877, closes #34462, closes #30705, closes #34448
2018-11-07 15:08:03 -06:00
Tim Vernum 574ec6686e
Include realm type in Security Realm setting keys (#30241)
This moves all Realm settings to an Affix definition.
However, because different realm types define different settings
(potentially conflicting settings) this requires that the realm type
become part of the setting key.

Thus, we now need to define realm settings as:

    xpack.security.authc.realms:
      file.file1:
        order: 0

      native.native1:
        order: 1

- This is a breaking change to realm config
- This is also a breaking change to custom security realms (SecurityExtension)
2018-11-06 14:56:50 +11:00
Jake Landis c2766b65cf
ingest: raise visibility of ingest plugin documentation (#35048)
* move the set security user processor to the main documentation
* link to plugin processors

part of #33188
2018-11-05 11:44:10 -06:00
Igor Motov 7b13d0592f
Watcher: fix metric stats names (#34951)
* Watcher: fix metric stats names

The current watcher stats metric names doesn't match the current
documentation. This commit fixes the behavior of `queued_watches`
metric, deprecates `pending_watches` metric and adds `current_watches`
to match the documented behavior. It also fixes the documentation, which
introduced `executing_watches` metric that was never added.

Fixes #34865
2018-11-01 10:12:05 -04:00
Albert Zaharovits 3f1fec1813
[Docs] audit logfile structured format (#34584)
Documents the new structured logfile format for auditing
that was introduced by #31931. Most changes herein
are for 6.x . In 7.0 the deprecated format is gone and a
follow-up PR is in order.
2018-10-26 15:19:35 +03:00
Jason Tedor 8bed0c68e2
Fix remote clusters section link
This commit fixes the remote clusters section link by changing some
underscores to dashes.
2018-10-20 11:16:39 -04:00
Jason Tedor 77d23a2007
Fix remote clusters section link
This fixes a link to the configuring remote clusters section of the
docs.
2018-10-20 10:28:25 -04:00