Commit Graph

270 Commits

Author SHA1 Message Date
Mayya Sharipova 22ab389531 Clarify that FLS/DLS disable shard request cache (#45462) 2019-08-13 09:05:57 -04:00
Lisa Cawley 00235bbecd [DOCS] Reformats the security APIs (#45124) 2019-08-02 11:32:47 -07:00
Alison Goryachev b607148ae9 [DOCS] Fix watcher email action docs (#44877) 2019-08-02 14:02:08 -04:00
Ioannis Kakavas 56da35b706 Indicate that some user APIs handle built-in users (#44857)
The Get Users API also returns users form the restricted realm or built-in users,
 as we call them in our docs. One can also change the passwords of built-in 
users with the Change Password API
2019-07-31 17:55:28 +03:00
Ioannis Kakavas ac131f986b Document xpack.security.authc.saml.realm for Kibana (#44705)
Since 7.3, it's possible to explicitly configure the SAML realm to
be used in Kibana's configuration. This in turn, eliminates the need
of properly setting `xpack.security.public.*` settings in Kibana
and largely simplifies relevant documentation.
This also changes `xpack.security.authProviders` to
`xpack.security.authc.providers` as the former was deprecated in
favor of the latter in 7.3 in Kibana
2019-07-26 09:38:49 +03:00
Lisa Cawley c9909b09b5 [DOCS] Adds command reference for elasticsearch-croneval (#43946) 2019-07-25 11:41:05 -07:00
James Rodewig ad7c164dd0 [DOCS] Rewrite `regexp` query (#42711) 2019-07-24 08:38:41 -04:00
James Rodewig d46545f729 [DOCS] Update anchors and links for Elasticsearch API relocation (#44500) 2019-07-19 09:18:23 -04:00
Lisa Cawley 621ec7cb84 [DOCS] Removes out-dated info from Watcher limitations (#44252) 2019-07-17 13:24:02 -07:00
James Rodewig ac07eef86c [DOCS] Remove :edit_url: overrides. (#44445)
These overrides do not work in Asciidoctor and are no longer needed.
2019-07-16 15:04:44 -04:00
Lee Hinman fb0461ac76
[7.x] Add Snapshot Lifecycle Management (#44382)
* Add Snapshot Lifecycle Management (#43934)

* Add SnapshotLifecycleService and related CRUD APIs

This commit adds `SnapshotLifecycleService` as a new service under the ilm
plugin. This service handles snapshot lifecycle policies by scheduling based on
the policies defined schedule.

This also includes the get, put, and delete APIs for these policies

Relates to #38461

* Make scheduledJobIds return an immutable set

* Use Object.equals for SnapshotLifecyclePolicy

* Remove unneeded TODO

* Implement ToXContentFragment on SnapshotLifecyclePolicyItem

* Copy contents of the scheduledJobIds

* Handle snapshot lifecycle policy updates and deletions (#40062)

(Note this is a PR against the `snapshot-lifecycle-management` feature branch)

This adds logic to `SnapshotLifecycleService` to handle updates and deletes for
snapshot policies. Policies with incremented versions have the old policy
cancelled and the new one scheduled. Deleted policies have their schedules
cancelled when they are no longer present in the cluster state metadata.

Relates to #38461

* Take a snapshot for the policy when the SLM policy is triggered (#40383)

(This is a PR for the `snapshot-lifecycle-management` branch)

This commit fills in `SnapshotLifecycleTask` to actually perform the
snapshotting when the policy is triggered. Currently there is no handling of the
results (other than logging) as that will be added in subsequent work.

This also adds unit tests and an integration test that schedules a policy and
ensures that a snapshot is correctly taken.

Relates to #38461

* Record most recent snapshot policy success/failure (#40619)

Keeping a record of the results of the successes and failures will aid
troubleshooting of policies and make users more confident that their
snapshots are being taken as expected.

This is the first step toward writing history in a more permanent
fashion.

* Validate snapshot lifecycle policies (#40654)

(This is a PR against the `snapshot-lifecycle-management` branch)

With the commit, we now validate the content of snapshot lifecycle policies when
the policy is being created or updated. This checks for the validity of the id,
name, schedule, and repository. Additionally, cluster state is checked to ensure
that the repository exists prior to the lifecycle being added to the cluster
state.

Part of #38461

* Hook SLM into ILM's start and stop APIs (#40871)

(This pull request is for the `snapshot-lifecycle-management` branch)

This change allows the existing `/_ilm/stop` and `/_ilm/start` APIs to also
manage snapshot lifecycle scheduling. When ILM is stopped all scheduled jobs are
cancelled.

Relates to #38461

* Add tests for SnapshotLifecyclePolicyItem (#40912)

Adds serialization tests for SnapshotLifecyclePolicyItem.

* Fix improper import in build.gradle after master merge

* Add human readable version of modified date for snapshot lifecycle policy (#41035)

* Add human readable version of modified date for snapshot lifecycle policy

This small change changes it from:

```
...
"modified_date": 1554843903242,
...
```

To

```
...
"modified_date" : "2019-04-09T21:05:03.242Z",
"modified_date_millis" : 1554843903242,
...
```

Including the `"modified_date"` field when the `?human` field is used.

Relates to #38461

* Fix test

* Add API to execute SLM policy on demand (#41038)

This commit adds the ability to perform a snapshot on demand for a policy. This
can be useful to take a snapshot immediately prior to performing some sort of
maintenance.

```json
PUT /_ilm/snapshot/<policy>/_execute
```

And it returns the response with the generated snapshot name:

```json
{
  "snapshot_name" : "production-snap-2019.04.09-rfyv3j9qreixkdbnfuw0ug"
}
```

Note that this does not allow waiting for the snapshot, and the snapshot could
still fail. It *does* record this information into the cluster state similar to
a regularly trigged SLM job.

Relates to #38461

* Add next_execution to SLM policy metadata (#41221)

* Add next_execution to SLM policy metadata

This adds the next time a snapshot lifecycle policy will be executed when
retriving a policy's metadata, for example:

```json
GET /_ilm/snapshot?human
{
  "production" : {
    "version" : 1,
    "modified_date" : "2019-04-15T21:16:21.865Z",
    "modified_date_millis" : 1555362981865,
    "policy" : {
      "name" : "<production-snap-{now/d}>",
      "schedule" : "*/30 * * * * ?",
      "repository" : "repo",
      "config" : {
        "indices" : [
          "foo-*",
          "important"
        ],
        "ignore_unavailable" : true,
        "include_global_state" : false
      }
    },
    "next_execution" : "2019-04-15T21:16:30.000Z",
    "next_execution_millis" : 1555362990000
  },
  "other" : {
    "version" : 1,
    "modified_date" : "2019-04-15T21:12:19.959Z",
    "modified_date_millis" : 1555362739959,
    "policy" : {
      "name" : "<other-snap-{now/d}>",
      "schedule" : "0 30 2 * * ?",
      "repository" : "repo",
      "config" : {
        "indices" : [
          "other"
        ],
        "ignore_unavailable" : false,
        "include_global_state" : true
      }
    },
    "next_execution" : "2019-04-16T02:30:00.000Z",
    "next_execution_millis" : 1555381800000
  }
}
```

Relates to #38461

* Fix and enhance tests

* Figured out how to Cron

* Change SLM endpoint from /_ilm/* to /_slm/* (#41320)

This commit changes the endpoint for snapshot lifecycle management from:

```
GET /_ilm/snapshot/<policy>
```

to:

```
GET /_slm/policy/<policy>
```

It mimics the ILM path only using `slm` instead of `ilm`.

Relates to #38461

* Add initial documentation for SLM (#41510)

* Add initial documentation for SLM

This adds the initial documentation for snapshot lifecycle management.

It also includes the REST spec API json files since they're sort of
documentation.

Relates to #38461

* Add `manage_slm` and `read_slm` roles (#41607)

* Add `manage_slm` and `read_slm` roles

This adds two more built in roles -

`manage_slm` which has permission to perform any of the SLM actions, as well as
stopping, starting, and retrieving the operation status of ILM.

`read_slm` which has permission to retrieve snapshot lifecycle policies as well
as retrieving the operation status of ILM.

Relates to #38461

* Add execute to the test

* Fix ilm -> slm typo in test

* Record SLM history into an index (#41707)

It is useful to have a record of the actions that Snapshot Lifecycle
Management takes, especially for the purposes of alerting when a
snapshot fails or has not been taken successfully for a certain amount of
time.

This adds the infrastructure to record SLM actions into an index that
can be queried at leisure, along with a lifecycle policy so that this
history does not grow without bound.

Additionally,
SLM automatically setting up an index + lifecycle policy leads to
`index_lifecycle` custom metadata in the cluster state, which some of
the ML tests don't know how to deal with due to setting up custom
`NamedXContentRegistry`s.  Watcher would cause the same problem, but it
is already disabled (for the same reason).

* High Level Rest Client support for SLM (#41767)

* High Level Rest Client support for SLM

This commit add HLRC support for SLM.

Relates to #38461

* Fill out documentation tests with tags

* Add more callouts and asciidoc for HLRC

* Update javadoc links to real locations

* Add security test testing SLM cluster privileges (#42678)

* Add security test testing SLM cluster privileges

This adds a test to `PermissionsIT` that uses the `manage_slm` and `read_slm`
cluster privileges.

Relates to #38461

* Don't redefine vars

*  Add Getting Started Guide for SLM  (#42878)

This commit adds a basic Getting Started Guide for SLM.

* Include SLM policy name in Snapshot metadata (#43132)

Keep track of which SLM policy in the metadata field of the Snapshots
taken by SLM. This allows users to more easily understand where the
snapshot came from, and will enable future SLM features such as
retention policies.

* Fix compilation after master merge

* [TEST] Move exception wrapping for devious exception throwing

Fixes an issue where an exception was created from one line and thrown in another.

* Fix SLM for the change to AcknowledgedResponse

* Add Snapshot Lifecycle Management Package Docs (#43535)

* Fix compilation for transport actions now that task is required

* Add a note mentioning the privileges needed for SLM (#43708)

* Add a note mentioning the privileges needed for SLM

This adds a note to the top of the "getting started with SLM"
documentation mentioning that there are two built-in privileges to
assist with creating roles for SLM users and administrators.

Relates to #38461

* Mention that you can create snapshots for indices you can't read

* Fix REST tests for new number of cluster privileges

* Mute testThatNonExistingTemplatesAreAddedImmediately (#43951)

* Fix SnapshotHistoryStoreTests after merge

* Remove overridden newResponse functions that have been removed

* Fix compilation for backport

* Fix get snapshot output parsing in test

* [DOCS] Add redirects for removed autogen anchors (#44380)

* Switch <tt>...</tt> in javadocs for {@code ...}
2019-07-16 07:37:13 -06:00
Lisa Cawley 753da8feac [DOCS] Updates terminology for alerting features (#43945) 2019-07-15 14:47:33 -07:00
Lisa Cawley aa6b544fac [DOCS] Moves Watcher troubleshooting page (#44250) 2019-07-12 08:28:18 -07:00
Ioannis Kakavas 475752be75
Make plugin verification FIPS 140 compliant (#44266)
This change makes the process of verifying the signature of
official plugins FIPS 140 compliant by defaulting to use the
BouncyCastle FIPS provider and adding a dependency to bcpg-fips
that implement parts of openPGP in a FIPS compliant manner.

In already FIPS 140 enabled environments that use the 
BouncyCastle FIPS provider, the bcfips dependency is redundant
but doesn't cause an issue as it will be added only in the classpath
 of the cli-tools

This is a backport of #44224
2019-07-12 14:34:15 +03:00
Mark Vieira 263f76e5ea
Revert "[DOCS] Moves Watcher troubleshooting page (#44144)"
This reverts commit 11375926ec.
2019-07-11 17:13:08 -07:00
Lisa Cawley 11375926ec [DOCS] Moves Watcher troubleshooting page (#44144) 2019-07-11 14:42:32 -07:00
Lisa Cawley 5b71340f99 [DOCS] Moves Watcher limitations (#44141) 2019-07-10 11:17:12 -07:00
Albert Zaharovits 018d946bba [DOC] Backup & Restore Security Configuration (#42970)
This commit documents the backup and restore of a cluster's
security configuration.

It is not possible to only backup (or only restore) security
configuration, independent to the rest of the cluster's conf,
so this describes how a full configuration backup&restore
will include security as well. Moreover, it explains how part
of the security conf data resides on the special .security
index and how to backup that using regular data snapshot API.

Co-Authored-By: Lisa Cawley <lcawley@elastic.co>
Co-Authored-By: Tim Vernum <tim@adjective.org>
2019-07-10 14:53:56 +03:00
marcos ramos 88ee47c9ba Fix OIDC documentation settings (#44115)
Current kibana setting is xpack.security.auth.oidc.realm, 
but the correct  one is xpack.security.authc.oidc.realm
2019-07-09 18:44:35 +03:00
Sachin Frayne 389c923a82 [Docs] Fix json syntax in watcher compare condition (#44032) 2019-07-09 13:43:18 +02:00
Alexander Reelsen 9077c4402f Watcher: Allow to execute actions for each element in array (#41997)
This adds the ability to execute an action for each element that occurs
in an array, for example you could sent a dedicated slack action for
each search hit returned from a search.

There is also a limit for the number of actions executed, which is
hardcoded to 100 right now, to prevent having watches run forever.

The watch history logs each action result and the total number of actions
the were executed.

Relates #34546
2019-07-03 11:28:50 +02:00
Tim Vernum 2a8f30eb9a
Support builtin privileges in get privileges API (#43901)
Adds a new "/_security/privilege/_builtin" endpoint so that builtin
index and cluster privileges can be retrieved via the Rest API

Backport of: #42134
2019-07-03 19:08:28 +10:00
Tim Vernum 14884c871f
Document API-Key APIs require manage_api_key priv (#43869)
Add the "Authorization" section to the API key API docs.
These APIs require The new manage_api_key cluster privilege.

Relates: #43865
Backport of: #43811
2019-07-03 13:51:44 +10:00
Yogesh Gaikwad 480453aa24
Make role descriptors optional when creating API keys (#43481) (#43614)
This commit changes the `role_descriptors` field from required
to optional when creating API key. The default behavior in .NET ES
client is to omit properties with `null` value requiring additional
workarounds. The behavior for the API does not change.
Field names (`id`, `name`) in the invalidate api keys API documentation have been
corrected where they were wrong.

Closes #42053
2019-06-26 14:30:51 +10:00
Yogesh Gaikwad 2f173402ec
Add kerberos grant_type to get token in exchange for Kerberos ticket (#42847) (#43355)
Kibana wants to create access_token/refresh_token pair using Token
management APIs in exchange for kerberos tickets. `client_credentials`
grant_type requires every user to have `cluster:admin/xpack/security/token/create`
cluster privilege.

This commit introduces `_kerberos` grant_type for generating `access_token`
and `refresh_token` in exchange for a valid base64 encoded kerberos ticket.
In addition, `kibana_user` role now has cluster privilege to create tokens.
This allows Kibana to create access_token/refresh_token pair in exchange for
kerberos tickets.

Note:
The lifetime from the kerberos ticket is not used in ES and so even after it expires
the access_token/refresh_token pair will be valid. Care must be taken to invalidate
such tokens using token management APIs if required.

Closes #41943
2019-06-19 18:26:52 +10:00
jalvar08 b77be89c9a Remove Comma in Example (#41873)
The comma is there in error as there are no other parameter after 'value'
2019-06-07 08:39:27 -04:00
Chris Cho 1514d1a1be Change shard allocation filter property and api (#42602)
The current example is not working and a bit confused. This change tries
to match it with the sample of the watcher blog.
2019-06-04 12:30:39 -04:00
Ioannis Kakavas 440ec4d9f5
[Backport 7.x] OpenID Connect realm guide (#42836)
This commit adds a configuration guide for the newly introduced
OpenID Connect realm. The guide is similar to the style of the
SAML Guide and shares certain parts where applicable (role mapping)
It also contains a short section on how the realm can be used for
authenticating users without Kibana.

Co-Authored-By: Lisa Cawley <lcawley@elastic.co>

Backport of #41423 and #42555
2019-06-04 14:08:41 +03:00
Lisa Cawley fd2d4d761b [DOCS] Updates TLS configuration info (#41983) 2019-05-20 09:13:37 -04:00
Lisa Cawley 9284a70ec8 [DOCS] Updates security configuration overview (#41982) 2019-05-09 07:50:51 -07:00
Jason Tedor d7fd51a84e
Provide names for all artifact repositories (#41857)
This commit adds a name for each Maven and Ivy repository used in the
build.
2019-05-07 06:35:28 -04:00
James Rodewig 66d8549083 [DOCS] Move block delimiter and anchor in 'Configuring Security' for Asciidoctor migration (#41604) 2019-04-30 16:41:26 -04:00
Christoph Büscher 52495843cc [Docs] Fix common word repetitions (#39703) 2019-04-25 20:47:47 +02:00
RomainGeffraye a02f5544fe Update configuring-ldap-realm.asciidoc (#40427) 2019-04-25 13:29:46 -04:00
Albert Zaharovits fe5789ada1 Fix Has Privilege API check on restricted indices (#41226)
The Has Privileges API allows to tap into the authorization process, to validate
privileges without actually running the operations to be authorized. This commit
fixes a bug, in which the Has Privilege API returned spurious results when checking
for index privileges over restricted indices (currently .security, .security-6,
.security-7). The actual authorization process is not affected by the bug.
2019-04-25 12:03:27 +03:00
James Rodewig 122e727d28 [DOCS] Fix broken link to elasticsearch-php security page 2019-04-23 12:43:40 -04:00
Ioannis Kakavas b8d054e73b OpenID Connect realm settings and rest API docs (#40740)
This commit adds the relevant docs for the OpenID Connect
realm settings and the REST APIs that are exposed.
2019-04-22 15:50:18 +03:00
Lisa Cawley c03394c236 [DOCS] Fixes deprecation notice in pagerduty action (#41362) 2019-04-18 17:33:07 -07:00
Tim Vernum 2c770ba3cb
Support mustache templates in role mappings (#40571)
This adds a new `role_templates` field to role mappings that is an
alternative to the existing roles field.

These templates are evaluated at runtime to determine which roles should be
granted to a user.
For example, it is possible to specify:

    "role_templates": [
      { "template":{ "source": "_user_{{username}}" } }
    ]

which would mean that every user is assigned to their own role based on
their username.

You may not specify both roles and role_templates in the same role
mapping.

This commit adds support for templates to the role mapping API, the role
mapping engine, the Java high level rest client, and Elasticsearch
documentation.

Due to the lack of caching in our role mapping store, it is currently
inefficient to use a large number of templated role mappings. This will be
addressed in a future change.

Backport of: #39984, #40504
2019-04-02 20:55:10 +11:00
James Rodewig 39756fa538
[DOCS] Correct keystore commands for Email and Jira actions in Watcher (#40417) (#40612) 2019-04-01 08:25:26 -04:00
Sébastien Loix 59c98fbb66 [Docs] Fix command to save SMTP password for email account (#40444) 2019-03-27 10:36:53 -04:00
James Rodewig 03aaeb35cc Deprecate elasticsearch.yml as supported Slack config method (#40410) 2019-03-27 10:32:25 -04:00
Yogesh Gaikwad 280567da8d
Correct documentation link for authorization engine example (#40261) (#40292)
This commit fixes the link for authorization engine example.
2019-03-22 12:38:03 +11:00
Lisa Cawley 696cb22e4a [DOCS] Enable testing for API key examples (#39583) 2019-03-19 11:13:09 -07:00
Mark Vovchuk a054a9866e Update managing-roles.asciidoc 2019-03-19 08:21:29 -07:00
Lisa Cawley d093205b6a [DOCS] Fixes edit_url attributes that were externalized as text strings (#40161) 2019-03-18 14:32:02 -07:00
Lisa Cawley e050d50759 [DOCS] Removes X-Pack settings section (#39870) 2019-03-14 14:37:37 -07:00
Lisa Cawley efd7003ea9 [DOCS] Corrected API path for invalidate token and SSL certificate examples (#39530) 2019-03-12 13:28:37 -07:00
lcawl 4dadeba039 [DOCS] Sorts security APIs 2019-03-04 15:06:33 -08:00
Lisa Cawley f1a7166708 [DOCS] Adds link to list of built-in users (#39529) 2019-03-01 10:32:49 -08:00