This change disables security for trial licenses unless security is
explicitly enabled in the settings. This is done to facilitate users
getting started and not having to deal with some of the complexities
involved in getting security configured. In order to do this and avoid
disabling security for existing users that have gold or platinum
licenses, we have to disable security after cluster formation so that
the license can be retrieved.
relates elastic/x-pack-elasticsearch#4078
Original commit: elastic/x-pack-elasticsearch@96bdb889fc
Changes are done in SetupPasswordTool to load the keystore
and set security settings to load password protected keys in SSL configuration.
Check for the verification mode and appropriately use hostname verifier.
Close the keystore after setup-password is complete.
Unit test for CommandLineHttpClient
TODO: TLS documentation needs to be fixed, which will be taken up as a separate
fix due to documentation refactoring in progress.
relates elastic/x-pack-elasticsearch#3760
Original commit: elastic/x-pack-elasticsearch@72e8666e21
SAML is only available on a Platinum license. If you try and use SAML
on a Gold license, then the error message is misleading - it gives the
equivalent of "cannot find saml realm".
This change adds a standard license error of
"current license is non-compliant for [saml]"
if SAML rest actions are used when SAML is not licensed.
Original commit: elastic/x-pack-elasticsearch@7c0e26d58e
Trying to setup passwords on a red cluster (or a cluster that cannot
reach a quorum) is generally not a good idea.
This commit:
- Adds a check for RED cluster status
- Prompts to confirm execution if the cluster is red
- Prints out the reason/type is an error response is received
- Increases the HTTP read timeout so that master election failures are
reported correctly.
Original commit: elastic/x-pack-elasticsearch@4ffbda23db
This creates a new "beats_system" user and role with the same
privileges as the existing "logstash_system" user/role.
The "beat_system" user is also added as a managed user within
the "setup-passwords" command.
Users who upgrade from an earlier version of Elasticsearch/X-Pack
will need to manually set a password for the beats_system user via
the change password API (or Kibana UI)
Original commit: elastic/x-pack-elasticsearch@6087d3a18e
Add realm name to all authz audit events: accessDenied, accessGranted,
runAsDenied and runAsGranted.
These event types receive the following attributes: realm,
run_by_realm and run_as_realm to go along with with the existing
attributes: principal, run_by_principal and run_as_principal. The
'effective realm name' (run_as_realm or run_by_realm) is certainly
filterable by ignore policies.
Original commit: elastic/x-pack-elasticsearch@cb3801e197
Audit trail filter policies can now be updated by the cluster update
settings API. Previously, policies were static inside the conf file.
This is helpful because, in practice, coming up with a reasonable
policy set is an iterative process, adding and changing policies to
shave off yet another pesky audit event.
Original commit: elastic/x-pack-elasticsearch@e8a670c427
This commit adds a Samba4 test fixture that acts as a domain controller
and has the same contents as the cloud active directory instance that
we previously used for tests.
The tests also support reading information from environment variables
so that they can be run against a real active directory instance in our
CI builds.
In addition, this commit also fixes a few issues that surfaced when
making this change. The first is a change in the base DN that is
searched when performing down-level authentication. The base DN is
now the configuration object instead of the domain DN. This change was
required due to the original producing unnecessary referrals, which we
cannot easily follow when running against this test figure. Referrals
cannot easily be followed as they are returned by the ldap server with
an unresolvable DNS name unless the host points to the samba4 instance
for DNS. The port returned in the referral url is the one samba is bound
to, which differs from the port that is forwarded to the host by the
test fixture.
The other issue that is resolved by this change is the addition of
settings that allow specifying non-standard ports for active directory.
This is needed for down-level authentication as we may need to query
the regular port of active directory instead of the global catalog
port as the configuration object is not replicated to the global
catalog.
relates elastic/x-pack-elasticsearch#185
Relates elastic/x-pack-elasticsearch#3800
Original commit: elastic/x-pack-elasticsearch@883c742fba
* Decouple XContentBuilder from BytesReference
This commit handles the removal of all mentions of BytesReference from
XContentBuilder. This is needed so that we can completely decouple the XContent
code and move it into its own dependency.
This is the x-pack side of https://github.com/elastic/elasticsearch/pull/28972
Original commit: elastic/x-pack-elasticsearch@8ba2e97b26
This commit replaces the usage of Lucene IOUtils with Elasticsearch
IOUtils, the former of which is now forbidden.
Original commit: elastic/x-pack-elasticsearch@8e0554001f
SslConfiguration can depend on SecureSettings, so it must be
constructed during the correct lifecycle phase.
For PkiRealmBootstrapCheck, moved the construction of SslConfiguration
objets into the constructor rather than the check method
Original commit: elastic/x-pack-elasticsearch@1a4d147216
If the Assertion Consumer Service URL already contained query
parameters, we would incorrectly append an addtional '?' rather than
adding the SAML parameters to the end with '&'
Original commit: elastic/x-pack-elasticsearch@60b6a977d8
This wraps the stream (`.streamInput()`) that is passed to many of the
`createParser` instances in the enclosing (or a new) try-with-resources block.
This ensures the `BytesReference.streamInput()` is closed.
Relates to elastic/x-pack-elasticsearch#28504
Original commit: elastic/x-pack-elasticsearch@7546e3b4d4
* Pass InputStream when creating XContent parser
Rather than passing the raw `BytesReference` in when creating the xcontent
parser, this passes the StreamInput (which is an InputStream), this allows us to
decouple XContent from BytesReference.
This is the x-pack side of https://github.com/elastic/elasticsearch/pull/28754
* Use the streamInput variant, not sourceAsString
Original commit: elastic/x-pack-elasticsearch@dd5d8b1654
This adds a new Rollup module to XPack, which allows users to configure periodic "rollup jobs" to pre-aggregate data. That data is then available later for search through a special RollupSearch API, which mimics the DSL and functionality of regular search.
Rollups are used to drastically reduce the on-disk footprint of metric-based data (e.g. timestamped document with numeric and keyword fields). It can also be used to speed up aggregations over large datasets, since the rolled data will be considerably smaller and fewer documents to search.
The PR adds seven new endpoints to interact with Rollups; create/get/delete job, start/stop job, a capabilities API similar to field-caps, and a Rollup-enabled search.
Original commit: elastic/x-pack-elasticsearch@dcde91aacf
Although not frequently used in production, we make heavy use of the
FileRolesStore within integration tests. This change adds a little bit
more logging at INFO level when the roles.yml file is (re)loaded.
Original commit: elastic/x-pack-elasticsearch@bbacd46e28
This change makes the IndexAuditTrail update mappings for the current
index even when indexing locally. This is needed in a couple of
scenarios; the first being upgrades from 5x to 6x where the document
type was changed and the second is when upgrading minor versions and
a new field has been introduced. A new field was introduced in 6.2.
Additionally, writing tests for the rolling upgrade revealed a bug
where the actual node name was never being set as it was using the name
key instead of the node.name key.
relates elastic/x-pack-elasticsearch#3901
relates elastic/x-pack-elasticsearch#3718
Original commit: elastic/x-pack-elasticsearch@7374a62ff1
This commit removes some outdated logic in the SecurityTestUtils class
for writing files and creating directories. A long, long time ago there
was a global cluster for tests that was used across suites and because
of this there were calls to delete files if they already existed. The
global cluster has been removed, so we no longer need the code that
deletes the content of a directory if it already exists.
Additionally, the file writing used in SecurityTestUtils did not use
atomic moves when possible and this commit changes the code such that a
temp file is written and we try to atomically move it to the correct
path; if atomic moves are not supported a regular move is performed.
relates elastic/x-pack-elasticsearch#3912
Original commit: elastic/x-pack-elasticsearch@973fcfe2e1
This test recently failed and it appears that the cause was that
roles.yml was deleted while the cluster was still running.
This turns on logging for the FileRolesStore so that changes to file
based roles can traced more easily.
Original commit: elastic/x-pack-elasticsearch@fbce5a8027
* Additional settings for SAML NameID policy
We should not be populating SPNameQualifier by default as it is
intended to be used to specify an alternate SP EntityID rather than
our own. Some IdPs (ADFS) fail when presented with this value.
This commit
- makes the SPNameQualifier a setting that defaults to blank
- adds a setting for "AllowCreate"
- documents the above
Original commit: elastic/x-pack-elasticsearch@093557e88f
* X-Pack-Security: Improve error messaging during setup password
Changes are done to improve logging and exception messages when
setup passwords. Now it checks whether
- X-Pack plugin is available
- X-Pack Security is available
- X-Pack Security is enabled
by invoking /_xpack query and throws a specific error message.
1. Modifications to simple http client to return http response.
2. Changes in SetupPasswordTool for throwing appropriate exceptions.
2. Unit tests
relates elastic/x-pack-elasticsearch#3476
Original commit: elastic/x-pack-elasticsearch@eb0c5a19c5
java.time features it's own halted clock, called a fixed clock, we can
use that one.
On top of that the watcher xcontent parser does not need a clock at all,
just a timestamp when parsing happened.
Original commit: elastic/x-pack-elasticsearch@2061aeffe1
There were a number of leftover unnecessary elements in the module
build.gradle files that were holdovers from copying the original plugin
build.gradle. This commit removes these elements.
Original commit: elastic/x-pack-elasticsearch@08babbd520
Calling start() when already in the STARTING state doesn't do anything, so the component
gets stuck in STARTING state forever.
Also: wait on the required index name not just the cluster.
Also: added more logging to help diagnose such issues (either in RemoteIndexAuditTrailStartingTests or production)
Original commit: elastic/x-pack-elasticsearch@fb81214fe7
This commit adds special handling for null values when building a DistinguishedNamePredicate for
role mapping. Previously this would have resulted in an exception from the unboundid ldapsdk as
the DN is invalid.
relates elastic/x-pack-elasticsearch#3787
Original commit: elastic/x-pack-elasticsearch@9386dae03e
If any of the follow take place on security index, then any cached role mappings
are potentially invalid and the associated realms need to clear any cached users.
- Index recovers from red
- Index is deleted
- Index becomes out-of-date / not-out-of-date
Original commit: elastic/x-pack-elasticsearch@1bcd86fcd4
Lee Hinman
Jay Modi
Yogesh Gaikwad
Tim Vernum
Albert Zaharovits
Albert Zaharovits
Jason Tedor
Yannick Welsch
Ioannis Kakavas
Zachary Tong
Alexander Reelsen
Tim Brooks
Tim Vernum
Ryan Ernst
Nik Everett
Costin Leau
Lee Hinman
Simon Willnauer