When events are searched to be passed to the autodetect process, they
are currently calculated based on the latest record timestamp, when
a job opens, and `now` when the process is updated.
This commit changes both to be consistent and based on the earliest
valid timestamp for the job. The earliest valid timestamp is the
latest record timestamp minus the job latency.
Relates elastic/x-pack-elasticsearch#3016
Original commit: elastic/x-pack-elasticsearch@7f882ea053
* [Monitoring] Add uptime.ms field to Beats template
* Add cpu mapping [WIP]
* sort fields
* hierarchical - no dots in fields
* add rss, more event metrics, system cpu and os load
* fix some mapping errors
Original commit: elastic/x-pack-elasticsearch@59cbcb4ee5
This change makes the IndexLifecycleManager check for both index existence and up to date mappings
on the index prior to executing the provided runnable. Doing this provides a mechanism to make
non-breaking mapping updates to the security index in minor versions.
relates elastic/x-pack-elasticsearch#3462
Original commit: elastic/x-pack-elasticsearch@80f05d83b4
This commit adds both the security index template and the audit index template to the excluded
templates for the RemoteIndexAuditTrailStartingTests and IndexAuditTrailTests. By allowing the
templates to remain in place, we make it easier for subsequent tests in the suite to have their
audit trails started.
Relates elastic/x-pack-elasticsearch#435
Original commit: elastic/x-pack-elasticsearch@6e54de19f9
By moving tokenization for categorization to Java we give users access to considerably more options for tokenizing their log messages prior to using ML to categorize them. Now all Elasticsearch analyzer functionality is available, which opens up the possibility to sensibly categorize non-English log messages.
Relates elastic/machine-learning-cpp#491
Original commit: elastic/x-pack-elasticsearch@5d61b67614
In order to support buttons that can be clicked on within a slack
message, this commits adds support for so called actions within
attachments. This allows to create buttons, that are clicked and execute
a GET request, so actions must be idempotent according to the official
slack documentation.
Official slack documentation is available at
https://api.slack.com/docs/message-attachments#action_fields
Original commit: elastic/x-pack-elasticsearch@29ddc90b01
This commit introduces audit event filtering policies for the logging audit.
Until now the only way to temper the gush of audit events was to specifically
pick some event types that were reported.
This superposes a way (named policies) to filter events using lucene regexp filters on
the following event fields: users, realms, roles and indices. The policies are ignore
policies, ie when an event matches it is ignored (not reported).
Original commit: elastic/x-pack-elasticsearch@233f685121
This creates a cluster alert that triggers whenever a node is:
- Added
- Removed
- Restarted (aka Removed and Added between collections)
Unlike all previous cluster alerts, this cluster is always instantly resolved because there is no way to otherwise resolve it later (especially if nodes are replaced).
This will require a small change to the UI so that it does not simply ignore resolved alerts.
Original commit: elastic/x-pack-elasticsearch@6340bf7960
We need to push this flag down to the command line for BWC builds or the
artifacts in release tests will have the wrong version (being considered
snapshots instead of non-snapshots).
Original commit: elastic/x-pack-elasticsearch@279fd02aa7
This commit modifies the BWC build to invoke the Gradle wrapper. The
motivation for this is two-fold:
- BWC versions might be dependent on a different version of Gradle than
the current version of Gradle
- in a follow-up we are going to need to be able to set JAVA_HOME to a
different value than the current value of JAVA_HOME
Relates elastic/x-pack-elasticsearch#3502
Original commit: elastic/x-pack-elasticsearch@22062f635b
The HttpClient uses an Apache HTTP client class named StringEntity to
encode a HTTP request body. This one however assumes ISO-8859-1 as its
charset when encoding the string based body to bytes.
This commit switches to a byte array based body, then sets the content
type header and falls back to the old text/plain content type if no
content type header is specified.
relates elastic/x-pack-elasticsearch#3397
Original commit: elastic/x-pack-elasticsearch@d5a6e7f0c7
The categorization integration tests were picking up a bug in the C++
categorization code, but the expected results were set incorrectly to
match the incorrect C++ output. This change adjusts the expected
results to reflect what is really expected. It also mutes the tests
to avoid failures while the C++ fix propagates through the build
process. The tests will be unmuted tomorrow.
Relates elastic/machine-learning-cpp#512
Original commit: elastic/x-pack-elasticsearch@d2b57b6216
When creating a transport client for a remote index audit trail, we are
implicitly allowing the construction of this transport client to
initialize the number of processors that Netty thinks are on the
system. Since we never pushed down the number of processors, this will
always default to the number of cores on the machine. If the user has
also set the processors setting, when the server bootstraps it will try
to push the number of processors down to Netty too. If this value does
not match the number of cores, we will fail in bootstrap because we
guard against initializing the number of processors that Netty sees to
different values. Instead, the transport client should inherit the
number of processors too and push this down when it pushes the number of
processors down to Netty. We have to worry about another possibility: an
explicit setting for the number of processors for the transport client
so we require this matches the inherited value.
Relates elastic/x-pack-elasticsearch#3469
Original commit: elastic/x-pack-elasticsearch@032810bb0b
Minor refactoring on the reserved realm:
- Removed some duplicated code
- Added in some additional assertions
- Extended some testing
- Removed use of the obsolete "allow_default_passwords" from the test.
Original commit: elastic/x-pack-elasticsearch@584171d2bd
It is unsafe to clear passwords of bind requests if the connection is live
and might be used latter (for eg for group searches). This is a temporary
fix that exposes passwords in memory.
Original commit: elastic/x-pack-elasticsearch@e2e1f1a358
This commit hacks up the xpack repo in such a way that it moves all
actions that the transport client uses to plugin core. It also moves
any classes that those actions use to plugin core, with a few
exceptions. I tried to split up any classes that pulled in server side
logic into the client, but that was not always 100% possible. Consider
this commit a guide, and since I do not know the codebase for each
plugin, consider it a best guess for what should be moved. A few other
things were merged in below.
This commit extracts the parts of Condition and AlwaysCondition that are
needed by classes that will be moved into the client package. The only
odd thing here is that since there are two parent classes (Condition and
AlwaysCondition) it was not possible to make AlwaysConditionInteral
extend from AlwaysCondition and still parse a ConditionInternal object,
which is what the ConditionFactory expects.
This commit removes the use of internal users in the User class, and
instead moves them to Authentication, where they are used.
[insert obligatory chuck norris karate image here]
ref elastic/x-pack-elasticsearch#2925
Original commit: elastic/x-pack-elasticsearch@42d0b72209
This adds a traditional blacklist that can be set [dynamically] at the exporter level to block specific cluster alerts (watches) from being created. It goes further and actually removes any watch that has been created already for the current cluster.
Original commit: elastic/x-pack-elasticsearch@1f67bb9501
This modifies the default template to use auto_expand_replicas so that a one node cluster can stay green, even while using the Logstash management feature.
Original commit: elastic/x-pack-elasticsearch@b1ddf3af6f
In the previous pass, only the ".watch*" templates were impacted. This also should use `auto_expand_replicas`.
Original commit: elastic/x-pack-elasticsearch@5d3a4f1f7b
This creates all ML templates using the `index.auto_expand_replicas` set as `"0-1"` so that a single node cluster (e.g., dev) can startup as green.
Original commit: elastic/x-pack-elasticsearch@7a243c5b06
* Calendar event actions
* Add page params and date range tests
* Address review comments
* Support POSTing params in the body of a request
Original commit: elastic/x-pack-elasticsearch@22a7e17a8f
The test uses the bound address to determine how to speak to the node via http.
It currently takes the port but hard codes the host to `localhost`. This can lead
to mismatches where a port for ipv6 is used but localhost resolves to ipv4
relates elastic/x-pack-elasticsearch#3382
Original commit: elastic/x-pack-elasticsearch@e97363a521
This test is failing for days, possibly due to a change in core
Elasticsearch. This commit marks this test as awaits fix.
Original commit: elastic/x-pack-elasticsearch@8ed3965795
This test is failing for days, possibly due to a change in core
Elasticsearch. This commit marks this test as awaits fix.
Original commit: elastic/x-pack-elasticsearch@8d7db1c423
This test is failing for days, possibly due to a change in core
Elasticsearch. This commit marks this test as awaits fix.
Original commit: elastic/x-pack-elasticsearch@f519c2a7ee
Tests that rely on the security index and security index template being present should not remove
the template between tests as this can cause test failures. The template upgrade service relies
on cluster state updates to trigger the template being added after a delete, but there is a
scenario where the test will just wait for template that never shows up as there is no cluster
state update in that time. Instead of fighting ourselves, we should just leave the template in
place.
Relates elastic/x-pack-elasticsearch#2915
Relates elastic/x-pack-elasticsearch#2911
Original commit: elastic/x-pack-elasticsearch@3ca4aef0be
* Calendar jobs endpoints
* Refactor put and delete calendar job to use the same action
* Check jobs exist when creating the calendar
* Address review comments
* Add isGroupOrJobMethod
* Increase default page size for calendar query
Original commit: elastic/x-pack-elasticsearch@7484799fe9
# 2650 added the mapping for the interval_ms field in the Elasticsearch
template but not for Kibana,Logstash and Beats templates.
Original commit: elastic/x-pack-elasticsearch@44fb501bb3
The getShortPathName method can only be used on a directory that actually exists, otherwise it will fail with a cryptic message.
Original commit: elastic/x-pack-elasticsearch@44552dcfc8
The API JAR POM picks up the wrong artifact name for the :core:cli
dependency, using the project name instead of the archive base
name. This commit fixes this issue by explicitly referring to the
artifact as a runtime dependency. With this change, the correct artifact
name is used in the API JAR POM.
Relates elastic/x-pack-elasticsearch#3336
Original commit: elastic/x-pack-elasticsearch@68026168da
This changes the default behavior of .watch* indices to be green on one-node clusters, instead of constantly yellow.
Original commit: elastic/x-pack-elasticsearch@cdaee7cd72
The index action allowed to set the id of a document dynamically,
however this was not allowed for the index or the type.
If a user wants to execute a search, modify the found documents and
index them back, then this would only work across a single index and a
single type. This change allows the watch writer to just take a search
result, read index and type out of that and configure this as part of
the index action.
On top of that the integration tests have been changed to become fast
running unit tests.
Original commit: elastic/x-pack-elasticsearch@640b085dd4
Only the Logstash and Kibana version mismatch watches contain a time
filter, the others are only sorting by timestamp. In combination with
searching in all `.monitoring-es-*` indices, this is IMO pretty resource
intensive, as we cannot exit early on any search request.
This commit adds time based filters to remaining three watches, using
the same range than the other two.
Original commit: elastic/x-pack-elasticsearch@3eb6bf0de2
* Fix license messaging for Logstash functionality
With a Basic license, users are still able to perform CRUD operations on the `.logstash` index, therefore manage their Logstash pipelines. However, Logstash itself will not pick up any changes from this index and act on them. With an expired license Logstash functionality continues to operate as normal.
* Fixing messages after feedback
* Removing extraneous tabs at end of line
* Fixing typo
Original commit: elastic/x-pack-elasticsearch@bc069cf00f
Verify that the configuration directory `$ES_PATH_CONF/x-pack`
exists before attempting to run any of the `users` commands, and
return a helpful error message to the user if it doesn't.
Original commit: elastic/x-pack-elasticsearch@6d663b6654
This changes the default behavior of .monitoring indices to be green on one-node clusters, instead of constantly yellow.
Note: This only impacts .monitoring* indices. Watcher indices currently still require a replica.
Original commit: elastic/x-pack-elasticsearch@6eb8a48a9f
* Rename folder x-pack-core -> core
The jar remains 'x-pack-core-*.jar'
* Put group in top-level build.gradle instead of plugin/core/build.gradle
Original commit: elastic/x-pack-elasticsearch@b23452fa55
This commit adds additional checks around resize operations and alias creation operations to
add an extra layer of security around these APIs.
Original commit: elastic/x-pack-elasticsearch@b79f16673c
Upon selecting a node to run a datafeed we normally check that the
data indices exists and their primaries are active. However, these
checks cannot be applied for CCS to a remote cluster. This commit
skips these checks for remote indices.
This removes the last obstacle for running CCS datafeeds.
Relates elastic/x-pack-elasticsearch#1071
Original commit: elastic/x-pack-elasticsearch@092f44feee
SQL used to have some changes to security. We've since reverted them but
we have some leftover stuff like import reordering and spacing changes.
We may as well remove them so merging SQL to master is smaller.
Original commit: elastic/x-pack-elasticsearch@c632256ddd
This commits adds a new end point for closing in-flight cursors, it also ensures that all cursors are properly closed by adding after test checks that ensures that we don't leave any search context open.
relates elastic/x-pack-elasticsearch#2878
Original commit: elastic/x-pack-elasticsearch@1052ea28dc
This commit updates x-pack to be compatible with
elastic/elasticsearch#27711. That commit removed the need for channels
to be internally tracked inside transport implementations. This commit
removes a test mocking class that is not necessary after that change.
Original commit: elastic/x-pack-elasticsearch@75d99ba1d1
This creates a basic skeleton for the plugin split by adding folders and example
`build.gradle` files. It also includes a non-implemented `migrate-plugins.sh`
script that we can fill in at a later time.
Relates to elastic/x-pack-elasticsearch#2925
Original commit: elastic/x-pack-elasticsearch@2ab035d6b6
Dimitris Athanasiou
Tim Brooks
Albert Zaharovits
Alexander Reelsen
Tim Sullivan
Igor Motov
Jay Modi
David Kyle
jaymode
David Roberts
Albert Zaharovits
Chris Earle
Jason Tedor
Yannick Welsch
Ryan Ernst
Tim Vernum
Michael Basnight
Hendrik Muhs
Boaz Leskes
Simon Willnauer
Tanguy Leroux
Shaunak Kashyap
Colin Goodheart-Smithe
Ioannis Kakavas
Costin Leau
Lee Hinman
Nik Everett