Many files supplied to the upcoming ML data preparation
functionality will not be "log" files. For example,
CSV files are generally not "log" files. Therefore it
makes sense to rename library that determines the
structure of these files.
Although "file structure" could be considered too broad,
as the library currently only works with a few text
formats, in the future it may be extended to work with
more formats.
When index sorting is enabled, toXContent tried to serialize an
SortField object, resulting in an exception, when using the _segments
endpoint.
Relates #29120
With this commit we use the classic parent circuit breaker which does
not account for real memory usage. In those tests we want to have
reproducible results and hence it makes sense to disable the real memory
circuit breaker there.
Auto Following Patterns is a cross cluster replication feature that
keeps track whether in the leader cluster indices are being created with
names that match with a specific pattern and if so automatically let
the follower cluster follow these newly created indices.
This change adds an `AutoFollowCoordinator` component that is only active
on the elected master node. Periodically this component checks the
the cluster state of remote clusters if there new leader indices that
match with configured auto follow patterns that have been defined in
`AutoFollowMetadata` custom metadata.
This change also adds two new APIs to manage auto follow patterns. A put
auto follow pattern api:
```
PUT /_ccr/_autofollow/{{remote_cluster}}
{
"leader_index_pattern": ["logs-*", ...],
"follow_index_pattern": "{{leader_index}}-copy",
"max_concurrent_read_batches": 2
... // other optional parameters
}
```
and delete auto follow pattern api:
```
DELETE /_ccr/_autofollow/{{remote_cluster_alias}}
```
The auto follow patterns are directly tied to the remote cluster aliases
configured in the follow cluster.
Relates to #33007
Co-authored-by: Jason Tedor jason@tedor.me
With features like CCR building on the CCS infrastructure, the settings
prefix search.remote makes less sense as the namespace for these remote
cluster settings than does a more general namespace like
cluster.remote. This commit replaces these settings with cluster.remote
with a fallback to the deprecated settings search.remote.
There are two races in the testUpdateAndReadChangesConcurrently if the
following engines are created in the worker threads. We fixed the
translog issue in #33352, but there is still another race with
createStore.
This commit ensures that we create all engines in the main thread.
Relates #33352Closes#33344
Historically we have had a ESLoggingHandler in the netty module that
logs low-level connection operations. This class just extends the netty
logging handler with some (broken) message deserialization. This commit
fixes this message serialization and moves the class to server.
This new logger logs inbound and outbound messages. Eventually, we
should move other event logging to this class (connect, close, flush).
That way we will have consistent logging regards of which transport is
loaded.
Resolves#27306 on master. Older branches will need a different fix.
This commit is related to #32517. It allows an "server_name"
attribute on a DiscoveryNode to be propagated to the server using
the TLS SNI extentsion. This functionality is only implemented for
the netty security transport.
Re-implement the cache to avoid jackson JSON de-serialization for
every IP lookup. The built in maxmind cache caches JsonNode objects.
This requires de-serialization for every lookup, even if the object
is found in cache. Profiling shows that is very expensive (CPU).
The cache will now consist of the fully de-serialized objects.
Profiling shows that the new footprint for the CityDB is ~6KB per cache
entry. This may result in ~6MB increase with the 1000 entry default.
The performance has been measured up to 40% faster on a modern 4 core/8 thread
CPU for an ingest (minimal indexing) workflow.
Further, the since prior implementation cached the JsonNode objects,
and there is not a 1:1 relationship between an IP lookup / JsonNode
object, the default cache size was most likely too small to be very
effective. While this change does not change the 1000 default cache
size, it will now cache more since there is now a 1:1 relationship between
an IP lookup and value in the cache.
This change adds an expert index setting called `index.merge.policy.deletes_pct_allowed`.
It controls the maximum percentage of deleted documents that is tolerated in the index.
Lower values make the index more space efficient at the expense of increased CPU and I/O activity.
Values must be between `20` and `50`. Default value is `33`.
Drops and unused logging constructor, simplifies a rarely used one, and
removes `Settings` from a third. There is now only a single logging ctor
that takes `Settings` and we'll remove that one in a follow up change.
This commit adds a security client to the high level rest client, which
includes an implementation for the put user api. As part of these
changes, a new request and response class have been added that are
specific to the high level rest client. One change here is that the response
was previously wrapped inside a user object. The plan is to remove this
wrapping and this PR adds an unwrapped response outside of the user
object so we can remove the user object later on.
See #29827
The maximum number of fields per index is limited to 1000 by default by the
`index.mapping.total_fields.limit` setting to prevent accidental mapping
explosions due to too many fields. Currently all metadata fields also count
towards this limit, which can lead to some confusion when using lower limits.
It is not obvious for users that they cannot actually add as many fields as
are specified by the limit in this case.
This change takes the number of metadata fields out of the field count that we
check against the field limit. It also adds tests that check that we can add
fields up to the specified limit, but throw an exception for any additional field added.
Closes#24096
Deprecating a some setting specializations (e.g., list settings) does
not cause deprecation warning headers and deprecation log messages to
appear. This is due to a missed check for deprecation. This commit fixes
this for all setting specializations, and ensures that this can not be
missed again.
This allows tokenfilters to be applied selectively, depending on the status of the current token in the tokenstream. The filter takes a scripted predicate, and only applies its subfilter when the predicate returns true.
Solves all of the xpack line length suppressions and then merges the
remainder of the xpack checkstyle_suppressions.xml file into the core
checkstyle_suppressions.xml file. At this point that just means the
antlr generated files for sql.
It also adds an exclusion to the line length tests for javadocs that
are just a URL. We have one such javadoc and breaking up the line would
make the link difficult to use.
The log structure endpoint will return these in addition to
pure structure information so that it can be used to drive
pre-import data visualizer functionality.
The statistics for every field are count, cardinality
(distinct count) and top hits (most common values). Extra
statistics are calculated if the field is numeric: min, max,
mean and median.
Gradle triggers the build of artifacts even if assemble is disabled.
Most users will not need bwc distributions after running `./gradlew
assemble` so instead of forcing them to add `-x buildBwcVersion`, we
detect this and skip the configuration of the artifacts.
Adds a place for users to store cluster-wide data they wish to associate
with the cluster via the Cluster Settings API. This is strictly for
user-defined data, Elasticsearch makes no other other use of these
settings.
With the introduction of the default distribution, it means that by
default the query cache is wrapped in the security implementation of the
query cache. This cache does not allow caching if the request does not
carry indices permissions. Yet, this will not happen if authorization is
not allowed, which it is not by default. This means that with the
introduction of the default distribution, query caching was disabled by
default! This commit addresses this by checking if authorization is
allowed and if not, delegating to the default indices query
cache. Otherwise, we proceed as before with security. Additionally, we
clear the cache on license state changes.
Now that types are unique per mapping we can retrieve the document mapper
without referencing the type. This fixes an NPE when stored fields are disabled.
For 6x we'll need a different fix since mappings can still have multiple types.
Relates #32941
Extend SHOW TABLES, DESCRIBE and SHOW COLUMNS to support table
identifiers not just SQL LIKE pattern.
This allows both Elasticsearch-style multi-index patterns and SQL LIKE.
To disambiguate between the two (as the " vs ' can be easy to miss),
the grammar now requires LIKE keyword as a prefix for all LIKE-like
patterns.
Also added some docs comparing the two types of patterns.
Fix#33294
This is not changing the behaviour as when the sort field was set
to `influencer_score` the secondary sort would be used and that
was using the `record_score` at the highest priority.
1. The TOMCAT_DATESTAMP format needs to be checked before
TIMESTAMP_ISO8601, otherwise TIMESTAMP_ISO8601 will
match the start of the Tomcat datestamp.
2. Exclude more characters before and after numbers. For
example, in 1.2.3 we don't want to match 1.2 as a float.
Benjamin Trent
Martijn van Groningen
David Roberts
Alan Woodward
Alexander Reelsen
Daniel Mitterdorfer
Jason Tedor
Nhat Nguyen
Tim Brooks
Jake Landis
Armin Braun
Jim Ferenczi
Nik Everett
Adrien Grand
Jay Modi
Christoph Büscher
Michael Basnight
Dimitris Athanasiou
Paul Sanwald
Costin Leau
Alpar Torok
Gordon Brown
Lisa Cawley
Costin Leau
Sohaib Iftikhar
Nikolay Vasiliev
Julie Tibshirani