This change updates the WatcherLifecycleService to have its own single
thread executor that is used for lifecycle changes in order to have a
guarantee for the order that the changes are executed in.
Previously, a runnable would be submitted to the generic threadpool for
each lifecycle change that is needed. There was no guarantee of
ordering for these changes and no checks to see if a state change was
already in flight.
Relates elastic/x-pack-elasticsearch#4429
Original commit: elastic/x-pack-elasticsearch@14b73381db
The PKI realm has never been a caching realm as the need had not
presented itself until now. The PKI realm relies on role mappings to
map the DN from a certificate to roles so that the users have the
appropriate access permissions. Without caching, this role mapping will
happen on every request. For file based role mappings, this is not an
issue as the mappings are based on equality checks for the DN.
However, the design of the API based role mappings allows for more
complex matches. These matches are implemented using automata, which
are built on every request that needs role mappings. Building automata
is an expensive operation and in combination with the PKI realm's lack
of caching leads to a significant performance impact.
The change in this commit makes the PkiRealm a caching realm using the
same pattern as other caching realms. The cache provided by
elasticsearch core is used to map the fingerprint of a certificate to
the user that was resolved from this certificate. The semantics of
modifications to this cache during iteration requires that we use a
read-write lock to protect access. There can be multiple concurrent
modifications and retrievals but iteration must be protected from any
attempts to modify the cache.
Additionally, some PKI tests were converted to single node tests as
part of this change. One test only used a single node and the other did
not require multiple nodes.
relates elastic/x-pack-elasticsearch#4406
Original commit: elastic/x-pack-elasticsearch@214772e1c1
Adds a check in BlobstoreRepository.snapshot(...) that prevents duplicate snapshot names and fails
the snapshot before writing out the new index file. This ensures that you cannot end up in this
situation where the index file has duplicate names and cannot be read anymore .
Relates to #28906
The suggest stats were folded into the search stats as part of the
indices stats API in 5.0.0. However, the suggest metric remained as a
synonym for the search metric for BWC reasons. This commit deprecates
usage of the suggest metric on the indices stats API.
Similarly, due to the changes to fold the suggest stats into the search
stats, requesting the suggest index metric on the indices metric on the
nodes stats API has produced an empty object as the response since
5.0.0. This commit deprecates this index metric on the indices metric on
the nodes stats API.
This commit implements the ability to remove values from a Cache using
the values iterator. This brings the values iterator in line with the
keys iterator and adds support for removing items in the cache that are
not easily found by the key used for the cache.
Remove indentation so that the two lines are not interpreted as
code block and the link is rendered correctly
Original commit: elastic/x-pack-elasticsearch@a73d52dfba
Add missing Atan2 & Power(and introduce BinaryMath operations), similar
to MathOperation.
Also align arithmetic package with binary math for code reuse.
Original commit: elastic/x-pack-elasticsearch@311961815e
When deployed, JDBC version needs to look at jars inside the classpath
not on the file system. Failing to do that, causes the version
information to be missing
Original commit: elastic/x-pack-elasticsearch@d35e8abb29
* es/master: (32 commits)
TEST: Unmute testPrimaryRelocationWhileIndexing
Remove remaining tribe node references (#29574)
Never leave stale delete tombstones in version map (#29619)
Do not serialize common stats flags using ordinal (#29600)
Remove stale comment from JVM stats (#29625)
TEST: Mute testPrimaryRelocationWhileIndexing
Remove bulk fallback for write thread pool (#29609)
Fix an incorrect reference to 'zero_terms_docs' in match_phrase queries.
Update the version compatibility for zero_terms_query in match_phrase.
Account translog location to ram usage in version map
Remove extra spaces from changelog
Add support to match_phrase query for zero_terms_query. (#29598)
Fix incorrect references to 'zero_terms_docs' in query parsing error messages. (#29599)
Build: Move java home checks to pre-execution phase (#29548)
Avoid side-effect in VersionMap when assertion enabled (#29585)
[Tests] Remove accidental logger usage
Add tests for ranking evaluation with aliases (#29452)
Deprecate use of `htmlStrip` as name for HtmlStripCharFilter (#27429)
Update plan for the removal of mapping types. (#29586)
[Docs] Add rankEval method for Jva HL client
...
Previously we did not put an indexing to a version map if that map does
not require safe access but removed the existing delete tombstone only
if assertion enabled. In #29585, we removed the side-effect caused by
assertion then this test started failing. This failure can be explained
as follows:
- Step 1: Index a doc then delete that doc
- Step 2: The version map can switch to unsafe mode because of
concurrent refreshes (implicitly called by flushes)
- Step 3: Index a document - the version map won't add this version
value and won't prune the tombstone (previously it did)
- Step 4: Delete a document - this will return NOT_FOUND instead of
DELETED because of the stale delete tombstone
This failure is actually fixed by #29619 in which we never leave stale
delete tombstones
Closes#29626
Today the VersionMap does not clean up a stale delete tombstone if it
does not require safe access. However, in a very rare situation due to
concurrent refreshes, the safe-access flag may be flipped over then an
engine accidentally consult that stale delete tombstone.
This commit ensures to never leave stale delete tombstones in a version
map by always pruning delete tombstones when putting a new index entry
regardless of the value of the safe-access flag.
Today we can use the soft-deletes feature from Lucene to maintain a
history of a document. This change simply replaces hard-deletes by
soft-deletes in Engine.
Besides marking a document as deleted, we also index a tombstone
associated with that delete operation. Storing delete tombstones allows
us to have a history of sequence-based operations which can serve in
recovery or rollback.
Relates #29530
This commit remove serializing of common stats flags via its enum
ordinal and uses an explicit index defined on the enum. This is to
enable us to remove an unused flag (Suggest) without ruining the
ordering and thus breaking serialization.
We removed catched throwable from the code base and left behind was a
comment about catching InternalError in MemoryManagementMXBean. We are
not going to catch InternalError here as we expect that to be
fatal. This commit removes that stale comment.
The name of the bulk thread pool was renamed to "write" with "bulk" as a
fallback name. This change was made in 6.x for BWC reasons yet in 7.0.0
we are removing this fallback. This commit removes this fallback for the
write thread pool.
We grab all the NumberFieldMappers to determine the whitelist of allowed
number types, but `scaled_float` is in a module so it isn't picked up
automatically. This commit adds `scaled_float` to the whitelist
manually.
Original commit: elastic/x-pack-elasticsearch@fb35440315
This commit allows license installation without TLS being enabled when
the cluster is in dev mode. The main difference this change enables is
the ability to install a production license on a single node cluster
that is bound to localhost and does not have the single-node discovery
enabled.
relates elastic/x-pack-elasticsearch#4123
Original commit: elastic/x-pack-elasticsearch@04ebcc0fab
The SQL action supports several text outputs and used to default to an
output that looked like the SQL CLI. It is a lovely output format but
this changes output selection behavior to mimick Elasticsearch's
standard behavior: it'll now default to the same format as the request.
That means that if you want the pretty text format then you need to ask
for it. The way to do that is:
```
POST /_xpack/sql?format=text/plain
{
"query": "SELECT * FROM library ORDER BY page_count DESC LIMIT 5"
}
```
Original commit: elastic/x-pack-elasticsearch@4a15a23b18
* Move sql-proto and jdbc to java.util.TimeZone
This moves sql-proto and jdbc from Joda's `DateTimeZone` to
`java.util.TimeZone`, this will allow us in the future to be able to decouple
JDBC from Joda.
This does not decouple all of SQL from joda, it focuses on as small a piece as I
could for sql-proto and jdbc.
Requires https://github.com/elastic/elasticsearch/pull/29487 to be merged first.
Original commit: elastic/x-pack-elasticsearch@7c9d52e675
This commit moves the checks on JAVAX_HOME (where X is the java version
number) existing to the end of gradle's configuration phase, and based
on whether the tasks needing the java home are configured to execute.
relates #29519
Today when a version map does not require safe access, we will skip that
document. However, if the assertion is enabled, we remove the delete
tombstone of that document if existed. This side-effect may accidentally
hide bugs in which stale delete tombstone can be accessed.
This change ensures putAssertionMap not modify the tombstone maps.
This commit adds waits for two Netty threads that run in the background
and require explicit calls to await inactivity in them. This shows up
as an issue in security tests since we always use netty and do not have
these waits so these threads can trip the thread leak detector.
relates elastic/x-pack-elasticsearch#4367
Original commit: elastic/x-pack-elasticsearch@76d84553ba
The ranking evaluation requests so far were not tested against aliases
but they should run regardless of the targeted index is a real index or
an alias. This change adds cases for this to the integration and rest
tests.
The `create` privilege can be misleading based on a user's
interpretation of the meaning of the privilege. The create privilege
grants the user access to the index api and the put mapping api. The
index api allows for a document to be created but it also allows for
an existing document to be overwritten (assuming the CREATE operation
type is not used). However, the name `create` implies the ability to
only create a document and not be able to modify an existing document,
which may not be clear to users unfamiliar with the index API.
This commit adds a small note under the privilege in the documentation
that should add some clarity to the operations allowed by this
privilege.
Original commit: elastic/x-pack-elasticsearch@24596c41b0
The camel case name `htmlStip` should be removed in favour of `html_strip`, but
we need to deprecate it first. This change adds deprecation warnings for indices
with version starting with 6.3.0 and logs deprecation warnings in this cases.
* Remove copy-pasted code
An examination of the x-pack source code revealed copy-pasted code
in ActiveDirectoryGroupsResolver in the binarySidToStringSid
method. I have replaced this with an apache2 implementation from
the apache directory studio project. Furthermore, I have added
a test that leverages a real binary/string SID pair retrieved
from an active directory domain controller. The apache2-based
implementation is exempt for the license checker.
Original commit: elastic/x-pack-elasticsearch@81a7471261
Ryan Ernst
Jay Modi
Yannick Welsch
Jason Tedor
Costin Leau
David Kyle
Ioannis Kakavas
Martijn van Groningen
Nhat Nguyen
Zachary Tong
Julie Tibshirani
Nik Everett
Lisa Cawley
Lee Hinman
Christoph Büscher
Russ Cam
tomcallahan
Adrien Grand