This reverts commit c50e8c83a2476ca4ca2f1dd05fa5a608bc0e9ef6
which went should have merged to the rename branch instead of
the main branch.
Signed-off-by: Peter Nied <petern@amazon.com>
This commit refactors all instances of elasticsearch in
server/src/main/java/org/apache to opensearch.
Signed-off-by: Nicholas Knize <nknize@amazon.com>
Signed-off-by: Peter Nied <petern@amazon.com>
This commit changes the building, packaging, and testing framework to only support OSS on different distributions.
Next steps:
completely remove -oss flag dependencies in package and build tests
move 6.x bwc testing to be an explicit option
remove any references to elastic.co download site (or replace with downloads from the OSS website)
Co-authored-by: Himanshu Setia <setiah@amazon.com>
Co-authored-by: Rabi Panda <pandarab@amazon.com>
Co-authored-by: Himanshu Setia <58999915+setiah@users.noreply.github.com>
Co-authored-by: Sarat Vemulapalli <vemsarat@amazon.com>
Signed-off-by: Peter Nied <petern@amazon.com>
ReplicationOperation can notify the listener twice if the primary shard
is demoted after it has completed the primary operation.
Closes#68049
Signed-off-by: Peter Nied <petern@amazon.com>
This commit fixes DeleteDataStreamRequestTests.testDeleteSnapshottingDataStream unit test failure by passing SnapshotsInProgress.Entry.SUCCESS in the createEntrymethod.
Signed-off-by: Peter Nied <petern@amazon.com>
This commit cleans up the following:
* Remove unused imports
* Remove ILM settings in hlrc testCluster formation
* Comment out security users settings in ElasticsearchNode creation for build-tools tests
Signed-off-by: Peter Nied <petern@amazon.com>
This fixes the constructor for IndexNameExpressionResolver to pass in
Settings.EMPTY to a ThreadContext used by the resolver.
Signed-off-by: Peter Nied <petern@amazon.com>
Drops an assertion that in 1911 Paris rolled its clocks back past
midnight. This assertion is so far in the past that it isn't
consistently included in the JDK's tzdb. When we upgraded to 15.0.1 it
fell out of my tzdb locally. It doesn't seem to have happened to CI,
oddly, but it just doesn't seem worth keeping.
Closes#67930
We have seen a case where the memory of IndexingPressure was
re-adjusted twice. With this commit, we will log that error with a
stack trace so that we can figure out the source of the issue.
Our test for early termination would break if we made many small
segments because none of them would be large enough to trigger the early
termination. This makes sure have only a single segment for these test,
making sure we terminate early.
Closes#62769
This test failed on WindowsFS. We failed to remove the corrupted file if
it's being opened (for a short window by ListShardStore action) and the
pending delete files were clear when we restarted that node.
This commit fixes the issue by shutting down the node before removing
the corrupted file to avoid any access to that file.
Closes#66893
We started passing down the root document's _source when processing
nested hits, to avoid reloading and reparsing the root source for each hit.
Unfortunately the approach did not work when there are multiple layers of
`inner_hits`. In this case, the second-layer inner hit received its immediate
parent's source instead of the root source. This parent source is filtered to
just contain the parts corresponding to the nested document, but the source
parsing logic is designed to always operate on the top-level root source. This
caused failures when loading the second-layer inner hits.
This PR makes sure to always pass the root document's _source when processing
inner hits, even if there are multiple layers.
Previously we treated attribute filtering for _tier-prefixed attributes a pass-through, meaning
that they were essentially always treated as matching in DiscoveryNodeFilters.match, however, for
exclude settings, this meant that the node was considered to match the node if a _tier* filter was
specified.
This commit prunes these attributes from the DiscoveryNodeFilters when considering the filters for
FilterAllocationDecider so that they are only considered in DataTierAllocationDecider.
Resolves#66679
This PR fixes two bugs that can arise when _source is disabled and we fetch nested documents:
* Fix exception when highlighting `inner_hits` with disabled _source.
* Fix exception in nested `top_hits` with disabled _source.
* Add more tests for highlighting `inner_hits`.
For async searches (EQL included) the client's request headers were
erroneously stored in the .tasks index. This might expose the requesting
client's HTTP Authorization header. This PR fixes that by employing the
usual approach to store only the security-internal headers, which carry
the authentication result, instead of the original Authorization header,
which is commonly utilized to redo authentication for scheduled tasks.
This PR removes outdated overrides in some tests that prevent them from testing
older index versions. Also removes an old comment + logic from
AggregatorFactoriesTests.
This commit fixes the cat tasks api parameter specification and the
handler so that the parameters are consumed during request preparation.
Closes#59493
Backport of #66272
This commit ensures that the after key is parsed with the doc value formatter.
This is needed for unsigned longs that uses shifted longs internally.
Closes#65685
In the assertion mentioned in the new comment we first get the `endingSnapshots`
and then check that we don't have a listener that isn't referred to by it so we need
to remove from the listers map before removing from `endingSnapshots` to avoid rare, random
assertion tripping here with concurrent repository operations.
If creating the latest translog file and retrieving a translog stats
happen within the same millisecond, then the earliestLastModifiedAge
will be zero.
Closes#66092
This PR fixes a regression where fvh fragments could be loaded from the wrong
document _source.
Some `FragmentsBuilder` implementations contain a `SourceLookup` to load from
_source. The lookup should be positioned to load from the current hit document.
However, since `FragmentsBuilder` are cached and shared across hits, the lookup
is never updated to load from the new documents. This means we accidentally
load _source from a different document.
The regression was introduced in #60179, which started storing `SourceLookup`
on `FragmentsBuilder`.
Fixes#65533.
This commit adjusts the behavior when calculating the diff between two
`AbstractScopedSettings` objects, so that the default values of settings
whose default values depend on the values of other settings are
correctly calculated. Previously, when calculating the diff, the default
value of a depended setting would be calculated based on the default
value of the setting(s) it depends on, rather than the current value of
those settings.
In #61906 we added the possibility for the master node to fetch
the size of a shard snapshot before allocating the shard to a
data node with enough disk space to host it. When merging
this change we agreed that any failure during size fetching
should not prevent the shard to be allocated.
Sadly it does not work as expected: the service only triggers
reroutes when fetching the size succeed but never when it
fails. It means that a shard might stay unassigned until
another cluster state update triggers a new allocation
(as in #64372). More sadly, the test I wrote was wrong as
it explicitly triggered a reroute.
This commit changes the InternalSnapshotsInfoService
so that it also triggers a reroute when fetching the snapshot
shard size failed, ensuring that the allocation can move
forward by using an UNAVAILABLE_EXPECTED_SHARD_SIZE
shard size. This unknown shard size is kept around in the
snapshot info service until no corresponding unassigned
shards need the information.
Backport of #65436
This change simplifies logic and allow more legit cases in Metadata.Builder.validateDataStreams.
It will only show conflict on names that are in form of .ds-<data stream name>-<[0-9]+> and will allow any names like .ds-<data stream name>-something-else-<[0-9]+>.
This fixes problem with rollover when you have 2 data streams with names like a and a-b - currently if a-b has generation greater than a you won't be able to rollover a anymore.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
TransportService doesn't respond to the pending requests of proxy
connections when the underlying connections get disconnected because
proxy connections do not override the getCacheKey method. Some CCS
requests would never be completed because of this bug.
These strings are quite long individually and will be repeated
potentially up to the number of snapshots in the repository times.
Since these make up more than half of the size of the repository metadata
and are likely the same for all snapshots the savings from deduplicating them
can make up for more than half the size of `RepositoryData` easily in most real-world
cases.
Store stats can be `null` if e.g. the shard was already closed
when the stats where retrieved. Don't record those shards in the
sizes map to fix an NPE in this case.
When n=0 in TranslogTests.testTotalTests we never update earliestLastModifiedAge so it fails comparison with default value of total.getEarliestLastModifiedAge() which is 0.
In this change we always check this special case and then select n>0
Closes#65629
In the refactoring of TextFieldMapper, we lost the ability to define
a default search or search_quote analyzer in index settings. This
commit restores that ability, and adds some more comprehensive
testing.
Fixes#65434
In certain situations, such as when configured in FIPS 140 mode,
the Java security provider in use might throw a subclass of
java.lang.Error. We currently do not catch these and as a result
the JVM exits, shutting down elasticsearch.
This commit attempts to address this by catching subclasses of Error
that might be thrown for instance when a PBKDF2 implementation
is used from a Security Provider in FIPS 140 mode, with the password
input being less than 14 bytes (112 bits).
- In our PBKDF2 family of hashers, we catch the Error and
throw an ElasticsearchException while creating or verifying the
hash. We throw on verification instead of simply returning false
on purpose so that the message bubbles up and the cause becomes
obvious (otherwise it would be indistinguishable from a wrong
password).
- In KeyStoreWrapper, we catch the Error in order to wrap and re-throw
a GeneralSecurityException with a helpful message. This can happen when
using any of the keystore CLI commands, when the node starts or when we
attempt to reload secure settings.
- In the `elasticsearch-users` tool, we catch the ElasticsearchException that
the Hasher class re-throws and throw an appropriate UserException.
Tests are missing because it's not trivial to set CI in fips approved mode
right now, and thus any tests would need to be muted. There is a parallel
effort in #64024 to enable that and tests will be added in a followup.
KeyStoreAwareCommand attempted to deduce whether an error occurred
because of a wrong password by checking the cause of the
SecurityException that KeyStoreWrapper.decrypt() throws. Checking
for AEADBadTagException was wrong becase that exception could be
(and usually is) wrapped in an IOException. Furthermore, since we
are doing the check already in KeyStoreWrapper, we can just return
the message of the SecurityException to the user directly, as we do
in other places.
Tianli Feng
Sarat Vemulapalli
Nicholas Knize
Nick Knize
Nhat Nguyen
Rabi Panda
Nik Everett
James Rodewig
Rene Groeschke
Julie Tibshirani
Lee Hinman
Yang Wang
Gordon Brown
Albert Zaharovits
asce0705
Jay Modi
Jim Ferenczi
Armin Braun
Francisco Fernández Castaño
Tanguy Leroux
Przemko Robakowski
Howard
Alan Woodward
Ioannis Kakavas