* [Rename] o.e.common subpackages round 1
This commit refactors the following subpackages of o.e.common:
* o.e.common.joda
* o.e.common.lease
* o.e.common.metrics
* o.e.common.network
* o.e.common.path
* o.e.common.recycling
* o.e.common.regex
* o.e.common.rounding
* o.e.common.text
* o.e.common.time
* o.e.common.transport
to the o.opensearch namespace. All references throughout the codebase have been
refactored.
Signed-off-by: Nicholas Knize <nknize@amazon.com>
* fix imports 1
Signed-off-by: Nicholas Knize <nknize@amazon.com>
Rename `org.elasticserach.search.aggregations.pipeline` to `org.opensearch.search.aggregations.pipeline` in package names and references.
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
* [Rename] o.e.common.inject subpackages
This commit refactors the remaining o.e.common.inject subpackages to the
o.opensearch.common.inject namespace. All references throughout the codebase
have been refactored.
Signed-off-by: Nicholas Knize <nknize@amazon.com>
This commit refactors the following packages:
* o.e.common.geo
* o.e.common.hash
* o.e.common.io
into the o.opensearch.common namespace. All references throughout the codebase
have been refactored.
Signed-off-by: Nicholas Knize <nknize@amazon.com>
This commit refactors the classes in the o.e.inject module to the
o.opensearch.inject namespace. References throughout the codebase have also been
refactored.
Signed-off-by: Nicholas Knize <nknize@amazon.com>
This commit refactors the following:
* o.e.common.cache
* o.e.common.collect
* o.e.common.component
* o.e.common.compress
* o.e.common.document
to the o.opensearch namespace. All references throughout the codebase are also
refactored
Signed-off-by: Nicholas Knize <nknize@amazon.com>
As part of this commit we refactor the following in the o.e.search package:
- rename `org.elasticsearch.search.fetch` to `org.opensearch.search.fetch`
- rename `org.elasticsearch.search.internal` to `org.opensearch.search.internal`
- rename `org.elasticsearch.search.profile` to `org.opensearch.search.profile`
- rename `org.elasticsearch.search.query` to `org.opensearch.search.query`
- rename `org.elasticsearch.search.suggest` to `org.opensearch.search.suggest`
- rename other instances of Elasticsearch to OpenSearch in these packages.
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
This commit refactors the following packages:
* o.e.common.blobstore
* o.e.common.breaker
* o.e.common.bytes
to the o.opensearch.common namespace. All references throughout the codebase
have been refactored.
Signed-off-by: Nicholas Knize <nknize@amazon.com>
This commit refactors all classes in o.e.discovery to the o.opensearch.discovery
namespace. All references throughout the codebase are also refactored.
Signed-off-by: Nicholas Knize <nknize@amazon.com>
This commit refactors classes under o.e.common to o.opensearch.common. All
references throughout the codebase have also been refactored.
Signed-off-by: Nicholas Knize <nknize@amazon.com>
This commit refactors the o.e.bootstrap package to o.opensearch.bootstrap. All
references throughout the code are also refactored.
Signed-off-by: Nicholas Knize <nknize@amazon.com>
This commit refactors the remaining o.e.cluster packages to
o.opensearch.cluster. All references throughout the codebase are also
refactored.
Signed-off-by: Nicholas Knize <nknize@amazon.com>
This commit refactors the remaining o.e.action.support subpackages to
o.opensearch.support. All references throughout the codebase are also refactored
Signed-off-by: Nicholas Knize <nknize@amazon.com>
Refactoring:
- rename `org.elasticsearch.search.builder` to `org.opensearch.search.builder`
- rename `org.elasticsearch.search.collapse` to `org.opensearch.search.collapse`
- rename `org.elasticsearch.search.dfs` to `org.opensearch.search.dfs`
- rename `org.elasticsearch.search.lookup` to `org.opensearch.search.lookup`
- rename `org.elasticsearch.search.lookup` to `org.opensearch.search.lookup`
- rename `org.elasticsearch.search.rescore` to `org.opensearch.search.rescore`
- rename `org.elasticsearch.search.searchafter` to `org.opensearch.search.searchafter`
- rename `org.elasticsearch.search.slice` to `org.opensearch.search.slice`
- rename `org.elasticsearch.search.sort` to `org.opensearch.search.sort`
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
Refactor the transport package in the server module to rename the package from `org.elasticsearch.transport` to `org.opensearch.transport`
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
This commit refactors o.e.Version to o.opensearch.Version. This is retained in a
single commit to serve as a reference for re-versioning the opensearch codebase
from legacy 7.10 to 1.0.
Signed-off-by: Nicholas Knize <nknize@amazon.com>
This commit refactors all OpenSearch classes in the root server package to
o.opensearch. All references throughout the codebase are also refactored.
Signed-off-by: Nicholas Knize <nknize@amazon.com>
This commit refactors the o.e.cli and o.e.client packages from elasticsearch to
o.opensearch.cli and o.opensearch.client packages in the server module,
respectively.
Signed-off-by: Nicholas Knize <nknize@amazon.com>
This commit refactors the following subpackages:
* o.e.cluster.health
* o.e.cluster.metadata
* o.e.cluster.node
to o.opensearch.cluster.*. All other references throughout the codebase are
updated.
Signed-off-by: Nicholas Knize <nknize@amazon.com>
This commit refactors the heavily used ESPolicy, Elasticsearch (main class), and Elasticsearch
prefixed test classes used in the bootstrap package under the server module. Refactoring the
namespace will come in a separate commit.
Signed-off-by: Nicholas Knize <nknize@amazon.com>
Refactor the repositories package in the server module to rename the package from `org.elasticsearch.repositories` to `org.opensearch.repositories`
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
This commit refactors the following:
* o.e.cluster.ack
* o.e.cluster.action
* o.e.cluster.block
* o.e.cluster.coordination
to o.opensearch package. all other references are also refactored.
Signed-off-by: Nicholas Knize <nknize@amazon.com>
This commit refactors all classes in o.e.cluster to o.opensearch.cluster.
Refereences throughtout the code base are updated.
Signed-off-by: Nicholas Knize <nknize@amazon.com>
Refactor the `libs/cli` module to rename the package name from `org.elasticsearch.cli` to `org.opensearch.cli` as part of the rename to OpenSearch work.
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
Refactor the `org.elasticsearch.usage` package in the server module to rename it to `org.opensearch.usage`.
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
Refactor the package`org.elasticsearch.script` in server module to rename it to`org.opensearch.script`.
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
Refactor the `libs/geo` module to rename the package name from `org.elasticsearch.geometry` to `org.opensearch.geometry` as part of the rename to OpenSearch work.
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
Refactor the libs/plugin-cli and libs/secure-sm modules to rename the package names
- `org.elasticsearch.plugins` to `org.opensearch.plugins`
- `org.elasticsearch.secure_sm` to `org.opensearch.secure_sm`
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
Refactor the server/tasks package to rename the package names from`org.elasticsearch.tasks` to `org.opensearch.tasks`.
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
Refactor the server/threadpool package to rename the package names from`org.elasticsearch.threadpool` to `org.opensearch.threadpool`.
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
This commit refactors the classes in o.e.action.support to
o.opensearch.action.support. The remaining directories will be refactored in a
separate commit.
Signed-off-by: Nicholas Knize <nknize@amazon.com>
Refactor `server/snapshots` to rename the package names from `org.elasticsearch.snapshots` to `org.opensearch.snapshots` as part of the rename to OpenSearch work.
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
This commit refactors all classes in o.e.action.bulk to o.opensearch.action.bulk
all references throughout the rest of the codebase are updated.
Signed-off-by: Nicholas Knize <nknize@amazon.com>
This commit refactors all classes in o.e.action.admin.cluster to
org.opensearch.action.admin.cluster. References are updated
throughout the codebase.
Signed-off-by: Nicholas Knize <nknize@amazon.com>
This commit refactors top level classes in o.e.action to o.opensearch.action.
References throughout the rest of the codebase have been updated.
Signed-off-by: Nicholas Knize <nknize@amazon.com>
This commit refactors o.e.action.admin.indices package to
o.opensearch.action.admin.indices. References through out the codebase have been
updated to reflect the new package location.
Signed-off-by: Nicholas Knize <nknize@amazon.com>
This commit refactors ElasticsearchParseException class in the server module to
OpenSearchParseException. References and usages throughout the rest of the
codebase are fully refactored.
Signed-off-by: Nicholas Knize <nknize@amazon.com>
This commit refactors the ElasticsearchDirectoryReader class located in the
server module to OpenSearchDirectoryReader. References and usages, along with
method names, throughout the rest of the codebase are fully refactored.
Signed-off-by: Nicholas Knize <nknize@amazon.com>
This commit refactors the ElasticsearchNodeCommand class located in the server
module to OpenSearchNodeCommand. References and usages throughout the rest of
the codebase are fully refactored.
Signed-off-by: Nicholas Knize <nknize@amazon.com>
This commit refactors the ElasticsearchException class located in the server module
to OpenSearchException. References and usages throughout the rest of the
codebase are fully refactored.
Signed-off-by: Nicholas Knize <nknize@amazon.com>
This commit changes the building, packaging, and testing framework to only support OSS on different distributions.
Next steps:
completely remove -oss flag dependencies in package and build tests
move 6.x bwc testing to be an explicit option
remove any references to elastic.co download site (or replace with downloads from the OSS website)
Co-authored-by: Himanshu Setia <setiah@amazon.com>
Co-authored-by: Rabi Panda <pandarab@amazon.com>
Co-authored-by: Himanshu Setia <58999915+setiah@users.noreply.github.com>
Co-authored-by: Sarat Vemulapalli <vemsarat@amazon.com>
Signed-off-by: Peter Nied <petern@amazon.com>
This commit adds parse logic to correctly parse feature flags in the test framework. It also fixes a test failure in cat.templates/Sort Templates yaml test.
Signed-off-by: Peter Nied <petern@amazon.com>
This commit removes all trace of the security high level rest client and other reference to x-pack security
Co-authored-by: Rabi Panda <rabipanda@icloud.com>
Signed-off-by: Peter Nied <petern@amazon.com>
The selective muting implemented for autoscaling in #67159
is extended to the ML tests that also fail when machine
memory is reported as 0.
Most of the logic to determine when memory will not be
accurately reported is now in a utility method in the
base class.
Relates #66885
Backport of #67422
TransportService doesn't respond to the pending requests of proxy
connections when the underlying connections get disconnected because
proxy connections do not override the getCacheKey method. Some CCS
requests would never be completed because of this bug.
In the refactoring of TextFieldMapper, we lost the ability to define
a default search or search_quote analyzer in index settings. This
commit restores that ability, and adds some more comprehensive
testing.
Fixes#65434
This change fixes the equals and hashCode methods of the custom FieldValuesSource
that is used internally to extract the value from a doc value field.
Using the field data instance to check equality prevented the query to be cached in
previous versions. Switching to the field name should make the query eligible for
caching again.
In aa1ea96b8698aa12bed1c4e8d704882a2a639791 I made all
`testReduceRandom` tests for aggs mimick production more precisely.
More precisely, they pick the correct "lead" result when performing
partial reduction. This is great, but, sadly, some tests assumed that we
always reduced against the "first" aggregator. This fixes those tests.
Closes#65163
This changes has `InternalAggregationTestCase` performs partial
reductions to better mimick `InternalAggregations#reduce`. In
particular, this fixes a bug when we'd attempt a partial reduction of
aggregations that were entirely unmapped but use a mapped aggregation as
the "lead" of the reduction. This was possible in the test but isn't
possible in production. The reduction "lead" is unmapped if all agg
results being reduced are unmapped.
Closes#65118
Backport of #64454
- Add LongRareTerms and StringRareTerms to the DefaultNamedXContents,
ensure that the response of RareTerms aggregation can be parsed
correctly.
- Add testSearchWithRareTermsAgg method to test the response of
RareTerms aggregation can be parsed correctly.
- Add some test code to ensure the AggregationsTests can execute
successfully.
Co-authored-by: bellengao <gbl_long@163.com>
We were correctly dealing with boosts that had an effect, but mappers
that had a silently accepted but ignored boost parameter were throwing
an error instead of continuing to ignore the boost but emitting a
warning.
Fixes#64982
Currently a rejected execution exception can be swallowed when async
actions return during transport bulk actions. This includes scenarios
where we went async to perform ingest pipelines or index creation. This
commit resolves the issue by propagating a rejected exception.
10s is pretty tight here considering that some x-pack related tests
might have a running create-index tasks in the backgroud after a test
that require multiple CS updates and can randomly take a few seconds on
slow IO etc.
closes#64580
This change fixes a bug introduced in #61779 that uses a compound order to
compare buckets when merging. The bug is triggered when the compound order
uses a primary sort ordered by key (asc or desc).
This commit ensures that we always extract the primary sort when comparing keys
during merging.
The PR is marked as no-issue since the bug has not been released in any official version.
We have to wait for no more operations here not for `1`. This mostly worked
because the test thread would add the listener quickly enough so that it sees the
state where either the snapshot or clone but not both have already finished
but randomly the test thread would be slow and time out on a state without snaphots in it.
We had and an error when serializing fully reduced scripted metrics.
Small typo and sever lack of tests..... Anyway, this fixed the one
character typo and adds a bunch more tests.
Today indexing to a shard with 2147483519 documents will fail that
shard. We should check the number of documents and reject the write
requests instead.
Closes#51136
This fixes a gap in testing and a bug that can occur in various forms:
When we would start a snapshot or clone related to a shard that was done
snapshotting/cloning but its overall operation was not yet finalized
at the time of starting the operation, we would base the operation off of
the wrong generation. This would not cause a corrupted repo, but would
cause the operation to be `PARTIAL`.
This commit fixes the state machine to take into account the correct generation
in this case.
Closes#63498
This PR implements value fetching for the following field types:
* `text` phrase and prefix subfields
* `search_as_you_type`, plus its subfields
* `token_count`, which is implemented by fetching doc values
Supporting these types helps ensure that retrieving all fields through
`"fields": ["*"]` doesn't fail because of unsupported value fetchers.
Plugins are loaded in isolated child class loaders of the root class loader. However, some libraries depend on the context class loader being set. This commit sets the context class loader for the duration of calling each plugins constructor.
relates #52320
Co-authored-by: Ryan Ernst <ryan@iernst.net>
This PR adds deprecation warnings when accessing System Indices via the REST layer. At this time, these warnings are only enabled for Snapshot builds by default, to allow projects external to Elasticsearch additional time to adjust their access patterns.
Deprecation warnings will be triggered by all REST requests which access registered System Indices, except for purpose-specific APIs which access System Indices as an implementation detail a few specific APIs which will continue to allow access to system indices by default:
- `GET _cluster/health`
- `GET {index}/_recovery`
- `GET _cluster/allocation/explain`
- `GET _cluster/state`
- `POST _cluster/reroute`
- `GET {index}/_stats`
- `GET {index}/_segments`
- `GET {index}/_shard_stores`
- `GET _cat/[indices,aliases,health,recovery,shards,segments]`
Deprecation warnings for accessing system indices take the form:
```
this request accesses system indices: [.some_system_index], but in a future major version, direct access to system indices will be prevented by default
```
Determines the shard size of shards before allocating shards that are
recovering from snapshots. It ensures during shard allocation that the
target node that is selected as recovery target will have enough free
disk space for the recovery event. This applies to regular restores,
CCR bootstrap from remote, as well as mounting searchable snapshots.
The InternalSnapshotInfoService is responsible for fetching snapshot
shard sizes from repositories. It provides a getShardSize() method
to other components of the system that can be used to retrieve the
latest known shard size. If the latest snapshot shard size retrieval
failed, the getShardSize() returns
ShardRouting.UNAVAILABLE_EXPECTED_SHARD_SIZE. While
we'd like a better way to handle such failures, returning this value
allows to keep the existing behavior for now.
Note that this PR does not address an issues (we already have today)
where a replica is being allocated without knowing how much disk
space is being used by the primary.
Co-authored-by: Yannick Welsch <yannick@welsch.lu>
MapperService carries a lot of weight and is only used to determine if loading of field data for the id field is enabled, which can be done in a different way.
Just a few spots where we can dry up these tests using the snapshot test infrastructure
in core that I found while studying the existing searchable snapshot tests.
* Just some obvious drying up of these super complex tests.
* Mainly just shortening the diff of #61839 here by moving test utilities
to the abstract test case.
Also, making use of the now available functionality to simplify existing tests
and improve logging in them.
For runtime fields, we will want to do all search-time interaction with
a field definition via a MappedFieldType, rather than a FieldMapper, to
avoid interfering with the logic of document parsing. Currently, fetching
values for runtime scripts and for building top hits responses need to
call a method on FieldMapper. This commit moves this method to
MappedFieldType, incidentally simplifying the current call sites and freeing
us up to implement runtime fields as pure MappedFieldType objects.
Splitting some tests out of this class that has become a catch-all
for random snapshot related tests into either existing suits that fit
better for these tests or one of two new suits to prevent timeouts
in extreme cases (e.g. `WindowsFS` + many nodes + multiple data paths per node).
No other changes to tests were made whatsoever.
Closes#61541
Introduce 64-bit unsigned long field type
This field type supports
- indexing of integer values from [0, 18446744073709551615]
- precise queries (term, range)
- precise sort and terms aggregations
- other aggregations are based on conversion of long values
to double and can be imprecise for large values.
Backport for #60050Closes#32434
This commit adds a mechanism to MapperTestCase that allows implementing
test classes to check that their parameters can be updated, or throw conflict
errors as advertised. Child classes override the registerParameters method
and tell the passed-in UpdateChecker class about their parameters. Simple
conflicts can be checked, using the existing minimal mappings as a base to
compare against, or alternatively a particular initial mapping can be provided
to check edge cases (eg, norms can be updated from true to false, but not
vice versa). Updates are registered with a predicate that checks that the update
has in fact been applied to the resulting FieldMapper.
Fixes#61631
Most of our field types have the same implementation for their `existsQuery` method which relies on doc_values if present, otherwise it queries norms if available or uses a term query against the _field_names meta field. This standard implementation is repeated in many different mappers.
There are field types that only query doc_values, because they always have them, and field types that always query _field_names, because they never have norms nor doc_values. We could apply the same standard logic to all of these field types as `MappedFieldType` has the knowledge about what data structures are available.
This commit introduces a standard implementation that does the right thing depending on the data structure that is available. With that only field types that require a different behaviour need to override the existsQuery method.
At the same time, this no longer forces subclasses to override `existsQuery`, which could be forgotten when needed. To address this we introduced a new test method in `MapperTestCase` that verifies the `existsQuery` being generated and its consistency with the available data structures.
Nick Knize
Rabi Panda
Sarat Vemulapalli
Rabi Panda
Nick Knize
Abbas Hussain
Tianli Feng
David Roberts
Nhat Nguyen
Alan Woodward
Jim Ferenczi
Nik Everett
Sylvain Wallez
Alan Woodward
Tim Brooks
Armin Braun
Jason Tedor
Julie Tibshirani
Przemyslaw Gomulka
Dawid Weiss
Gordon Brown
Tanguy Leroux
Luca Cavanna
Howard
Francisco Fernández Castaño
Mayya Sharipova