* Moving RecoveryState.Index to a top-level class and renaming
This class is a building block of replication and will be re-used between peer recovery and segment replication. Thus, the inner class has been extracted to a top-level class and moved to the replication.common package. It has been renamed to ReplicationLuceneIndex to better reflect what it represents. It has two dependent inner classes from RecoveryState that have also been moved along with it - these remain inner classes since they are not currently used anywhere else. The RecoveryFilesDetails class has been renamed to FilesDetails and the FileDetail class has been renamed to FileMetadata.
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Incorporate PR comments
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Revert Project_Default.xml
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Update REST Action test to no longer mock a final class
Instead, the test now populates dummy data.
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Removing mocking of ReplicationLuceneIndex in RecoverySourceHandlerTests
The class has been marked final, so it can no longer be mocked. Instead, the test class sets up the lucene index class by adding the smae file metadata that is set up for the store.
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Configure @opensearch.internal as custom Javadoc tag
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Revert "Configure @opensearch.internal as custom Javadoc tag"
This reverts commit 2077d76e421fc2ceea7c1c9714b6a1906a555a31.
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Bump HdrHistogram from 2.1.9 to 2.1.12 in /server
Bumps [HdrHistogram](https://github.com/HdrHistogram/HdrHistogram) from 2.1.9 to 2.1.12.
- [Release notes](https://github.com/HdrHistogram/HdrHistogram/releases)
- [Commits](https://github.com/HdrHistogram/HdrHistogram/compare/HdrHistogram-2.1.9...HdrHistogram-2.1.12)
---
updated-dependencies:
- dependency-name: org.hdrhistogram:HdrHistogram
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* Updating SHAs
Signed-off-by: dependabot[bot] <support@github.com>
* Update test to match new behavior of HdrHistogram
There was a [change in behavior in HdrHistogram][1] that causes the
tests to fail with the new version. The fix here is to add another value
in the test data set for the failing test, which has the result of
making both versions of HdrHistogram agree on the percentile values.
[1]: 5c7226c4ff
Signed-off-by: Andrew Ross <andrross@amazon.com>
* Remove unnecessary exclusion on third party audit
Signed-off-by: Andrew Ross <andrross@amazon.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
Co-authored-by: Andrew Ross <andrross@amazon.com>
Removes last trace of TypeFieldMapper since types have been removed and
NestedPathFieldMapper is now available.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Adds javadocs to classes in the org.opensearch.monitor, persistence, plugins,
repository, script, threadpool, usage, and watcher packages.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Adds javadocs to classes in the org.opensearch.discovery, env, gateway, http,
ingest, lucene, and node packages.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* Upgrading Tika from 1.24.1 to 2.1.0 and bumping xmlbeans version
This major version upgrade requires an explicit dependency on tika-parsers-standard-package to import the parser implementations, and an update to the namespace of RTFParser. Also, LanguageIdentifier has been deprecated and replaced by LanguageDetector.
This change includes a bump in xmlbeans version from 3.0.1 to 3.1.0
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Upgrade Tika libraries from 2.1.0 to 2.2.0
This also requires a update of Apache Commons-IO from 2.7 to 2.11.0
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Upgrade Tika libraries from 2.2.0 to 2.2.1
Also update PDFBox to 2.0.25 as per Tika release notes
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Upgraded Tika and xmlbeans libraries
Tika libraries have been upgraded from 2.2.1 to 2.3.0. xmlbeans is now a subproject of POI, so POI was upgraded from 4.1.2 to 5.2.2. With POI 5.x the ooxml-schemas library has been moved to ooxml-lite/ooxml-full. Since ooxml-schemas no longer exists, the LICENSE and NOTICE files in the licenses/ directory have been removed. Finally, xmlbeans has been updated from 3.1.0 to 5.0.2
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* (In progress) Added tika-langdetect
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Upgrading tika libraries to 2.4.0
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Switched from tika-langdetect to tika-langdetect-optimaize
To fix the license check, the mapping regex was expanded to tika-.*
This now means the tika-core LICENSE and NOTICE files are no longer needed.
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* (Work in progress) Switching AttachmentProcessor to use OptimaizeLangDetector
This is a concrete implementation of LanguageDetector. Using this requires bringing in the optimaize dependency.
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Manually added LICENSE and NOTICE files for Optimaize language-detector
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Move Optimaize dependency to runtimeOnly
Also bring in transitive Guava dependency. This requires manual addition of LICENSE and NOTICE files as with other plugins.
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Fix Optimaize langDetector to load models first before detecting
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Fallback logic, and test updates
Following the Tika library upgrade, some fallback logic is necessary:
1. "Author" is deprecated for MSOffice document parsing. It is recommended to use CREATOR from Tika Core Properties instead.
2. EPUB parsing no longer automatically extracts keywords. The convention to fall back to SUBJECT is now manually implemented in AttachmentProcessor
Finally, unit tests have been upgraded to account for non-deterministic language results across library upgrades.
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Drop Guava version from 31.1 to 18.0
This is the version that Optimaize 0.6 depends on, and it allows for a smaller ignoreViolations list
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Fix ingest-attachment integration test to assert correct language
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
The types exist transport action can be removed now that the TransportClient has
been removed and types support has been removed.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
In preparation for re-enabling the missingJavadoc gradle task this change adds
in the missing package-info.java files to the server folder. For now general
javadocs are added to these files with the intent to clean up with better
descriptions over time.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
When using package distribution
```
./gradlew :distribution:packages:no-jdk-deb:assemble
```
When `true` this include jdk switch the boolean to the correct value
fix https://github.com/opensearch-project/OpenSearch/issues/3024
Signed-off-by: Laurent Arnoud <laurent.arnoud@platform.sh>
Refactors XContentType.fromMediaTypeOrFormat to fromMediaType so Accept headers
and Content-Type headers can be parsed separately. This helps in reusing the
same parse logic in for REST Versioning API support.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* Replace internal usages of 'master' term in 'client' directory
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Add a uni test for NodeSelector to test deprecated master role
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Replace internal usages of 'master' terminology in server/src/main directory
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Restore rename DISCOVERED_MASTER in ClusterHealthResponse
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Rename two methods in unit tests
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Replace master word in ClusterState
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Replace master word in LeaderChecker JoinHelper JoinTaskExecutor
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Replace master word in more classes
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Replace master word in more classes
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Replace master word in more classes
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Replace master word in more classes
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Replace master word in more classes
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Replace master word in more classes
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Replace master word in DiscoveryNodes classes
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Replace master word in more classes
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Correct mistakes
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Adjust format by spotlessApply task
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Change MASTER__NODE_BOOTSTRAPPED_MSG in test
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Fix SnapshotDisruptionIT by renaming to cluster-manager
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* [Remove] Type from nested fields using new metadata field mapper
types support is removed yet nested documents use the _type field to store the
path for nested documents. A new _nested_path metadata field mapper is added to
take the place of the _type field in order to remove the type dependency in
nested documents. BWC is handled in the new field mapper to ensure compatibility
with older versions.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* pr fixes
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* add test to merge same mapping with empty index settings
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* This change formalizes the notion of feature flags, and adds a "replication type" setting that will differentiate between document and segment replication, gated by a feature flag.
Since seg-rep is currently an incomplete implementation, the feature flag ensures that the setting is not visible to users without explicitly setting a system property. We can then continue to merge seg-rep related changes from the feature branch to `main` safely hidden behind the feature flag gate.
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Update security policy for testing feature flags
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Co-authored-by: Nicholas Walter Knize <nknize@apache.org>
Daniel Doubrovkine (dB.)
Nick Knize
Sarat Vemulapalli
Prudhvi Godithi
opensearch-trigger-bot[bot]
Kartik Ganesh
Andrew Ross
dependabot[bot]
Tianli Feng
Laurent Arnoud
Poojita Raj
Suraj Singh
Peng Huo
Daniel Doubrovkine (dB.)