🔎 Open source distributed and RESTful search engine.
Go to file
Marc Handalian a0030dfb47
Add a new Engine implementation for replicas with segment replication enabled. (#3240)
* Change fastForwardProcessedSeqNo method in LocalCheckpointTracker to persisted checkpoint.

This change inverts fastForwardProcessedSeqNo to fastForwardPersistedSeqNo for use in
Segment Replication.  This is so that a Segrep Engine can match the logic of InternalEngine
where the seqNo is incremented with each operation, but only persisted in the tracker on a flush.
With Segment Replication we bump the processed number with each operation received index/delete/noOp, and
invoke this method when we receive a new set of segments to bump the persisted seqNo.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Extract Translog specific engine methods into an abstract class.

This change extracts translog specific methods to an abstract engine class so that other engine
implementations can reuse translog logic.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Add a separate Engine implementation for replicas with segment replication enabled.

This change adds a new engine intended to be used on replicas with segment replication enabled.
This engine does not wire up an IndexWriter, but still writes all operations to a translog.
The engine uses a new ReaderManager that refreshes from an externally provided SegmentInfos.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Fix spotless checks.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Fix :server:compileInternalClusterTestJava compilation.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Fix failing test naming convention check.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* PR feedback.

- Removed isReadOnlyReplica from overloaded constructor and added feature flag checks.
- Updated log msg in NRTReplicationReaderManager
- cleaned up store ref counting in NRTReplicationEngine.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Fix spotless check.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Remove TranslogAwareEngine and build translog in NRTReplicationEngine.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Fix formatting

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Add missing translog methods to NRTEngine.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Remove persistent seqNo check from fastForwardProcessedSeqNo.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* PR feedback.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Add test specific to translog trimming.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Javadoc check.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Add failEngine calls to translog methods in NRTReplicationEngine.
Roll xlog generation on replica when a new commit point is received.

Signed-off-by: Marc Handalian <handalm@amazon.com>
2022-05-24 10:18:18 -07:00
.ci Added bwc version 1.3.3 (#3205) 2022-05-05 16:19:04 -04:00
.github Fix Lucene-snapshots repo for jdk 17. (#3396) 2022-05-19 09:46:03 -05:00
.idea Add IssueNavigationLink (#1964) 2022-01-27 09:04:03 -08:00
benchmarks Add a new node role 'cluster_manager' as the alternative for 'master' role and deprecate 'master' role (#2424) 2022-03-18 15:06:06 -07:00
buildSrc [Upgrade] Lucene-9.2.0-snapshot-ba8c3a8 (#3416) 2022-05-20 16:29:51 -05:00
client [Type removal] Remove redundant _type in pipeline simulate action (#3371) 2022-05-18 13:43:32 -04:00
dev-tools Used set to make shell scripts more strict (#3278) 2022-05-16 14:05:13 -04:00
distribution Used set to make shell scripts more strict (#3278) 2022-05-16 14:05:13 -04:00
doc-tools Remove binary (class) files from the repository (#2969) 2022-04-19 12:34:42 -04:00
gradle Replace 'whitelist' terminology in Java API (#3350) 2022-05-17 13:34:05 -07:00
jenkins Changed JAVA_HOME to jdk-17 (#2656) 2022-03-29 18:26:04 -07:00
libs [Upgrade] Lucene-9.2.0-snapshot-ba8c3a8 (#3416) 2022-05-20 16:29:51 -05:00
licenses Remove the Elastic license file, all checks for this license and the license REST APIs. (#12) 2021-03-13 10:36:07 -06:00
modules [Upgrade] Lucene-9.2.0-snapshot-ba8c3a8 (#3416) 2022-05-20 16:29:51 -05:00
plugins Bump gax-httpjson from 0.101.0 to 0.103.1 in /plugins/repository-gcs (#3426) 2022-05-23 11:25:57 -05:00
qa Used set to make shell scripts more strict (#3278) 2022-05-16 14:05:13 -04:00
release-notes Fix release notes for 2.0.0-rc1 version (#3418) 2022-05-20 15:29:56 -07:00
rest-api-spec [Type removal] Remove _type from _bulk yaml test, scripts, unused constants (#3372) 2022-05-18 17:04:29 -04:00
sandbox [Upgrade] Lucene-9.2-snapshot (#2924) 2022-04-21 14:36:01 -07:00
server Add a new Engine implementation for replicas with segment replication enabled. (#3240) 2022-05-24 10:18:18 -07:00
test Add a new Engine implementation for replicas with segment replication enabled. (#3240) 2022-05-24 10:18:18 -07:00
.dir-locals.el Go back to 140 column limit in .dir-locals.el 2017-04-14 08:50:53 -06:00
.editorconfig Remove default indent from .editorconfig (#49183) 2019-11-18 08:05:53 +00:00
.gitattributes Updating .gitattributes for additional file types (#1727) 2021-12-17 16:09:34 -05:00
.gitignore Removed binary file from linelint (#3015) 2022-04-20 22:36:58 -05:00
.linelint.yml Removed binary file from linelint (#3015) 2022-04-20 22:36:58 -05:00
.lychee.excludes Upgrade and fix link checker to 1.2. (#1811) 2021-12-27 11:23:52 -06:00
.whitesource Change whitesource integration to scan on 1.x branch (#1786) 2022-01-03 17:49:09 -08:00
ADMINS.md Updated READMEs on releasing, maintaining, admins and security. (#853) 2021-06-17 18:03:11 -04:00
CODE_OF_CONDUCT.md Update CODE_OF_CONDUCT.md 2021-03-13 10:36:16 -06:00
CONTRIBUTING.md changed work-in-progress language (#1275) 2021-10-26 10:39:25 -07:00
DEVELOPER_GUIDE.md Add Github Workflow to build and publish lucene snapshots. (#2906) 2022-04-15 15:09:33 -05:00
LICENSE.txt Update LICENSE.txt 2021-03-22 09:25:46 -05:00
MAINTAINERS.md Adding @Bukhtawar to OpenSearch maintainers. (#3231) 2022-05-06 15:49:46 -04:00
NOTICE.txt Fixed copyright to OpenSearch (#1175) 2021-08-31 12:55:28 -04:00
README.md Added a new line linter (#2875) 2022-04-13 14:14:18 -04:00
RELEASING.md Added a new line linter (#2875) 2022-04-13 14:14:18 -04:00
SECURITY.md Added a new line linter (#2875) 2022-04-13 14:14:18 -04:00
TESTING.md Replace blacklist in Gradle build environment configuration (#2752) 2022-04-05 17:48:39 -07:00
Vagrantfile [FEATURE] Add OPENSEARCH_JAVA_HOME env to override JAVA_HOME (#2001) 2022-02-02 13:55:22 -05:00
build.gradle Bump com.diffplug.spotless from 6.5.2 to 6.6.1 (#3356) 2022-05-18 10:31:26 -07:00
gradle.properties Adding support for JDK17 and removing JDK8 (#2025) 2022-02-02 20:59:10 -05:00
gradlew Update gradle wrapper to 6.6 (#59909) (#60949) 2020-08-11 11:03:19 +02:00
gradlew.bat Update gradle wrapper to 6.6 (#59909) (#60949) 2020-08-11 11:03:19 +02:00
settings.gradle Bump com.gradle.enterprise from 3.10 to 3.10.1 (#3425) 2022-05-23 11:24:07 -05:00
whitesource.config Change whitesource integration to scan on 1.x branch (#1786) 2022-01-03 17:49:09 -08:00

README.md

Welcome!

OpenSearch is a community-driven, open source fork of Elasticsearch and Kibana following the licence change in early 2021. We're looking to sustain (and evolve!) a search and analytics suite for the multitude of businesses who are dependent on the rights granted by the original, Apache v2.0 License.

Project Resources

Code of Conduct

This project has adopted the Amazon Open Source Code of Conduct. For more information see the Code of Conduct FAQ, or contact opensource-codeofconduct@amazon.com with any additional questions or comments.

Security

If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our vulnerability reporting page or directly via email to aws-security@amazon.com. Please do not create a public GitHub issue.

License

This project is licensed under the Apache v2.0 License.

Copyright OpenSearch Contributors. See NOTICE for details.

Trademark

OpenSearch is a registered trademark of Amazon Web Services.

OpenSearch includes certain Apache-licensed Elasticsearch code from Elasticsearch B.V. and other source code. Elasticsearch B.V. is not the source of that other source code. ELASTICSEARCH is a registered trademark of Elasticsearch B.V.