mirror of
https://github.com/honeymoose/OpenSearch.git
synced 2025-02-11 15:35:05 +00:00
7cab29760d
SQL: Introduce PreAnalyze phase to resolve catalogs async The new preanalyze phase collects all unresolved relations and tries to resolve them as indices through typical async calls _before_ starting the analysis process. The result is loaded into a catalog which is then passed to the analyzer. While at it, the analyzer was made singleton and state across the engine is done through SqlSession#currentContext(). Commit missing fix Fix typo Fix license Fix line length remove redundant static modifier Remove redundant generics type Rename catalogResolver instance member to indexResolver Fix translate action to return a response through the listener, it hangs otherwise IndexResolver improvements Make sure that get index requests calls are locally executed by providing local flag. Don't replace index/alias name with concrete index name in asCatalog response conversion. We need to preserve the original alias name for security, so it is reused in the subsequent search. Update roles and actions names for security tests Get index is now executed instead of sql get indices, and sql get indices has been removed. Also made cluster privileges more restrictive to make sure that cluster state calls are no longer executed. Fix most of the security IT tests indices options are now unified, always lenient. The only situation where we get authorization exception back is when the user is not authorized for the sql action (besides for which indices). Improve SessionContext handling Fix context being invalid in non-executable phases Make Explain & Debug command fully async Resolve checkstyle error about redundant modifiers Temporarily restore SqlGetIndicesAction SqlGetIndicesAction action is still needed in RestSqlJdbcAction (metaTable and metaColumn methods), where we can't at the moment call IndexResolver directly, as security (FLS) needs index resolver to be called as part of the execution of an indices action. Once mappings are returned filtered, delayed action and the security filter will go away, as well as SqlGetIndicesAction. SqlGetIndicesAction doesn't need to be a delayed action, my bad [TEST] remove unused expectSqlWithAsyncLookup and rename expectSqlWithSyncLookup to expectSqlCompositeAction Polish and feedback Add unit test for PreAnalyzer Original commit: elastic/x-pack-elasticsearch@57846ed613
= Elasticsearch X-Pack A set of Elastic's commercial plugins for Elasticsearch: - License - Security - Watcher - Monitoring - Machine Learning - Graph = Setup You must checkout `x-pack-elasticsearch` and `elasticsearch` with a specific directory structure. The `elasticsearch` checkout will be used when building `x-pack-elasticsearch`. The structure is: - /path/to/elastic/elasticsearch - /path/to/elastic/elasticsearch-extra/x-pack-elasticsearch == Vault Secret The build requires a Vault Secret ID. You can use a GitHub token by following these steps: 1. Go to https://github.com/settings/tokens 2. Click *Generate new token* 3. Set permissions to `read:org` 4. Copy the token into `~/.elastic/github.token` 5. Set the token's file permissions to `600` ``` $ mkdir ~/.elastic $ vi ~/.elastic/github.token # Add your_token exactly as it is into the file and save it $ chmod 600 ~/.elastic/github.token ``` If you do not create the token, then you will see something along the lines of this as the failure when trying to build X-Pack: ``` * What went wrong: Missing ~/.elastic/github.token file or VAULT_SECRET_ID environment variable, needed to authenticate with vault for secrets ``` === Offline Mode When running the build in offline mode (`--offline`), it will not required to have the vault secret setup. == Native Code **This is mandatory as tests depend on it** Machine Learning requires platform specific binaries, build from https://github.com/elastic/machine-learning-cpp via CI servers. The native artifacts are stored in S3. To retrieve them infra's team Vault service is utilized, which requires a github token. Please setup a github token as documented: https://github.com/elastic/infra/blob/master/docs/vault.md#github-auth The github token has to be put into ~/.elastic/github.token, while the file rights must be set to 0600. = Build - Run unit tests: + [source, txt] ----- gradle clean test ----- - Run all tests: + [source, txt] ----- gradle clean check ----- - Run integration tests: + [source, txt] ----- gradle clean integTest ----- - Package X-Pack (without running tests) + [source, txt] ----- gradle clean assemble ----- - Install X-Pack (without running tests) + [source, txt] ----- gradle clean install ----- = Building documentation The source files in this repository can be included in either the X-Pack Reference or the Elasticsearch Reference. NOTE: In 5.5 and later, the Elasticsearch Reference includes X-Pack-specific content when it is built from this repo. To build the Elasticsearch Reference on your local machine: * Use the `index.asciidoc` file in the docs/en directory. * Specify the location of the `elasticsearch/docs` directory with the `--resource` option when you run `build_docs.pl`. For example: [source, txt] ----- ./docs/build_docs.pl --doc elasticsearch-extra/x-pack-elasticsearch/docs/en/index.asciidoc --resource=elasticsearch/docs --chunk 1 ----- For information about building the X-Pack Reference, see the README in the x-pack repo. To build a release notes page for the pull requests in this repository: * Use the dev-tools/xes-release-notes.pl script to pull PRs from the x-pack-elasticsearch repo. Alternatively, use the dev-tools/xescpp_release_notes.pl script to pull PRs from both the x-pack-elasticsearch and machine-learning-cpp repos. * Specify the version label for which you want the release notes. * Redirect the output to a new local file. NOTE: You must have a personal access token called ~/.github_auth with "repo" scope. Use steps similar to "Vault Secret" to create this file. For example: [source, txt] ----- ./dev-tools/xes_release_notes.pl v5.5.2 > ~/tmp/5.5.2.asciidoc ----- == Adding Images When you include an image in the documentation, specify the path relative to the location of the asciidoc file. By convention, we put images in an `images` subdirectory. For example to insert `watcher-ui-edit-watch.png` in `watcher/limitations.asciidoc`: . Add an `images` subdirectory to the watcher directory if it doesn't already exist. . In `limitations.asciidoc` specify: + [source, txt] ----- image::images/watcher-ui-edit-watch.png["Editing a watch"] ----- Please note that image names and anchor IDs must be unique within the book, so do not use generic identifiers.