OpenSearch/settings.gradle

187 lines
6.5 KiB
Groovy
Raw Normal View History

String dirName = rootProject.projectDir.name
rootProject.name = dirName
List projects = [
'build-tools',
'rest-api-spec',
'docs',
'client:rest',
'client:rest-high-level',
'client:sniffer',
'client:transport',
'client:test',
'client:client-benchmark-noop-api-plugin',
'client:benchmark',
'benchmarks',
2015-12-03 17:52:51 -05:00
'distribution:integ-test-zip',
'distribution:zip',
'distribution:tar',
'distribution:deb',
'distribution:rpm',
'distribution:tools:launchers',
'distribution:tools:plugin-cli',
'server',
'server:cli',
'test:framework',
'test:fixtures:example-fixture',
'test:fixtures:hdfs-fixture',
'test:fixtures:krb5kdc-fixture',
'test:fixtures:old-elasticsearch',
'test:logger-usage',
'libs:elasticsearch-core',
'libs:elasticsearch-nio',
'modules:aggs-matrix-stats',
'modules:analysis-common',
'modules:ingest-common',
'modules:lang-expression',
'modules:lang-mustache',
'modules:lang-painless',
'modules:mapper-extras',
'modules:parent-join',
'modules:percolator',
'modules:rank-eval',
'modules:reindex',
'modules:repository-url',
'modules:transport-netty4',
'modules:tribe',
'plugins:analysis-icu',
'plugins:analysis-kuromoji',
'plugins:analysis-phonetic',
'plugins:analysis-smartcn',
'plugins:analysis-stempel',
'plugins:analysis-ukrainian',
'plugins:discovery-azure-classic',
'plugins:discovery-ec2',
'plugins:discovery-file',
'plugins:discovery-gce',
'plugins:ingest-geoip',
'plugins:ingest-attachment',
'plugins:ingest-user-agent',
'plugins:mapper-murmur3',
'plugins:mapper-size',
'plugins:repository-azure',
'plugins:repository-gcs',
'plugins:repository-hdfs',
'plugins:repository-s3',
'plugins:jvm-example',
'plugins:store-smb',
'plugins:transport-nio',
'qa:auto-create-index',
Cross Cluster Search: make remote clusters optional (#27182) Today Cross Cluster Search requires at least one node in each remote cluster to be up once the cross cluster search is run. Otherwise the whole search request fails despite some of the data (either local and/or remote) is available. This happens when performing the _search/shards calls to find out which remote shards the query has to be executed on. This scenario is different from shard failures that may happen later on when the query is actually executed, in case e.g. remote shards are missing, which is not going to fail the whole request but rather yield partial results, and the _shards section in the response will indicate that. This commit introduces a boolean setting per cluster called search.remote.$cluster_alias.skip_if_disconnected, set to false by default, which allows to skip certain clusters if they are down when trying to reach them through a cross cluster search requests. By default all clusters are mandatory. Scroll requests support such setting too when they are first initiated (first search request with scroll parameter), but subsequent scroll rounds (_search/scroll endpoint) will fail if some of the remote clusters went down meanwhile. The search API response contains now a new _clusters section, similar to the _shards section, that gets returned whenever one or more clusters were disconnected and got skipped: "_clusters" : { "total" : 3, "successful" : 2, "skipped" : 1 } Such section won't be part of the response if no clusters have been skipped. The per cluster skip_unavailable setting value has also been added to the output of the remote/info API.
2017-11-21 05:41:47 -05:00
'qa:ccs-unavailable-clusters',
'qa:evil-tests',
'qa:full-cluster-restart',
'qa:integration-bwc',
'qa:mixed-cluster',
'qa:multi-cluster-search',
'qa:no-bootstrap-tests',
'qa:reindex-from-old',
2016-08-31 19:45:03 -04:00
'qa:rolling-upgrade',
'qa:smoke-test-client',
'qa:smoke-test-http',
'qa:smoke-test-ingest-with-all-dependencies',
'qa:smoke-test-ingest-disabled',
'qa:smoke-test-multinode',
'qa:smoke-test-rank-eval-with-mustache',
'qa:smoke-test-plugins',
'qa:smoke-test-reindex-with-all-modules',
'qa:smoke-test-tribe-node',
[test] Gradle-ify vagrant tests This gets the tar and tar_plugins tests working in gradle. It does so by adding a subproject, qa/vagrant, which adds the following tasks: Verification ------------ checkPackages - Check the packages against a representative sample of the linux distributions we have in our Vagrantfile checkPackagesAllDistros - Check the packages against all the linux distributions we have in our Vagrantfile Package Verification -------------------- checkCentos6 - Run packaging tests against centos-6 checkCentos7 - Run packaging tests against centos-7 checkDebian8 - Run packaging tests against debian-8 checkFedora22 - Run packaging tests against fedora-22 checkOel7 - Run packaging tests against oel-7 checkOpensuse13 - Run packaging tests against opensuse-13 checkSles12 - Run packaging tests against sles-12 checkUbuntu1204 - Run packaging tests against ubuntu-1204 checkUbuntu1404 - Run packaging tests against ubuntu-1404 checkUbuntu1504 - Run packaging tests against ubuntu-1504 Vagrant ------- smokeTestCentos6 - Smoke test the centos-6 VM smokeTestCentos7 - Smoke test the centos-7 VM smokeTestDebian8 - Smoke test the debian-8 VM smokeTestFedora22 - Smoke test the fedora-22 VM smokeTestOel7 - Smoke test the oel-7 VM smokeTestOpensuse13 - Smoke test the opensuse-13 VM smokeTestSles12 - Smoke test the sles-12 VM smokeTestUbuntu1204 - Smoke test the ubuntu-1204 VM smokeTestUbuntu1404 - Smoke test the ubuntu-1404 VM smokeTestUbuntu1504 - Smoke test the ubuntu-1504 VM vagrantHaltCentos6 - Shutdown the vagrant VM running centos-6 vagrantHaltCentos7 - Shutdown the vagrant VM running centos-7 vagrantHaltDebian8 - Shutdown the vagrant VM running debian-8 vagrantHaltFedora22 - Shutdown the vagrant VM running fedora-22 vagrantHaltOel7 - Shutdown the vagrant VM running oel-7 vagrantHaltOpensuse13 - Shutdown the vagrant VM running opensuse-13 vagrantHaltSles12 - Shutdown the vagrant VM running sles-12 vagrantHaltUbuntu1204 - Shutdown the vagrant VM running ubuntu-1204 vagrantHaltUbuntu1404 - Shutdown the vagrant VM running ubuntu-1404 vagrantHaltUbuntu1504 - Shutdown the vagrant VM running ubuntu-1504 vagrantSmokeTest - Smoke test some representative distros from the Vagrantfile vagrantSmokeTestAllDistros - Smoke test all distros from the Vagrantfile vagrantUpCentos6 - Startup a vagrant VM running centos-6 vagrantUpCentos7 - Startup a vagrant VM running centos-7 vagrantUpDebian8 - Startup a vagrant VM running debian-8 vagrantUpFedora22 - Startup a vagrant VM running fedora-22 vagrantUpOel7 - Startup a vagrant VM running oel-7 vagrantUpOpensuse13 - Startup a vagrant VM running opensuse-13 vagrantUpSles12 - Startup a vagrant VM running sles-12 vagrantUpUbuntu1204 - Startup a vagrant VM running ubuntu-1204 vagrantUpUbuntu1404 - Startup a vagrant VM running ubuntu-1404 vagrantUpUbuntu1504 - Startup a vagrant VM running ubuntu-1504 It does not make the "check" task depend on "checkPackages" so running the vagrant tests is still optional. They are slow and depend on vagrant and virtualbox. The Package Verification tasks are useful for testing individual distros. The Vagrant tasks are listed in `gradle tasks` primarily for discoverability.
2015-11-02 12:40:47 -05:00
'qa:vagrant',
'qa:verify-version-constants',
'qa:wildfly',
'qa:query-builder-bwc'
]
projects.add("libs")
File libsDir = new File(rootProject.projectDir, 'libs')
for (File libDir : new File(rootProject.projectDir, 'libs').listFiles()) {
if (libDir.isDirectory() == false) continue;
if (libDir.name.startsWith('build') || libDir.name.startsWith('.')) continue;
projects.add("libs:${libDir.name}".toString())
}
/* Create projects for building BWC snapshot distributions from the heads of other branches */
final List<String> branches = ['5.6', '6.0', '6.1', '6.x']
for (final String branch : branches) {
projects.add("distribution:bwc-snapshot-${branch}".toString())
}
boolean isEclipse = System.getProperty("eclipse.launcher") != null || gradle.startParameter.taskNames.contains('eclipse') || gradle.startParameter.taskNames.contains('cleanEclipse')
if (isEclipse) {
// eclipse cannot handle an intermediate dependency between main and test, so we must create separate projects
// for server-src and server-tests
projects << 'server-tests'
projects << 'libs:elasticsearch-nio-tests'
}
include projects.toArray(new String[0])
project(':build-tools').projectDir = new File(rootProject.projectDir, 'buildSrc')
/* The BWC snapshot projects share the same build directory and build file,
* but apply to different backwards compatibility branches. */
for (final String branch : branches) {
project(":distribution:bwc-snapshot-${branch}").projectDir = new File(rootProject.projectDir, 'distribution/bwc')
}
if (isEclipse) {
project(":server").projectDir = new File(rootProject.projectDir, 'server/src/main')
project(":server").buildFileName = 'eclipse-build.gradle'
project(":server-tests").projectDir = new File(rootProject.projectDir, 'server/src/test')
project(":server-tests").buildFileName = 'eclipse-build.gradle'
project(":libs:elasticsearch-nio").projectDir = new File(rootProject.projectDir, 'libs/elasticsearch-nio/src/main')
project(":libs:elasticsearch-nio").buildFileName = 'eclipse-build.gradle'
project(":libs:elasticsearch-nio-tests").projectDir = new File(rootProject.projectDir, 'libs/elasticsearch-nio/src/test')
project(":libs:elasticsearch-nio-tests").buildFileName = 'eclipse-build.gradle'
}
/**
* Iterates over sub directories, looking for build.gradle, and adds a project if found
* for that dir with the given path prefix. Note that this requires each level
* of the dir hierarchy to have a build.gradle. Otherwise we would have to iterate
* all files/directories in the source tree to find all projects.
*/
void addSubProjects(String path, File dir, List<String> projects, List<String> branches) {
if (dir.isDirectory() == false) return;
if (dir.name == 'buildSrc') return;
if (new File(dir, 'build.gradle').exists() == false) return;
final String projectName = "${path}:${dir.name}"
include projectName
if (dir.name == 'bwc-snapshot-dummy-projects') {
for (final String branch : branches) {
final String snapshotProjectName = "${projectName}:bwc-snapshot-${branch}"
projects.add(snapshotProjectName)
include snapshotProjectName
project("${snapshotProjectName}").projectDir = dir
}
// TODO do we want to assert that there's nothing else in the bwc directory?
} else {
Add the ability to bundle multiple plugins into a meta plugin (#28022) This commit adds the ability to package multiple plugins in a single zip. The zip file for a meta plugin must contains the following structure: |____elasticsearch/ | |____ <plugin1> <-- The plugin files for plugin1 (the content of the elastisearch directory) | |____ <plugin2> <-- The plugin files for plugin2 | |____ meta-plugin-descriptor.properties <-- example contents below The meta plugin properties descriptor is mandatory and must contain the following properties: description: simple summary of the meta plugin. name: the meta plugin name The installation process installs each plugin in a sub-folder inside the meta plugin directory. The example above would create the following structure in the plugins directory: |_____ plugins | |____ <name_of_the_meta_plugin> | | |____ meta-plugin-descriptor.properties | | |____ <plugin1> | | |____ <plugin2> If the sub plugins contain a config or a bin directory, they are copied in a sub folder inside the meta plugin config/bin directory. |_____ config | |____ <name_of_the_meta_plugin> | | |____ <plugin1> | | |____ <plugin2> |_____ bin | |____ <name_of_the_meta_plugin> | | |____ <plugin1> | | |____ <plugin2> The sub-plugins are loaded at startup like normal plugins with the same restrictions; they have a separate class loader and a sub-plugin cannot have the same name than another plugin (or a sub-plugin inside another meta plugin). It is also not possible to remove a sub-plugin inside a meta plugin, only full removal of the meta plugin is allowed. Closes #27316
2018-01-09 12:28:43 -05:00
if (path.isEmpty() || path.startsWith(':example-plugins')) {
project(projectName).projectDir = dir
}
for (File subdir : dir.listFiles()) {
addSubProjects(projectName, subdir, projects, branches)
}
}
}
Add the ability to bundle multiple plugins into a meta plugin (#28022) This commit adds the ability to package multiple plugins in a single zip. The zip file for a meta plugin must contains the following structure: |____elasticsearch/ | |____ <plugin1> <-- The plugin files for plugin1 (the content of the elastisearch directory) | |____ <plugin2> <-- The plugin files for plugin2 | |____ meta-plugin-descriptor.properties <-- example contents below The meta plugin properties descriptor is mandatory and must contain the following properties: description: simple summary of the meta plugin. name: the meta plugin name The installation process installs each plugin in a sub-folder inside the meta plugin directory. The example above would create the following structure in the plugins directory: |_____ plugins | |____ <name_of_the_meta_plugin> | | |____ meta-plugin-descriptor.properties | | |____ <plugin1> | | |____ <plugin2> If the sub plugins contain a config or a bin directory, they are copied in a sub folder inside the meta plugin config/bin directory. |_____ config | |____ <name_of_the_meta_plugin> | | |____ <plugin1> | | |____ <plugin2> |_____ bin | |____ <name_of_the_meta_plugin> | | |____ <plugin1> | | |____ <plugin2> The sub-plugins are loaded at startup like normal plugins with the same restrictions; they have a separate class loader and a sub-plugin cannot have the same name than another plugin (or a sub-plugin inside another meta plugin). It is also not possible to remove a sub-plugin inside a meta plugin, only full removal of the meta plugin is allowed. Closes #27316
2018-01-09 12:28:43 -05:00
// include example plugins
File examplePluginsDir = new File(rootProject.projectDir, 'plugins/examples')
for (File example : examplePluginsDir.listFiles()) {
if (example.isDirectory() == false) continue;
if (example.name.startsWith('build') || example.name.startsWith('.')) continue;
addSubProjects(':example-plugins', example, projects, [])
}
project(':example-plugins').projectDir = new File(rootProject.projectDir, 'plugins/examples')
// look for extra plugins for elasticsearch
File extraProjects = new File(rootProject.projectDir.parentFile, "${dirName}-extra")
if (extraProjects.exists()) {
for (File extraProjectDir : extraProjects.listFiles()) {
addSubProjects('', extraProjectDir, projects, branches)
}
}
include 'libs'