OpenSearch/x-pack/plugin/sql/qa/build.gradle

129 lines
3.9 KiB
Groovy
Raw Normal View History

description = 'Integration tests for SQL'
apply plugin: 'elasticsearch.build'
archivesBaseName = 'qa-sql'
group = "org.elasticsearch.x-pack.qa.sql"
dependencies {
compile project(":test:framework")
SQL: Replace the cli fixture with in-process testing (elastic/x-pack-elasticsearch#3889) I'm really really sad to be removing the cli-fixture but I've had trouble with it leaking recently it is pretty slow. Beyond that, we'd prefer that our test fixture only fixture things that are external depndencies. So, yeah, I'm removing it. So we get faster tests and no chance of leaking processes. We lose some "realness" in the tests. Instead of interacting with the CLI like a real user we embed it in the test process. That means we don't test the forking, we don't test the executable jar, and we don't test the jLine console detection stuff. On the other hand we were kind of forcing the jLine console detection stuff in a funky way with the fixture anyway. And we test the executable jar in the packaging tests. And that'll have to do. I haven't renamed `RemoteCli` because it'd bloat this commit with mechanical changes that'd make it hard to review. I'll rename it in a followup commit. This also updates jLine so we can disable blinking to matching parentheses during testing. I have no clue why, but this wasn't happening when we used the fixture. The trouble with the blinking is that it is based on *time* so it slows things down. Worse, it works inconsistently! Sometimes it spits out sensible ascii codes and sometimes it, well, spits out weird garbage. When you use it in person it works fine though. So we keep it on when not testing. Cleans up some redundancy in when testing CLI errors. Less copy and paste good. I was tempted to disable the xterm emulation entirely while working on this because upgrading jLine changed a few things and it was a real pain to update. But If we turned that off then we'd have *nothing* testing the colors and such. That'd be a shame because we use color in the output to commicate stuff. I like it so I don't want to break it. While I was there, I replaces the cli connector's `PrintWriter` with a `BufferedWriter`. The `PrintWriter` was kind of a trap because `println` would fail to work properly on windows because we force the terminal into xterm mode and it doesn't know what to do with windows line endings. Windows..... Additionally I fixed a race condition between disabling echo when reading passwords and fast writers. We were disabling the echo shortly after sending the prompt. A fast enough writer could send us text before the echo disable kicked in. Now I delegate to `LineReader#readLine` with a special echo mask that disables echo. This is both easier to test and doesn't seem to have the race condition. This race condition was failing the tests because they are so much faster now. Yay! Original commit: elastic/x-pack-elasticsearch@d0ec0273964630a3a113ab27009fdff6a182ecb2
2018-02-27 12:24:16 -05:00
// JDBC testing dependencies
compile project(path: xpackModule('sql:jdbc'))
compile "net.sourceforge.csvjdbc:csvjdbc:${csvjdbcVersion}"
// CLI testing dependencies
compile project(path: xpackModule('sql:sql-cli'))
SQL: Add initial geo support (#42031) (#42135) Adds an initial limited implementations of geo features to SQL. This implementation is based on the [OpenGIS® Implementation Standard for Geographic information - Simple feature access](http://www.opengeospatial.org/standards/sfs), which is the current standard for GIS system implementation. This effort is concentrate on SQL option AKA ISO 19125-2. Queries that are supported as a result of this initial implementation Metadata commands - `DESCRIBE table` - returns the correct column types `GEOMETRY` for geo shapes and geo points. - `SHOW FUNCTIONS` - returns a list that includes supported `ST_` functions - `SYS TYPES` and `SYS COLUMNS` display correct types `GEO_SHAPE` and `GEO_POINT` for geo shapes and geo points accordingly. Returning geoshapes and geopoints from elasticsearch - `SELECT geom FROM table` - returns the geoshapes and geo_points as libs/geo objects in JDBC or as WKT strings in console. - `SELECT ST_AsWKT(geom) FROM table;` and `SELECT ST_AsText(geom) FROM table;`- returns the geoshapes ang geopoints in their WKT representation; Using geopoints to elasticsearch - The following functions will be supported for geopoints in queries, sorting and aggregations: `ST_GeomFromText`, `ST_X`, `ST_Y`, `ST_Z`, `ST_GeometryType`, and `ST_Distance`. In most cases when used in queries, sorting and aggregations, these function are translated into script. These functions can be used in the SELECT clause for both geopoints and geoshapes. - `SELECT * FROM table WHERE ST_Distance(ST_GeomFromText(POINT(1 2), point) < 10;` - returns all records for which `point` is located within 10m from the `POINT(1 2)`. In this case the WHERE clause is translated into a range query. Limitations: Geoshapes cannot be used in queries, sorting and aggregations as part of this initial effort. In order to fully take advantage of geoshapes we would need to have access to geoshape doc values, which is coming in #37206. `ST_Z` cannot be used on geopoints in queries, sorting and aggregations since we don't store altitude in geo_point doc values. Relates to #29872 Backport of #42031
2019-05-14 19:57:12 -04:00
// H2GIS testing dependencies
compile("org.orbisgis:h2gis:${h2gisVersion}") {
SQL: Add initial geo support (#42031) (#42135) Adds an initial limited implementations of geo features to SQL. This implementation is based on the [OpenGIS® Implementation Standard for Geographic information - Simple feature access](http://www.opengeospatial.org/standards/sfs), which is the current standard for GIS system implementation. This effort is concentrate on SQL option AKA ISO 19125-2. Queries that are supported as a result of this initial implementation Metadata commands - `DESCRIBE table` - returns the correct column types `GEOMETRY` for geo shapes and geo points. - `SHOW FUNCTIONS` - returns a list that includes supported `ST_` functions - `SYS TYPES` and `SYS COLUMNS` display correct types `GEO_SHAPE` and `GEO_POINT` for geo shapes and geo points accordingly. Returning geoshapes and geopoints from elasticsearch - `SELECT geom FROM table` - returns the geoshapes and geo_points as libs/geo objects in JDBC or as WKT strings in console. - `SELECT ST_AsWKT(geom) FROM table;` and `SELECT ST_AsText(geom) FROM table;`- returns the geoshapes ang geopoints in their WKT representation; Using geopoints to elasticsearch - The following functions will be supported for geopoints in queries, sorting and aggregations: `ST_GeomFromText`, `ST_X`, `ST_Y`, `ST_Z`, `ST_GeometryType`, and `ST_Distance`. In most cases when used in queries, sorting and aggregations, these function are translated into script. These functions can be used in the SELECT clause for both geopoints and geoshapes. - `SELECT * FROM table WHERE ST_Distance(ST_GeomFromText(POINT(1 2), point) < 10;` - returns all records for which `point` is located within 10m from the `POINT(1 2)`. In this case the WHERE clause is translated into a range query. Limitations: Geoshapes cannot be used in queries, sorting and aggregations as part of this initial effort. In order to fully take advantage of geoshapes we would need to have access to geoshape doc values, which is coming in #37206. `ST_Z` cannot be used on geopoints in queries, sorting and aggregations since we don't store altitude in geo_point doc values. Relates to #29872 Backport of #42031
2019-05-14 19:57:12 -04:00
exclude group: "org.locationtech.jts"
}
// select just the parts of JLine that are needed
compile("org.jline:jline-terminal-jna:${jlineVersion}") {
exclude group: "net.java.dev.jna"
}
compile "org.jline:jline-terminal:${jlineVersion}"
compile "org.jline:jline-reader:${jlineVersion}"
compile "org.jline:jline-style:${jlineVersion}"
testRuntime "org.elasticsearch:jna:${versions.jna}"
}
/* disable unit tests because these are all integration tests used
* other qa projects. */
test.enabled = false
dependencyLicenses.enabled = false
Fix unknown licenses (#31223) The goal of this commit is to address unknown licenses when producing the dependencies info report. We have two different checks that we run on licenses. The first check is whether or not we have stashed a copy of the license text for a dependency in the repository. The second is to map every dependency to a license type (e.g., BSD 3-clause). The problem here is that the way we were handling licenses in the second check differs from how we handle licenses in the first check. The first check works by finding a license file with the name of the artifact followed by the text -LICENSE.txt. Yet in some cases we allow mapping an artifact name to another name used to check for the license (e.g., we map lucene-.* to lucene, and opensaml-.* to shibboleth. The second check understood the first way of looking for a license file but not the second way. So in this commit we teach the second check about the mappings from artifact names to license names. We do this by copying the configuration from the dependencyLicenses task to the dependenciesInfo task and then reusing the code from the first check in the second check. There were some other challenges here though. For example, dependenciesInfo was checking too many dependencies. For now, we should only be checking direct dependencies and leaving transitive dependencies from another org.elasticsearch artifact to that artifact (we want to do this differently in a follow-up). We also want to disable dependenciesInfo for projects that we do not publish, users only care about licenses they might be exposed to if they use our assembled products. With all of the changes in this commit we have eliminated all unknown licenses. A follow-up will enforce that when we add a new dependency it does not get mapped to unknown, these will be forbidden in the future. Therefore, with this change and earlier changes are left having no unknown licenses and two custom licenses; custom here means it does not map to an SPDX license type. Those two licenses are xz and ldapsdk. A future change will not allow additional custom licenses unless they are explicitly whitelisted. This ensures that if a new dependency is added it is mapped to an SPDX license or mapped to custom because it does not have an SPDX license.
2018-06-09 07:28:41 -04:00
dependenciesInfo.enabled = false
// the main files are actually test files, so use the appropriate forbidden api sigs
tasks.named('forbiddenApisMain').configure {
replaceSignatureFiles 'es-all-signatures', 'es-test-signatures'
}
SQL: Add initial geo support (#42031) (#42135) Adds an initial limited implementations of geo features to SQL. This implementation is based on the [OpenGIS® Implementation Standard for Geographic information - Simple feature access](http://www.opengeospatial.org/standards/sfs), which is the current standard for GIS system implementation. This effort is concentrate on SQL option AKA ISO 19125-2. Queries that are supported as a result of this initial implementation Metadata commands - `DESCRIBE table` - returns the correct column types `GEOMETRY` for geo shapes and geo points. - `SHOW FUNCTIONS` - returns a list that includes supported `ST_` functions - `SYS TYPES` and `SYS COLUMNS` display correct types `GEO_SHAPE` and `GEO_POINT` for geo shapes and geo points accordingly. Returning geoshapes and geopoints from elasticsearch - `SELECT geom FROM table` - returns the geoshapes and geo_points as libs/geo objects in JDBC or as WKT strings in console. - `SELECT ST_AsWKT(geom) FROM table;` and `SELECT ST_AsText(geom) FROM table;`- returns the geoshapes ang geopoints in their WKT representation; Using geopoints to elasticsearch - The following functions will be supported for geopoints in queries, sorting and aggregations: `ST_GeomFromText`, `ST_X`, `ST_Y`, `ST_Z`, `ST_GeometryType`, and `ST_Distance`. In most cases when used in queries, sorting and aggregations, these function are translated into script. These functions can be used in the SELECT clause for both geopoints and geoshapes. - `SELECT * FROM table WHERE ST_Distance(ST_GeomFromText(POINT(1 2), point) < 10;` - returns all records for which `point` is located within 10m from the `POINT(1 2)`. In this case the WHERE clause is translated into a range query. Limitations: Geoshapes cannot be used in queries, sorting and aggregations as part of this initial effort. In order to fully take advantage of geoshapes we would need to have access to geoshape doc values, which is coming in #37206. `ST_Z` cannot be used on geopoints in queries, sorting and aggregations since we don't store altitude in geo_point doc values. Relates to #29872 Backport of #42031
2019-05-14 19:57:12 -04:00
// just a test fixture: we aren't using this jars in releases and H2GIS requires disabling a lot of checks
thirdPartyAudit.enabled = false
subprojects {
if (subprojects.isEmpty()) {
// leaf project
apply plugin: 'elasticsearch.standalone-rest-test'
} else {
apply plugin: 'elasticsearch.build'
}
configurations.testRuntimeClasspath {
resolutionStrategy.force "org.slf4j:slf4j-api:1.7.25"
}
configurations.testRuntime {
// This is also required to make resolveAllDependencies work
resolutionStrategy.force "org.slf4j:slf4j-api:1.7.25"
}
dependencies {
/* Since we're a standalone rest test we actually get transitive
* dependencies but we don't really want them because they cause
* all kinds of trouble with the jar hell checks. So we suppress
* them explicitly for non-es projects. */
testCompile(xpackProject('plugin:sql:qa')) {
transitive = false
}
testCompile project(":test:framework")
// JDBC testing dependencies
testRuntime "net.sourceforge.csvjdbc:csvjdbc:${csvjdbcVersion}"
testRuntime "com.h2database:h2:${h2Version}"
SQL: Add initial geo support (#42031) (#42135) Adds an initial limited implementations of geo features to SQL. This implementation is based on the [OpenGIS® Implementation Standard for Geographic information - Simple feature access](http://www.opengeospatial.org/standards/sfs), which is the current standard for GIS system implementation. This effort is concentrate on SQL option AKA ISO 19125-2. Queries that are supported as a result of this initial implementation Metadata commands - `DESCRIBE table` - returns the correct column types `GEOMETRY` for geo shapes and geo points. - `SHOW FUNCTIONS` - returns a list that includes supported `ST_` functions - `SYS TYPES` and `SYS COLUMNS` display correct types `GEO_SHAPE` and `GEO_POINT` for geo shapes and geo points accordingly. Returning geoshapes and geopoints from elasticsearch - `SELECT geom FROM table` - returns the geoshapes and geo_points as libs/geo objects in JDBC or as WKT strings in console. - `SELECT ST_AsWKT(geom) FROM table;` and `SELECT ST_AsText(geom) FROM table;`- returns the geoshapes ang geopoints in their WKT representation; Using geopoints to elasticsearch - The following functions will be supported for geopoints in queries, sorting and aggregations: `ST_GeomFromText`, `ST_X`, `ST_Y`, `ST_Z`, `ST_GeometryType`, and `ST_Distance`. In most cases when used in queries, sorting and aggregations, these function are translated into script. These functions can be used in the SELECT clause for both geopoints and geoshapes. - `SELECT * FROM table WHERE ST_Distance(ST_GeomFromText(POINT(1 2), point) < 10;` - returns all records for which `point` is located within 10m from the `POINT(1 2)`. In this case the WHERE clause is translated into a range query. Limitations: Geoshapes cannot be used in queries, sorting and aggregations as part of this initial effort. In order to fully take advantage of geoshapes we would need to have access to geoshape doc values, which is coming in #37206. `ST_Z` cannot be used on geopoints in queries, sorting and aggregations since we don't store altitude in geo_point doc values. Relates to #29872 Backport of #42031
2019-05-14 19:57:12 -04:00
// H2GIS testing dependencies
testRuntime("org.orbisgis:h2gis:${h2gisVersion}") {
exclude group: "org.locationtech.jts"
exclude group: "com.fasterxml.jackson.core"
SQL: Add initial geo support (#42031) (#42135) Adds an initial limited implementations of geo features to SQL. This implementation is based on the [OpenGIS® Implementation Standard for Geographic information - Simple feature access](http://www.opengeospatial.org/standards/sfs), which is the current standard for GIS system implementation. This effort is concentrate on SQL option AKA ISO 19125-2. Queries that are supported as a result of this initial implementation Metadata commands - `DESCRIBE table` - returns the correct column types `GEOMETRY` for geo shapes and geo points. - `SHOW FUNCTIONS` - returns a list that includes supported `ST_` functions - `SYS TYPES` and `SYS COLUMNS` display correct types `GEO_SHAPE` and `GEO_POINT` for geo shapes and geo points accordingly. Returning geoshapes and geopoints from elasticsearch - `SELECT geom FROM table` - returns the geoshapes and geo_points as libs/geo objects in JDBC or as WKT strings in console. - `SELECT ST_AsWKT(geom) FROM table;` and `SELECT ST_AsText(geom) FROM table;`- returns the geoshapes ang geopoints in their WKT representation; Using geopoints to elasticsearch - The following functions will be supported for geopoints in queries, sorting and aggregations: `ST_GeomFromText`, `ST_X`, `ST_Y`, `ST_Z`, `ST_GeometryType`, and `ST_Distance`. In most cases when used in queries, sorting and aggregations, these function are translated into script. These functions can be used in the SELECT clause for both geopoints and geoshapes. - `SELECT * FROM table WHERE ST_Distance(ST_GeomFromText(POINT(1 2), point) < 10;` - returns all records for which `point` is located within 10m from the `POINT(1 2)`. In this case the WHERE clause is translated into a range query. Limitations: Geoshapes cannot be used in queries, sorting and aggregations as part of this initial effort. In order to fully take advantage of geoshapes we would need to have access to geoshape doc values, which is coming in #37206. `ST_Z` cannot be used on geopoints in queries, sorting and aggregations since we don't store altitude in geo_point doc values. Relates to #29872 Backport of #42031
2019-05-14 19:57:12 -04:00
}
testRuntime project(path: xpackModule('sql:jdbc'))
testRuntime xpackProject('plugin:sql:sql-client')
SQL: Replace the cli fixture with in-process testing (elastic/x-pack-elasticsearch#3889) I'm really really sad to be removing the cli-fixture but I've had trouble with it leaking recently it is pretty slow. Beyond that, we'd prefer that our test fixture only fixture things that are external depndencies. So, yeah, I'm removing it. So we get faster tests and no chance of leaking processes. We lose some "realness" in the tests. Instead of interacting with the CLI like a real user we embed it in the test process. That means we don't test the forking, we don't test the executable jar, and we don't test the jLine console detection stuff. On the other hand we were kind of forcing the jLine console detection stuff in a funky way with the fixture anyway. And we test the executable jar in the packaging tests. And that'll have to do. I haven't renamed `RemoteCli` because it'd bloat this commit with mechanical changes that'd make it hard to review. I'll rename it in a followup commit. This also updates jLine so we can disable blinking to matching parentheses during testing. I have no clue why, but this wasn't happening when we used the fixture. The trouble with the blinking is that it is based on *time* so it slows things down. Worse, it works inconsistently! Sometimes it spits out sensible ascii codes and sometimes it, well, spits out weird garbage. When you use it in person it works fine though. So we keep it on when not testing. Cleans up some redundancy in when testing CLI errors. Less copy and paste good. I was tempted to disable the xterm emulation entirely while working on this because upgrading jLine changed a few things and it was a real pain to update. But If we turned that off then we'd have *nothing* testing the colors and such. That'd be a shame because we use color in the output to commicate stuff. I like it so I don't want to break it. While I was there, I replaces the cli connector's `PrintWriter` with a `BufferedWriter`. The `PrintWriter` was kind of a trap because `println` would fail to work properly on windows because we force the terminal into xterm mode and it doesn't know what to do with windows line endings. Windows..... Additionally I fixed a race condition between disabling echo when reading passwords and fast writers. We were disabling the echo shortly after sending the prompt. A fast enough writer could send us text before the echo disable kicked in. Now I delegate to `LineReader#readLine` with a special echo mask that disables echo. This is both easier to test and doesn't seem to have the race condition. This race condition was failing the tests because they are so much faster now. Yay! Original commit: elastic/x-pack-elasticsearch@d0ec0273964630a3a113ab27009fdff6a182ecb2
2018-02-27 12:24:16 -05:00
// TODO check if needed
testRuntime("org.antlr:antlr4-runtime:${antlrVersion}") {
transitive = false
}
// CLI testing dependencies
testRuntime project(path: xpackModule('sql:sql-cli'))
testRuntime(xpackProject('plugin:sql:sql-action')) {
transitive = false
}
testRuntime("org.jline:jline-terminal-jna:${jlineVersion}") {
exclude group: "net.java.dev.jna"
}
testRuntime "org.jline:jline-terminal:${jlineVersion}"
testRuntime "org.jline:jline-reader:${jlineVersion}"
testRuntime "org.jline:jline-style:${jlineVersion}"
testRuntime "org.elasticsearch:jna:${versions.jna}"
// spatial dependency
testRuntime project(path: xpackModule('spatial'))
}
if (project.name != 'security') {
// The security project just configures its subprojects
apply plugin: 'elasticsearch.testclusters'
apply plugin: 'elasticsearch.rest-test'
testClusters.integTest {
testDistribution = 'DEFAULT'
setting 'xpack.ml.enabled', 'false'
setting 'xpack.watcher.enabled', 'false'
}
task runqa {
doFirst {
println "Run with `-Dtestclusters.inspect.failure=true integTest` to leave the cluster running after failure"
}
}
}
}