Commit Graph

74 Commits

Author SHA1 Message Date
Harold Wang a011c15bc8 Rename files under qa folder (#212)
* Rename directory elasticsearch to opensearch
Rename EvilElasticsearchCliTests EvilOpenSearchCliTests

Signed-off-by: Harold Wang <harowang@amazon.com>

* Rename org.elasticsearch to org.opensearch

Signed-off-by: Harold Wang <harowang@amazon.com>

* Rename waitForElasticsearch to waitForOpenSearch

Signed-off-by: Harold Wang <harowang@amazon.com>

* Rename OpensearchNode to OpenSearchNode

Signed-off-by: Harold Wang <harowang@amazon.com>

* Rename "elasticsearch.version" to "opensearch.version"

Signed-off-by: Harold Wang <harowang@amazon.com>

* Rename elasticsearchVersionString to opensearchVersionString

Signed-off-by: Harold Wang <harowang@amazon.com>

* Rename elasticsearch.yml to opensearch.yml

Signed-off-by: Harold Wang <harowang@amazon.com>

* Rename runElasticsearchTests to runOpenSearchTests

Signed-off-by: Harold Wang <harowang@amazon.com>

* Rename elasticsearchVersion to opensearchVersion

Signed-off-by: Harold Wang <harowang@amazon.com>

* Rename elasticsearch to opensearch in gradle files

Signed-off-by: Harold Wang <harowang@amazon.com>

* Rename ElasticsearchAssertions to OpenSearchAssertions

Signed-off-by: Harold Wang <harowang@amazon.com>

* Rename folder share/elasticsearch to share/opensearch

Signed-off-by: Harold Wang <harowang@amazon.com>

* Rename elasticsearch to opensearch

Signed-off-by: Harold Wang <harowang@amazon.com>

* Rename elasticsearch-service-x64 to opensearch-service-x64

Signed-off-by: Harold Wang <harowang@amazon.com>

* Rename elasticsearch-service.bat to opensearch-service.bat

Signed-off-by: Harold Wang <harowang@amazon.com>

* Rename Elasticsearch to Opensearch
Rename elasticsearch to opensearch

Signed-off-by: Harold Wang <harowang@amazon.com>

* Rename ELASTIC_PASSWORD_FILE to OPENSEARCH_PASSWORD_FILE

Signed-off-by: Harold Wang <harowang@amazon.com>

* Rename ELASTICSEARCH_PASSWORD to OPENSEARCH_PASSWORD
Rename elasticsearch to opensearch

Signed-off-by: Harold Wang <harowang@amazon.com>

* Rename elasticsearch to opensearch

Signed-off-by: Harold Wang <harowang@amazon.com>

* Rename elasticsearch.log to opensearch.log

Signed-off-by: Harold Wang <harowang@amazon.com>

* Rename es-repo to opensearch-repo

Signed-off-by: Harold Wang <harowang@amazon.com>

* Rename ESTestCase to OpenSearchTestCase

Signed-off-by: Harold Wang <harowang@amazon.com>

* Rename ESRestTestCase to OpenSearchRestTestCase

Signed-off-by: Harold Wang <harowang@amazon.com>

* Rename elasticsearch to opensearch
Rename "Starts ElasticSearch" to "Starts OpenSearch"

Signed-off-by: Harold Wang <harowang@amazon.com>

* Rename ESElasticsearchCliTestCase to BaseOpenSearchCliTestCase

Signed-off-by: Harold Wang <harowang@amazon.com>

* Rename "elasticsearch:test" to "opensearch:test"

Signed-off-by: Harold Wang <harowang@amazon.com>

* Rename test91ElasticsearchShardCliPackaging to test91OpenSearchShardCliPackaging

Signed-off-by: Harold Wang <harowang@amazon.com>

* Rename elasticsearch.toString to opensearch.toString

Signed-off-by: Harold Wang <harowang@amazon.com>

* Rename elasticsearch.pid to opensearch.pid

Signed-off-by: Harold Wang <harowang@amazon.com>

* Rename "Opensearch" to "OpenSearch"
Rename "elasticsearch" to "opensearch"

Signed-off-by: Harold Wang <harowang@amazon.com>

* Rename Elasticsearch to OpenSearch
Remove unecessary dot after opensearch.

Signed-off-by: Harold Wang <harowang@amazon.com>
2021-03-21 20:56:34 -05:00
Harold Wang 844e56abf3 [Rename] Rename files under "qa" folder (#194)
* [Rename] Rename qa folder

Signed-off-by: Harold Wang <harowang@amazon.com>

* Remove the dot in the end of "package org.opensearch."

Signed-off-by: Harold Wang <harowang@amazon.com>

* Add semicolon

Signed-off-by: Harold Wang <harowang@amazon.com>
2021-03-21 20:56:34 -05:00
Jake Landis 604c6dd528
7.x - Create plugin for yamlTest task (#56841) (#59090)
This commit creates a new Gradle plugin to provide a separate task name
and source set for running YAML based REST tests. The only project
converted to use the new plugin in this PR is distribution/archives/integ-test-zip.
For which the testing has been moved to :rest-api-spec since it makes the most
sense and it avoids a small but awkward change to the distribution plugin.

The remaining cases in modules, plugins, and x-pack will be handled in followups.

This plugin is distinctly different from the plugin introduced in #55896 since
the YAML REST tests are intended to be black box tests over HTTP. As such they
should not (by default) have access to the classpath for that which they are testing.

The YAML based REST tests will be moved to separate source sets (yamlRestTest).
The which source is the target for the test resources is dependent on if this
new plugin is applied. If it is not applied, it will default to the test source
set.

Further, this introduces a breaking change for plugin developers that
use the YAML testing framework. They will now need to either use the new source set
and matching task, or configure the rest resources to use the old "test" source set that
matches the old integTest task. (The former should be preferred).

As part of this change (which is also breaking for plugin developers) the
rest resources plugin has been removed from the build plugin and now requires
either explicit application or application via the new YAML REST test plugin.

Plugin developers should be able to fix the breaking changes to the YAML tests
by adding apply plugin: 'elasticsearch.yaml-rest-test' and moving the YAML tests
under a yamlRestTest folder (instead of test)
2020-07-06 14:16:26 -05:00
Rene Groeschke d952b101e6
Replace compile configuration usage with api (7.x backport) (#58721)
* Replace compile configuration usage with api (#58451)

- Use java-library instead of plugin to allow api configuration usage
- Remove explicit references to runtime configurations in dependency declarations
- Make test runtime classpath input for testing convention
  - required as java library will by default not have build jar file
  - jar file is now explicit input of the task and gradle will ensure its properly build

* Fix compile usages in 7.x branch
2020-06-30 15:57:41 +02:00
Rene Groeschke 01e9126588
Remove deprecated usage of testCompile configuration (#57921) (#58083)
* Remove usage of deprecated testCompile configuration
* Replace testCompile usage by testImplementation
* Make testImplementation non transitive by default (as we did for testCompile)
* Update CONTRIBUTING about using testImplementation for test dependencies
* Fail on testCompile configuration usage
2020-06-14 22:30:44 +02:00
Jason Tedor 5fcda57b37
Rename MetaData to Metadata in all of the places (#54519)
This is a simple naming change PR, to fix the fact that "metadata" is a
single English word, and for too long we have not followed general
naming conventions for it. We are also not consistent about it, for
example, METADATA instead of META_DATA if we were trying to be
consistent with MetaData (although METADATA is correct when considered
in the context of "metadata"). This was a simple find and replace across
the code base, only taking a few minutes to fix this naming issue
forever.
2020-03-31 17:24:38 -04:00
Rory Hunter c46a0e8708
Apply 2-space indent to all gradle scripts (#49071)
Backport of #48849. Update `.editorconfig` to make the Java settings the
default for all files, and then apply a 2-space indent to all `*.gradle`
files. Then reformat all the files.
2019-11-14 11:01:23 +00:00
Alpar Torok 23f739b513 Testclusters: Convert additional projects (#43625)
* Testclusters: Convert additional projects

Found some more that were not using testclusters from elasticsearch-ci/1

* Allow IOException too

* Make the client more resilient
2019-06-28 16:44:42 +03:00
Julie Tibshirani c2e9d13ebd
Default include_type_name to false in the yml test harness. (#38058)
This PR removes the temporary change we made to the yml test harness in #37285
to automatically set `include_type_name` to `true` in index creation requests
if it's not already specified. This is possible now that the vast majority of
index creation requests were updated to be typeless in #37611. A few additional
tests also needed updating here.

Additionally, this PR updates the test harness to set `include_type_name` to
`false` in index creation requests when communicating with 6.x nodes. This
mirrors the logic added in #37611 to allow for typeless document write requests
in test set-up code. With this update in place, we can remove many references
to `include_type_name: false` from the yml tests.
2019-02-01 11:44:13 -08:00
David Roberts 2f7776c8b7
Switch default time format for ingest from Joda to Java for v7 (#37934)
Date formats with and without the "8" prefix are now all treated
as Java time formats, so that ingest does the same as mappings
in this respect.
2019-01-30 16:26:28 +00:00
Alexander Reelsen 9e350d027e
Add BWC compatible processing to ingest date processors (#37407)
The ingest date processor is currently only able to parse joda formats.
However it is not using the existing elasticsearch classes but access
joda directly. This means that our existing BWC layer does not notify
the user about deprecated formats. This commit switches to use the
exising Elasticsearch Joda methods to acquire a date format, that
includes the BWC check and the ability to parse java 8 dates.

The date parsing in ingest has also another extra feature, that the
fallback year, when a date format without a year is used, is the current
year, and not 1970 like usual. This is currently not properly supported
in the DateFormatter class. As this is the only case for this feature
and java time can take care of this using the toZonedDateTime() method,
a workaround just for the joda time parser has been created, that can be
removed soon again from 7.0.
2019-01-25 13:50:19 +01:00
Alexander Reelsen daa2ec8a60
Switch mapping/aggregations over to java time (#36363)
This commit moves the aggregation and mapping code from joda time to
java time. This includes field mappers, root object mappers, aggregations with date
histograms, query builders and a lot of changes within tests.

The cut-over to java time is a requirement so that we can support nanoseconds
properly in a future field mapper.

Relates #27330
2019-01-23 10:40:05 +01:00
Alpar Torok 6344e9a3ce
Testing conventions: add support for checking base classes (#36650) 2019-01-08 13:39:03 +02:00
Jason Tedor e1717df0ac
Package ingest-geoip as a module (#36898)
This commit moves ingest-geoip from being a plugin to being a module
that is packaged with Elasticsearch distributions.
2018-12-22 07:21:49 -05:00
Mayya Sharipova 2f18325384
Deprecate types in update_by_query and delete_by_query (#36365)
Relates to #35190
2018-12-11 17:09:59 -05:00
Jim Ferenczi 18866c4c0b
Make hits.total an object in the search response (#35849)
This commit changes the format of the `hits.total` in the search response to be an object with
a `value` and a `relation`. The `value` indicates the number of hits that match the query and the
`relation` indicates whether the number is accurate (in which case the relation is equals to `eq`)
or a lower bound of the total (in which case it is equals to `gte`).
This change also adds a parameter called `rest_total_hits_as_int` that can be used in the
search APIs to opt out from this change (retrieve the total hits as a number in the rest response).
Note that currently all search responses are accurate (`track_total_hits: true`) or they don't contain
`hits.total` (`track_total_hits: true`). We'll add a way to get a lower bound of the total hits in a
follow up (to allow numbers to be passed to `track_total_hits`).

Relates #33028
2018-12-05 19:49:06 +01:00
Armin Braun 986c55b830
INGEST: Add Configuration Except. Data to Metdata (#32322)
* closes #27728
2018-08-15 19:02:19 +02:00
Christoph Büscher 1ea9f11b03
Change ScriptException status to 400 (bad request) (#30861)
Currently failures to compile a script usually lead to a ScriptException, which
inherits the 500 INTERNAL_SERVER_ERROR from ElasticsearchException if it does
not contain another root cause. Instead, this should be a 400 Bad Request error.
This PR changes this more generally for script compilation errors by changing 
ScriptException to return 400 (bad request) as status code.

Closes #12315
2018-05-30 14:00:07 +02:00
Adrien Grand 4918924fae
Remove legacy mapping code. (#29224)
Some features have been deprecated since `6.0` like the `_parent` field or the
ability to have multiple types per index. This allows to remove quite some
code, which in-turn will hopefully make it easier to proceed with the removal
of types.
2018-04-11 09:41:37 +02:00
Yu 7d8fb69d50 version set in ingest pipeline (#27573)
Add support version and version_type in ingest pipelines

Add support for setting document version and version type in set
processor of an ingest pipeline.
2018-02-21 09:34:51 +01:00
Martijn van Groningen dd0cb11d53
Fixed test to be up to date with the new database files. 2017-12-20 14:41:41 +01:00
kel 2e36f19051 Add support for parsing inline script (#23824) (#26846)
* Add support for parsing inline script (#23824)

* Fix test
2017-10-11 09:15:37 -07:00
Alexander Reelsen 80d0a32f8e ScriptService: Replace max compilation per minute setting with max compilation rate (#26399)
The current script service has a script compilation limit for a one
minute window. This is set to a small default value of 15. Instead of
increasing that default value, this commit introduces a new setting 
that allows to configure a rate per time unit, so that the script service can deal with bursts better.

The new setting is named `script.max_compilations_rate`,
requires a nonnegative number and a positive time value.

The default is `75/5m`, which is equivalent to the existing 15 per minute.
2017-09-01 10:15:27 +02:00
Tal Levy e04be73ad5 remove ingest.new_date_format (#25583) 2017-07-10 13:07:50 -07:00
Ryan Ernst a03b6c2fa5 Scripting: Change keys for inline/stored scripts to source/id (#25127)
This commit adds back "id" as the key within a script to specify a
stored script (which with file scripts now gone is no longer ambiguous).
It also adds "source" as a replacement for "code". This is in an attempt
to normalize how scripts are specified across both put stored scripts and script usages, including search template requests. This also deprecates the old inline/stored keys.
2017-06-09 08:29:25 -07:00
Tal Levy 340909582f remove Ingest's Internal Template Service (#25085)
Ingest was using it's own wrapper around TemplateScripts and the ScriptService.
This commit removes that abstraction
2017-06-08 15:24:03 -07:00
Ryan Ernst 8aaea51a0a Scripting: Move context definitions to instance type classes (#24883)
This is a simple refactoring to move the context definitions into the
type that they use. While we have multiple context names for the same
class at the moment, this will eventually become one ScriptContext per
instance type, so the pattern of a static member on the interface called
CONTEXT can be used. This commit also moves the consolidated list of
contexts provided by core ES into ScriptModule.
2017-05-25 12:18:45 -07:00
Ryan Ernst 52d504bb5f Scripting: Simplify ScriptContext (#24818)
As we work towards contexts implying the return type of compilation, we
first need ScriptContext to not be an enum. This commit removes the
Standard enum and Plugin subclass of ScriptContext.
2017-05-22 13:11:15 -07:00
Ryan Ernst 2de748859f Scripting: Remove "inline script enabled" on script engines (#24815)
ScriptEngine implementations have an overridable method to indicate they
are safe to use as inline scripts. Since groovy was removed fro 6.0,
there are no longer any implementations which used the default false
value. Furthermore, the value was not actually read anywhere. This
commit removes the method. The ScriptEngineRegistry was also no longer
necessary as it only was used to build a map from language to engine.
2017-05-20 12:01:25 -07:00
Jack Conradson 1196dfb6bb Remove Deprecated Script Settings (#24756)
Removes all fine-grained script settings replaced by scripts.types_allowed and scripts.contexts_allowed.
2017-05-18 13:32:46 -07:00
Ryan Ernst 463fe2f4d4 Scripting: Remove file scripts (#24627)
This commit removes file scripts, which were deprecated in 5.5.

closes #21798
2017-05-17 14:42:25 -07:00
Ryan Ernst 2a65bed243 Tests: Change rest test extension from .yaml to .yml (#24659)
This commit renames all rest test files to use the .yml extension
instead of .yaml. This way the extension used within all of
elasticsearch for yaml is consistent.
2017-05-16 17:24:35 -07:00
Ryan Ernst 9ca7d28552 Scripting: Remove "service" from ScriptEngine interface name (#24574)
This commit renames ScriptEngineService to ScriptEngine.  It is often
confusing because we have the ScriptService, and then
ScriptEngineService implementations, but the latter are not services as
we see in other places in elasticsearch.
2017-05-10 00:47:33 -07:00
Ryan Ernst ebd3e5f73f Scripting: Deprecate file script settings (#24555)
File scripts have 2 related settings: the path of file scripts, and
whether they can be dynamically reloaded. This commit deprecates those
settings.

relates #21798
2017-05-09 16:14:57 -07:00
Tal Levy 945b3cd4ca fix ingest pipeline timestamp timezone format check to be even more generic 2017-05-09 15:31:32 -07:00
Tal Levy d2416a3a40 modify ingest timestamp format check to remove timezone 2017-05-09 14:33:43 -07:00
Jason Tedor f222748506 Increase compilation limit in ingest tests
These tests are running into the script compilation limit. This commit
increases the limit for these tests.
2017-05-09 09:42:02 -04:00
Tal Levy 423b0f5e3d add option for _ingest.timestamp to use new ZonedDateTime (#24030)
Previously, Mustache would call `toString` on the `_ingest.timestamp`
field and return a date format that did not match Elasticsearch's
defaults for date-mapping parsing. The new ZonedDateTime class in Java 8
happens to do format itself in the same way ES is expecting.

This commit adds support for a feature flag that enables the usage of this new date format 
that has more native behavior.

Fixes #23168.

This new fix can be found in the form of a cluster setting called
`ingest.new_date_format`. By default, in 5.x, the existing behavior
will remain the same. One will set this property to `true` in order to
take advantage of this update for ingest-pipeline convenience.
2017-05-08 15:06:39 -07:00
Ryan Ernst 212f24aa27 Tests: Clean up rest test file handling (#21392)
This change simplifies how the rest test runner finds test files and
removes all leniency.  Previously multiple prefixes and suffixes would
be tried, and tests could exist inside or outside of the classpath,
although outside of the classpath never quite worked. Now only classpath
tests are supported, and only one resource prefix is supported,
`/rest-api-spec/tests`.

closes #20240
2017-04-18 15:07:08 -07:00
Nik Everett 2dcdaa1c9d Mustache: don't extend AbstractComponent (#23419)
Don't extend `AbstractComponent` in `MustacheScriptEngine` because
it doesn't buy anything.
2017-03-01 14:54:27 -05:00
Ryan Ernst 175bda64a0 Build: Rework integ test setup and shutdown to ensure stop runs when desired (#23304)
Gradle's finalizedBy on tasks only ensures one task runs after another,
but not immediately after. This is problematic for our integration tests
since it allows multiple project's integ test clusters to be
simultaneously. While this has not been a problem thus far (gradle 2.13
happened to keep the finalizedBy tasks close enough that no clusters
were running in parallel), with gradle 3.3 the task graph generation has
changed, and numerous clusters may be running simultaneously, causing
memory pressure, and thus generally slower tests, or even failure if the
system has a limited amount of memory (eg in a vagrant host).

This commit reworks how integ tests are configured. It adds an
`integTestCluster` extension to gradle which is equivalent to the current
`integTest.cluster` and moves the rest test runner task to
`integTestRunner`.  The `integTest` task is then just a dummy task,
which depends on the cluster runner task, as well as the cluster stop
task. This means running `integTest` in one project will both run the
rest tests, and shut down the cluster, before running `integTest` in
another project.
2017-02-22 12:43:15 -08:00
Jack Conradson 3d2626c4c6 Change Namespace for Stored Script to Only Use Id (#22206)
Currently, stored scripts use a namespace of (lang, id) to be put, get, deleted, and executed. This is not necessary since the lang is stored with the stored script. A user should only have to specify an id to use a stored script. This change makes that possible while keeping backwards compatibility with the previous namespace of (lang, id). Anywhere the previous namespace is used will log deprecation warnings.

The new behavior is the following:

When a user specifies a stored script, that script will be stored under both the new namespace and old namespace.

Take for example script 'A' with lang 'L0' and data 'D0'. If we add script 'A' to the empty set, the scripts map will be ["A" -- D0, "A#L0" -- D0]. If a script 'A' with lang 'L1' and data 'D1' is then added, the scripts map will be ["A" -- D1, "A#L1" -- D1, "A#L0" -- D0].

When a user deletes a stored script, that script will be deleted from both the new namespace (if it exists) and the old namespace.

Take for example a scripts map with {"A" -- D1, "A#L1" -- D1, "A#L0" -- D0}. If a script is removed specified by an id 'A' and lang null then the scripts map will be {"A#L0" -- D0}. To remove the final script, the deprecated namespace must be used, so an id 'A' and lang 'L0' would need to be specified.

When a user gets/executes a stored script, if the new namespace is used then the script will be retrieved/executed using only 'id', and if the old namespace is used then the script will be retrieved/executed using 'id' and 'lang'
2017-01-31 13:27:02 -08:00
Nik Everett 232af512f4 Switch from standalone-test to standalone-rest-test
standalone-rest-test doesn't configure unit tests and for these
integ test only tests, that is what we want.
2017-01-05 10:55:47 +01:00
Nik Everett 812f63e5ef Require either BuildPlugin or StandaloneTestBasePlugin to use RestTestPlugin
It used to be that RestTestPlugin "came with" StandaloneTestBasePlugin
but we'd like to use it with BuildPlugin for the high level rest client.
2017-01-05 10:55:47 +01:00
Nik Everett f5f2149ff2 Remove much ceremony from parsing client yaml test suites (#22311)
* Remove a checked exception, replacing it with `ParsingException`.
* Remove all Parser classes for the yaml sections, replacing them with static methods.
* Remove `ClientYamlTestFragmentParser`. Isn't used any more.
* Remove `ClientYamlTestSuiteParseContext`, replacing it with some static utility methods.

I did not rewrite the parsers using `ObjectParser` because I don't think it is worth it right now.
2016-12-22 11:00:34 -05:00
Tal Levy eaf82a6e7e compile ScriptProcessor inline scripts when creating ingest pipelines (#21858)
Inline scripts defined in Ingest Pipelines are now compiled at creation time to preemptively catch errors on initialization of the pipeline.

Fixes #21842.
2016-12-14 17:26:51 -08:00
Adrien Grand 6231009a8f Remove 2.x backward compatibility of mappings. (#21670)
For the record, I also had to remove the geo-hash cell and geo-distance range
queries to make the code compile. These queries already throw an exception in
all cases with 5.x indices, so that does not hurt any more.

I also had to rename all 2.x bwc indices from `index-${version}` to
`unsupported-${version}` to make `OldIndexBackwardCompatibilityIT`
happy.
2016-11-30 13:34:46 +01:00
Ryan Ernst 7a2c984bcc Test: Remove multi process support from rest test runner (#21391)
At one point in the past when moving out the rest tests from core to
their own subproject, we had multiple test classes which evenly split up
the tests to run. However, we simplified this and went back to a single
test runner to have better reproduceability in tests. This change
removes the remnants of that multiplexing support.
2016-11-07 15:07:34 -08:00
Tal Levy 38c650f376 make painless the default scripting language for ScriptProcessor (#20981)
- fixes a bug in the docs that mentions `lang` as optional
- now `lang` defaults to "painless"
2016-10-18 16:22:01 -07:00
Igor Motov b36fbc4452 Add support for parameters to the script ingest processor
The script processor should support `params` to be consistent with all other script consumers.
2016-08-24 16:49:48 -04:00