Commit Graph

34 Commits

Author SHA1 Message Date
dependabot[bot] 6d52ac7334
Bump gson from 2.8.9 to 2.9.0 in /test/fixtures/hdfs-fixture (#2190)
Bumps [gson](https://github.com/google/gson) from 2.8.9 to 2.9.0.
- [Release notes](https://github.com/google/gson/releases)
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md)
- [Commits](https://github.com/google/gson/compare/gson-parent-2.8.9...gson-parent-2.9.0)

---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-21 11:48:57 -08:00
dependabot[bot] 8d69c77081
Bump json-smart from 2.4.7 to 2.4.8 in /test/fixtures/hdfs-fixture (#2136)
Bumps [json-smart](https://github.com/netplex/json-smart-v2) from 2.4.7 to 2.4.8.
- [Release notes](https://github.com/netplex/json-smart-v2/releases)
- [Commits](https://github.com/netplex/json-smart-v2/commits/2.4.8)

---
updated-dependencies:
- dependency-name: net.minidev:json-smart
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-17 14:31:45 -08:00
Tianli Feng db23f72a2a
Update FIPS API libraries of Bouncy Castle (#1853)
* Update bc-fips to 1.0.2.1

Signed-off-by: Tianli Feng <ftl94@live.com>

* Update bcpg-fips to 1.0.5.1

Signed-off-by: Tianli Feng <ftl94@live.com>

* Update bctls-fips to 1.0.12.2

Signed-off-by: Tianli Feng <ftl94@live.com>

* Use the unified bouncycastle version for bcpkix-jdk15on in HDFS testing fixture

Signed-off-by: Tianli Feng <ftl94@live.com>
2022-01-05 16:59:55 -08:00
Vacha fd87f3a2d7
Upgrading commons-codec in hdfs-fixture and cleaning up dependencies in repository-hdfs (#1603)
Signed-off-by: Vacha <vachshah@amazon.com>
2021-11-23 13:32:33 -05:00
Vacha c6dd484ce3
Upgrading gson to 2.8.9 (#1541)
Signed-off-by: Vacha <vachshah@amazon.com>
2021-11-15 14:10:29 -05:00
Vacha af6ae752b4
Upgrading dependencies (#1491)
Signed-off-by: Vacha <vachshah@amazon.com>
2021-11-02 13:47:54 -07:00
Vacha 389b7dfa18
Upgrading dependencies in hdfs plugin (#1466)
Signed-off-by: Vacha <vachshah@amazon.com>
2021-11-01 12:58:07 -07:00
Vacha d151082832
Upgrade hadoop dependencies for hdfs plugin (#1335)
* Upgrade hadoop dependencies for hdfs plugin

Signed-off-by: Vacha <vachshah@amazon.com>

* Fixing gradle check failures

Signed-off-by: Vacha <vachshah@amazon.com>

* Upgrading htrace-core4 to 4.1.0

Signed-off-by: Vacha <vachshah@amazon.com>
2021-10-14 14:43:49 -04:00
Himanshu Setia 6f893ed1cd
Merging javadoc feature branch changes to main (#715)
* Adds a gradle plugin to validate missing javadocs

Use `./gradlew missingJavadoc` to validate missing javadocs.
Currently this task fails because several modules are missing
appropriate javadocs. Once added, this should pass.
Also, precommit PomValidation check currently fails with missing Javadoc
plugin, that needs to be fixed -
https://github.com/opensearch-project/OpenSearch/issues/449
Thus keeping this in a separate feature branch.

Signed-off-by: Himanshu Setia <setiah@amazon.com>

* Fix Javadoc errors in module `client/rest` (#685)

* Fix Javadoc errors in client/rest module

Signed-off-by: Gregor Zurowski <gregor@zurowski.org>

* Add package info file in client/rest module

Signed-off-by: Gregor Zurowski <gregor@zurowski.org>

* Fix typos

Signed-off-by: Gregor Zurowski <gregor@zurowski.org>

* Add exception documentation to Javadoc

Signed-off-by: Gregor Zurowski <gregor@zurowski.org>

* Fixes precommit task configuration failures due to newly added missin… (#707)

* Fixes precommit task configuration failures due to newly added missingJavadoc task

Signed-off-by: Himanshu Setia <setiah@amazon.com>

* Fixes javadoc task errors due to PR#685

Signed-off-by: Himanshu Setia <setiah@amazon.com>

* Updated CONTRIBUTING.md for info on javadocs

Signed-off-by: Himanshu Setia <setiah@amazon.com>

* Correcting licenses and naming

Signed-off-by: Himanshu Setia <setiah@amazon.com>

* Correcting version info

Signed-off-by: Himanshu Setia <setiah@amazon.com>

Co-authored-by: Gregor Zurowski <gregor@zurowski.org>
2021-05-18 13:21:41 -07:00
Rabi Panda 943c778a7f
[CVE-2018-11765] Upgrade hadoop dependencies for hdfs plugin (#654)
Hadoop 2.8.5 has been reported to have CVEs (https://bugzilla.redhat.com/show_bug.cgi?id=1883549). We need to upgrade this to 2.10.1. This also updates the hadoop-minicluster version to 2.10.1 as well. This upgrade also brings in two additional dependencies, woodstox-core and stax2-api that are added along with the sha1s, licenses and notices.

Also upgrade guava to the latest as per the CVE https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-8908

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-05-13 14:56:47 -07:00
Rabi Panda 67460c7804
Update hadoop-minicluster version for test fixture. (#645)
The `hadoop-minicluster:2.8.5` which is used for integration tests, has dependencies which have been flagged for having security vulnerabilities. Update this to the latest version `3.3.0`

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-05-04 10:48:43 -07:00
Nick Knize 9168f1fb43
[License] Add SPDX and OpenSearch Modification license header (#509)
This commit adds the SPDX Apache-2.0 license header along with an additional
copyright header for all modifications.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-04-09 14:28:18 -05:00
Himanshu Setia b5e00e1e58 Refactors test/fixtures as per OpenSearch naming (#370)
Signed-off-by: Himanshu Setia <setiah@amazon.com>
2021-03-21 20:56:34 -05:00
Ryan Ernst 6d3b691048
Add snapshot only test modules (#61954)
This commit adds external test modules. These are modules meant for
external systems to test edge cases in elasticsearch, but only within
snapshots. They are not meant to be used in production, so protections
are also added from their accidental inclusion in release builds.

Note that this commit does not actually add any new modules, it only
adds the infrastructure for the new modules, under
`test/external-modules`.
2020-09-04 16:35:18 -07:00
Rene Groeschke d952b101e6
Replace compile configuration usage with api (7.x backport) (#58721)
* Replace compile configuration usage with api (#58451)

- Use java-library instead of plugin to allow api configuration usage
- Remove explicit references to runtime configurations in dependency declarations
- Make test runtime classpath input for testing convention
  - required as java library will by default not have build jar file
  - jar file is now explicit input of the task and gradle will ensure its properly build

* Fix compile usages in 7.x branch
2020-06-30 15:57:41 +02:00
James Baiera b84c74cf70
Update the HDFS version used by HDFS Repo (#53693) (#54125) 2020-03-25 14:01:29 -04:00
Yogesh Gaikwad 91c342a888
fix and enable repository-hdfs secure tests (#44044) (#44199)
Due to recent changes are done for converting `repository-hdfs` to test
clusters (#41252), the `integTestSecure*` tasks did not depend on
`secureHdfsFixture` which when running would fail as the fixture
would not be available. This commit adds the dependency of the fixture
to the task.

The `secureHdfsFixture` is a `AntFixture` which is spawned a process.
Internally it waits for 30 seconds for the resources to be made available.
For my local machine, it took almost 45 seconds to be available so I have
added the wait time as an input to the `AntFixture` defaults to 30 seconds
 and set it to 60 seconds in case of secure hdfs fixture.

The integ test for secure hdfs was disabled for a long time and so
the changes done in #42090 to fix the tests are also done in this commit.
2019-07-12 12:44:01 +10:00
Yogesh Gaikwad 4ae1e30a98
Enable krb5kdc-fixture, kerberos tests mount urandom for kdc container (#41710) (#43178)
Infra has fixed #10462 by installing `haveged` on CI workers.
This commit enables the disabled fixture and tests, and mounts
`/dev/urandom` for the container so there is enough
entropy required for kdc.
Note: hdfs-repository tests have been disabled, will raise a separate issue for it.

Closes #40624 Closes #40678
2019-06-13 13:02:16 +10:00
Mark Vieira 1287c7d91f
[Backport] Replace usages RandomizedTestingTask with built-in Gradle Test (#40978) (#40993)
* Replace usages RandomizedTestingTask with built-in Gradle Test (#40978)

This commit replaces the existing RandomizedTestingTask and supporting code with Gradle's built-in JUnit support via the Test task type. Additionally, the previous workaround to disable all tasks named "test" and create new unit testing tasks named "unitTest" has been removed such that the "test" task now runs unit tests as per the normal Gradle Java plugin conventions.

(cherry picked from commit 323f312bbc829a63056a79ebe45adced5099f6e6)

* Fix forking JVM runner

* Don't bump shadow plugin version
2019-04-09 11:52:50 -07:00
Alpar Torok 293297ae3d Fix repository-hdfs when no docker and unnecesary fixture
The hdfs-fixture is actually executed in plugin/repository-hdfs as a
dependency. The fixture is not needed and actually causes a failure
because we have two copies now and both use the same ports.
2019-03-29 16:55:12 +02:00
Alpar Torok 2b91fb1cc0 Avoid building hdfs-fixure use an image that works instead
Avoid the additional requirement for the debian package repos to be up,
and depend on dockerhub only instead.
2019-03-29 16:55:11 +02:00
Alpar Torok d791e08932 Test fixtures krb5 (#40297)
Replaces the vagrant based kerberos fixtures with docker based test fixtures plugin.
The configuration is now entirely static on the docker side and no longer driven by Gradle,
also two different services are being configured since there are two different consumers of the fixture that can run in parallel and require different configurations.
2019-03-28 17:26:58 +02:00
Alpar Torok e9ef5bdce8
Converting randomized testing to create a separate unitTest task instead of replacing the builtin test task (#36311)
- Create a separate unitTest task instead of Gradle's built in 
- convert all configuration to use the new task 
- the  built in task is now disabled
2018-12-19 08:25:20 +02:00
Alpar Torok c00d0fc814
Test fixtures improovements (#36037)
* Upgrae plugin to latest and expose udp
* Explicit check for windows
* Rename the properties for the port numbers
* Tasks for pre and pos container actions
2018-12-12 12:00:47 +02:00
Tanguy Leroux bf58660482
Remove all unused imports and fix CRLF (#31207)
The X-Pack opening and the recent other refactorings left a lot of 
unused imports in the codebase. This commit removes them all.
2018-06-11 15:12:12 +02:00
James Baiera e16f1271b6
Fix SecurityException when HDFS Repository used against HA Namenodes (#27196)
* Sense HA HDFS settings and remove permission restrictions during regular execution.

This PR adds integration tests for HA-Enabled HDFS deployments, both regular and secured. 
The Mini HDFS fixture has been updated to optionally run in HA-Mode. A new test suite has 
been added for reproducing the effects of a Namenode failing over during regular repository 
usage. Going forward, the HDFS Repository will still be subject to its self imposed permission 
restrictions during normal use, but will no longer restrict them when running against an HA 
enabled HDFS cluster. Instead, the plugin will rely on the provided security policy and not 
further restrict the permissions so that the transparent operation to failover to a different 
Namenode in the client does not raise security exceptions. Additionally, we are now testing the 
secure mode with SASL based wire encryption of data between Elasticsearch and HDFS. This 
includes a missing library (commons codec) in order to support this change.
2017-12-01 14:26:05 -05:00
James Baiera c760eec054 Add permission checks before reading from HDFS stream (#26716)
Add checks for special permissions before reading hdfs stream data. Also adds test from 
readonly repository fix. MiniHDFS will now start with an existing repository with a single snapshot 
contained within. Readonly Repository is created in tests and attempts to list the snapshots 
within this repo.
2017-09-21 11:55:07 -04:00
James Baiera 74f4a14d82 Upgrading HDFS Repository Plugin to use HDFS 2.8.1 Client (#25497)
Hadoop 2.7.x libraries fail when running on JDK9 due to the version string changing to a single 
character. On Hadoop 2.8, this is no longer a problem, and it is unclear on whether the fix will be 
backported to the 2.7 branch. This commit upgrades our dependency of Hadoop for the HDFS 
Repository to 2.8.1.
2017-06-30 17:57:56 -04:00
James Baiera 6a113ae499 Introduce Kerberos Test Fixture for Repository HDFS Security Tests (#24493)
This PR introduces a subproject in test/fixtures that contains a Vagrantfile used for standing up a 
KRB5 KDC (Kerberos). The PR also includes helper scripts for provisioning principals, a few 
changes to the HDFS Fixture to allow it to interface with the KDC, as well as a new suite of 
integration tests for the HDFS Repository plugin.

The HDFS Repository plugin senses if the local environment can support the HDFS Fixture 
(Windows is generally a restricted environment). If it can use the regular fixture, it then tests if 
Vagrant is installed with a compatible version to determine if the secure test fixtures should be 
enabled. If the secure tests are enabled, then we create a Kerberos KDC fixture, tasks for adding 
the required principals, and an HDFS fixture configured for security. A new integration test task is 
also configured to use the KDC and secure HDFS fixture and to run a testing suite that uses 
authentication. At the end of the secure integration test the fixtures are torn down.
2017-05-10 17:42:20 -04:00
Robert Muir f67390e0c8 in the plugin: guard against HADOOP_HOME in environment on any platform.
hdfs fixture: minihdfs works on windows now, if things are properly set
but our test fixture still cannot launch this on windows.
2015-12-21 02:21:53 -05:00
Robert Muir e93c491dbe simplify hdfs fixture 2015-12-20 23:50:27 -05:00
Robert Muir 99f2cde225 Fail fast if HDFS cluster shuts itself down 2015-12-20 22:30:41 -05:00
Robert Muir 08d1d2f192 add simple javaexec for now until test fixtures can be debugged when they do not start 2015-12-20 19:04:28 -05:00
Robert Muir 12a8428dfb Add MiniHDFS test fixture, started before integTest and shut down after.
Currently uses a hardcoded port (9999), need to apply MavenFilteringHack after it starts.
2015-12-20 16:00:37 -05:00