Add support for hadoop 3 profiles . Most of the details are captured in #11791 .
We use a combination of maven profiles and resource filtering to achieve this. Hadoop2 is supported by default and a new maven profile with the name hadoop3 is created. This will allow the user to choose the profile which is best suited for the use case.
The CVE details are here - https://nvd.nist.gov/vuln/detail/CVE-2021-26291. I am marking it suppressed since we are only using kafka-clients jar in druid. We use maven-artifact jar ourselves but it is only used for comparing versions
Dependency on hadoop 2.8.5 is preventing us form updating this dependency to a later version. We don't believe that this is a major concern since Druid eats uncaught exceptions, and only displays them in logs. This issue also should only affect ingestion jobs, which can only be run by admin type users.
* Ignore CVEs from htrace and ambari transitive deps
htrace CVEs are suppressed for now as addressing them requires updating
the hadoop version.
ambari CVEs are suppressed for now since ambari is updated to the latest
version and is no longer actively maintained.
* Fix compilation issue from ambari upgrade
* Add missing test coverage
* Document possible vulnerabilities for the druid-ranger-security
In certain configurations the ranger plugin can expose vulnerabilities due
to some of its dependencies having CVEs.
* Spelling checker is a bit tight
CVE-2013-7109 can be ignored for openstack-swift as it is for the python
SDK and druid uses the java SDK.
The jackson-databind:2.4.0 CVEs via htrace-core4 are all suppressed for
now as fixing them requires updating the hadoop version.
CVE-2020-8840 was updated on 19 Feb 2020, which now gets flagged by the
security vulnerability scan. Since the CVE is for jackson-databind, via
htrace-core-4.0.1, it can be added to the existing list of security
vulnerability suppressions for that dependency.
* Suppress netty 3 vulnerabilites and upgrade netty 4 version
* Upgrade netty 4 version to fix vulnerabilities CVE-2019-20445
and CVE-2019-20444
* suppress these CVEs for netty 3
* * simplify suppression xml file
* update licenses file with new version of netty
* * fix type in licenses.yaml
CVE-2019-20330 was updated on 14 Jan 2020, which now gets flagged by the
security vulnerability scan. Since the CVE is for jackson-databind, via
htrace-core-4.0.1, it can be added to the existing list of security
vulnerability suppressions for that dependency.
* Address security vulnerabilities CVSS >= 7
Update dependencies to address security vulnerabilities with CVSS scores
of 7 or higher. A new Travis CI job is added to prevent new
high/critical security vulnerabilities from being added.
Updated dependencies:
- api-util 1.0.0 -> 1.0.3
- jackson 2.9.10 -> 2.10.1
- kafka 2.1.0 -> 2.1.1
- libthrift 0.10.0 -> 0.13.0
- protobuf 3.2.0 -> 3.11.0
The following high/critical security vulnerabilities are currently
suppressed (so that the new Travis CI job can be added now) and are left
as future work to fix:
- hibernate-validator:5.2.5
- jackson-mapper-asl:1.9.13
- libthrift:0.6.1
- netty:3.10.6
- nimbus-jose-jwt:4.41.1
* Rename EDL1 license file
* Fix inspection errors
