Commit Graph

110 Commits

Author SHA1 Message Date
Robert Muir 935c2c75f6 Remove slf4j hack 2015-12-20 22:08:18 -05:00
Robert Muir 03a2b6b01b Disable HDFS fixture on windows, it requires native libraries. 2015-12-20 16:30:19 -08:00
Robert Muir 12a8428dfb Add MiniHDFS test fixture, started before integTest and shut down after.
Currently uses a hardcoded port (9999), need to apply MavenFilteringHack after it starts.
2015-12-20 16:00:37 -05:00
Robert Muir 02fbd55118 enable thirdPartyAudit so you can see the crazy shit hadoop does 2015-12-18 23:45:05 -05:00
Robert Muir 75ef9da53f get up to connectexception 2015-12-18 22:11:58 -05:00
Ryan Ernst 91fe99a7f6 Make hdfs plugin not use transitive deps 2015-12-18 16:52:22 -08:00
Costin Leau 7584810ff4 * Make plugin hadoop2-only
Polish MiniDFS cluster to be Hadoop2 (instead of Hadoop1) based
2015-12-19 01:35:53 +02:00
Robert Muir 6692e42d9a thirdPartyAudit round 2
This fixes the `lenient` parameter to be `missingClasses`. I will remove this boolean and we can handle them via the normal whitelist.
It also adds a check for sheisty classes (jar hell with the jdk).
This is inspired by the lucene "sheisty" classes check, but it has false positives. This check is more evil, it validates every class file against the extension classloader as a resource, to see if it exists there. If so: jar hell.

This jar hell is a problem for several reasons:

1. causes insanely-hard-to-debug problems (like bugs in forbidden-apis)
2. hides problems (like internal api access)
3. the code you think is executing, is not really executing
4. security permissions are not what you think they are
5. brings in unnecessary dependencies
6. its jar hell

The more difficult problems are stuff like jython, where these classes are simply 'uberjared' directly in, so you cant just fix them by removing a bogus dependency. And there is a legit reason for them to do that, they want to support java 1.4.
2015-12-17 02:35:00 -05:00
Robert Muir ee79d46583 Add gradle thirdPartyAudit to precommit tasks 2015-12-16 16:38:16 -05:00
Costin Leau 7bca97bba6 HDFS Snapshot/Restore plugin
Migrated from ES-Hadoop. Contains several improvements regarding:

* Security
Takes advantage of the pluggable security in ES 2.2 and uses that in order
to grant the necessary permissions to the Hadoop libs. It relies on a
dedicated DomainCombiner to grant permissions only when needed only to the
libraries installed in the plugin folder
Add security checks for SpecialPermission/scripting and provides out of
the box permissions for the latest Hadoop 1.x (1.2.1) and 2.x (2.7.1)

* Testing
Uses a customized Local FS to perform actual integration testing of the
Hadoop stack (and thus to make sure the proper permissions and ACC blocks
are in place) however without requiring extra permissions for testing.
If needed, a MiniDFS cluster is provided (though it requires extra
permissions to bind ports)
Provides a RestIT test

* Build system
Picks the build system used in ES (still Gradle)
2015-12-14 21:50:09 +02:00