Commit Graph

146 Commits

Author SHA1 Message Date
Ryan Ernst d104d6d652 Refactor hdfs unit tests to be simple and check every configuration
error condition
2015-12-21 19:38:54 -08:00
Ryan Ernst af7d6b629c Change hdfs unit tests to be a single node test instead of integ test 2015-12-21 18:32:28 -08:00
Robert Muir c54d53c8d5 streamline these classes a bit 2015-12-21 21:11:06 -05:00
Robert Muir 795869c345 remove filecontextfactory 2015-12-21 20:55:17 -05:00
Robert Muir 956281f039 remove shitton of permissions 2015-12-21 20:26:13 -05:00
Ryan Ernst d0e9306413 Remove reading node settings as defaults for hdfs repository settings 2015-12-21 16:25:55 -08:00
Robert Muir 3c07a427dc fix exc handling 2015-12-21 19:06:56 -05:00
Ryan Ernst 26eaa16a89 Remove "additional config" from hdfs repositories 2015-12-21 15:55:13 -08:00
Robert Muir 2cbfc54a81 avoid too-long classpath so it works on windows 2015-12-21 18:25:08 -05:00
Robert Muir 7065639a26 add test for listing 2015-12-21 17:25:15 -05:00
Robert Muir 5ebcf183e5 tests 2015-12-21 17:02:50 -05:00
Robert Muir b8524bdb11 add tests 2015-12-21 16:16:24 -05:00
Robert Muir 3a2464b80e improve build logic on windows without native libraries 2015-12-21 15:37:34 -05:00
Robert Muir 0ed45c5bfb remove filesystem leniency 2015-12-21 14:16:53 -05:00
Robert Muir deaf8884e9 Fix exc handling 2015-12-21 13:04:22 -05:00
Robert Muir 3ffd1a5219 final 2015-12-21 12:54:33 -05:00
Robert Muir f81b12e327 minimize accessiblity, remove unused threadpool 2015-12-21 12:39:40 -05:00
Robert Muir f67390e0c8 in the plugin: guard against HADOOP_HOME in environment on any platform.
hdfs fixture: minihdfs works on windows now, if things are properly set
but our test fixture still cannot launch this on windows.
2015-12-21 02:21:53 -05:00
Robert Muir 53530f1243 remove hacks, test fixtures are clean before each execution 2015-12-20 22:23:30 -05:00
Robert Muir 935c2c75f6 Remove slf4j hack 2015-12-20 22:08:18 -05:00
Robert Muir 04966bcc3e contain and improve hack 2015-12-20 21:02:03 -05:00
Robert Muir 03a2b6b01b Disable HDFS fixture on windows, it requires native libraries. 2015-12-20 16:30:19 -08:00
Robert Muir a37417085d blind stab at unit test issues on windows 2015-12-20 18:31:55 -05:00
Robert Muir ee546ff655 try to get windows working 2015-12-20 17:10:01 -05:00
Robert Muir 2347e3c373 Get forbidden apis passing again, this needs to be investigated 2015-12-20 16:17:17 -05:00
Robert Muir 7ac49bb278 Merge branch 'hdfs2-only' of github.com:costin/elasticsearch into hdfs2-only 2015-12-20 16:12:23 -05:00
Robert Muir 12a8428dfb Add MiniHDFS test fixture, started before integTest and shut down after.
Currently uses a hardcoded port (9999), need to apply MavenFilteringHack after it starts.
2015-12-20 16:00:37 -05:00
Costin Leau 3204e87220 Restrict usage to HDFS only 2015-12-20 15:53:18 +02:00
Robert Muir d171773bdb remove leniency in tests 2015-12-19 04:39:01 -05:00
Robert Muir e2b2ee24fa Add licensing for dependencies 2015-12-19 03:06:40 -05:00
Robert Muir 9df447295c Fix unit tests (also works from IDE). 2015-12-19 02:43:27 -05:00
Robert Muir 3269beeb4d don't throw exceptions from ctor, guice is hell 2015-12-19 02:09:14 -05:00
Robert Muir f174e96a14 explicitly initialize some hadoop classes elevated, so we don't rely on classloading order.
maybe this allows us to do less stuff in doPriv later, we will see. at least it makes things
like unit testing easier.
2015-12-19 00:21:01 -05:00
Robert Muir 2e8c68d09b Remove no-longer needed domaincombiner stuff 2015-12-18 23:51:41 -05:00
Robert Muir 02fbd55118 enable thirdPartyAudit so you can see the crazy shit hadoop does 2015-12-18 23:45:05 -05:00
Robert Muir bc11962438 get full snapshot restore tests passing 2015-12-18 23:16:41 -05:00
Robert Muir fbe3d64ea4 add passing test that takes snapshot 2015-12-18 22:55:15 -05:00
Robert Muir 75ef9da53f get up to connectexception 2015-12-18 22:11:58 -05:00
Ryan Ernst c2c5081830 Remove uneeded class loading stuff from hdfs plugin 2015-12-18 17:01:38 -08:00
Ryan Ernst 91fe99a7f6 Make hdfs plugin not use transitive deps 2015-12-18 16:52:22 -08:00
Costin Leau 7584810ff4 * Make plugin hadoop2-only
Polish MiniDFS cluster to be Hadoop2 (instead of Hadoop1) based
2015-12-19 01:35:53 +02:00
Ryan Ernst 4ea19995cf Remove wildcard imports 2015-12-18 12:43:47 -08:00
Robert Muir 447729f0e1 add missing license headers 2015-12-18 13:08:17 -05:00
Robert Muir 6692e42d9a thirdPartyAudit round 2
This fixes the `lenient` parameter to be `missingClasses`. I will remove this boolean and we can handle them via the normal whitelist.
It also adds a check for sheisty classes (jar hell with the jdk).
This is inspired by the lucene "sheisty" classes check, but it has false positives. This check is more evil, it validates every class file against the extension classloader as a resource, to see if it exists there. If so: jar hell.

This jar hell is a problem for several reasons:

1. causes insanely-hard-to-debug problems (like bugs in forbidden-apis)
2. hides problems (like internal api access)
3. the code you think is executing, is not really executing
4. security permissions are not what you think they are
5. brings in unnecessary dependencies
6. its jar hell

The more difficult problems are stuff like jython, where these classes are simply 'uberjared' directly in, so you cant just fix them by removing a bogus dependency. And there is a legit reason for them to do that, they want to support java 1.4.
2015-12-17 02:35:00 -05:00
Robert Muir ee79d46583 Add gradle thirdPartyAudit to precommit tasks 2015-12-16 16:38:16 -05:00
Costin Leau 7bca97bba6 HDFS Snapshot/Restore plugin
Migrated from ES-Hadoop. Contains several improvements regarding:

* Security
Takes advantage of the pluggable security in ES 2.2 and uses that in order
to grant the necessary permissions to the Hadoop libs. It relies on a
dedicated DomainCombiner to grant permissions only when needed only to the
libraries installed in the plugin folder
Add security checks for SpecialPermission/scripting and provides out of
the box permissions for the latest Hadoop 1.x (1.2.1) and 2.x (2.7.1)

* Testing
Uses a customized Local FS to perform actual integration testing of the
Hadoop stack (and thus to make sure the proper permissions and ACC blocks
are in place) however without requiring extra permissions for testing.
If needed, a MiniDFS cluster is provided (though it requires extra
permissions to bind ports)
Provides a RestIT test

* Build system
Picks the build system used in ES (still Gradle)
2015-12-14 21:50:09 +02:00