Commit Graph

386 Commits

Author SHA1 Message Date
Robert Muir ee79d46583 Add gradle thirdPartyAudit to precommit tasks 2015-12-16 16:38:16 -05:00
Robert Muir 2741888498 Remove RuntimePermission("accessDeclaredMembers")
Upgrades lucene to 5.5.0-1719088, randomizedtesting to 2.3.2, and securemock to 1.2
2015-12-10 14:26:55 -05:00
David Pilato c1f7171e61 Add support for path_style_access
From https://github.com/elastic/elasticsearch-cloud-aws/pull/159

Add a new option `path_style_access` for S3 buckets. It adds support for path style access for [virtual hosting of buckets](http://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html).
Defaults to `false`.

Closes https://github.com/elastic/elasticsearch-cloud-aws/issues/124.
2015-12-10 09:00:31 +01:00
David Pilato 7dcb40bcac Add support for proxy authentication for s3 and ec2
When using S3 or EC2, it was possible to use a proxy to access EC2 or S3 API but username and password were not possible to be set.

This commit adds support for this. Also, to make all that consistent, proxy settings for both plugins have been renamed:

* from `cloud.aws.proxy_host` to `cloud.aws.proxy.host`
* from `cloud.aws.ec2.proxy_host` to `cloud.aws.ec2.proxy.host`
* from `cloud.aws.s3.proxy_host` to `cloud.aws.s3.proxy.host`
* from `cloud.aws.proxy_port` to `cloud.aws.proxy.port`
* from `cloud.aws.ec2.proxy_port` to `cloud.aws.ec2.proxy.port`
* from `cloud.aws.s3.proxy_port` to `cloud.aws.s3.proxy.port`

New settings are `proxy.username` and `proxy.password`.

```yml
cloud:
    aws:
        protocol: https
        proxy:
            host: proxy1.company.com
            port: 8083
            username: myself
            password: theBestPasswordEver!
```

You can also set different proxies for `ec2` and `s3`:

```yml
cloud:
    aws:
        s3:
            proxy:
                host: proxy1.company.com
                port: 8083
                username: myself1
                password: theBestPasswordEver1!
        ec2:
            proxy:
                host: proxy2.company.com
                port: 8083
                username: myself2
                password: theBestPasswordEver2!
```

Note that `password` is filtered with `SettingsFilter`.

We also fix a potential issue in S3 repository. We were supposed to accept key/secret either set under `cloud.aws` or `cloud.aws.s3` but the actual code never implemented that.

It was:

```java
account = settings.get("cloud.aws.access_key");
key = settings.get("cloud.aws.secret_key");
```

We replaced that by:

```java
String account = settings.get(CLOUD_S3.KEY, settings.get(CLOUD_AWS.KEY));
String key = settings.get(CLOUD_S3.SECRET, settings.get(CLOUD_AWS.SECRET));
```

Also, we extract all settings for S3 in `AwsS3Service` as it's already the case for `AwsEc2Service` class.

Closes #15268.
2015-12-07 23:10:54 +01:00
Robert Muir b0c64910b0 ban RuntimePermission("getClassLoader")
this gives more isolation between modules and plugins.
2015-12-04 15:58:02 -05:00
Robert Muir 30529c008d Ban write access to system properties
* Forbid System.setProperties & co in forbidden APIs.
* Ban property write access at runtime with security manager.

Plugins that need to modify system properties will need to request permission in their plugin-security.policy
2015-11-21 22:33:06 -05:00
David Pilato d48d8ef863 Merge branch 's3-storage-class-support' of https://github.com/socialrank/elasticsearch into socialrank-s3-storage-class-support
# Please enter a commit message to explain why this merge is necessary,
# especially if it merges an updated upstream into a topic branch.
#
# Lines starting with '#' will be ignored, and an empty message aborts
# the commit.
2015-11-19 13:10:08 +01:00
Robert Muir 914cee213e Upgrade to aws 1.10.33
Security issues have been fixed. This removes our hacks.
2015-11-10 22:51:33 -05:00
Ryan Ernst 4b5f87cb7d Build: Remove transitive dependencies
Transitive dependencies can be confusing and hard to deal with when
conflicts arise between them. This change removes transitive
dependencies from elasticsearch, and forces any dependency conflicts to
be resolved manually, instead of automatically by gradle.

closes #14627
2015-11-10 15:01:41 -08:00
David Pilato a61fcfd8ca Remove AbstractLegacyBlobContainer
`AbstractLegacyBlobContainer` was kept for historical reasons (see #13434).
We can migrate Azure and S3 repositories to use the new methods added in #13434 so we can remove `AbstractLegacyBlobContainer` class.
2015-11-10 16:04:44 +01:00
David Pilato 1b98e1b1ad Adding US-Gov-West for S3
Follow up for #14358
2015-11-02 14:38:03 +01:00
Ryan Ernst 542522531a Build: Remove maven pom files and supporting ant files
This change removes the leftover pom files. A couple files were left for
reference, namely in qa tests that have not yet been migrated (vagrant
and multinode). The deb and rpm assemblies also still exist for
reference when finishing their setup in gradle.

See #13930
2015-10-29 23:53:49 -07:00
xuzha b3090d654a Add AWS Canned ACL support
User could set cannedACL, S3 repository would use this cannedACL to
create s3 object and bucket.
2015-10-29 11:48:02 -07:00
Ryan Ernst c86100f636 Switch build system to Gradle
See #13930
2015-10-29 11:40:19 -07:00
Nik Everett 2cc97a0d3e Remove and ban @Test
There are three ways `@Test` was used. Way one:

```java
@Test
public void flubTheBlort() {
```

This way was always replaced with:

```java
public void testFlubTheBlort() {
```

Or, maybe with a better method name if I was feeling generous.

Way two:

```java
@Test(throws=IllegalArgumentException.class)
public void testFoo() {
    methodThatThrows();
}
```

This way of using `@Test` is actually pretty OK, but to get the tools to ban
`@Test` entirely it can't be used. Instead:

```java
public void testFoo() {
    try {
        methodThatThrows();
        fail("Expected IllegalArgumentException");
    } catch (IllegalArgumentException e ) {
        assertThat(e.getMessage(), containsString("something"));
    }
}
```

This is longer but tests more than the old ways and is much more precise.
Compare:

```java
@Test(throws=IllegalArgumentException.class)
public void testFoo() {
    some();
    copy();
    and();
    pasted();
    methodThatThrows();
    code();  // <---- This was left here by mistake and is never called
}
```

to:

```java
@Test(throws=IllegalArgumentException.class)
public void testFoo() {
    some();
    copy();
    and();
    pasted();
    try {
        methodThatThrows();
        fail("Expected IllegalArgumentException");
    } catch (IllegalArgumentException e ) {
        assertThat(e.getMessage(), containsString("something"));
    }
}
```

The final use of test is:

```java
@Test(timeout=1000)
public void testFoo() {
    methodThatWasSlow();
}
```

This is the most insidious use of `@Test` because its tempting but tragically
flawed. Its flaws are:
1. Hard and fast timeouts can look like they are asserting that something is
faster and even do an ok job of it when you compare the timings on the same
machine but as soon as you take them to another machine they start to be
invalid. On a slow VM both the new and old methods fail. On a super-fast
machine the slower and faster ways succeed.
2. Tests often contain slow `assert` calls so the performance of tests isn't
sure to predict the performance of non-test code.
3. These timeouts are rude to debuggers because the test just drops out from
under it after the timeout.

Confusingly, timeouts are useful in tests because it'd be rude for a broken
test to cause CI to abort the whole build after it hits a global timeout. But
those timeouts should be very very long "backstop" timeouts and aren't useful
assertions about speed.

For all its flaws `@Test(timeout=1000)` doesn't have a good replacement __in__
__tests__. Nightly benchmarks like http://benchmarks.elasticsearch.org/ are
useful here because they run on the same machine but they aren't quick to check
and it takes lots of time to figure out the regressions. Sometimes its useful
to compare dueling implementations but that requires keeping both
implementations around. All and all we don't have a satisfactory answer to the
question "what do you replace `@Test(timeout=1000)`" with. So we handle each
occurrence on a case by case basis.

For files with `@Test` this also:
1. Removes excess blank lines. They don't help anything.
2. Removes underscores from method names. Those would fail any code style
checks we ever care to run and don't add to readability. Since I did this manually
I didn't do it consistently.
3. Make sure all test method names start with `test`. Some used to end in `Test` or start
with `verify` or `check` and they were picked up using the annotation. Without the
annotation they always need to start with `test`.
4. Organizes imports using the rules we generate for Eclipse. For the most part
this just removes `*` imports which is a win all on its own. It was "required"
to quickly remove `@Test`.
5. Removes unneeded casts. This is just a setting I have enabled in Eclipse and
forgot to turn off before I did this work. It probably isn't hurting anything.
6. Removes trailing whitespace. Again, another Eclipse setting I forgot to turn
off that doesn't hurt anything. Hopefully.
7. Swaps some tests override superclass tests to make them empty with
`assumeTrue` so that the reasoning for the skips is logged in the test run and
it doesn't "look like" that thing is being tested when it isn't.
8. Adds an oxford comma to an error message.

The total test count doesn't change. I know. I counted.
```bash
git checkout master && mvn clean && mvn install | tee with_test
git no_test_annotation master && mvn clean && mvn install | tee not_test
grep 'Tests summary' with_test > with_test_summary
grep 'Tests summary' not_test > not_test_summary
diff with_test_summary not_test_summary
```

These differ somewhat because some tests are skipped based on the random seed.
The total shouldn't differ. But it does!
```
1c1
< [INFO] Tests summary: 564 suites (1 ignored), 3171 tests, 31 ignored (31 assumptions)
---
> [INFO] Tests summary: 564 suites (1 ignored), 3167 tests, 17 ignored (17 assumptions)
```

These are the core unit tests. So we dig further:
```bash
cat with_test | perl -pe 's/\n// if /^Suite/;s/.*\n// if /IGNOR/;s/.*\n// if /Assumption #/;s/.*\n// if /HEARTBEAT/;s/Completed .+?,//' | grep Suite > with_test_suites
cat not_test | perl -pe 's/\n// if /^Suite/;s/.*\n// if /IGNOR/;s/.*\n// if /Assumption #/;s/.*\n// if /HEARTBEAT/;s/Completed .+?,//' | grep Suite > not_test_suites
diff <(sort with_test_suites) <(sort not_test_suites)
```

The four tests with lower test numbers are all extend `AbstractQueryTestCase`
and all have a method that looks like this:

```java
@Override
public void testToQuery() throws IOException {
    assumeTrue("test runs only when at least a type is registered", getCurrentTypes().length > 0);
    super.testToQuery();
}
```

It looks like this method was being double counted on master and isn't anymore.

Closes #14028
2015-10-20 17:37:36 -04:00
javanna a6e7a5f307 Java api: remove the count api
Closes #14166
Closes #13928
2015-10-19 14:40:52 +02:00
Robert Muir 5d001d1578 Decentralize plugin security
* Add ability for plugins to declare additional permissions with a custom plugin-security.policy file and corresponding AccessController logic. See the plugin author's guide for more information.
* Add warning messages to users for extra plugin permissions in bin/plugin.
* When bin/plugin is run interactively (stdin is a controlling terminal and -b/--batch not supplied), require user confirmation.
* Improve unit test and IDE support for plugins with additional permissions by exposing plugin's metadata as a maven test resource.

Closes #14108

Squashed commit of the following:

commit cf8ace65a7397aaccd356bf55f95d6fbb8bb571c
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Oct 14 13:36:05 2015 -0400

    fix new unit test from master merge

commit 9be3c5aa38f2d9ae50f3d54924a30ad9cddeeb65
Merge: 2f168b8 7368231
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Oct 14 12:58:31 2015 -0400

    Merge branch 'master' into off_my_back

commit 2f168b8038e32672f01ad0279fb5db77ba902ae8
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Oct 14 12:56:04 2015 -0400

    improve plugin author documentation

commit 6e6c2bfda68a418d92733ac22a58eec35508b2d0
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Oct 14 12:52:14 2015 -0400

    move security confirmation after 'plugin already installed' check, to prevent user from answering unnecessary questions.

commit 08233a2972554afef2a6a7521990283102e20d92
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Oct 14 05:36:42 2015 -0400

    Add documentation and pluginmanager support

commit 05dad86c51488ba43ccbd749f0164f3fbd3aee62
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Oct 14 02:22:24 2015 -0400

    Decentralize plugin permissions (modulo docs and pluginmanager work)
2015-10-14 14:46:45 -04:00
Robert Muir 6d8c035f70 Add SpecialPermission to guard exceptions to security policy.
Closes #13854

Squashed commit of the following:

commit 42c1166efc55adda0d13fed77de583c0973e44b3
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Sep 29 11:59:43 2015 -0400

    Add paranoia

    Groovy holds on to a classloader, so check it before compilation too.
    I have not reviewed yet what Rhino is doing, but just be safe.

commit b58668a81428e964dd5ffa712872c0a34897fc91
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Sep 29 11:46:06 2015 -0400

    Add SpecialPermission to guard exceptions to security policy.

    In some cases (e.g. buggy cloud libraries, scripting engines), we must
    grant dangerous permissions to contained cases. Those AccessController blocks
    are dangerous, since they truncate the stack, and can allow privilege escalation.

    This PR adds a simple permission to check before each one, so that unprivileged code
    like groovy scripts, can't do anything they shouldn't be allowed to do otherwise.
2015-09-29 17:32:56 -04:00
Robert Muir b582de79ae Merge pull request #13702 from rmuir/broke_javadocs
Fix all javadocs issues, re-enable compiler warnings (but disable on java 9 where maven is broken)
2015-09-22 00:46:31 -04:00
Robert Muir 2f67cacaa3 Fix all javadocs issues, re-enable compiler warnings (but disable on java9 where maven is broken) 2015-09-21 23:35:32 -04:00
Ryan Ernst 8aa6aec344 Merge pull request #13663 from rjernst/license_cleanup
Remove unnecessary copies of license and notice files
2015-09-21 18:04:57 -07:00
Simon Willnauer 66cb36e9b3 update repository-s3 licenses 2015-09-19 07:41:30 +02:00
Ryan Ernst 18c519145d Remove unnecessary copies of license and notice files
We moved a lot of repositories into elasticsearch, but in their new
location they retained their LICENSE.txt and NOTICE.txt files. These are
all the same, and having the license and notice and the root of the
repository should be sufficient.
2015-09-18 17:48:30 -07:00
Michael Schonfeld cd33ff6fd8 move the aws sdk version parameter to plugins/pom.xml 2015-09-18 10:10:51 -04:00
Michael Schonfeld b94f32116c update s3 sdk version 2015-09-18 09:43:35 -04:00
Ryan Ernst b14326d494 Merge pull request #13611 from rjernst/spec_in_resources
Move rest-api-spec for plugins into test resources
2015-09-16 11:15:35 -07:00
Ryan Ernst 45f757de6d Test: Move rest-api-spec for plugins into test resources
Plugin tests require having rest-api tests, and currently copy that spec
from a directory in the root of the plugin source into the test
resources. This change moves the rest-api-spec dir into test resources
so it is like any other test resources. It also removes unnecessary
configuration for resources from the shared plugin pom.
2015-09-16 03:04:53 -07:00
Robert Muir 01e6d8e3dc Remove java.lang.reflect.ReflectPermission "suppressAccessChecks"
Closes #13603

Squashed commit of the following:

commit 8799fb42d80297a79285beaf407b1bbecdb5854d
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Sep 16 03:32:29 2015 -0400

    Add randomizedtesting snapshot note

commit 0d874d9f0f5fddaeab8f48f9816a052dcaa691be
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Sep 16 03:11:01 2015 -0400

    Add a mechanism for insecure plugins and get all tests passing

commit 80540aeb9a264f6f299aaa3bc89df7f9b7923a60
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Sep 15 22:59:29 2015 -0400

    Really remove, we are killing this

commit 884818c1ad44ca2e7572a6998c086580be919657
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Sep 15 22:57:22 2015 -0400

    fill in TODOs

commit 34f4cb81f249edfec4d8d211da892f8c987e5948
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Sep 15 22:31:43 2015 -0400

    Publish snapshots of RR and lucene and cutover

commit d68eb9d66ce059761805c64d67e41a29098c9afa
Merge: f27e208 f62da59
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Sep 15 12:32:41 2015 -0400

    Merge branch 'master' into kill-setaccessible

commit f27e20855216dab6a6ad035d41018d8c67f3144c
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Sep 15 12:32:21 2015 -0400

    make a real lucene snapshot
2015-09-16 04:08:31 -04:00
David Pilato a38bcc5d62 [test] plugins simple RestIT tests don't work from IDE
When running a RestIT test from the IDE, you actually start an internal node which does not automatically load the plugin you would like to test.

We need to add:

```java
    @Override
    protected Collection<Class<? extends Plugin>> nodePlugins() {
        return pluginList(PLUGIN_HERE.class);
    }
```

Everything works fine when running from maven because each test basically:

* installs elasticsearch
* installs one plugin
* starts elasticsearch with this plugin loaded
* runs the test

Note that this PR only fixes the fact we run an internal cluster with the expected plugin.

Cloud tests will still fail when run from the IDE because is such a case you actually start an internal node with many mock plugins.
And REST test suite for cloud plugins basically checks if the plugin is running by checking the output of NodesInfo API.

And we check:

```yml
- match:  { nodes.$master.plugins.0.name: cloud-azure  }
- match:  { nodes.$master.plugins.0.jvm: true  }
```

But in that case, this condition is certainly false as we started also `mock-transport-service`, `mock-index-store`, `mock-engine-factory`, `node-mocks`, `asserting-local-transport`, `mock-search-service`.

Closes #13479
2015-09-15 10:10:05 +02:00
Robert Muir 0b16552367 Revert "Disable this test (we should fix rest framework to disable based on java versions)"
This reverts commit 6a0381b582.
2015-09-12 12:05:47 -04:00
Igor Motov 39ca45050a Simplify the BlobContainer blob writing interface
Instead of asking blob store to create output for posting blob content, this change provides that content of the blob to the blob store for writing. This will significantly simplify the  interface for S3 and Azure plugins.
2015-09-11 16:39:58 -04:00
Robert Muir 6a0381b582 Disable this test (we should fix rest framework to disable based on java versions) 2015-09-11 00:13:42 -04:00
Ryan Ernst 9e8a90a657 Add xlint ignores for warning classes, where appropriate. 2015-09-09 12:47:07 -07:00
xuzha e69551d8df This commit remove the block for user to set invalid Signer Type.
Instead of throw an exception, this change log a warning message
and accept any signer type.
2015-09-05 22:57:05 -07:00
David Pilato 6953193f5b [test] fix test names
For aws s3 and ec2
2015-09-03 12:21:26 +02:00
David Pilato 30aa231f8e [plugin] split cloud-aws in repository-s3 and discovery-ec2
Until now we had a cloud-aws plugin which is providing 2 disctinct features:

* discovery on EC2
* snapshot/restore on S3

This commit splits the plugin by feature so people can use either one or the other or both features.

Doc is updated accordingly.
2015-09-03 11:12:20 +02:00