Commit Graph

880 Commits

Author SHA1 Message Date
Adrien Grand d94bba2d9c Remove back compat for the `path` option.
The `path` option allowed to index/store a field `a.b.c` under just `c` when
set to `just_name`. This "feature" has been removed in 2.0 in favor of `copy_to`
so we can remove the back compat in 3.x.
2015-12-15 14:55:23 +01:00
Adrien Grand 50eeafa75c Make mappings immutable.
Today mappings are mutable because of two APIs:
 - Mapper.merge, which expects changes to be performed in-place
 - IncludeInAll, which allows to change whether values should be put in the
   `_all` field in place.

This commit changes both APIs to return a modified copy instead of modifying in
place so that mappings can be immutable. For now, only the type-level object is
immutable, but in the future we can imagine making them immutable at the
index-level so that mapping updates could be completely atomic at the index
level.

Close #9365
2015-12-15 10:20:28 +01:00
Ryan Ernst 60d35c81af Plugins: Expose http.type setting, and collapse al(most all) modules relating to transport/http
This change adds back the http.type setting. It also cleans up all the
transport related guice code to be consolidated within the
NetworkModule (as transport and http related stuff is what and how ES
exposes over the network). The setter methods previously used by some
plugins to override eg the TransportService or HttpServerTransport are
removed, and those plugins should now register a custom implementation
of the class with a name and set that using the appropriate config
setting. Note that I think ActionModule should also be moved into here,
to sit along side the rest actions, but I left that for a followup.

closes #14148
2015-12-14 22:01:04 -08:00
Costin Leau 7bca97bba6 HDFS Snapshot/Restore plugin
Migrated from ES-Hadoop. Contains several improvements regarding:

* Security
Takes advantage of the pluggable security in ES 2.2 and uses that in order
to grant the necessary permissions to the Hadoop libs. It relies on a
dedicated DomainCombiner to grant permissions only when needed only to the
libraries installed in the plugin folder
Add security checks for SpecialPermission/scripting and provides out of
the box permissions for the latest Hadoop 1.x (1.2.1) and 2.x (2.7.1)

* Testing
Uses a customized Local FS to perform actual integration testing of the
Hadoop stack (and thus to make sure the proper permissions and ACC blocks
are in place) however without requiring extra permissions for testing.
If needed, a MiniDFS cluster is provided (though it requires extra
permissions to bind ports)
Provides a RestIT test

* Build system
Picks the build system used in ES (still Gradle)
2015-12-14 21:50:09 +02:00
Jason Tedor 3383c24be0 Remove and forbid use of Collections#shuffle(List) and Random#<init>()
This commit removes and now forbids all uses of
Collections#shuffle(List) and Random#<init>() across the codebase. The
rationale for removing and forbidding these methods is to increase test
reproducibility. As these methods use non-reproducible seeds, production
code and tests that rely on these methods contribute to
non-reproducbility of tests.

Instead of Collections#shuffle(List) the method
Collections#shuffle(List, Random) can be used. All that is required then
is a reproducible source of randomness. Consequently, the utility class
Randomness has been added to assist in creating reproducible sources of
randomness.

Instead of Random#<init>(), Random#<init>(long) with a reproducible seed
or the aforementioned Randomess class can be used.

Closes #15287
2015-12-11 11:16:38 -05:00
Robert Muir 2741888498 Remove RuntimePermission("accessDeclaredMembers")
Upgrades lucene to 5.5.0-1719088, randomizedtesting to 2.3.2, and securemock to 1.2
2015-12-10 14:26:55 -05:00
Boaz Leskes fafeb3abdd Introduce a common base response class to all single doc write ops
IndexResponse, DeleteResponse and UpdateResponse share some logic. This can be unified to a single DocWriteResponse base class. On top, some replication actions are now not about write operations anymore. This commit renames ActionWriteResponse to ReplicationResponse

Last some toXContent is moved from the Rest layer to the actual response classes, for more code re-sharing.

Closes #15334
2015-12-10 15:14:02 +01:00
Jack Conradson da5b07ae13 Added a new scripting language (PlanA).
Closes #15136
2015-12-09 16:32:37 -08:00
David Pilato 414fccb7d1 Merge branch 'fix/15268-proxy-auth' 2015-12-09 23:21:00 +01:00
Simon Willnauer a49120bfc1 fix compilation 2015-12-09 12:26:28 +01:00
Simon Willnauer 85a1b54867 fix compilation 2015-12-09 11:41:14 +01:00
Simon Willnauer c9d7c92243 fold ClusterSettingsService into ClusterSettings 2015-12-09 09:57:39 +01:00
Britta Weber e0aa481bf5 Merge pull request #15213 from brwe/copy-to-in-multi-fields-exception
throw exception if a copy_to is within a multi field

Copy to within multi field is ignored from 2.0 on, see #10802.
Instead of just ignoring it, we should throw an exception if this
is found in the mapping when a mapping is added. For already
existing indices we should at least log a warning.
We remove the copy_to in any case.

related to #14946
2015-12-08 14:41:07 +01:00
Simon Willnauer fbbb04b87e Add infrastructure to transactionally apply and reset dynamic settings
This commit adds the infrastructure to make settings that are updateable
resetable and changes the application of updates to be transactional. This means
setting updates are either applied or not. If the application failes all values are rejected.

This initial commit converts all dynamic cluster settings to make use of the new infrastructure.
All cluster level dynamic settings are not resettable to their defaults or to the node level settings.
The infrastructure also allows to list default values and descriptions which is not fully implemented yet.

Values can be reset using a list of key or simple regular expressions. This has only been implemented on the java
layer yet. For instance to reset all recovery settings to their defaults a user can just specify `indices.recovery.*`.

This commit also adds strict settings validation, if a setting is unknown or if a setting can not be applied the entire
settings update request will fail.
2015-12-08 14:39:15 +01:00
David Pilato 7dcb40bcac Add support for proxy authentication for s3 and ec2
When using S3 or EC2, it was possible to use a proxy to access EC2 or S3 API but username and password were not possible to be set.

This commit adds support for this. Also, to make all that consistent, proxy settings for both plugins have been renamed:

* from `cloud.aws.proxy_host` to `cloud.aws.proxy.host`
* from `cloud.aws.ec2.proxy_host` to `cloud.aws.ec2.proxy.host`
* from `cloud.aws.s3.proxy_host` to `cloud.aws.s3.proxy.host`
* from `cloud.aws.proxy_port` to `cloud.aws.proxy.port`
* from `cloud.aws.ec2.proxy_port` to `cloud.aws.ec2.proxy.port`
* from `cloud.aws.s3.proxy_port` to `cloud.aws.s3.proxy.port`

New settings are `proxy.username` and `proxy.password`.

```yml
cloud:
    aws:
        protocol: https
        proxy:
            host: proxy1.company.com
            port: 8083
            username: myself
            password: theBestPasswordEver!
```

You can also set different proxies for `ec2` and `s3`:

```yml
cloud:
    aws:
        s3:
            proxy:
                host: proxy1.company.com
                port: 8083
                username: myself1
                password: theBestPasswordEver1!
        ec2:
            proxy:
                host: proxy2.company.com
                port: 8083
                username: myself2
                password: theBestPasswordEver2!
```

Note that `password` is filtered with `SettingsFilter`.

We also fix a potential issue in S3 repository. We were supposed to accept key/secret either set under `cloud.aws` or `cloud.aws.s3` but the actual code never implemented that.

It was:

```java
account = settings.get("cloud.aws.access_key");
key = settings.get("cloud.aws.secret_key");
```

We replaced that by:

```java
String account = settings.get(CLOUD_S3.KEY, settings.get(CLOUD_AWS.KEY));
String key = settings.get(CLOUD_S3.SECRET, settings.get(CLOUD_AWS.SECRET));
```

Also, we extract all settings for S3 in `AwsS3Service` as it's already the case for `AwsEc2Service` class.

Closes #15268.
2015-12-07 23:10:54 +01:00
Robert Muir 2169a123a5 Filter classes loaded by scripts
Since 2.2 we run all scripts with minimal privileges, similar to applets in your browser.
The problem is, they have unrestricted access to other things they can muck with (ES, JDK, whatever).
So they can still easily do tons of bad things

This PR restricts what classes scripts can load via the classloader mechanism, to make life more difficult.
The "standard" list was populated from the old list used for the groovy sandbox: though
a few more were needed for tests to pass (java.lang.String, java.util.Iterator, nothing scary there).

Additionally, each scripting engine typically needs permissions to some runtime stuff.
That is the downside of this "good old classloader" approach, but I like the transparency and simplicity,
and I don't want to waste my time with any feature provided by the engine itself for this, I don't trust them.

This is not perfect and the engines are not perfect but you gotta start somewhere. For expert users that
need to tweak the permissions, we already support that via the standard java security configuration files, the
specification is simple, supports wildcards, etc (though we do not use them ourselves).
2015-12-05 21:46:52 -05:00
Robert Muir 46377778a9 Merge branch 'master' into getClassLoader 2015-12-04 15:58:36 -05:00
Robert Muir b0c64910b0 ban RuntimePermission("getClassLoader")
this gives more isolation between modules and plugins.
2015-12-04 15:58:02 -05:00
Ryan Ernst 01d48e2062 Merge branch 'master' into jigsaw 2015-12-04 11:29:49 -08:00
David Pilato 619fb998e8 Update Azure Service Management API to 0.9.0
Azure team released new versions of their Java SDK.

According to https://github.com/Azure/azure-sdk-for-java/wiki/Azure-SDK-for-Java-Features, it comes with 2 versions.
We should at least update to `0.9.0` of V1 but also consider moving to the new APIs (V2).

This commit first updates to latest API V1.

```xml
<dependency>
    <groupId>com.microsoft.azure</groupId>
    <artifactId>azure-svc-mgmt-compute</artifactId>
    <version>0.9.0</version>
</dependency>
```

Closes #15209
2015-12-04 17:32:11 +01:00
Adrien Grand 3f86adddbf Remove MergeMappingException.
Failures to merge a mapping can either come as a MergeMappingException if they
come from Mapper.merge or as an IllegalArgumentException if they come from
FieldTypeLookup.checkCompatibility. I think we should settle on one: this pull
request replaces all usage of MergeMappingException with
IllegalArgumentException.
2015-12-04 12:56:26 +01:00
Ryan Ernst 0a4a81afaf Added modules, distributions now include them (just plugins installed in
a diff dir)
2015-12-03 14:18:26 -08:00
Jason Tedor fbe736c9bb Cleaner type-inference assistance 2015-12-02 10:49:35 -05:00
Jason Tedor 05430a788a Remove and forbid use of the type-unsafe empty Collections fields
This commit removes and now forbids all uses of the type-unsafe empty
Collections fields Collections#EMPTY_LIST, Collections#EMPTY_MAP, and
Collections#EMPTY_SET. The type-safe methods Collections#emptyList,
Collections#emptyMap, and Collections#emptySet should be used instead.
2015-12-02 10:41:59 -05:00
David Pilato d23d8a891f Remove "empty" licenses dir
Follow up #15168

We don't need to have "fake" licenses dir anymore.
2015-12-02 10:22:52 +01:00
Ryan Ernst d68c6673a2 Build: Cleanup precommit task gradle code
This change attempts to simplify the gradle tasks for precommit. One
major part of that is using a "less groovy style", as well as being more
consistent about how tasks are created and where they are configured. It
also allows the things creating the tasks to set up inter task
dependencies, instead of assuming them (ie decoupling from tasks
eleswhere in the build).
2015-12-01 22:36:54 -08:00
javanna c67a332486 Query DSL: Enforce distance is greater than 0 in geo distance query
Validation is not done as part of the distance setter method and tested in GeoDistanceQueryBuilderTests. Fixed GeoDistanceTests to adapt to the new validation.

Closes #15135
2015-12-01 14:07:32 +01:00
Britta Weber d8a1a4bd43 fix toXContent() for mapper attachments field
We must use simpleName() instead of name() because otherwise when the mapping
is generated as a string the field name will be the full path with dots
and that is illegal from es 2.0 on.

closes https://github.com/elastic/elasticsearch-mapper-attachments/issues/169
2015-11-30 15:28:12 +01:00
Jim Ferenczi e182072b6f Merge pull request #15017 from jimferenczi/fields_option
Refuse to load fields from _source when using the `fields` option and support wildcards.
2015-11-30 11:01:21 +01:00
Jim Ferenczi 731833cfc6 Fixes #14489
Do not to load fields from _source when using the `fields` option.
  Non stored (non existing) fields are ignored by the fields visitor when using the `fields` option.

Fixes #10783
  Support * wildcard to retrieve stored fields when using the `fields` option.
  Supported pattern styles are "xxx*", "*xxx", "*xxx*" and "xxx*yyy".
2015-11-30 11:00:32 +01:00
Robert Muir a2816ec574 don't assert exact expected lengths for documents.
This will change depending on newline of the operating system
2015-11-29 11:48:54 -05:00
Robert Muir 415c37340a do not assert charset for mapper-attachments tests.
Its enough to test the content type for what we are testing.

Currently tests are flaky if charset is detected as e.g. windows-1252 vs iso-8859-1 and so on.
In fact, they fail on windows 100% of the time.

We are not trying to test charset detection heuristics (which might be different even due to newlines in tests or other things).
If we want to do test that, we should test it separately.
2015-11-29 11:40:27 -05:00
Simon Willnauer 65b661b1f4 [TEST] Fix MapperUpgrade tests to use a dedicated master to ensure dangeling index import works predictably
When importing dangling indices on a single node that is data and master eligable the async dangling index
call can still be in-flight when the cluster is checked for green / yellow. Adding a dedicated master node
and a data only node that does the importing fixes this issus just like we do in OldIndexBackwardsCompatibilityIT
2015-11-27 10:32:21 +01:00
Simon Willnauer 9f6598b18d Fix compile errors 2015-11-26 13:41:00 +01:00
Adrien Grand e8520bf519 Tests: For single data path for *FieldMapperUpgradeTests. 2015-11-25 11:46:19 +01:00
Adrien Grand aad84395c9 Add a test that upgrades succeed even if a mapping contains fields that come from a plugin. 2015-11-24 19:14:19 +01:00
Adrien Grand 5f33fbdb75 Register field mappers at the node level.
This moves the registration of field mappers from the index level to the node
level and also ensures that mappers coming from plugins are treated no
differently from core mappers.
2015-11-24 08:59:37 +01:00
Michael McCandless e13b0d4bde upgrade lucene to 5.4.0-snapshot-1715952 2015-11-23 17:13:49 -05:00
David Pilato 5b0e2823b1 Merge branch 'docs/mapper-attachments' 2015-11-23 12:14:31 +01:00
Robert Muir 30529c008d Ban write access to system properties
* Forbid System.setProperties & co in forbidden APIs.
* Ban property write access at runtime with security manager.

Plugins that need to modify system properties will need to request permission in their plugin-security.policy
2015-11-21 22:33:06 -05:00
Ryan Ernst 88ee2e23a6 Merge branch 'master' into javadocs 2015-11-20 12:20:59 -08:00
markharwood c3a50d7ca2 Search: shift SearchTimeoutTests into core tests minus the Groovy dependency and renamed with IT test suffix 2015-11-20 18:50:17 +00:00
markharwood 5a5f05a0e9 Aggregations: Shift significant terms score test back into core without the Groovy dependency 2015-11-20 18:32:04 +00:00
Colin Goodheart-Smithe 5a18f740ba Aggregations: Moves SumTests out of lang-groovy and back into core
Renames SumTests to SumIT and puts it in the same package as the other aggregation tests. Also updates the tests to not require Groovy
2015-11-20 12:42:08 +00:00
Daniel Mitterdorfer c95c142829 Merge remote-tracking branch 'danielmitterdorfer/fix/ValueCountTests' 2015-11-20 13:21:15 +01:00
Daniel Mitterdorfer 7d6b0fc481 Move ValueCountTests to core
This script moves ValueCountTests from plugins back to core
by using a mock script engine instead of Groovy.
2015-11-20 13:19:55 +01:00
Adrien Grand 4602b8177b Move AvgTests back to core.
This makes AvgTests use a mock plugin engine. I also removed the
textScriptExplicit* methods for the base class since they only make sense for
a groovy script, not a mock script.
2015-11-20 11:58:38 +01:00
Ryan Ernst c3cb1fd08c Merge branch 'master' into javadocs 2015-11-19 10:43:43 -08:00
David Pilato 149279fbee Filter cloud azure credentials
Bug introduced in #13779: we don't filter anymore credentials because we were filtering `cloud.azure.storage.account` and `cloud.azure.storage.key` but now credentials are like `cloud.azure.storage.XXX.account` and `cloud.azure.storage.XXX.key` where `XXX` can be a storage setting id.

Closes #14843.
2015-11-19 15:40:56 +01:00
David Pilato 33b0e662cc Merge branch 'pr/phonetic_daitch_mokotoff' 2015-11-19 14:48:11 +01:00
David Pilato d48d8ef863 Merge branch 's3-storage-class-support' of https://github.com/socialrank/elasticsearch into socialrank-s3-storage-class-support
# Please enter a commit message to explain why this merge is necessary,
# especially if it merges an updated upstream into a topic branch.
#
# Lines starting with '#' will be ignored, and an empty message aborts
# the commit.
2015-11-19 13:10:08 +01:00
Nicholas Knize 0d349854d3 Add CONTAINS relation to geo_shape query
At the time of geo_shape query conception, CONTAINS was not yet a supported spatial operation in Lucene. Since it is now available this commit adds ShapeRelation.CONTAINS to GeoShapeQuery. Randomized testing is included and documentation is updated.
2015-11-18 14:15:35 -06:00
David Pilato 052285ac88 Revert 3cacec4468
We actually want to keep the test when using deprecated setting in 3.0.
We will keep this setting deprecated as well so people will be able to update in a smoother way.

Also add the deprecating information to the migration documentation.
2015-11-18 19:14:26 +01:00
David Pilato 3cacec4468 Fix test in Azure
Related to #13656.

We need to remove deprecated settings in another PR.
2015-11-18 17:14:10 +01:00
David Pilato dc17ab31d3 Merge remote-tracking branch 'origin/master' 2015-11-18 16:37:02 +01:00
Christoph Büscher 3e483b5626 Merge pull request #14587 from cbuescher/move-datehistogramtests-core
Move DateHistogramTests back to core module
2015-11-18 16:35:59 +01:00
craigwi 79a4d9ce36 Add support for secondary azure storage account
Follow up for #13228.

This commit adds support for a secondary storage account:

```yml
cloud:
    azure:
        storage:
            my_account1:
                account: your_azure_storage_account1
                key: your_azure_storage_key1
                default: true
            my_account2:
                account: your_azure_storage_account2
                key: your_azure_storage_key2
```

When creating a repository, you can choose which azure account you want to use for it:

```sh
curl -XPUT localhost:9200/_snapshot/my_backup1?pretty -d '{
  "type": "azure"
}'

curl -XPUT localhost:9200/_snapshot/my_backup2?pretty -d '{
  "type": "azure",
  "settings": {
    "account" : "my_account2",
    "location_mode": "secondary_only"
  }
}'
```

`location_mode` supports `primary_only` or `secondary_only`. Defaults to `primary_only`. Note that if you set it
to `secondary_only`, it will force `read_only` to true.
2015-11-18 16:31:48 +01:00
David Pilato 52bf365013 Add support for `daitch_mokotoff`
[Daitch Mokotoff](https://en.wikipedia.org/wiki/Daitch%E2%80%93Mokotoff_Soundex) support has been added in Lucene 5.
So we can now support it as well.
2015-11-18 15:41:39 +01:00
Michael McCandless a0bf253d16 upgrade lucene 5.4 snapshot 2015-11-16 14:38:05 -05:00
Michael McCandless 9d7ca53022 upgrade lucene 5.4 snapshot 2015-11-16 14:35:17 -05:00
Boaz Leskes fa55a940f7 Test: move SearchQueryIT back to core
There is no groovy dependency there (it's now in lang groovy)

Closes #14770
2015-11-16 14:43:36 +01:00
Christoph Büscher e3f9824931 Tests: Move DateHistogramTests back to core module
DateHistogramTests had some dependency on groovy scripts and
were moved to the lang-groovy module. This PR moves it back
and replaces use of groovy scripts by a mock script engine.
Removing three test cases that were testing doing some date
manipulation using script, since these are more groovy script
tests than testing the DateHistogram aggregation.
2015-11-16 11:58:28 +01:00
Boaz Leskes 18a75fb30c Test: wrong index and query in testConstantScoreQuery 2015-11-15 20:34:40 +01:00
Ryan Ernst 4b17492456 Build: Add javadocs jars
This change adds javadoc jars to core, test-framework and plugins. There
were a couple issues which javadoc found, but doclint did not already
find.
2015-11-15 01:44:42 -08:00
Colin Goodheart-Smithe 12bb1b79f6 Aggregations: Pass extended bounds into HistogramAggregator when creating an unmapped aggregator
This fixes an issue where if the field for the aggregation was unmapped the extended bounds would get dropped and the resulting buckets would not cover the extended bounds requested.

Closes #14735
2015-11-13 14:49:15 +00:00
Boaz Leskes ac0da91bf7 Extend usage of IndexSetting class
I decided to leave external listeners (used by plugins) alone, for now.

Closes #14731
2015-11-13 14:30:23 +01:00
Colin Goodheart-Smithe be64bfb135 Revert "Aggregations: Pass extended bounds into HistogramAggregator when creating an unmapped aggregator"
This reverts commit e1bf9798bf.
2015-11-13 12:48:51 +00:00
Colin Goodheart-Smithe e1bf9798bf Aggregations: Pass extended bounds into HistogramAggregator when creating an unmapped aggregator
This fixes an issue where if the field for the aggregation was unmapped the extended bounds would get dropped and the resulting buckets would not cover the extended bounds requested.

Closes #14735
2015-11-13 12:33:06 +00:00
Nicholas Knize dc77815744 Enable GeoPointV2 with backward compatibility testing
This commit removes all noreleases and cuts over to Lucene 5.4 GeoPointField type. Included are randomized testing updates to unit and integration test suites for ensuring full backward compatability with existing geo_point indexes.
2015-11-12 23:01:51 -06:00
Robert Muir 914cee213e Upgrade to aws 1.10.33
Security issues have been fixed. This removes our hacks.
2015-11-10 22:51:33 -05:00
Ryan Ernst 2cc89e8587 Fix concrete references to shared deps 2015-11-10 15:13:54 -08:00
Ryan Ernst 4b5f87cb7d Build: Remove transitive dependencies
Transitive dependencies can be confusing and hard to deal with when
conflicts arise between them. This change removes transitive
dependencies from elasticsearch, and forces any dependency conflicts to
be resolved manually, instead of automatically by gradle.

closes #14627
2015-11-10 15:01:41 -08:00
Tanguy Leroux 4063354dbe Fix Delete-by-Query with Shield
closes #14527
2015-11-10 20:39:38 +01:00
Ryan Ernst 7bb4e10353 Merge pull request #14645 from rjernst/version_cleanup
Consolidate dependencies specified in multiple places
2015-11-10 07:49:10 -08:00
David Pilato a61fcfd8ca Remove AbstractLegacyBlobContainer
`AbstractLegacyBlobContainer` was kept for historical reasons (see #13434).
We can migrate Azure and S3 repositories to use the new methods added in #13434 so we can remove `AbstractLegacyBlobContainer` class.
2015-11-10 16:04:44 +01:00
Ryan Ernst 7a6155e12f Build: Consolidate dependencies specified in multiple places
Some dependencies must be specified in a couple places in the build.
e.g. randomized runner is specified both in buildSrc (for the gradle
wrapper plugin), as well as in the test-framework.

This change creates buildSrc/versions.properties which acts similar to
the set of shared version properties we used to have in the maven parent
pom.
2015-11-09 23:59:56 -08:00
Adrien Grand d0853e9253 Merge pull request #14619 from jpountz/upgrade/lucene-5.4.0-snapshot-1712973
Upgrade to lucene-5.4.0-snapshot-1712973.
2015-11-09 22:38:44 +01:00
Robert Muir ac612208b7 minor attachments cleanups: IDE test support and EPUB format
The documentation says we support EPUB, but the parser is not enabled.
This parser does not require any external dependencies, so I think its ok?

Separately, test-framework drags in an ancient commons-codec (via httpclient), which gradle
"upgrades", but IDEs can't handle this case and just hit jar hell. So just wire that to 1.9,
this allows running tests in the IDE for this plugin.
2015-11-09 11:30:58 -05:00
Adrien Grand d6d7af0a6c Upgrade to lucene-5.4.0-snapshot-1712973. 2015-11-09 15:53:27 +01:00
David Pilato e993c6a862 Migrate mapper attachements plugin to asciidoc
Followup for #14605
2015-11-09 15:35:06 +01:00
Robert Muir e88896e40e apply a few post-merge cleanups 2015-11-08 02:31:26 -05:00
Robert Muir a164586796 Merge branch 'mapper_migrate' into migrate-mapper-attachments 2015-11-08 02:19:50 -05:00
Robert Muir 550f462c47 migrate mapper attachments 2015-11-08 02:09:36 -05:00
Ryan Ernst 8d78f970b0 Build: Enforce name is set for all plugins
The plugin name currently defaults to the gradle project name. But the
gradle project name for standalone repo (like an external plugin would
be) defaults to the directory name of the repo. This is trappy, since it
depends on how the repo was checked out.

This change enforces the plugin name is always set.

closes #14603
2015-11-07 16:57:11 -08:00
Robert Muir e06cae84f3 only allow code to bind to the user's configured port numbers/ranges
Random code shouldn't be listening on sockets elsewhere.

Today its the wild west, but we only need to grant access to what the user configured.

This means e.g. multicast plugin has to declare its intentions in its security.policy

Closes #14549
2015-11-07 12:22:12 -05:00
Christoph Büscher 40f82de4a1 Tests: make sure to load timezones in parent classloader
When GroovySecurityTests are run before any other test using
time zones, joda ZoneInfoProvider fails to load the time zones
correctly and never tries again later. This makes sure we load it
correctly on startup.

Relates to #14524
2015-11-06 11:11:28 +01:00
Simon Willnauer 94bed42213 Simplify plugin API and fix IndexService internal allocation 2015-11-05 13:16:35 +01:00
Simon Willnauer a311491c8e decouple IndexingMemoryController from IndexShard 2015-11-05 11:18:11 +01:00
Simon Willnauer 487af301ea Remove guice from the index level
This commit removes guice from the index level and adds a simple extension point
to add class instances with an index-lifecycle scope.
2015-11-05 11:18:11 +01:00
Robert Muir 79467a138a Add property permissions so groovy scripts can serialize json
Allowing read to these properties is not really dangerous, even if
the code surrounding them is.

Closes #14488
2015-11-03 15:21:55 -05:00
Robert Muir 7e6008f0b9 refactor GroovySecurityTests into a unit test.
This was basically a resurrected form of the tests for the old sandbox.
We use it to check that groovy scripts some degree of additional containment.

The other scripting plugins (javascript, python) already have this as a unit test,
its much easier to debug any problems that way.

closes #14484
2015-11-03 13:39:17 -05:00
Robert Muir b6a2318563 upgrade rhino for plugins/lang-javascript
the current jar is over 3 years old, we should upgrade it for bugfixes.
the current integration could be more secure: set a global policy and enforce additional (compile-time) checks

closes #14466
2015-11-03 09:53:07 -05:00
David Pilato 7bbd2a1513 Fix test for ec2 discovery
See #14155
2015-11-02 15:33:36 +01:00
David Pilato 77521560ed Merge branch 'ec2-improve-disco-nodes' of https://github.com/chaudum/elasticsearch into chaudum-ec2-improve-disco-nodes 2015-11-02 14:52:08 +01:00
David Pilato 1b98e1b1ad Adding US-Gov-West for S3
Follow up for #14358
2015-11-02 14:38:03 +01:00
David Pilato bcc372a29e Merge branch 'aws-gov-west' of https://github.com/amos6224/elasticsearch-1 into amos6224-aws-gov-west 2015-11-02 14:32:23 +01:00
Robert Muir 3961c9b58b fix eclipse (again) 2015-10-31 02:39:35 -04:00
Ryan Ernst b6dee6bd43 Merge pull request #14375 from rjernst/sweep_up_maven
Remove maven pom files and supporting ant files
2015-10-30 18:59:11 -07:00
Areek Zillur 13b60e1b92 update to lucene-5.4.x-snapshot-1711508 2015-10-30 15:42:02 -04:00
Nik Everett 28633fae21 [mapping] Remove transform
Removes the mapping transform feature which when used made debugging very
difficult. Users should transform their documents on the way into
Elasticsearch rather than having Elasticsearch do it.

Closes #12674
2015-10-30 11:46:54 -04:00
Simon Willnauer aa38d053d7 Simplify Analysis registration and configuration
This change moves all the analysis component registration to the node level
and removes the significant API overhead to register tokenfilter, tokenizer,
charfilter and analyzer. All registration is done without guice interaction such
that real factories via functional interfaces are passed instead of class objects
that are instantiated at runtime.

This change also hides the internal analyzer caching that was done previously in the
IndicesAnalysisService entirely and decouples all analysis registration and creation
from dependency injection.
2015-10-30 11:40:18 +01:00
Ryan Ernst 542522531a Build: Remove maven pom files and supporting ant files
This change removes the leftover pom files. A couple files were left for
reference, namely in qa tests that have not yet been migrated (vagrant
and multinode). The deb and rpm assemblies also still exist for
reference when finishing their setup in gradle.

See #13930
2015-10-29 23:53:49 -07:00
xuzha b3090d654a Add AWS Canned ACL support
User could set cannedACL, S3 repository would use this cannedACL to
create s3 object and bucket.
2015-10-29 11:48:02 -07:00
Ryan Ernst c86100f636 Switch build system to Gradle
See #13930
2015-10-29 11:40:19 -07:00
Robert Muir 1194cd0bdc Add threadgroup isolation.
Closes #14353

Squashed commit of the following:

commit edae0729f71ea3d3f9fa9c0d27c9effc042eb5a9
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Oct 29 14:13:42 2015 -0400

    update sha1 and simplify test

commit 635c4f245d66ad353a16267c810e02b725553fad
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Oct 29 07:01:26 2015 -0400

    Add threadgroup isolation.

    Code with `modifyThread` and `modifyThreadGroup` may only modify
    its own threadgroup (or an ancestor of that). This enforces
    what is intended by the ThreadGroup class.

    This has two immediate implications:
    1. Code without these permissions (scripts) may not create or mess with threads
    2. ES application threads cannot mess with Java system threads

    ES puts all application threads in one single group today, but in the future
    this can be organized better, and we will have more isolation in the system.
2015-10-29 14:14:27 -04:00
Jeff Destine a8268ba37d Adding US-Gov-West 2015-10-29 13:09:11 -04:00
Christian Haudum 3334160003 improved building of disco nodes
* improved retry policy of ec2 client
* cache results for 10s
2015-10-29 18:07:52 +01:00
Nicholas Knize 8c535e0f6e Refactor Geo utilities to Lucene 5.4
Remove local lucene/XGeo* classes and refactor to Lucene 5.4 Geo Utility classes.
2015-10-28 11:41:56 -05:00
Adrien Grand 2ed02f9eb3 Merge pull request #14320 from jpountz/upgrade/lucene-5.4-snapshot-1710880
Upgrade to lucene-5.4-snapshot-1710880.
2015-10-28 14:45:45 +01:00
javanna ea750de39f Explain api: move query parsing to the coordinating node
Similarly to what we did with the search api, we can now also move query parsing on the coordinating node for the explain api. Given that the explain api is a single shard operation (compared to search which is instead a broadcast operation), this doesn't change a lot in how the api works internally. The main benefit is that we can simplify the java api by requiring a structured query object to be provided rather than a bytes array that will get parsed on the data node. Previously if you specified a QueryBuilder it would be serialized in json format and would get reparsed on the data node, while now it doesn't go through parsing anymore (as expected), given that after the query-refactoring we are able to properly stream queries natively.

Closes #14270
2015-10-28 10:42:03 +01:00
Adrien Grand 43958db10b Upgrade to lucene-5.4-snapshot-1710880. 2015-10-28 09:34:54 +01:00
Simon Willnauer 935a8fc3d4 Merge pull request #14306 from s1monw/no_lenient_on_module
Don't be lenient in PluginService#processModule(Module)
2015-10-27 16:52:21 +01:00
Simon Willnauer 6fbfdb0e12 remove broken Stream registration API 2015-10-27 16:25:58 +01:00
javanna dc900a08a6 Remove "query" query and fix related parsing bugs
We have two types of parse methods for queries: one for the inner query, to be used once the parser is positioned within the query element, and one for the whole query source, including the query element that wraps the actual query.

With the search refactoring we ended up using the former in count, cat count and delete by query, whereas we should have used the former.  It ends up working properly given that we have a registered (deprecated) query called "query", which used to allow to wrap a filter into a query, but this has the following downsides:
1) prevents us from removing the deprecated "query" query
2) we end up supporting a top level query that is not wrapped within a query element (pre 1.0 syntax iirc that shouldn't be supported anymore)

This commit finally removes the "query" query and fixes the related parsing bugs. We also had some tests that were providing queries in the wrong format, those have been fixed too.

Closes #13326
Closes #14304
2015-10-27 14:54:30 +01:00
Simon Willnauer a5e5a5025b Fold IndexCacheModule into IndexModule
This commit brings all the registration etc. from IndexCacheModule into
IndexModule. As a side-effect to remove a circular dependency between
IndicesService and IndicesWarmer this commit also cleans up IndicesWarmer and
separates the Engine from the warmer.
2015-10-27 11:21:30 +01:00
Simon Willnauer cc3532f593 Remove circular dependency between IndicesService and IndicesStore
This commit introduces a new IndexStoreConfig that is passed to
IndexStore instances instead it's pretty messy parent service.
2015-10-26 20:56:01 +01:00
Simon Willnauer cdcc5da756 Remove unused import 2015-10-26 14:23:28 +01:00
Simon Willnauer 5ffdf15f02 Remove guice injection from IndexStore and friends
This commit replaces dependency injection from IndexStore and subclasses
and replaces it with dedicated set of dependencies.
2015-10-26 12:32:34 +01:00
Simon Willnauer beac4b17be Merge pull request #14251 from s1monw/trash_index_settings
Replace IndexSettings annotation with a full-fledged class
2015-10-24 15:38:46 +02:00
Robert Muir 6c8e290322 Allow binding to multiple addresses
* Allow for multiple host specifications (e.g. _en0_,192.168.1.2,_site_).
* Add _site_ and _global_ scopes as counterparts to _local_.
* Warn on heuristic selection of publish address.
* Remove the arbitrary _non_loopback_ setting.

Closes #13954
2015-10-23 23:43:37 -04:00
Simon Willnauer 8a9dd871d3 Make IndexSettings also own the IndexMetaData and separate node settings 2015-10-23 10:53:39 +02:00
Simon Willnauer 66d5d0c4f2 Replace IndexSettings annotation with a full-fledged class
The @IndexSettings annoationat has been used to differentiate between node-level
and index level settings. It was also decoupled from realtime-updates such that
the settings object that a class got injected when it was created was static and
not subject to change when an update was applied. This change removes the annoation
and replaces it with a full-fledged class that adds type-safety and encapsulates additional
functionality as well as checks on the settings.
2015-10-22 20:43:41 +02:00
javanna 75cedca0da Remove search exists api
Closes #13682
Closes #13911
2015-10-21 17:39:32 +02:00
Adrien Grand 76231c89da Remove "uninverted" and "binary" fielddata support for numeric and boolean fields.
Numeric and boolean fields have doc values enabled by default as of
elasticsearch 2.0. This commit removes support for uninverted/in-memory
fielddata, as well as numeric fields encoded in binary doc values which was
the way that elasticsearch stored doc values in a Lucene index before the
1.4 release.

As a consequence, you will only be able to sort and aggregate on numeric and
boolean fields in Elasticsearch 3.0 if doc values have not been switched off.
2015-10-21 12:15:40 +02:00
Lee Hinman b3646f9bfc Merge remote-tracking branch 'dakrone/use-the-force-luke' 2015-10-20 21:13:18 -06:00
Nik Everett 2cc97a0d3e Remove and ban @Test
There are three ways `@Test` was used. Way one:

```java
@Test
public void flubTheBlort() {
```

This way was always replaced with:

```java
public void testFlubTheBlort() {
```

Or, maybe with a better method name if I was feeling generous.

Way two:

```java
@Test(throws=IllegalArgumentException.class)
public void testFoo() {
    methodThatThrows();
}
```

This way of using `@Test` is actually pretty OK, but to get the tools to ban
`@Test` entirely it can't be used. Instead:

```java
public void testFoo() {
    try {
        methodThatThrows();
        fail("Expected IllegalArgumentException");
    } catch (IllegalArgumentException e ) {
        assertThat(e.getMessage(), containsString("something"));
    }
}
```

This is longer but tests more than the old ways and is much more precise.
Compare:

```java
@Test(throws=IllegalArgumentException.class)
public void testFoo() {
    some();
    copy();
    and();
    pasted();
    methodThatThrows();
    code();  // <---- This was left here by mistake and is never called
}
```

to:

```java
@Test(throws=IllegalArgumentException.class)
public void testFoo() {
    some();
    copy();
    and();
    pasted();
    try {
        methodThatThrows();
        fail("Expected IllegalArgumentException");
    } catch (IllegalArgumentException e ) {
        assertThat(e.getMessage(), containsString("something"));
    }
}
```

The final use of test is:

```java
@Test(timeout=1000)
public void testFoo() {
    methodThatWasSlow();
}
```

This is the most insidious use of `@Test` because its tempting but tragically
flawed. Its flaws are:
1. Hard and fast timeouts can look like they are asserting that something is
faster and even do an ok job of it when you compare the timings on the same
machine but as soon as you take them to another machine they start to be
invalid. On a slow VM both the new and old methods fail. On a super-fast
machine the slower and faster ways succeed.
2. Tests often contain slow `assert` calls so the performance of tests isn't
sure to predict the performance of non-test code.
3. These timeouts are rude to debuggers because the test just drops out from
under it after the timeout.

Confusingly, timeouts are useful in tests because it'd be rude for a broken
test to cause CI to abort the whole build after it hits a global timeout. But
those timeouts should be very very long "backstop" timeouts and aren't useful
assertions about speed.

For all its flaws `@Test(timeout=1000)` doesn't have a good replacement __in__
__tests__. Nightly benchmarks like http://benchmarks.elasticsearch.org/ are
useful here because they run on the same machine but they aren't quick to check
and it takes lots of time to figure out the regressions. Sometimes its useful
to compare dueling implementations but that requires keeping both
implementations around. All and all we don't have a satisfactory answer to the
question "what do you replace `@Test(timeout=1000)`" with. So we handle each
occurrence on a case by case basis.

For files with `@Test` this also:
1. Removes excess blank lines. They don't help anything.
2. Removes underscores from method names. Those would fail any code style
checks we ever care to run and don't add to readability. Since I did this manually
I didn't do it consistently.
3. Make sure all test method names start with `test`. Some used to end in `Test` or start
with `verify` or `check` and they were picked up using the annotation. Without the
annotation they always need to start with `test`.
4. Organizes imports using the rules we generate for Eclipse. For the most part
this just removes `*` imports which is a win all on its own. It was "required"
to quickly remove `@Test`.
5. Removes unneeded casts. This is just a setting I have enabled in Eclipse and
forgot to turn off before I did this work. It probably isn't hurting anything.
6. Removes trailing whitespace. Again, another Eclipse setting I forgot to turn
off that doesn't hurt anything. Hopefully.
7. Swaps some tests override superclass tests to make them empty with
`assumeTrue` so that the reasoning for the skips is logged in the test run and
it doesn't "look like" that thing is being tested when it isn't.
8. Adds an oxford comma to an error message.

The total test count doesn't change. I know. I counted.
```bash
git checkout master && mvn clean && mvn install | tee with_test
git no_test_annotation master && mvn clean && mvn install | tee not_test
grep 'Tests summary' with_test > with_test_summary
grep 'Tests summary' not_test > not_test_summary
diff with_test_summary not_test_summary
```

These differ somewhat because some tests are skipped based on the random seed.
The total shouldn't differ. But it does!
```
1c1
< [INFO] Tests summary: 564 suites (1 ignored), 3171 tests, 31 ignored (31 assumptions)
---
> [INFO] Tests summary: 564 suites (1 ignored), 3167 tests, 17 ignored (17 assumptions)
```

These are the core unit tests. So we dig further:
```bash
cat with_test | perl -pe 's/\n// if /^Suite/;s/.*\n// if /IGNOR/;s/.*\n// if /Assumption #/;s/.*\n// if /HEARTBEAT/;s/Completed .+?,//' | grep Suite > with_test_suites
cat not_test | perl -pe 's/\n// if /^Suite/;s/.*\n// if /IGNOR/;s/.*\n// if /Assumption #/;s/.*\n// if /HEARTBEAT/;s/Completed .+?,//' | grep Suite > not_test_suites
diff <(sort with_test_suites) <(sort not_test_suites)
```

The four tests with lower test numbers are all extend `AbstractQueryTestCase`
and all have a method that looks like this:

```java
@Override
public void testToQuery() throws IOException {
    assumeTrue("test runs only when at least a type is registered", getCurrentTypes().length > 0);
    super.testToQuery();
}
```

It looks like this method was being double counted on master and isn't anymore.

Closes #14028
2015-10-20 17:37:36 -04:00
Nik Everett 2eff063341 Merge pull request #14215 from nik9000/retry_gce_test_to_estestcase
Convert GCE test to ESTestCase
2015-10-20 17:21:25 -04:00
Nik Everett 6074f631d0 [test] Convert GCE test to ESTestCase
And work around GCE's annoying security issues that that exposed.

Required for #14069
2015-10-20 14:55:34 -04:00
Christoph Büscher 5d25bc30cd Query DSL: Remove NotQueryBuilder
The NotQueryBuilder has been deprecated on the 2.x branches
and can be removed with the next major version. It can be
replaced by boolean query with added mustNot() clause.

Closes #13761
2015-10-20 19:43:16 +02:00
Lee Hinman 9ea4909035 Add Force Merge API, deprecate Optimize API
This adds an API for force merging lucene segments. The `/_optimize` API is now
deprecated and replaced by the `/_forcemerge` API, which has all the same flags
and action, just a different name.
2015-10-20 09:00:24 -06:00
Jason Tedor 1b461a7f9d Remove mistakenly committed build output files
This commit removes some build output files from the
burn_maven_with_fire_branch that appear to have been mistakenly
committed to master in bfb9054a11.
2015-10-20 08:34:51 -04:00
xuzha 1ae524bace Fix test bug to match windows /r 2015-10-19 12:41:52 -07:00
xuzha 1069ad9fad fix a test bug 2015-10-19 12:06:20 -07:00
xuzha 6a6e168b8b Adding backoff from retries on GCE errors
In case of any error while trying to get GCE instances list from GCE
API, elasticsearch will slow down its API calls.
2015-10-19 10:28:08 -07:00
javanna a6e7a5f307 Java api: remove the count api
Closes #14166
Closes #13928
2015-10-19 14:40:52 +02:00
Adrien Grand 5ae810991c Upgrade to lucene-5.4-snapshot-1708254. 2015-10-16 09:41:36 +02:00
Colin Goodheart-Smithe 63c51b78b2 review comment fixes 2015-10-15 11:34:31 +01:00
Colin Goodheart-Smithe c618f75b76 Merge branch 'master' into feature/search-request-refactoring
# Conflicts:
#	core/src/main/java/org/elasticsearch/search/SearchService.java
2015-10-15 11:02:34 +01:00
Robert Muir 5d001d1578 Decentralize plugin security
* Add ability for plugins to declare additional permissions with a custom plugin-security.policy file and corresponding AccessController logic. See the plugin author's guide for more information.
* Add warning messages to users for extra plugin permissions in bin/plugin.
* When bin/plugin is run interactively (stdin is a controlling terminal and -b/--batch not supplied), require user confirmation.
* Improve unit test and IDE support for plugins with additional permissions by exposing plugin's metadata as a maven test resource.

Closes #14108

Squashed commit of the following:

commit cf8ace65a7397aaccd356bf55f95d6fbb8bb571c
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Oct 14 13:36:05 2015 -0400

    fix new unit test from master merge

commit 9be3c5aa38f2d9ae50f3d54924a30ad9cddeeb65
Merge: 2f168b8 7368231
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Oct 14 12:58:31 2015 -0400

    Merge branch 'master' into off_my_back

commit 2f168b8038e32672f01ad0279fb5db77ba902ae8
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Oct 14 12:56:04 2015 -0400

    improve plugin author documentation

commit 6e6c2bfda68a418d92733ac22a58eec35508b2d0
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Oct 14 12:52:14 2015 -0400

    move security confirmation after 'plugin already installed' check, to prevent user from answering unnecessary questions.

commit 08233a2972554afef2a6a7521990283102e20d92
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Oct 14 05:36:42 2015 -0400

    Add documentation and pluginmanager support

commit 05dad86c51488ba43ccbd749f0164f3fbd3aee62
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Oct 14 02:22:24 2015 -0400

    Decentralize plugin permissions (modulo docs and pluginmanager work)
2015-10-14 14:46:45 -04:00
Colin Goodheart-Smithe 6819224d2c Merge branch 'master' into feature/search-request-refactoring
# Conflicts:
#	core/src/main/java/org/elasticsearch/search/SearchService.java
#	core/src/test/java/org/elasticsearch/cluster/ClusterStateDiffIT.java
#	core/src/test/java/org/elasticsearch/search/fetch/FetchSubPhasePluginIT.java
2015-10-12 12:54:14 +01:00
Jason Tedor 9a9a6a4b3b Remove Guava as a dependency
This commit removes Guava as a dependency. Note that Guava will remain
as a test-only dependency (transitively through Jimfs).

Closes #13224
2015-10-09 14:19:22 -04:00
Nik Everett d9e11e4b39 Merge branch 'master' into immutable_map_be_gone 2015-10-09 12:04:03 -04:00
Nik Everett bfb9054a11 Remove addAll(Collection, Iterable<ObjectCursor>)
It was used just one time and didn't add any value.
2015-10-09 10:12:54 -04:00
Adrien Grand 7400cbe5fd Remove UpdateTests' dependency on groovy.
This test had to be moved to lang-groovy when groovy has been made a plugin.
I refactored it a bit to use mock plugins instead so that groovy is not
necessary anymore and it can come back to core.
2015-10-08 16:19:00 +02:00
Colin Goodheart-Smithe 4557d1b560 Merge branch 'master' into feature/search-request-refactoring
# Conflicts:
#	plugins/lang-groovy/src/test/java/org/elasticsearch/messy/tests/FunctionScoreTests.java
2015-10-08 11:39:34 +01:00
David Pilato c73ab50df1 Rename cloud-gce plugin to discovery-gce plugin
Follow up azure and aws splits, we need to be consistent and rename `cloud-gce` to `discovery-gce`.
2015-10-08 06:53:37 +02:00
David Pilato 289cd5dcf4 [discovery-gce] add _gce_ network host setting
When running in GCE platform, an instance has access to:

http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/ip

Which gives back the private IP address, for example `10.240.0.2`.

http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/externalIp

Gives back the public Ip address, for example `130.211.108.21`.

As we have for `ec2`, we can support new network host settings:

* `_gce:privateIp:X_`: The private IP address of the machine for a given network interface.
* `_gce:hostname_`: The hostname of the machine.
* `_gce_`: Same as `_gce:privateIp:0_` (recommended).

Closes #13605.
Closes #13590.

BTW resolveIfPossible now throws IOException so code is also updated for ec2 discovery and
some basic tests have been added.
2015-10-07 22:04:34 +02:00
Colin Goodheart-Smithe ee5d2847b5 more clean up 2015-10-07 15:34:21 +01:00
Colin Goodheart-Smithe 5b23d9abdc more test clean ups 2015-10-07 15:29:13 +01:00
Britta Weber 871df97e18 Merge pull request #13961 from brwe/function-score-unit-tests
Convert some messy function score tests to unit tests
2015-10-07 14:51:22 +02:00