Commit Graph

1765 Commits

Author SHA1 Message Date
Robert Muir 3e9daa1040 Merge pull request #13324 from rmuir/lucene_1701068
Upgrade master to lucene 5.4-snapshot r1701068
2015-09-03 15:28:52 -04:00
javanna ea83007c7e [TEST] adapt FilteredQueryBuilderTests to upstream changes 2015-09-03 21:18:24 +02:00
Robert Muir f216d92d19 Upgrade to lucene 5.4-snapshot r1701068 2015-09-03 15:13:33 -04:00
javanna 8bd7cdbdad Merge branch 'master' into feature/query-refactoring
Conflicts:
	core/src/main/java/org/elasticsearch/index/query/FilteredQueryParser.java
2015-09-03 20:56:06 +02:00
javanna da554fc30c Query DSL: match_phrase_prefix to take boost into account
The match_phrase_prefix query properly parses the boost etc. but it loses it in its rewrite method. Fixed that by setting the orginal boost to the rewritten query before returning it. Also cleaned up some warning in MultiPhrasePrefixQuery.

Closes #13129
Closes #13142
2015-09-03 19:46:42 +02:00
Jason Tedor 5c4b864a42 Workaround pitfall in Java 8 target-type inference
Target-type inference has been improved in Java 8. This leads to these
lines now being interpreted as invoking String#valueOf(char[]) whereas
they previously were interpreted as invoking String#valueOf(Object).
This change leads to ClassCastExceptions during test execution. Simply
casting the parameter to Object restores the old invocation.

Closes #13315
2015-09-03 13:10:01 -04:00
Robert Muir 11314336b6 bump master (3.0-snapshot) to java 8
Closes #13314

Squashed commit of the following:

commit 0e2d6c8e55e07957d0bea08f3c80f51f481ba6d4
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Sep 3 12:49:51 2015 -0400

    re-enable the rest of this test, as its one method that hangs

commit 04dc6b62d4800f574babcdbbf0e44f5c3400e0f0
Merge: 2b12805 52945b2
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Sep 3 12:48:55 2015 -0400

    Merge branch 'master' into java8

commit 2b128056ded741cddaf1d92c55920d7baf80dc95
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Sep 3 12:28:07 2015 -0400

    more jsr166e removal

commit 5826feba484389a87dbcad9a53ee5bf9a306eb61
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Sep 3 12:15:13 2015 -0400

    disable broken test

commit aabd88b85f2dd7b9b2b9308ffd52bc71aef8ab82
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Sep 3 12:11:12 2015 -0400

    Disable broken test

commit 863463303a12415dfecd6a46a6767eb07a2c7415
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Sep 3 11:38:32 2015 -0400

    bump master (3.0-snapshot) to java 8
2015-09-03 12:55:21 -04:00
Britta Weber 52945b2b46 [test] mute testReadOnPostRecoveryShards 2015-09-03 18:27:09 +02:00
javanna 6e00a613ee Internal: simplify filtered query conversion to lucene query
We have some optimization in FilteredQueryParser that tries to mimic what the rewrite method in lucene does, based on what gets parsed we return the simplest query possible. That might cause issues with boost values though, if specified in both the main query and the inner query that we shortcut to. We should rather rely on lucene's rewrite method to simplify the lucene representation of the query, and always build a filtered query instead.

relates to #13272
Closes #13312
2015-09-03 17:13:00 +02:00
Lee Hinman ffb2f75305 Remove DisableAllocationDecider
The `EnableAllocatiorDecider` has replaced the
`DisableAllocationDecider`, which has been deprecated. It can now be
removed.
2015-09-03 09:10:48 -06:00
Adrien Grand 745b977ce7 Optimize scrolls for constant-score queries.
We currently optimize scroll when sort=_doc because docs are returned in order.
But documents are also returned in order when sorting by score and the query
gives constant scores. This optimization has the nice side-effect of also
optimizing scrolls with the default `match_all` query.
2015-09-03 16:12:04 +02:00
Robert Muir 4b0182932a Merge pull request #13299 from rmuir/network_config_improvements
Improve situation when network.host is set to wildcard (e.g. 0.0.0.0)
2015-09-03 08:18:46 -04:00
javanna 107a4b14e0 Merge branch 'master' into feature/query-refactoring 2015-09-03 14:09:39 +02:00
javanna 4559db9cd9 [TEST] renamed tests according to updated naming conventions (Tests suffix enforced) 2015-09-03 13:30:07 +02:00
javanna fa93cc7b89 Merge branch 'master' into feature/query-refactoring
Conflicts:
	core/src/test/java/org/elasticsearch/ExceptionSerializationTests.java
2015-09-03 13:02:49 +02:00
David Pilato ac9682dd00 [test] cloud-aws has been replaced by repository-s3 and discovery-ec2 2015-09-03 12:52:02 +02:00
David Pilato a117fc85a6 Update help install for discovery-ec2 and repository-s3 2015-09-03 11:29:45 +02:00
David Pilato 2931687b46 Merge remote-tracking branch 'origin/master' 2015-09-03 11:13:19 +02:00
David Pilato 30aa231f8e [plugin] split cloud-aws in repository-s3 and discovery-ec2
Until now we had a cloud-aws plugin which is providing 2 disctinct features:

* discovery on EC2
* snapshot/restore on S3

This commit splits the plugin by feature so people can use either one or the other or both features.

Doc is updated accordingly.
2015-09-03 11:12:20 +02:00
Colin Goodheart-Smithe ddd064ce1f [TESTS] Changed message in NamingConventionTests to reflect code checks
Before this change the check would check that all test classes end in Tests but the message would say they need to end in Test or Tests which was confusing.
2015-09-03 10:08:25 +01:00
Simon Willnauer 796701d52e Move version to 3.0.0-SNAPSHOT 2015-09-03 10:43:28 +02:00
Adrien Grand 5ff9ca9965 Optimize counts on simple queries.
Today we always collect in order to compute counts, but some of them can be
easily optimized by using pre-computed index statistics. This is especially
true in the case that there are no deletions, which should be common for the
time-based data use-case.

Counts on match_all queries can always be optimized, so requests like

```
GET index/_search?size=0

GET index/_search
{
  "size": 0,
  "query" : {
    "match_all": {}
  }
}
```

should now return almost instantly. Additionally, when there are no deletions,
term queries are also optimized, so the below queries which all boil down to a
single term query would also return almost immediately:

```
GET index/type/_search?size=0

GET index/_search
{
  "size": 0,
  "query" : {
    "match": {
      "foo": "bar"
    }
  }
}

GET index/_search
{
  "size": 0,
  "query" : {
    "constant_score": {
      "filter": {
        "exists": {
          "field": "foo"
        }
      }
    }
  }
}
```
2015-09-03 09:36:51 +02:00
Adrien Grand d85cbd9f37 Merge pull request #13295 from sksamuel/patch-1
Updated param name to match type
2015-09-03 09:23:20 +02:00
Robert Muir 6bdf89c47e Improve situation when network.host is set to wildcard (e.g. 0.0.0.0)
Users might specify something like -Des.network.host=0.0.0.0, as that
was the old default with previous versions of elasticsearch. This means
to bind to all interfaces, but it makes no sense as a publish address.

Pick a good one in this case, just like we do in other cases where
publish isn't explicitly specified and we are bound to multiple (e.g.
when configured by interface, or dns hostname with multiple addresses).
However, in this case warn the user about it: since its arbitrarily
picking the first non-loopback address like the old versions
did, thats a little too heuristical, but lets make the cutover easy.

Separately, fail hard if things like multicast or broadcast addresses are
configured as bind or publish addresses, as that is simply invalid.

Closes #13274
2015-09-02 23:03:31 -04:00
Igor Motov e288c9b0ce Tests: set the upper bound on the number of simulated failures in a restore test
The number and distribution of errors in some restore test may cause restore process to continue to fail for a prolong time. This test caps the total number of simulated failures to make sure that restore is guaranteed to eventually succeed after a limited number of retries.
2015-09-02 19:13:45 -04:00
Ryan Ernst f202ecdc84 Merge pull request #13294 from rjernst/no_test_exclusions3
Rename test suffix so we only use "Tests"
2015-09-02 15:19:17 -07:00
Stephen Samuel 8057f71c1d Updated param name to match type 2015-09-02 23:07:17 +01:00
Ryan Ernst 06d5e24548 Tests: Rename test suffix so we only use "Tests"
We currently have a small number of test classes with the suffix "Test",
yet most use the suffix "Tests". This change renames all the "Test"
classes, so that we have a simple rule: "Non-inner classes ending with
Tests".
2015-09-02 14:38:43 -07:00
Jason Tedor fe46227db8 Remove sole usgae of com.google.common.annotations.Beta 2015-09-02 17:31:11 -04:00
Simon Willnauer d10f80aea0 Merge pull request #13203 from kiryam/master
Add listeners for postIndex, postCreate, and postDelete
2015-09-02 22:52:29 +02:00
Ryan Ernst 0caa473353 Remove benchmark/stress too 2015-09-02 13:44:42 -07:00
Ryan Ernst 425e4c3492 Tests: Remove stress tests
These are not actually tests, but command line applications that must be
run manually. This change removes the entire stresstest package. We can
add back individual tests that we find necessary, and make them real
tests (whether integ or not).
2015-09-02 13:07:15 -07:00
Ivannikov Kirill 92a6968f2a Catch exception in callback of exception 2015-09-02 23:56:37 +05:00
Ryan Ernst e3304a19af Merge pull request #13282 from rjernst/no_test_exclusions
Remove test class exclusion for Abstract prefix and rename classes accordingly
2015-09-02 11:13:30 -07:00
Ryan Ernst 205675ad19 Addressed PR comments 2015-09-02 11:13:05 -07:00
Ryan Ernst 51db6cc358 Tests: Remove test class exclusion for Abstract prefix and rename classes accordingly
While the list of having exclusions is small, it shouldn't be necessary
at all. Base test cases should be suffixed with TestCase so they are not
picked up by the test class name pattern. This same rule works for
abstract classes as well.

This change renames abstract tests to use the TestCase suffix, adds a
check in naming convention tests, and removes the exclusion from our
test runner configuration. It also excludes inner classes (the only
exclude we should have IMO), so that we have no need to @Ignore the
inner test classes for naming convention tests.
2015-09-02 11:00:17 -07:00
Britta Weber 3d5128db07 Merge pull request #13268 from brwe/test-start-end
[test] print test start and end of test setup and cleanup
2015-09-02 19:44:40 +02:00
Britta Weber d47857b66f [test] don't call optimize while shard is relocating
In this test we assume that after waitForRelocation() has returned shards
are no more relocated and optimize will therefore succeed always.
However, because the test does not wait for green status, relocations can
still start after waitForRelocation() has returned successfully.

see #13266 for a detailed explanation
2015-09-02 19:15:46 +02:00
javanna 52be313c69 Merge branch 'master' into feature/query-refactoring
Conflicts:
	core/src/main/java/org/elasticsearch/index/query/BoolQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/IdsQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/SimpleQueryStringParser.java
	core/src/main/java/org/elasticsearch/index/query/TemplateQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/TermsQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/functionscore/fieldvaluefactor/FieldValueFactorFunctionParser.java
	core/src/main/java/org/elasticsearch/search/aggregations/bucket/significant/heuristics/GND.java
	core/src/main/java/org/elasticsearch/search/aggregations/bucket/significant/heuristics/JLHScore.java
	core/src/main/java/org/elasticsearch/search/aggregations/bucket/significant/heuristics/NXYSignificanceHeuristic.java
	core/src/main/java/org/elasticsearch/search/aggregations/bucket/significant/heuristics/PercentageScore.java
	core/src/main/java/org/elasticsearch/search/aggregations/bucket/significant/heuristics/ScriptHeuristic.java
	core/src/test/java/org/elasticsearch/search/aggregations/bucket/SignificantTermsSignificanceScoreIT.java
2015-09-02 16:00:35 +02:00
Ivannikov Kirill 0d2675e80b fix typo 2015-09-02 18:22:15 +05:00
Ivannikov Kirill cddf198c83 postCreateUnderLock try catch listeners 2015-09-02 18:21:17 +05:00
Ivannikov Kirill 39fad8c374 try catch in post listeners 2015-09-02 18:18:49 +05:00
Ivannikov Kirill 30b6a29e0b remove try in posIndex 2015-09-02 18:04:26 +05:00
Ivannikov Kirill 32b304b43c fix skipped variable setting 2015-09-02 18:02:26 +05:00
Ivannikov Kirill a097375a6f strict listeners 2015-09-02 17:59:41 +05:00
Colin Goodheart-Smithe 821021f0e4 Scripting: Propagate Headers and Context through to ScriptService
At the moment if an index script is used in a request, the spawned request to get the indexed script from the `.scripts` index does not get the headers and context copied to it from the original request. This change makes the calls to the `ScriptService` pass in a `HasContextAndHeaders` object that can provide the headers and context. For the `search()` method the context and headers are retrieved from `SearchContext.current()`.

Closes #12891
2015-09-02 13:49:20 +01:00
Ivannikov Kirill fd8476dbd7 make more strict 2015-09-02 17:48:37 +05:00
Ivannikov Kirill d33f7dba61 assertNotNull 2015-09-02 16:59:49 +05:00
Ivannikov Kirill d6bae589c8 Fail if ex null 2015-09-02 15:09:18 +05:00
Britta Weber f11bec071b [test] print test start and end of test setup, execution and cleanup 2015-09-02 11:57:33 +02:00
David Pilato 34ee4c2d66 [build] remove shaded elasticsearch version
The shaded version of elasticsearch was built at the very beginning to avoid dependency conflicts in a specific case where:

* People use elasticsearch from Java
* People needs to embed elasticsearch jar within their own application (as it's today the only way to get a `TransportClient`)
* People also embed in their application another (most of the time older) version of dependency we are using for elasticsearch, such as: Guava, Joda, Jackson...

This conflict issue can be solved within the projects themselves by either upgrade the dependency version and use the one provided by elasticsearch or by shading elasticsearch project and relocating some conflicting packages.

Example
-------

As an example, let's say you want to use within your project `Joda 2.1` but elasticsearch `2.0.0-beta1` provides `Joda 2.8`.
Let's say you also want to run all that with shield plugin.

Create a new maven project or module with:

```xml
<groupId>fr.pilato.elasticsearch.test</groupId>
<artifactId>es-shaded</artifactId>
<version>1.0-SNAPSHOT</version>

<properties>
    <elasticsearch.version>2.0.0-beta1</elasticsearch.version>
</properties>

<dependencies>
    <dependency>
        <groupId>org.elasticsearch</groupId>
        <artifactId>elasticsearch</artifactId>
        <version>${elasticsearch.version}</version>
    </dependency>
    <dependency>
        <groupId>org.elasticsearch.plugin</groupId>
        <artifactId>shield</artifactId>
        <version>${elasticsearch.version}</version>
    </dependency>
</dependencies>
```

And now shade and relocate all packages which conflicts with your own application:

```xml
<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>2.4.1</version>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                    <configuration>
                        <relocations>
                            <relocation>
                                <pattern>org.joda</pattern>
                                <shadedPattern>fr.pilato.thirdparty.joda</shadedPattern>
                            </relocation>
                        </relocations>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>
```

You can create now a shaded version of elasticsearch + shield by running `mvn clean install`.

In your project, you can now depend on:

```xml
<dependency>
    <groupId>fr.pilato.elasticsearch.test</groupId>
    <artifactId>es-shaded</artifactId>
    <version>1.0-SNAPSHOT</version>
</dependency>
<dependency>
    <groupId>joda-time</groupId>
    <artifactId>joda-time</artifactId>
    <version>2.1</version>
</dependency>
```

Build then your TransportClient as usual:

```java
TransportClient client = TransportClient.builder()
        .settings(Settings.builder()
                        .put("path.home", ".")
                        .put("shield.user", "username:password")
                        .put("plugin.types", "org.elasticsearch.shield.ShieldPlugin")
        )
        .build();
client.addTransportAddress(new InetSocketTransportAddress(new InetSocketAddress("localhost", 9300)));

// Index some data
client.prepareIndex("test", "doc", "1").setSource("foo", "bar").setRefresh(true).get();
SearchResponse searchResponse = client.prepareSearch("test").get();
```

If you want to use your own version of Joda, then import for example `org.joda.time.DateTime`. If you want to access to the shaded version (not recommended though), import `fr.pilato.thirdparty.joda.time.DateTime`.

You can run a simple test to make sure that both classes can live together within the same JVM:

```java
CodeSource codeSource = new org.joda.time.DateTime().getClass().getProtectionDomain().getCodeSource();
System.out.println("unshaded = " + codeSource);

codeSource = new fr.pilato.thirdparty.joda.time.DateTime().getClass().getProtectionDomain().getCodeSource();
System.out.println("shaded = " + codeSource);
```

It will print:

```
unshaded = (file:/path/to/joda-time-2.1.jar <no signer certificates>)
shaded = (file:/path/to/es-shaded-1.0-SNAPSHOT.jar <no signer certificates>)
```

This PR also removes fully-loaded module.

By the way, the project can now build with Maven 3.3.3 so we can relax a bit our maven policy.
2015-09-02 11:57:10 +02:00
Ivannikov Kirill dbf42ba297 AtomicBoolean 2015-09-02 14:46:39 +05:00
Ivannikov Kirill 17089992e2 testPostIndexWithException listener test 2015-09-02 13:57:57 +05:00
Isabel Drost-Fromm 17959871ff Merge pull request #12414 from MaineC/feature/10885
Adds template support to _msearch resource

Much like we already do with search this adds templating support to the _msearch resource.

Closes #10885
2015-09-02 10:21:36 +02:00
Ivannikov Kirill 8f0ae51939 preIndex and postIndex listener test 2015-09-02 13:05:26 +05:00
Britta Weber 9f49e0e7c1 Merge pull request #13246 from brwe/read-post-recovery
Allow reads on shards that are in POST_RECOVERY
2015-09-02 09:46:01 +02:00
Britta Weber 89ac6a83f1 Allow reads on shards that are in POST_RECOVERY
Currently, we do not allow reads on shards which are in POST_RECOVERY which
unfortunately can cause search failures on shards which just recovered if there no replicas (#9421).
The reason why we did not allow reads on shards that are in POST_RECOVERY is
that after relocating a shard might miss a refresh if the node that executed the
refresh is behind with cluster state processing. If that happens, a user might execute
index/refresh/search but still not find the document that was indexed.

We changed how refresh works now in #13068 to make sure that shards cannot miss a refresh this
way by sending refresh requests the same way that we send write requests.

This commit changes IndexShard to allow reads on POST_RECOVERY now.
In addition it adds two test:

- test for issue #9421 (After relocation shards might temporarily not be searchable if still in POST_RECOVERY)
- test for visibility issue with relocation and refresh if reads allowed when shard is in POST_RECOVERY

closes #9421
2015-09-02 09:42:36 +02:00
Robert Muir abec2679f0 Improve jacoco coverage
Upgrade jacoco version and allow it to run with security manager enabled.
2015-09-01 22:14:49 -04:00
Robert Muir 930a5ee170 Merge pull request #13244 from rmuir/lt
Lithuanian analysis
2015-09-01 15:42:58 -04:00
Ryan Ernst 768bfb7d1d Merge pull request #13206 from rjernst/mappings_broken
Fix numerous checks for equality and compatibility in mapper field types
2015-09-01 11:44:18 -07:00
Simon Willnauer 90c2b3a384 Add simple comparator tests Relates to #13249 2015-09-01 20:11:10 +02:00
Britta Weber 7cbd0a1bf5 [test] delete index after test for repetition
DateHistogramIT has suite scope and therefore must take care of cleaning up after
each test. Otherwise we cannot run tests in several times with tests.iters.
2015-09-01 17:58:34 +02:00
Adrien Grand 656fa69f2d Merge pull request #13243 from llasram/hll-estimate-bias-nn-k-6
Estimate HyperLogLog bias via k-NN regression
2015-09-01 14:59:17 +02:00
Robert Muir 0d3e3f81fc Lithuanian analysis 2015-09-01 08:52:10 -04:00
Marshall Bockrath-Vandegrift 1c773e235a Estimate HyperLogLog bias via k-NN regression
The implementation this commit replaces was almost k-NN regression with
k=2, but had two bugs: (a) it depends on the empirical raw estimates
being in strictly non-decreasing order for the binary search (which they
are not); and (b) it weights the biases positively with increased
distance from the corresponding raw estimate.

“HyperLogLog in Practice” leaves the choice of exact algorithm here
fairly vague, just noting: “We use k-nearest neighbor interpolation to
get the bias for a given raw estimate (for k = 6).”  The majority of
other open source HyperLogLog++ implementations appear to use k-NN
regression with uniform weights (and generally k = 6).  Uniform
weighting does decrease variance, but also introduces bias at the domain
extrema.  This problem, plus the use of the word “interpolation” in the
original paper, suggests (inverse) distance-weighted k-NN, as
implemented here.
2015-09-01 08:42:29 -04:00
Adrien Grand 7bc1acf956 Merge pull request #13239 from jpountz/upgrade/lucene-5.3.0
Upgrade to lucene-5.3.0.
2015-09-01 14:03:29 +02:00
Britta Weber d386d909fc rename actions back to admin/* and add suffix [s] instead 2015-09-01 12:53:07 +02:00
Britta Weber 05b48b904d set timeout for refresh and flush to default
Since #13068 refresh and flush requests go to the primary first and are then replicated.
One difference to before is though that if a shard is  not available (INITIALIZING for example)
we wait a little for an indexing request but for refresh we don't and just give up immediately.
Before, refresh requests were just send to the shards regardless of what their state is.

In tests we sometimes create an index, issue an indexing request, refresh and
then get the document. But we do not wait until all nodes know that all primaries have ben assigned.
Now potentially one node can be one cluster state behind and not know yet that
the shards have ben started. If the refresh is executed through this node then the
refresh request will silently fail on shards that are started already because from
the nodes perspective they are still initializing. As a consequence, documents
that expected to be available in the test are now not.
Example test failures are here: http://build-us-00.elastic.co/job/elasticsearch-20-oracle-jdk7/395/

This commit changes the timeout to 1m (default) to make sure we don't miss shards
when we refresh. This will trigger the same retry mechanism as for indexing requests.
We still have to make a decision if this change of behavior is acceptable.

see #13238
2015-09-01 12:20:05 +02:00
Robert Muir 7caed74d5d Merge pull request #13232 from rmuir/nullcheck_policy
Add missing null check in ESPolicy.
2015-09-01 06:03:35 -04:00
Adrien Grand 5d9fb2e8a6 Upgrade to lucene-5.3.0.
From a user perspective, the main benefit from this upgrade is that the new
Lucene53Codec has disk-based norms. The elasticsearch directory has been fixed
to load these norms through mmap instead of nio.

Other changes include the removal of `max_thread_states`, the fact that
PhraseQuery and BooleanQuery are now immutable, and that deleted docs are now
applied on top of the Scorer API.

This change introduces a couple of `AwaitsFix`s but I don't think it should
hold us from merging.
2015-09-01 11:58:45 +02:00
Isabel Drost-Fromm 8cd86a615a Adds template support to _msearch resource
Much like we already do with search this adds templating support to the _msearch resource.

Closes #10885
2015-09-01 11:54:43 +02:00
Adrien Grand f0b7fa2f31 Merge pull request #13060 from andrestc/enhancement/functionscore-unmapped
Make FunctionScore work on unmapped field with `missing` parameter
2015-09-01 11:05:30 +02:00
Simon Willnauer 7571276b84 Pass in relevant disk usage map for early termination 2015-09-01 10:35:56 +02:00
Ivannikov Kirill 2fe2c7fef8 Add listeners to postCreate etc 2015-09-01 12:45:40 +05:00
xuzha f46e66e7d0 Remove the experimental indices.fielddata.cache.expire
closes #10781
2015-09-01 00:40:04 -07:00
Britta Weber 333831c126 Merge pull request #13068 from brwe/broadcast_replication
Make refresh a replicated action
2015-09-01 09:21:54 +02:00
Robert Muir a58c5dba89 Add missing null check in ESPolicy.
This allows reducing privileges with doPrivileged to work,
otherwise it will fail with NPE.

In general, if some code wants to do that, let it. The null
check is needed, even though ProtectionDomain(CodeSource, PermissionCollection)
is more than a bit misleading: "the current Policy will not be consulted".

Additionally add a defensive check for location, since the docs
there are even more confusing: https://bugs.openjdk.java.net/browse/JDK-8129972

The jdk policy impl has both these checks.
2015-09-01 00:34:34 -04:00
Jason Tedor aea00a62f3 Merge pull request #13227 from jasontedor/immutable-lists-be-gone
Remove and forbid use of com.google.common.collect.ImmutableList
2015-08-31 15:29:35 -04:00
Martijn van Groningen 238b56dedf Merge pull request #13046 from jimhooker2002/issue-4665-clean
Turn DestructiveOperations into a Guice module.

To share the same instance between component inside a node.

Closes #4665
2015-08-31 21:22:55 +02:00
Martijn van Groningen 30ffa9a61b test: Allow tests to override whether mock modules are used 2015-08-31 21:02:49 +02:00
Jason Tedor a8bace9f97 Remove and forbid final uses of ImmutableList 2015-08-31 14:35:23 -04:00
Jason Tedor b0af7a1426 Fix NettyTransport 2015-08-31 14:29:00 -04:00
Jason Tedor e39a3bae2c Merge branch 'master' into lists_are_simple 2015-08-31 14:07:00 -04:00
Britta Weber d81f426b68 Make refresh a replicated action
prerequisite to #9421
see also #12600
2015-08-31 19:44:00 +02:00
javanna 95f3817182 Internal: simplify SimpleQueryStringBuilder#doXContent method
No need to test for null values in fieldsAndWeights map, we use 1.0f a default boost all the time.
2015-08-31 17:42:19 +02:00
javanna 11b41f1d68 Java api: don't mutate fields in SimpleQueryStringBuilder during toQuery execution
We used to mutate the map of fields and weights while transforming to lucene query (toQuery method), by adding the default field if the map is empty. That is an anti-pattern, we should leave the original query as is instead, but still generate the proper lucene query. Also took the chance to fix a couple of issues in SimpleQueryStringBuilderTest#doAssertLuceneQuery.
2015-08-31 17:15:55 +02:00
Martijn van Groningen 1b84cadb7b test: The transport client that interacts with the external cluster shoud be provided a list of transport client plugins. 2015-08-31 16:58:03 +02:00
Britta Weber a7e240077d Merge pull request #13218 from brwe/resolve-index-default-impl
add default impl for resolveIndex()
2015-08-31 15:53:57 +02:00
Michael McCandless a49217949f Merge pull request #13199 from mikemccand/remove_merge_docs
Move expert segment merge settings documentation off site into javadocs.
2015-08-31 09:52:19 -04:00
Britta Weber 73785e075e add default impl for resolveIndex() 2015-08-31 15:48:32 +02:00
Tanguy Leroux dbbecce8f2 Sort thread pools by name in Nodes Stats 2015-08-31 14:30:43 +02:00
javanna faf526dccb Merge branch 'master' into feature/query-refactoring 2015-08-31 13:28:48 +02:00
javanna f624e350fb Internal: remove unused method from QueryParseContext and add comments 2015-08-31 13:14:53 +02:00
Jason Tedor 6e2dc73023 Merge pull request #13205 from jasontedor/feature/13204
Convert upgrade action to broadcast by node
2015-08-31 06:02:08 -04:00
Jason Tedor d1223b7369 Convert upgrade action to broadcast by node
Several shard-level operations that previously broadcasted a request
per shard were converted to broadcast a request per node. This commit
converts upgrade action to this new model as well.

Closes #13204
2015-08-31 05:59:57 -04:00
Alexander Reelsen 856b040a0a Plugins: Replace HTTP urls with HTTPS
Switch to use HTTPS by default for all hardcoded plugin URLs.
If users want to install via HTTP they can still specify a HTTP
URL manually.

Closes #12748
2015-08-31 11:45:38 +02:00
Alexander Reelsen 00902207a6 Tests: Ensure binding on localhost host is consistently ipv4/v6
The current netty multiport tests bind on localhost and then try to connect
to 127.0.0.1, which may fail, if localhost is resolved to ipv6 by default.

This randomly chooses between 127.0.0.1, localhost and ::1 (if available) for
binding and then uses this throughout the test.
2015-08-31 10:56:42 +02:00
javanna c41843afec Internal: call context.handleTermsLookup rather than context.indexQueryParserService().handleTermsLookup() in TermsQueryBuilder#toQuery 2015-08-31 10:42:38 +02:00
javanna dd65bd0fe8 Internal: add comments to XContentStructure
Added comments to make sure we don't forget to remove the class once the refactoring is done.
2015-08-31 10:41:38 +02:00
Simon Willnauer a17d7500d3 Take Shard data path into account in DiskThresholdDecider
The path that a shard is allocated on is not taken into account when
we decide to move a shard away from a node because it passed a watermark.
Even worse we potentially moved away (relocated) a shard that was not even
allocated on that disk but on another on the node in question. This commit
adds a ShardRouting -> dataPath mapping to ClusterInfo that allows to identify
on which disk the shards are allocated on.

Relates to #13106
2015-08-31 10:40:42 +02:00
javanna 84b51fc237 [TEST] don't use deprecated methods in AbstractChildTestCase 2015-08-31 10:15:05 +02:00
javanna c3070062db Internal: small QueryShardContext cleanup
Added some comments about changes that we might be able to make later on in the refactoring. Also exposed handleTermsLookup in the context directly, removed unused similarityService getter and corrected simpleMatchToIndexNames call according to changes happened on master (the method variant that accept types will go away, types are ignored anyway).
2015-08-31 10:05:47 +02:00
javanna c403bdbd62 Internal: MoreLikeThisQueryParser to refer to index taken from QueryShardContext rather than QueryParseContext
The reason behind this change is that we will soon remove the index member from QueryParseContext, as parsing will be independent of the index it happens against.
2015-08-31 10:01:56 +02:00
javanna cc408ddd45 [TEST] refactor DummyQueryBuilder and corresponding parser
DummyQueryParser properly implements now fromXContent and the corresponding builder supports converting to the corresponding lucene query through toQuery method.
2015-08-31 10:00:09 +02:00
javanna 1420482f86 Simplify XContent representation of BoolQueryBuilder to always print out an array 2015-08-31 09:58:22 +02:00
javanna 2958751a2d [TEST] parse queries through QueryParseContext in BaseQueryTestCase#parseQuery
This allows us to remove the string argument from the parseQuery method and the need for removing the first few tokens from the query, as the selected parser knows only about his portion of dsl.
2015-08-31 09:55:37 +02:00
Ryan Ernst c01b377ea8 Mappings: Fix numerous checks for equality and compatibility
The field type tests for mappings had a huge hole: check compatibility
was not tested directly at all! I had meant for this to happen in a
follow up after #8871, and was relying on existing mapping tests.
However, there were a number of issues.

This change reworks the fieldtype tests to be able to check all settable
properties on a field type work with checkCompatibility. It fixes a
handful of small bugs in various field types. In particular, analyzer
comparison was just wrong: it was comparing reference equality for
search analyzer instead of the analyzer name. There was also no check
for search quote analyzer.

closes #13112
2015-08-30 23:05:38 -07:00
Michael McCandless 7ad2222ccc copy over merge docs as javadocs 2015-08-30 18:14:47 -04:00
Ryan Ernst 6295f8e795 Merge branch 'master' into tell_me_your_plugins 2015-08-30 14:20:54 -07:00
Ryan Ernst 2539b779c8 Merge pull request #13137 from rjernst/empty_doc_again
Fix doc parser to still pre/post process metadata fields on disabled type
2015-08-30 12:14:18 -07:00
Ivannikov Kirill 38805f3cbd Fix 13202 2015-08-30 23:56:34 +05:00
Jason Tedor aa26b66e96 Remove leftover debugging statement 2015-08-30 14:19:30 -04:00
Simon Willnauer 86a8a0a570 IndicesStatsAction is now a per node operation 2015-08-30 12:48:13 +02:00
David Pilato 03bb28514e Installing plugin without checksums ends up downloading from github
```sh
bin/plugin install lmenezes/elasticsearch-kopf/develop
-> Installing lmenezes/elasticsearch-kopf/develop...
Trying http://download.elastic.co/lmenezes/elasticsearch-kopf/elasticsearch-kopf-develop.zip ...
Trying http://search.maven.org/remotecontent?filepath=lmenezes/elasticsearch-kopf/develop/elasticsearch-kopf-develop.zip ...
Trying https://oss.sonatype.org/service/local/repositories/releases/content/lmenezes/elasticsearch-kopf/develop/elasticsearch-kopf-develop.zip ...
Trying https://github.com/lmenezes/elasticsearch-kopf/archive/develop.zip ...
Downloading .................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................DONE
Verifying https://github.com/lmenezes/elasticsearch-kopf/archive/develop.zip checksums if available ...
Trying https://github.com/lmenezes/elasticsearch-kopf/archive/master.zip ...
Downloading ....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................DONE
Verifying https://github.com/lmenezes/elasticsearch-kopf/archive/master.zip checksums if available ...
```

This happens because we don't have anymore ElasticsearchWrapperException here but standard java exceptions.

Closes #13196.
2015-08-29 23:00:45 +02:00
Jason Tedor 5cb86130ec Add mechanism for transporting shard-level actions by node
Currently, many shard-level operations are transported with a request
per shard via TransportBroadcastAction. These shard-level requests are
then submitted to unbounded execution queues for asynchronous execution
on the receiving node. This transport mechanism and stuffing of the
execution queues can be problematic on large clusters. A better
mechanism would be to aggregate the shard-level requests, transport
them via a single request per node, and execute the shard-level
operations serially on the receiving node.

This commit introduces TransportNodeBroadcastAction which is the
high-level mechanism for transporting the shard-level operations in a
single request per node. The shard-level operations are executed
serially on the receiving node and per-node shard-level results are
aggregated into a single response per node. These node-level results
are then aggregated into a single response to the initial request.

One item of note is a new mechanism for registering request handlers.
This mechanism enables registrants to provide a callback for
instantiating new instances of the request class. Doing this enables
the inner class to be instantiated with the context of its outer class.
This is done so that a single NodeRequest class can be defined rather
than defining a class per operation.

Closes #7990
2015-08-29 16:15:12 -04:00
Jim Hooker 05aa1d90b8 Extend AbstractComponent and remove logger 2015-08-29 07:43:17 +01:00
Jason Tedor 0fa8ee1edd Fix logging statement in o.e.a.s.m.TransportMasterNodeAction 2015-08-28 13:39:56 -04:00
Jason Tedor 532d100c22 Fix logging statement 2015-08-28 13:34:22 -04:00
Nik Everett c180defb10 [CAT] Default verbose to false
Closes #13156
2015-08-28 11:15:44 -04:00
Simon Willnauer 0c71328186 Expand ClusterInfo to provide min / max disk usage forn allocation decider
Today we sum up the disk usage for the allocation decider which is broken since
we don't stripe across multiple data paths. Each shard has it's own private path
now but the allocation deciders still treat all paths as one big disk. This commit
adds allows allocation deciders to access the least used and most used path to make
better allocation decidsions upon canRemain and canAllocate calls.

Yet, this commit doesn't fix all the issues since we still can't tell which shard
can remain and which can't. This problem is out of scope in this commit and will be solved
in a followup commit.

Relates to #13106
2015-08-28 14:04:25 +02:00
Boaz Leskes 218979da1b remove committedOrFailed and use committedOrFailedLatch for state 2015-08-28 12:31:46 +02:00
Boaz Leskes 10e8c410ea more feedback 2015-08-28 12:31:46 +02:00
Boaz Leskes 0668e0d623 more feedback 2015-08-28 12:31:46 +02:00
Boaz Leskes c9ee8dbd16 tighten up FailedToCommitClusterStateException semantics and other feedback 2015-08-28 12:31:45 +02:00
Boaz Leskes 98ed133dd7 reduce log chatter 2015-08-28 12:31:45 +02:00
Boaz Leskes c7c65b626f commit timeout default should never be larger than publishing timeout 2015-08-28 12:31:45 +02:00
Boaz Leskes 6208248215 fix defaults in DiscoverySettings 2015-08-28 12:31:44 +02:00
Boaz Leskes 91dee8b311 reject older cluster state from the same master 2015-08-28 12:31:44 +02:00
Boaz Leskes a56d67d8d7 force mock transport in testCanNotPublishWithoutMinMastNodes 2015-08-28 12:31:44 +02:00
Boaz Leskes e3e0aa5049 Improved concurrency controls In SendingController to make sure that a CS is never committed after publishing is marked out as timed out 2015-08-28 12:31:44 +02:00
Boaz Leskes 234a3794e5 improved timeout handling 2015-08-28 12:31:44 +02:00
Boaz Leskes 4d31681057 added constructor to FailedToCommitException 2015-08-28 12:31:43 +02:00
Boaz Leskes 7d3a36b20f fix ZenDiscoveryUnitTest.testShouldIgnoreNewClusterState 2015-08-28 12:31:43 +02:00
Boaz Leskes 7390bcf833 add FailedToCommitException to registration 2015-08-28 12:31:43 +02:00
Boaz Leskes b702843fe9 beefed up testing... 2015-08-28 12:31:43 +02:00
Boaz Leskes 81e07e81e0 simplified PublishClusterStateActionTests infra 2015-08-28 12:31:42 +02:00
Boaz Leskes 3815a41626 initial copy over from POC 2015-08-28 12:31:42 +02:00
Boaz Leskes 35f9ee7a62 Tests: better isolation of cluster ports
Previously multiple clusters in the same JVM reused the same port ranges, leading to potential big gaps in port selection, which in turns causes unicast based discovery to fail, missing to find another node in the default 5 port range.

Also the previous logic had http use a range that is assigned to another JVMs.
2015-08-28 11:39:30 +02:00
Michael McCandless 07b5d22d91 disable new test on windows 2015-08-28 05:06:35 -04:00
javanna 4ed198fef0 [TEST] fix compilation error in IndexShardTests 2015-08-28 10:54:14 +02:00
javanna 78d097de39 Merge branch 'master' into feature/query-refactoring
Conflicts:
	core/src/main/java/org/apache/lucene/queryparser/classic/MapperQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/AndQueryBuilder.java
	core/src/main/java/org/elasticsearch/index/query/AndQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/BoolQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/DisMaxQueryBuilder.java
	core/src/main/java/org/elasticsearch/index/query/DisMaxQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/GeoPolygonQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/HasChildQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/HasParentQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/MoreLikeThisQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/MultiMatchQueryBuilder.java
	core/src/main/java/org/elasticsearch/index/query/NestedQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/OrQueryBuilder.java
	core/src/main/java/org/elasticsearch/index/query/OrQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/PrefixQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/RegexpQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/SpanNearQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/SpanOrQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/TermQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/TermsQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/support/InnerHitsQueryParserHelper.java
	core/src/test/java/org/elasticsearch/index/query/SimpleIndexQueryParserTests.java
2015-08-28 10:49:28 +02:00
Michael McCandless fb703845dd Merge pull request #13158 from mikemccand/new_path_for_shard_test
Add unit test for ShardPath.selectNewPathForShard
2015-08-28 04:15:15 -04:00
javanna 084a6101fe [TEST] add serialization tests for QueryShardException 2015-08-28 10:13:41 +02:00
Michael McCandless b646ed9cd8 try to work on Windows too 2015-08-28 04:13:21 -04:00
Michael McCandless 8dbc1fbdbd use ShardPath.getRootStatePath; allow forbidden API 2015-08-28 03:59:02 -04:00
javanna 6307738406 Internal: clean up named writeable registry handling in netty transport classes
There are some leftovers around usage of NamedWriteableRegistry in our branch compared to master, just because the registry was added first here, then backported to master, and something didn't go completely right during the final merge.
2015-08-28 09:57:39 +02:00
Boaz Leskes db5e225a25 Discovery: fix `discovery.zen.join_timeout` default value logic
We default the value to be 20x the value of a ping timeout, however we only use the legacy ping timeout settings value for the calculation.

Closes #13162
2015-08-28 09:47:15 +02:00
javanna 9b2e77903d Internal: make ValidationException methods final and fix javadocs 2015-08-28 09:41:47 +02:00
javanna 37ec221df5 Internal: remove unused MapperQueryParser constructor 2015-08-28 09:38:29 +02:00
javanna 24e53b5f8f Internal: remove code duplication in QueryValidationException
The class can now simply extend ValidationException and rely on the base class.
2015-08-28 09:35:43 +02:00
Jason Tedor 90bc784194 Work around for JDK-8039214 on JDK 9 2015-08-27 23:29:22 -04:00
Jason Tedor 3067cacb66 Remove and forbid use of com.google.common.collect.Lists
This commit removes and now forbids all uses of
com.google.common.collect.Lists across the codebase. This is the first
of many steps in the eventual removal of Guava as a dependency.
2015-08-27 22:14:33 -04:00
Igor Motov 2b87d7d919 Add `readonly` option for repositories
Closes #7831
Closes #11753
2015-08-27 18:21:29 -04:00
Nik Everett 19a79c99f9 [test] Use proper comparison operator
lessThanOrEqualTo is more appropriate when comparing _ttl than lessThan
because in rare cases, when tests run very fast, the ttl you fetch will
still equal the one you sent.
2015-08-27 16:43:10 -04:00
Britta Weber e6eeadd171 [test] make sure that the scripts in testScoreAccessWithinScript never compute log(0) 2015-08-27 22:02:51 +02:00
Ryan Ernst 448d3498b1 Merge branch 'master' into fix/bwc_creation 2015-08-27 10:16:45 -07:00
Michael McCandless e2e1b7f76a reference original issue 2015-08-27 13:06:00 -04:00
Michael McCandless 30a3e431ec polish 2015-08-27 13:01:36 -04:00
Michael McCandless 11f09f0a68 add basic unit test 2015-08-27 12:33:04 -04:00
Michael McCandless 4d38856f70 simplify API for ShardPath.selectNewPathForShard to enable unit testing: don't pass IndexShard 2015-08-27 12:32:21 -04:00
Lee Hinman 9f03f8cf44 Call `beforeIndexShardCreated` listener earlier in `createShard`
Some listeners may need to do work before a shard's path is
accessed (such as creating the directory in a plugin), so the listener
should be called before anything happens (as its name implies).
2015-08-27 10:05:27 -06:00
Michael McCandless 8f2ae59316 add asserts to make sure mocking 'took' 2015-08-27 11:19:55 -04:00
Michael McCandless 7a8a608d50 initial mock filesystem setup for test case 2015-08-27 10:55:04 -04:00
Nik Everett 9eb684da51 Default detect_noop to true
detect_noop is pretty cheap and noop updates compartively expensive so this
feels like a sensible default.

Also had to do some testing and documentation around how _ttl works with
detect_noop.

Closes #11282
2015-08-27 10:34:18 -04:00
Simon Willnauer 9a1b5cf966 [TEST] comparing paths seems to be hard on windonws 2015-08-27 13:20:22 +02:00
javanna dc807f294e Query DSL: deprecate _name and boost in short variants of queries
As discussed in #11744 this is the last step to unify parsing of boost and _name. Those fields are supported only in long version of queries, while we sometimes parse them when wwe shouldn't, inconsistently.

Closes #11744
Closes #12966
2015-08-27 12:24:42 +02:00
André Carvalho 793fcb6998 Warmers: delete _all should not throw exception when no warmers are registered
Closes #8991
Closes #13058
2015-08-27 12:24:42 +02:00
Jason Tedor a867a950ab Merge pull request #13135 from jasontedor/fix/eliminate-tuple-as-a-method-parameter
Remove usage of tuple as a method parameter
2015-08-27 04:57:26 -04:00
Simon Willnauer 03ccb99cd5 Expose shards data and state path via ShardStats
Since we now don't stripe shards across data paths we need a way
to access the information on which path a shard is allocated to
eventually do better allocation decisions based on disk usage etc.
This commit exposes the shard paths as part of the shard stats.

Relates to #13106
2015-08-27 09:42:56 +02:00
Ryan Ernst a5bae3bdd8 Mappings: Fix doc parser to still pre/post process metadata fields on disabled type
closes #13017
2015-08-26 20:21:22 -07:00
Ryan Ernst 1029b119d9 Tests: Fix generation scripts for bwc indexes, and add 2.0 beta1 index 2015-08-26 19:36:26 -07:00
Jason Tedor a1408ef622 Remove usage of tuple as a method parameter
This commit removes all instances of
org.elasticsearch.common.collect.Tuple as a method parameter.

Closes #10787
2015-08-26 22:07:33 -04:00
xuzha fb2be6d6a1 The name "position_offset_gap" is confusing because Lucene has three
similar sounding things:

* Analyzer#getPositionIncrementGap
* Analyzer#getOffsetGap
* IndexOptions.DOCS_AND_FREQS_AND_POSITIONS_AND_OFFSETS and
* FieldType#storeTermVectorOffsets

Rename position_offset_gap to position_increment_gap
closes #13056
2015-08-26 14:56:35 -07:00
javanna 350cba98cf [TEST] make MockTermsLookupFetchService non static 2015-08-26 15:56:47 +02:00
javanna 94dd6c5f0e [TEST] unify parsing code in tests
Also clean up static method in MoreLikeThisFetchService, fix randomly failing TermsQueryBuilderTest#testNullValues and rename mappedFieldNamesSmall to MAPPED_LEAF_FIELD_NAMES.
2015-08-26 15:50:01 +02:00
Martijn van Groningen a0565f00dd Removed the `operation_threaded` option.
This low level option isn't worth the complexity and an operation should never happen on the network thread.
2015-08-26 15:21:50 +02:00
Martijn van Groningen 3b400aafd6 parent/child: use the index searcher from engine searcher directly for JoinUtil based impl
and use the provided index searcher directly for the pre 2.0 parent/child imp.
2015-08-26 15:03:11 +02:00
Colin Goodheart-Smithe bd0753eaa3 Merge pull request #13033 from xuzha/xu-12391
Print field name when meet unexpected token.
2015-08-26 14:41:08 +02:00
Alex Ksikes 1af0a39221 Refactors TermsQueryBuilder and Parser
Refactors TermsQueryBuilder and Parser for #10217.

This PR is against the query-refactoring branch.

Closes #12042
2015-08-26 13:58:46 +02:00
Boaz Leskes 95c097bf24 Tests: use assertHitCount instead of direct assertion on hits in metric tests
So we will get more info on failures
2015-08-26 11:00:48 +02:00
Christoph Büscher efadf87371 Merge branch 'master' into feature/query-refactoring 2015-08-26 10:23:03 +02:00
Nik Everett 3f6108ab74 [Mapping] Move position_offset_gap default change
Until a couple of hours ago we expected the position_offset_gap to default
to 0 in 2.0 and 100 in 2.1. We decided it was worth backporting that new
default to 2.0. So now that its backported we need to teach 2.1 that 2.0
also defaults to 100.

Closes #7268
2015-08-25 18:01:10 -04:00
Nik Everett 4b9664beeb Mapping: Default position_offset_gap to 100
This is much more fiddly than you'd expect it to be because of the way
position_offset_gap is applied in StringFieldMapper. Instead of setting
the default to 100 its simpler to make sure that all the analyzers default
to 100 and that StringFieldMapper doesn't override the default unless the
user specifies something different. Unless the index was created before
2.1, in which case the old default of 0 has to take.

Also postition_offset_gaps less than 0 aren't allowed at all.

New tests test that:
1. the new default doesn't match phrases across values with reasonably low
slop (5)
2. the new default doest match phrases across values with reasonably high
slop (50)
3. you can override the value and phrases work as you'd expect
4. if you leave the value undefined in the mapping and define it on a
custom analyzer the the value from the custom analyzer shines through

Closes #7268
2015-08-25 14:21:50 -04:00
Ryan Ernst d77bcb0d9b Merge pull request #13085 from rjernst/fix/13017
Fix document parsing to properly ignore entire type when disabled
2015-08-25 08:58:07 -07:00
javanna 96b5cebfb1 Merge branch 'master' into feature/query-refactoring
Conflicts:
	core/src/main/java/org/elasticsearch/index/query/QueryStringQueryParser.java
2015-08-25 13:16:46 +02:00
javanna cc12501cef Internal: remove unused code from query_string parser and settings
Closes #13098
2015-08-25 12:53:41 +02:00
Christoph Büscher 59cb67c7bd Merge branch 'master' into feature/query-refactoring
Conflicts:
	core/src/main/java/org/elasticsearch/index/query/support/NestedInnerQueryParseSupport.java
2015-08-25 11:59:40 +02:00
Jason Tedor 8ff1efbcf0 Merge pull request #13082 from jasontedor/feature/13081
Consolidate duplicate logic in RoutingTable all*ShardsGrouped
2015-08-25 05:06:26 -04:00
Martijn van Groningen 9f3043fc78 Merge pull request #13087 from martijnvg/nested/child_filter_can_just_be_regular_filter
Nested query should only use bitset cache for parent filter
2015-08-25 10:13:36 +02:00
Ryan Ernst 01cfa95d1c do real copies where copyOf was used before 2015-08-24 21:22:21 -07:00
Ryan Ernst 69ba45a797 Merge branch 'master' into lists_are_simple 2015-08-24 20:23:59 -07:00
Ryan Ernst 311faa822a Merge branch 'master' into lists_are_simple 2015-08-24 19:06:58 -07:00
Jason Tedor 51ae1c4d53 Merge pull request #13086 from jasontedor/fix/no-duplicate-settings
Detect duplicate settings keys on startup
2015-08-24 20:51:22 -04:00
Jason Tedor a7af34022c Detect duplicate settings keys on startup
This commit changes the startup behavior of Elasticsearch to throw an
exception if duplicate settings keys are detected in the Elasticsearch
configuration file.

Closes #13079
2015-08-24 20:50:46 -04:00
Martijn van Groningen ee2a5fc2bc nested: child level filter should be a regular filter! 2015-08-24 22:19:56 +02:00
Ryan Ernst 297f0f22d7 Merge branch 'master' into tell_me_your_plugins 2015-08-24 12:03:25 -07:00
Ryan Ernst f7b3fe05e4 Mappings: Fix document parsing to properly ignore entire type when disabled
Currently when an entire type is disabled, our document parser will end
parsing on the first field of the document. This blows up the recently
added check that parsing did not silently skip any tokens (ie whether
there was garbage leftover).

This change fixes the parser to correctly skip the entire document when
the type is disabled.

closes #13017
2015-08-24 11:08:38 -07:00
Jason Tedor 29a03b9c8a Consolidate duplicate logic in RoutingTable all*ShardsGrouped
This commit consolidates the logic in
RoutingTable#allActiveShardsGrouped and
RoutingTable#allAssignedShardsGrouped into a single method that merely
applies a predicate to each ShardRouting.

Closes #13081
2015-08-24 12:26:54 -04:00
Christoph Büscher 119e9ba138 Merge pull request #13072 from cbuescher/plugin-install-help-update
Update the plugin install CLI help to the new short name for commercial plugins
2015-08-24 18:16:00 +02:00
Christoph Büscher 888989b3f3 Changed help text 2015-08-24 17:56:59 +02:00
Christoph Büscher 339486b943 Merge pull request #13073 from cbuescher/rm-rewrite-multiMatchQueryBuilder
Remove unsupported `rewrite` from multi_match query builder
2015-08-24 16:12:11 +02:00
Christoph Büscher 5f85349080 Minor fix in plugin install CLI help 2015-08-24 15:42:10 +02:00
André Carvalho c2c4742f65 proper creation of values for unmapped field on score function 2015-08-24 10:02:18 -03:00
Simon Willnauer 7aa855da51 Merge pull request #13057 from rmuir/easier_mlockall
Make mlockall configuration easier.
2015-08-24 14:49:22 +02:00
Simon Willnauer 707c6ca1d1 Merge pull request #13054 from rmuir/favicon
Add favicon
2015-08-24 14:48:57 +02:00
Simon Willnauer 23dc785f3f Merge pull request #13049 from s1monw/fix_race_in_meta_state
Don't check if directory is present to prevent races
2015-08-24 14:47:59 +02:00
Christoph Büscher 8757af2d92 Merge branch 'master' into feature/query-refactoring 2015-08-24 14:36:46 +02:00
Christoph Büscher 150b4747b4 Query DSL: Remove unsupported `rewrite` option from multi match query builder
The `rewrite` option has been removed from the parser with
commit da5fa6c4 and won't parse anymore, however we still
have a setter for it in the builder that gets rendered out
when used and potentially leads to parsing errors. This PR
removes the setter for the unsupported `rewrite` option.
2015-08-24 14:28:21 +02:00
André Carvalho 60ab72e322 using emptySortedNumericDoubles instead of null in score function 2015-08-24 09:19:45 -03:00
Christoph Büscher c7a8386928 Merge pull request #13069 from cbuescher/rm-matchQuery-rewrite
Remove unsupported `rewrite` option from match query builder
2015-08-24 14:04:52 +02:00
Martijn van Groningen d7327b7b31 test: changed the join timeout from the default 60s to 10s and
also wait for the cluster state to contain 4 nodes instead of just waiting for a green state.
2015-08-24 13:43:08 +02:00
Martijn van Groningen 8862348e4f test: moved ensureStableCluster(...) to base integ test class 2015-08-24 13:43:03 +02:00
Christoph Büscher 059ef5c749 Query DSL: Remove unsupported `rewrite` option from match query builder
The `rewrite` option has been removed from the parser with
commit da5fa6c4 and won't parse anymore, however we still
have a setter for it in the builder that gets rendered out
when used and potentially leads to parsing errors. This PR
removes the setter for the unsupported `rewrite` option.
2015-08-24 13:01:43 +02:00
xuzha f77b349c23 The queue_size value should be shown as an integer,
closes #10404
closes #13063
2015-08-24 10:04:55 +02:00
Ryan Ernst 76c472973e Also fix order of injection with plugins for transport client
See #13061
2015-08-23 11:18:38 -07:00
Ryan Ernst b6d3fa27e2 Internal: Add plugin modules before (almost all) others
This change makes modules added by plugins come before others, as it was
before #12783. The order of configuration, and thereby binding, happens
in the order modules are received, and without this change, some plugins
can get *insane* guice errors (500mb stack trace).
2015-08-23 11:08:36 -07:00
André Carvalho 345b2b98cf FunctionScore should work on unampped fields when missing parameter is specified 2015-08-22 23:36:13 -03:00
Ryan Ernst ef592a8e17 Fix test plugin to correctly add action 2015-08-22 16:08:35 -07:00
Ryan Ernst d2606feb3d Remove PreProcessModule 2015-08-22 09:44:04 -07:00
Ryan Ernst 07807f320a Merge branch 'master' into module_culling4 2015-08-22 09:40:17 -07:00
Simon Willnauer febc7f5d4c [TEST] Provide unicast hosts for Tribe nodes in TribeIT 2015-08-22 18:37:58 +02:00
Robert Muir f2db1023c1 Make mlockall configuration easier. 2015-08-22 09:55:09 -04:00
Jason Tedor 0407782baa Clearer logic in checking testing version validity 2015-08-22 09:27:56 -04:00
Ryan Ernst 89e1a0fb7d Plugins: Removed plugin.types
The setting `plugin.types` is currently used to load plugins from the
classpath. This is necessary in tests, as well as the transport client.

This change removes the setting, and replaces it with the ability to
directly add plugins when building a transport client, as well as
infrastructure in the integration tests to specify which plugin classes
should be loaded on each node.
2015-08-22 01:37:43 -07:00
Robert Muir 91d16f45c6 Add favicon 2015-08-22 00:16:36 -04:00
Robert Muir 35689f0d02 Cleanup bootstrap package.
* makes most classes final and package private
* removes duplicate and confusing multiple entry points
* adds javadocs to some classes like JarHell,Security
* adds a public class BootStrapInfo that exposes any stats
  needed by outside code.
2015-08-21 23:28:56 -04:00
Robert Muir be366c729f Lucene SPI support for plugins.
When we create a plugin's classloader, we should allow it to register things in
the lucene SPI (registry of Tokenizers, TokenFilters, CharFilters, Codec,
PostingsFormat, DocValuesFormat).

Plugins should be able to do this so they can extend Lucene.
2015-08-21 20:12:30 -04:00
Robert Muir d96af934db Improve startup exceptions (especially file permissions etc)
Squashed commit of the following:

commit 0014d31c1c478977cc8c50eb635eab0fb91449e4
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Aug 21 18:20:35 2015 -0400

    Add missing paren to javadocs

commit bb46142785c67b849cfa5aa08fe5a3877a550b3d
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Aug 21 18:08:45 2015 -0400

    Improve startup exceptions (especially file permissions etc)
2015-08-21 18:21:01 -04:00
Simon Willnauer 846cf620db Don't check if directory is present to prevent races
We do check if a directory is present and then open a dir stream on
it. Yet the file can be concurrrently deleted which is OK but we fail
with a hard exception. This change tries to open the dir directly (listing via stream)
and catches NoSuchFileEx | FNFEx.
2015-08-21 21:44:54 +02:00
Jim Hooker 57a83fce07 Turn DestructiveOperations.java into a Guice module.
https://github.com/elastic/elasticsearch/issues/4665

Inject DestructiveOperations object rather than use new.

Use constant rather than hard-coded string
2015-08-21 19:42:54 +01:00
Jason Tedor fe8eb80b4c Do not permit multiple settings files
This commit enforces that at most a single settings file is found. If
multiple settings files are found, a SettingsException will be thrown

Closes #13042
2015-08-21 11:37:35 -04:00
Robert Muir 5002f3b5ac Fix compilation... forgot to 'mvn clean' 2015-08-21 11:11:47 -04:00
Robert Muir 2ecf9af71d Merge pull request #13041 from rmuir/consistent_formatting
Use StartupError to format all exceptions hitting the console
2015-08-21 11:09:02 -04:00
Robert Muir 287789be53 Use StartupError to format all exceptions hitting the console 2015-08-21 11:05:31 -04:00
Jason Tedor f4774d17a6 Do not swallow exceptions thrown while parsing settings
This commit fixes an issue that was causing Elasticsearch to silently
ignore settings files that contain garbage. The underlying issue was
swallowing an SettingsException under the assumption that the only
reason that an exception could be throw was due to the settings file
not existing (in this case the IOException would be the cause of the
swallowed SettingsException). This assumption is mistaken as an
IOException could also be thrown due to an access error or a read
error. Additionally, a SettingsException could be thrown exactly
because garbage was found in the settings file. We should instead
explicitly check that the settings file exists, and bomb on an
exception thrown for any reason.

Closes #13028
2015-08-21 10:43:48 -04:00
javanna 37b2007eb4 [TEST] added tests for alternate queries formats
We currently test that our query parsers can parse the format that our query builder outputs in XContent format, but in some cases the parser supports more than that, hence we need more specific tests otherwise we have no coverage for alternate formats.
2015-08-21 15:59:13 +02:00
javanna 2d42839eef Merge branch 'master' into feature/query-refactoring
Conflicts:
	core/src/main/java/org/elasticsearch/index/query/QueryStringQueryBuilder.java
2015-08-21 15:58:36 +02:00
Jason Tedor 13c1c27d5a Merge pull request #13036 from jasontedor/limit-type-name-length
Limit type name length
2015-08-21 09:19:11 -04:00
Jason Tedor 1579e491d8 Limit type name length
This commit will limit type name length to be at most 255 characters.

Closes #13021
2015-08-21 09:18:30 -04:00
Jason Tedor f0aae96f11 Utility method for getting the first version prior to given version 2015-08-21 09:18:29 -04:00
Simon Willnauer d96e8634ac Suppress rest exceptions by default and log them instead
Today we are very verbose when rendering exceptions on the rest layer.
Yet, this isn't necessarily very easy to read and way too much infromation most
of the time. This change suppresses the stacktrace rendering by default but instead
adds a `rest.suppressed` logger that logs the suppressed stacktrace or rather the entire
exception on the node that renderes the exception
The log message looks like this:

```
[2015-08-19 16:21:58,427][INFO ][rest.suppressed          ] /test/_search/ Params: {index=test}
[test] IndexNotFoundException[no such index]
	at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:551)
```
2015-08-21 15:17:45 +02:00
javanna ded442952b Java api: add missing support for escape to QueryStringQueryBuilder
QueryStringQueryParser parses `escape`, but java api users had no chance to actually set it.

Closes #13016
2015-08-21 15:04:53 +02:00
Jason Tedor 31b80e4f3f Merge pull request #13010 from jasontedor/feature/improve-java-version-comparison
Improve java version comparison and explicitly enforce a version format
2015-08-21 08:30:50 -04:00
Jason Tedor 5ff1f8a058 Enforce version format for java.version for plugins
Currently we implicitly enforce a version format on the java.version
property for plugins via JarHell.checkJavaVersion. We should explicitly
enforce this version format and specify it in the documentation.

This commit adds an explicit enforcement of the version format on the
java.version property for plugins and updates the documentation for
plugin-descriptor.properties accordingly.

Closes #13009
2015-08-21 08:28:41 -04:00
Jason Tedor 126e8e4aee Improve Java version comparison in JarHell
This commit improves Java version comparison in JarHell.

The first improvement is the addition of a method to check the version
format of a target version string. This method will reject target
version strings that are not a sequence of nonnegative decimal integers
separated by “.”s, possibly with leading zeros (0*[0-9]+(\.[0-9]+)?).
This version format is the version format used for Java specification
versioning (cf. Java Product Versioning, 1.5.1 Specification Versioning
and the Javadocs for java.lang.Package.)

The second improvement is a clean method for checking that a target
version is compatible with the runtime version of the JVM. This is done
using the system property java.specification.version and comparing the
versions lexicograpically. This method of comparison has been tested on
JDK 9 builds that include JEP-220 (the Project Jigsaw JEP concerning
modular runtime images) and JEP-223 (the version string JEP). The class
that encapsulates the methods for parsing and comparing versions is
written in a way that can easily be converted to use the Version class
from JEP-223 if that class is ultimately incorporated into mainline JDK
9.

Closes #12441
2015-08-21 08:27:47 -04:00
Jason Tedor 8fca951be0 Merge pull request #12901 from jasontedor/feature/12900
Accumulate validation errors when validating index templates
2015-08-21 08:08:50 -04:00
Jason Tedor 4a5040bb32 Accumulate validation errors when validating index templates
This commit changes the behavior when validating index templates to
accumulate all validation errors before reporting failure to the user.
This addresses a usability issue when creating index templates.

Closes #12900
2015-08-21 08:06:42 -04:00
Colin Goodheart-Smithe 38085cf90a Aggregation: Fix AggregationPath.subPath() to not throw ArrayStoreException
Aggregation.subPath() always threw an ArrayStoreException because we were trying to pass a List into System.arraycopy(). This change fixes that bug and adds a test to prevent regression
2015-08-21 12:50:05 +01:00
Simon Willnauer 71cbcea3c2 [TEST] Add discovery-multicast to PluginManagerIT 2015-08-21 13:46:25 +02:00
Ryan Ernst 2a726fe4a1 Internal: Remove SpawnModules
The last use case of spawn modules was for plugins. This change handles
plugin modules directly, and removes SpawnModules.

closes #12783
2015-08-21 04:01:50 -07:00
Christoph Büscher b18e470278 Merge branch 'master' into feature/query-refactoring 2015-08-21 12:27:31 +02:00
xuzha 2af6f3fbff Print field name when meet unexpected token.
closes #12391
2015-08-21 01:16:38 -07:00
Adrien Grand f3265095ab Never cache match_all queries.
This commit backports https://issues.apache.org/jira/browse/LUCENE-6748 to make
sure that we never cache trivial queries like MatchAllDocsQuery.
2015-08-21 09:31:53 +02:00
Ryan Ernst 16e5c7c21c Add multicast plugin to plugin manager official list of plugins
See #13027
2015-08-20 23:45:36 -07:00
Simon Willnauer 3fb2d8e448 Merge pull request #12947 from s1monw/expected_shard_size
Add `expectedShardSize` to ShardRouting and use it in path.data allocation
2015-08-21 08:34:38 +02:00
Ryan Ernst 703a4b3f40 Merge pull request #13027 from rjernst/cast_away_multicast
Move multicast discovery to a plugin
2015-08-20 23:21:12 -07:00
Simon Willnauer 3dd6c4ab80 Use constant to determin if expected size is available 2015-08-21 08:18:59 +02:00
Simon Willnauer d2507c4ac0 Add `expectedShardSize` to ShardRouting and use it in path.data allocation
Today we only guess how big the shard will be that we are allocating on a node.
Yet, we have this information on the master but it's not available on the data nodes
when we pick a data path for the shard. We use some rather simple heuristic based on
existing shard sizes on this node which might be complete bogus. This change adds
the expected shard size to the ShardRouting for RELOCATING and INITIALIZING shards
to be used on the actual node to find the best data path for the shard.

Closes #11271
2015-08-21 08:18:59 +02:00
Robert Muir a2507a8f1c test: fix and simplify logic 2015-08-21 02:06:34 -04:00
Robert Muir d0835715c2 Fix formatting of startup/configuration errors.
Try to strike a balance between usability and debuggability.
2MB stacktrace is not gonna work... we print message and find
"likely root cause" and identify it as such, limit it to 30 stack
frames, and filter out guice. When frames are truncated we identify
that too, and always inform the user full exception is available from
logs.

Closes #13029
2015-08-21 01:19:29 -04:00
Ryan Ernst d0f5ce58d8 Add multicast plugin to help and qa 2015-08-20 22:04:17 -07:00
Ryan Ernst 2b1e4acafe Remove logging message from testing 2015-08-20 17:08:44 -07:00
Ryan Ernst bcd58ad02f Merge branch 'master' into cast_away_multicast 2015-08-20 17:03:02 -07:00
Ryan Ernst 164efaecbe Networking: Move multicast discovery to a plugin
Multicast has known issues (see #12999 and #12993). This change moves
multicast into a plugin, and deprecates it in the docs.  It also allows
for plugging in multiple zen ping implementations.

closes #13019
2015-08-20 16:43:25 -07:00
Robert Muir bd539e7f98 simplify this 2015-08-20 18:56:26 -04:00
Robert Muir 7298be474d fix test failure 2015-08-20 17:31:06 -04:00
Robert Muir 478bf1c9ef Merge pull request #13026 from rmuir/too_noisy
Don't print lots of noise on IPv4 only hosts.
2015-08-20 17:11:40 -04:00
Simon Willnauer aea596241c Merge pull request #13020 from s1monw/issues/13014
Remove support for address resolving in InetSocketTransportAddress
2015-08-20 23:10:50 +02:00
Robert Muir 5beabdd0ec Don't print lots of noise on IPv4 only hosts.
If the machine doesn't support IPv6, or if the user disabled it
with -Djava.net.preferIPv4Stack, or if the user disabled it with
ES_USE_IPV4, we will still try to bind a socket to ::1 out of box,
and we will see lots of exceptions logged to the console.

It is harmless noise but not a great experience.
2015-08-20 17:05:30 -04:00
Simon Willnauer d9089d56aa Don't use networking in InternalTestClusterTests 2015-08-20 22:55:04 +02:00
Simon Willnauer c15d4c33b2 Add back private default ctor 2015-08-20 21:42:14 +02:00
Simon Willnauer 214d303e6a Remove support for address resolving in InetSocketTransportAddress
this commit removes all support for reverse host name resolving from
InetSocketTransportAddress. This class now only returns IP addresses.

Closes #13014
2015-08-20 20:57:26 +02:00
Robert Muir e2ab62596f Default to unicast discovery, with default host list of 127.0.0.1, [::1]
Fix unicast discovery to work when a host has multiple addresses.
Ban dangerous methods in java.net with forbidden APIs.
Fix ipv6 bugs and formatting of network addresses everywhere.

Closes #12999
Closes #12993

Squashed commit of the following:

commit 6c1aa001d091c5cf25212a53dc701fb704337f1e
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 14:25:43 2015 -0400

    Fix these to be correct with addresses just in case

commit 648215627e84abf58a71400e7dc9ae775efb71d6
Merge: d00561b 41d8fbe
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 13:23:09 2015 -0400

    Merge branch 'master' into unicast_all_the_way_down

commit d00561b76fd1aa5850699f7901f3dae3d4d402b7
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 16:38:50 2015 +0200

    limit local ports to 5 in UnicastZenPing

commit e2e15c594006746cbe24432694294a71cc99deb8
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 10:32:47 2015 -0400

    fix port limiting

commit 10153cb7adadda81a1f482445e703836b65cf5e2
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 10:18:37 2015 -0400

    don't serialize scopeids: that's broken

commit 2aa63d43db2baec68a2e9bc227cfeb85dfeb4f83
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 16:06:51 2015 +0200

    restore @Network

commit c840f1d1ef438826ae1ecfd5e45942a0e30dc9c0
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 16:02:30 2015 +0200

     Use NetworkAddress.formatAddress where applicable in plugins

commit 374ce878852b35d626b7a29c8c4773545b0e9ddd
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 15:34:06 2015 +0200

    Use NetworkAddress.formatAddress where applicable

commit e7a606d63f1bc43c1b62b6e17adf707c76d43a15
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 10:17:57 2015 +0200

    Add @Multicast annotation to disable multicast tests by default.

    We only run multicast tests now when we explicitly state it. A working
    multicast env is required which is not always the case.

commit 2d7d2d0347179696ab41f71f048b13305014c85b
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 09:51:28 2015 +0200

    Remove extra check for local mode in InternalTestCluster

commit dda59ac39aa136d4687b9274c2692cd77f8b8f66
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 09:37:03 2015 +0200

    Handle node mode across entire test cluster

    We used static methods reading sys properties to define the node mode
    per cluster. this had lots of problems when tests couldn't cope with
    mixed or only local mode. Now we are passing it down to the cluster from the test
    which allows to @SuppressNetworkMode / @SupressLocalMode on the test to force
    consistent node configurations.

commit 058197b7a408318995c88ce7f6762e32348de0de
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 03:19:14 2015 -0400

    really ban InetSocketAddress's trappy method and break build and go to sleep, sorry

commit ac8779185aee1e17e6f5a81766290fdfc9c603ba
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 03:16:52 2015 -0400

    Ban methods that might surprisingly cause DNS lookups

commit e64fe3dff2b11503e5f2831eb9863d64f56c5538
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 02:59:05 2015 -0400

    Add unit test

commit f15434f20fb1a3691b1cc16028597d8fae937e05
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 02:39:02 2015 -0400

    fix ipv6 formatting bugs

commit 05c2c74098052c75fbb79ea1818a295ef2e03e30
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 02:12:05 2015 -0400

    format addresses correctly so I can actually read what comes out of our logs and stats apis

commit 4f9389dcf1e8925f23153c5eb271b4ce2294dbaf
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Aug 19 21:26:52 2015 -0400

    ban dangerous methods in java.net

commit 6aacd4d9925f324903d1d099a6cf5f862aeaf677
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Aug 19 20:59:24 2015 -0400

    ban lenient method

commit f466a842c60163d1f4554bdce8a4163edb534c2c
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 00:29:00 2015 +0200

    fix tests to not mix local transport and zen unicast disco

commit 0de007a33b33fb68cf85cd86db4ca4f8ce10bbc9
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 00:10:07 2015 +0200

    fix tests to not mix local transport and zen unicast disco

commit 539f6ca6e5137e0d496239adc8684688dedcc824
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 00:02:01 2015 +0200

    fix tests to not mix local transport and zen unicast disco

commit 004c2881b25467f332acc8c9f9e92b1f0f9d314e
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Aug 19 17:51:45 2015 -0400

    Fix multinode

commit 54113af325ce31571811c49fdaae89d5687be4ba
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Aug 19 17:36:45 2015 -0400

    fix integration tests

commit 0156a77a56319d6b9737ec6a531992052e50bd59
Author: Simon Willnauer <simonw@apache.org>
Date:   Wed Aug 19 23:32:18 2015 +0200

    enable multicast in MulticastZenPingIT.java

commit 1791caa35da853ce0122485fa3fd4674c671ec6e
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Aug 19 17:23:16 2015 -0400

    Fix constant

commit 22820b53e0b2dc9fd47145c2bc29ce912a8fd484
Author: Simon Willnauer <simonw@apache.org>
Date:   Wed Aug 19 22:59:09 2015 +0200

    give it some extra ids for local transport crazyness

commit b2138fafa94a8a085813fd48356df63e57ade5b3
Author: Simon Willnauer <simonw@apache.org>
Date:   Wed Aug 19 22:51:42 2015 +0200

    pass on local addresses from configured transport rather than hard code IP addresses

commit 1bf5de1f457b081e0ce262b57d2b55d39c434156
Author: Simon Willnauer <simonw@apache.org>
Date:   Wed Aug 19 22:04:31 2015 +0200

    fix PluggableTransportModuleIT.java to use local disco and detach port limit for node local disco

commit b6706eddfa04c43947c16551359ae98a463d34aa
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Aug 19 14:16:03 2015 -0400

    Default to unicast discovery, with default host list of 127.0.0.1, [::1]
2015-08-20 14:26:40 -04:00
Christoph Büscher 345a30a2a7 Merge branch 'master' into feature/query-refactoring
core/src/test/java/org/elasticsearch/plugins/PluginManagerIT.java
	core/src/test/java/org/elasticsearch/search/aggregations/bucket/SignificantTermsSignificanceScoreIT.java
2015-08-20 12:48:05 +02:00
Adrien Grand 6fa258b8fa Deprecate the `scan` search type.
This commit deprecates the `scan` search type in favour of regular scroll
requests sorted by `_doc`.

Related to #12983
2015-08-20 12:47:23 +02:00
Adrien Grand 78c2f1063a Optimize sorted scroll when sorting by `_doc`.
This change means that we will be able to remove the `SCAN` search type in 3.0
and recommend users to use sorted scrolls instead.
2015-08-20 11:51:59 +02:00
Jason Tedor a0243200e0 Fix a documentation typo and a code comment typo to path.repo 2015-08-19 22:14:50 -04:00
Robert Muir 829b858c1d Merge pull request #12995 from rmuir/dedup_resolver_for_binding
Deduplicate addresses from resolver.
2015-08-19 16:17:45 -04:00
Martijn van Groningen f2f95ea115 Removed CachedDfSource and move the dfs logic into the ContextIndexSearcher 2015-08-19 20:25:41 +02:00
Christoph Büscher 0fc96ede69 Merge pull request #12886 from cbuescher/fix/12278
Make ValueParser.DateMath aware of timezone setting
2015-08-19 18:42:19 +02:00
Ryan Ernst 6520395c1c tweaks 2015-08-19 09:37:35 -07:00
Robert Muir ce120b9c71 Deduplicate addresses from resolver.
In some cases this may contain duplicates, although its a misconfiguration,
lets not bind to multiple ports. Its no problem for us to dedup, this code
doesn't need to be huper-duper fast since its used only for logic around bind/publish
2015-08-19 12:06:54 -04:00
Simon Willnauer 22ae0a161f Log all interfaces we try to bind before binding 2015-08-19 17:20:48 +02:00
Colin Goodheart-Smithe 88d482437a Aggregations: Throw error if cardinality aggregator has a sub aggregation
The cardinality aggregation is a metric aggregation and therefore cannot accept sub-aggregations. It was previously possible to create a rest request with a cardinality aggregation that had sub-aggregations. Now such a request will throw an error in the response.

Close #12988
2015-08-19 14:31:57 +01:00
Simon Willnauer ca28acc373 Merge pull request #12986 from s1monw/fix_host_resolving
Only resolve host if explicitly allowed.
2015-08-19 15:25:55 +02:00
Simon Willnauer e5134317be Merge pull request #12936 from s1monw/add_build_hash_to_headers
Add build short hash to the download manager headers to identify staging builds
2015-08-19 15:10:46 +02:00
Simon Willnauer bd56b0a79e Only resolve host if explicitly allowed.
We have some settings that prevent host name resolution which should
be repected by InetSocketTransportAddress#getHost() to only resolve if
allowed or desired.
2015-08-19 14:54:56 +02:00
Robert Muir bbf669506b Merge pull request #12979 from rmuir/ifconfig
Log network configuration at debug level
2015-08-19 07:14:01 -04:00
Christoph Büscher 8454d49552 Plugins: Add 'name' property to plugin descriptor file to determine plugin name
At the moment, when installing from an url, a user provides the plugin name on
the command line like:

* bin/plugin install [plugin-name] --url [url]

This can lead to problems when picking an already existing name from another
plugin, and can potentially overwrite plugins already installed with that name.

This, this PR introduces a mandatory `name` property to the plugin descriptor
file which replaces the name formerly provided by the user.

With the addition of the `name` property to the plugin descriptor file, the user
does not need to specify the plugin name any longer when installing from a file
or url. Because of this, all arguments to `plugin install` command are now
either treated as a symbolic name, a URL or a file without the need to specify
this with an explicit option.

The new syntax for `plugin install` is now:

bin/plugin install [name or url]

* downloads official plugin
bin/plugin install analysis-kuromoji

* downloads github plugin
bin/plugin install lmenezes/elasticsearch-kopf

* install from URL or file
bin/plugin install http://link.to/foo.zip
bin/plugin install file:/path/to/foo.zip

If the argument does not parse to a valid URL, it is assumed to be a name and the
download location is resolved like before. Regardless of the source location of
the plugin, it is extracted to a temporary directory and the `name` property from
the descriptor file is used to determine the final install location.

Relates to #12715
2015-08-19 12:13:01 +02:00
Jason Tedor 66b0e7a6e1 Merge pull request #12977 from jasontedor/fix/12873
Add millisecond parser for dynamic date fields mapped from "yyyy/MM/dd"
2015-08-19 05:30:24 -04:00
Simon Willnauer ddd6be1047 Merge pull request #12981 from s1monw/serialize_interrupted_exceptions
Add serialization support for InterruptedException
2015-08-19 11:26:49 +02:00
Simon Willnauer 828e31ce32 Add serialization support for InterruptedException
it's an important exception to serialize and we see it often in tests
etc. but then it's wrapped in NotSerializableExceptionWrapper which is
odd. This commit adds native support for this exception.
2015-08-19 10:01:58 +02:00
Boaz Leskes 43fae91ab9 Test: ClusterDiscoveryConfiguration.UnicastZen should allow for port ranges
The cluster configuration allows to setup a cluster for use with unicast discovery. This means that nodes have to have pre-calculated known addresses which can be used to poplulate the unicast hosts setting. Despite of repeated attempts to select unused ports, we still see test failures where the node can not bind to it's assigned port due to it already being in use (most on CentOS). This commit changes it to allow each node to have a pre-set mutual exclusive range of ports and add all those ports to the unicast hosts list. That's OK because we know the node will only bind to one of those.
2015-08-19 10:01:38 +02:00
Robert Muir fa70162178 Log network configuration at debug level 2015-08-19 01:26:36 -04:00
Ryan Ernst c151f2137a Internal: Remove all uses of ImmutableList
We are in the process of getting rid of guava, and this removes a major
use. The replacement is mostly Collections.emptyList(), Arrays.asList
and Collections.unmodifiableList. While it is questionable whether we
need the last one (as these are usually placed in final members), we can
continue to refactor later by removing unnecessary wrappings.
2015-08-18 21:32:51 -07:00
Jason Tedor 4c9327d983 Add millisecond parser for dynamic date fields mapped from "yyyy/MM/dd"
Dynamic date fields mapped from dates of the form "yyyy-MM-dd"
automatically receive the millisecond paresr epoch_millis as an
alternative parsing format. However, dynamic date fields mapped from
dates of the form "yyyy/MM/dd" do not. This is a bug since the migration
documentation currently specifies that a dynamically added date field,
by default, includes the epoch_millis format. This commit adds
epoch_millis as an alternative parser to dynamic date fields mapped from
dates of the form "yyyy/MM/dd".

Closes #12873
2015-08-18 22:30:43 -04:00
Robert Muir f7f7fecafb Improve console logging on startup exception
Today we show the exception twice: once by the logger and then again
by the JVM. This is too noisy, and easy to avoid.
2015-08-18 22:27:39 -04:00
Ryan Ernst 32a097382c Merge pull request #12952 from rjernst/construct_it_yourself
Simplify Plugin API for constructing modules
2015-08-18 14:28:48 -07:00
Ryan Ernst 54ae55662c Remove leftover class and fix compile issues from master merge 2015-08-18 14:28:03 -07:00
Simon Willnauer 32bb49e829 [TEST] Make it clear what address we try to bind 2015-08-18 23:17:43 +02:00
Ryan Ernst 7393068417 Merge branch 'master' into construct_it_yourself 2015-08-18 14:15:08 -07:00
Simon Willnauer 0ffd99cca3 Drop commons-lang dependency
commons-lang really is only used by some core classes to join strings or modiy arrays.
It's not worth carrying the dependency. This commit removes the dependency on commons-lang
entirely.
2015-08-18 22:59:31 +02:00
Robert Muir e07f039659 Workaround JDK bug 8034057
This causes a FileSystemException when trying to retrieve FileStore for a path,
and falsely returns false for Files.isWritable

Squashed commit of the following:

commit d2cc0d966f3bc94aa836b316a42b3c5724bc01ef
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Aug 18 15:49:48 2015 -0400

    fixes for the non-bogus comments

commit 6e0a272f5f8ef7358654ded8ff4ffc31831fa5c7
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Aug 18 15:30:43 2015 -0400

    Fix isWritable too

commit 2a8764ca118fc4c950bfc60d0b97de873e0e82ad
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Aug 18 14:49:50 2015 -0400

    try to workaround filestore bug
2015-08-18 16:44:57 -04:00
Igor Motov 509edefb04 Improve stability of restoreIndexWithMissingShards test
Replaced awaitBusy with assertBusy and increased the timeout.
2015-08-18 14:27:24 -04:00
Isabel Drost-Fromm 7dcf0da0da Brings Lucene query assertions back to QB test.
This brings the Lucene query assertions back to the querybuilder test that were
removed in the ancient past when we tested Lucene queries through their
inherent equals method. As we no longer do that it makes sense to do at least
coarse sanity checking on the generated Lucene query. More such checks are
being added as part of this commit.

Relates to #10217
2015-08-18 20:06:11 +02:00
Ryan Ernst 2a57539a28 Tweak exception message 2015-08-18 10:35:37 -07:00
Ryan Ernst a0c41218ff Merge branch 'master' into no_more_toe_stepping 2015-08-18 10:34:29 -07:00
Ryan Ernst a3afe57792 Fix compile failure from bad merge after renaming of
ExtensionPoint.TypeExtensionPoint
2015-08-18 10:32:12 -07:00
Ryan Ernst c0444ccbc6 Merge branch 'master' into module_culling3 2015-08-18 10:16:40 -07:00
Ryan Ernst 4c5cfd02cc Add javadocs to repository types registry methods 2015-08-18 10:13:01 -07:00
Ryan Ernst c3a22e6f0e Merge branch 'master' into construct_it_yourself 2015-08-18 09:50:47 -07:00
Ryan Ernst e61c5ce5c3 Fix naming problem with test plugin 2015-08-18 09:49:58 -07:00
Simon Willnauer 60f273c891 Suppress forbiddenAPI in logger when using localhost 2015-08-18 17:32:20 +02:00
Simon Willnauer 1e511eda28 Remove usage or `InetAddress#getLocalHost`
this method is very confusing and if it's used it's likely the wrong thing
with respect to the actual bound / published address. This change discourages
it's use and removes all useage. It's replaced with the actual published address
most of the time.
2015-08-18 16:48:34 +02:00
Alex Ksikes 99ac70860f Refactoring of Indices Query
Relates to #10217

This PR is against the query-refactoring branch.

Closes #12031
2015-08-18 15:30:44 +02:00
David Pilato 20851a4e4a Merge branch 'pr/shorten-artifactids' 2015-08-18 15:25:38 +02:00
javanna 4a3faf1126 Merge branch 'master' into feature/query-refactoring
Conflicts:
	core/src/main/java/org/elasticsearch/index/query/NotQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/TermsQueryBuilder.java
2015-08-18 15:20:16 +02:00
javanna 501a1996a3 Query DSL: remove attemped (not working) support for array in not query parser
Closes #12890
2015-08-18 14:58:53 +02:00
javanna 8e93ac5d5c Java api: remove execution from TermsQueryBuilder as it has no effect
Also introduced ParseField for execution in TermsQueryParser so proper deprecation warnings get printed out when requested.

Closes #12884
2015-08-18 14:57:33 +02:00
David Pilato 692cc80523 [maven] also rename parent project artifactId
Also fixed bad scm links
2015-08-18 13:41:12 +02:00
David Pilato e19090b2ad [plugin] shorten plugin names
We can now simplify PluginHandle as `name` and `repo` are the same thing.
2015-08-18 13:41:12 +02:00
David Pilato 33e8fae824 [plugin] shorten plugin names
We don't need to prefix anymore plugin names with `elasticsearch-` or `es-`.
2015-08-18 13:41:12 +02:00
David Pilato 2c902a4d2f [maven] rename Elasticsearch names to Elasticsearch: 2015-08-18 13:38:49 +02:00
David Pilato 0bb9593596 Fix a typo in comment 2015-08-18 13:38:48 +02:00
javanna 3843ae484d Merge branch 'master' into feature/query-refactoring 2015-08-18 13:14:03 +02:00
javanna 3f04ee076e Internal: IndicesQueriesRegitry back to being created only once
With #12921 we refactored IndicesModule but we forgot to make sure we create IndicesQueriesRegistry once. IndicesQueriesModule used to do `bind(IndicesQueriesRegistry.class).asEagerSingleton();` otherwise we get multiple instances of the registry. This needs to be ported do the IndicesModule.
2015-08-18 13:04:30 +02:00
Martijn van Groningen e74f559fd4 parent/child: Explicitly disabled the query cache
It was already disabled, but during tests the test framework enabled the query cache by setting the static default query cache. The caching behaviour should be the same in production and in tests.
2015-08-18 12:56:37 +02:00
Adrien Grand c169386dd4 Merge pull request #12931 from jpountz/fix/murmur3_defaults
Move the `murmur3` field to a plugin and fix defaults.
2015-08-18 12:09:32 +02:00
Ryan Ernst dc1fa6736a Merged AbstractPlugin and Plugin. Also added Settings back to
indexModules and shardModules
2015-08-18 02:46:32 -07:00
Adrien Grand a91b3fcbb9 Move the `murmur3` field to a plugin and fix defaults.
This move the `murmur3` field to the `mapper-murmur3` plugin and fixes its
defaults so that values will not be indexed by default, as the only purpose
of this field is to speed up `cardinality` aggregations on high-cardinality
string fields, which only requires doc values.

I also removed the `rehash` option from the `cardinality` aggregation as it
doesn't bring much value (rehashing is cheap) and allowed to remove the
coupling between the `cardinality` aggregation and the `murmur3` field.

Close #12874
2015-08-18 11:41:52 +02:00
Martijn van Groningen 2b97f5d9eb Pass down the EngineConfig to IndexSearcherWrapper
If a new IndexSearcher gets created the engine config can be used to get the query cache and query cache policy from.
2015-08-18 10:41:24 +02:00
javanna de54671173 Merge branch 'master' into feature/query-refactoring
Conflicts:
	core/src/main/java/org/elasticsearch/index/query/GeoBoundingBoxQueryBuilder.java
	core/src/main/java/org/elasticsearch/index/query/GeoBoundingBoxQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/GeoDistanceQueryBuilder.java
	core/src/main/java/org/elasticsearch/index/query/GeoDistanceQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/GeoDistanceRangeQueryBuilder.java
	core/src/main/java/org/elasticsearch/index/query/GeoDistanceRangeQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/GeoPolygonQueryBuilder.java
	core/src/main/java/org/elasticsearch/index/query/GeoPolygonQueryParser.java
	core/src/main/java/org/elasticsearch/indices/query/IndicesQueriesModule.java
	core/src/test/java/org/elasticsearch/index/query/plugin/DummyQueryParserPlugin.java
2015-08-18 10:35:17 +02:00
javanna 260a929e3b [TEST] inject a random index to TestClusterService in BaseQueryTestCase#init
Some of our next queries to refactor rely on some state taken from the cluster state. That is why we need to mock cluster service and inject an index to it, the index that we simulate the execution of the queries against. The best would be to have multiple indices actually, but that would make our setup a lot more complicated, especially given that IndexQueryParseService is still per index. We might be able to improve that in the future though, for now this is as good as it gets.
2015-08-18 09:14:40 +02:00
Ryan Ernst 2bf84593e0 Plugins: Simplify Plugin API for constructing modules
The Plugin interface currently contains 6 different methods for
adding modules. Elasticsearch has 3 different levels of injectors,
and for each of those, there are two methods. The first takes no
arguments and returns a collection of class objects to construct. The
second takes a Settings object and returns a collection of module
objects already constructed. The settings argument is unecessary because
the plugin can already get the settings from its constructor. Removing
that, the only difference between the two versions is returning an
already constructed Module, or a module Class, and there is no reason
the plugin can't construct all their modules themselves.

This change reduces the plugin api down to just 3 methods for adding
modules. Each returns a Collection<Module>. It also removes the
processModule method, which was unnecessary since onModule
implementations fullfill the same requirement. And finally, it renames
the modules() method to nodeModules() so it is clear these are created
once for each node.
2015-08-17 20:41:45 -07:00
Robert Muir 34635a4b4f Merge pull request #12951 from rmuir/more_networking_cleanup
Use preferIPv6Addresses for sort order, not preferIPv4Stack
2015-08-17 23:30:35 -04:00
Robert Muir 3ca12889e5 Use preferIPv6Addresses for sort order, not preferIPv4Stack
java.net.preferIPv6Addresses is a better choice. preferIPv4Stack is a nuclear option
and you just won't even bind to any IPv6 addresses. This reduces confusion.
2015-08-17 22:52:22 -04:00
Igor Motov 4114a8359d Improve stability of Snapshot/Restore test
The restore portion of some snapshot/restore test is failing randomly due to #9421. This change suspends rebalance during snapshot/restore operations until #9421 is fixed.

Closes #12855
2015-08-17 21:30:46 -04:00
Ryan Ernst 2e90be77ff Plugins: Ensure additionalSettings() do not conflict
Plugins can preovide additional settings to be added to the settings
provided in elasticsearch.yml. However, if two different plugins supply
the same setting key, the last one to be loaded wins, which is
indeterminate. This change enforces plugins cannot have conflicting
settings, at startup time. As a followup, we should do this when
installing plugins as well, to give earlier errors when two plugins
collide.
2015-08-17 17:01:00 -07:00
Ryan Ernst 6f124e6eec Internal: Simplify custom repository type setup
Custom repository types are registered through the RepositoriesModule.
Later, when a specific repository type is used, the RespositoryModule
is installed, which in turn would spawn the module that was
registered for that repository type. However, a module is not needed
here. Each repository type has two associated classes, a Repository and
an IndexShardRepository.

This change makes the registration method for custom repository
types take both of these classes, instead of a module.

See #12783.
2015-08-17 15:08:08 -07:00
Nicholas Knize ee227efc62 move integration test dependency file gzippedmap.gz from sources to resources 2015-08-17 16:49:25 -05:00
Simon Willnauer 8624022222 Print es.node.mode if integration tests fail 2015-08-17 22:50:54 +02:00
Nicholas Knize b2ba3847f7 Refactor geo_point validate* and normalize* options to ignore_malformed and coerce*
For consistency geo_point mapper's validate and normalize options are converted to ignore_malformed and coerced
2015-08-17 14:46:23 -05:00
Robert Muir 68307aa9f3 Fix network binding for ipv4/ipv6
When elasticsearch is configured by interface (or default: loopback interfaces),
bind to all addresses on the interface rather than an arbitrary one.

If the publish address is not specified, default it from the bound addresses
based on the following sort ordering:

* ipv4/ipv6 (java.net.preferIPv4Stack, defaults to true)
* ordinary addresses
* site-local addresses
* link local addresses
* loopback addresses

One one address is published, and multicast is still always over ipv4: these
need to be future improvements.

Closes #12906
Closes #12915

Squashed commit of the following:

commit 7e60833312f329a5749f9a256b9c1331a956d98f
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 14:45:33 2015 -0400

    fix java 7 compilation oops

commit c7b9f3a42058beb061b05c6dd67fd91477fd258a
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 14:24:16 2015 -0400

    Cleanup/fix logic around custom resolvers

commit bd7065f1936e14a29c9eb8fe4ecab0ce512ac08e
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 13:29:42 2015 -0400

    Add some unit tests for utility methods

commit 0faf71cb0ee9a45462d58af3d1bf214e8a79347c
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 12:11:48 2015 -0400

    localhost all the way down

commit e198bb2bc0d1673288b96e07e6e6ad842179978c
Merge: b55d092 b93a75f
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 12:05:02 2015 -0400

    Merge branch 'master' into network_cleanup

commit b55d092811d7832bae579c5586e171e9cc1ebe9d
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 12:03:03 2015 -0400

    fix docs, fix another bug in multicast (publish host = bad here!)

commit 88c462eb302b30a82585f95413927a5cbb7d54c4
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 11:50:49 2015 -0400

    remove nocommit

commit 89547d7b10d68b23d7f24362e1f4782f5e1ca03c
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 11:49:35 2015 -0400

    fix http too

commit 9b9413aca8a3f6397b5031831f910791b685e5be
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 11:06:02 2015 -0400

    Fix transport / interface code

    Next up: multicast and then http
2015-08-17 15:43:07 -04:00
Alex Ksikes 3ebf7a43f5 Refactors WrapperQueryBuilder and Parser
Relates to #10217
Closes #12037

This PR is against the query-refactoring branch.
2015-08-17 21:32:53 +02:00
Ryan Ernst 75ced057ed Fix changed class name after bad merge, see #12921 2015-08-17 12:14:01 -07:00
Ryan Ernst 52f3eb5898 Merge pull request #12921 from rjernst/module_culling2
Flatten IndicesModule and add tests
2015-08-17 12:08:17 -07:00
Alex Ksikes d62128fc78 Revert "Refactors WrapperQueryBuilder and Parser"
This reverts commit afcbd2915d.
2015-08-17 20:28:16 +02:00
Alex Ksikes afcbd2915d Refactors WrapperQueryBuilder and Parser
Relates to #10217
Closes #12037

This PR is against the query-refactoring branch.
2015-08-17 19:52:11 +02:00
Nik Everett 4307e165c1 Tests: Make a test less flakey
EsExecutorsTests had a test that was failing spuriously due to threadpools
being threadpools. This weakens the assertions that the test makes to what
should always be true.
2015-08-17 08:20:18 -07:00
Simon Willnauer ea03e5dd17 Add build short hash to the download manager headers to identify staging builds
It might turn out to be useful to have the actual commit hash of the version we are
looking for if our download manager can just redirect to the right staging repository.
2015-08-17 15:17:45 +02:00
Tanguy Leroux 8e052f0da2 Make platform specific assumptions in OS & Process probes tests 2015-08-17 14:47:23 +02:00
Boaz Leskes e424701819 Merge pull request #12922 from xuzha/xu-network
Refactor, remove _node/network and _node/stats/network. 

Closes #12889 , Closes #12922
2015-08-17 14:46:05 +02:00
Boaz Leskes bb34b2fd85 Elasticsearch bootstrap help shouldn't mention plugins
We have a dedicated entry point for that.

Closes #12933
2015-08-17 14:36:13 +02:00
Boaz Leskes 6c4ef32160 Test: un-mute PluginManagerUnitTests.testSimplifiedNaming 2015-08-17 14:14:50 +02:00
Simon Willnauer 9608fe9dff Fix test - and don't use URL.equals() 2015-08-17 14:14:23 +02:00
Boaz Leskes cd9552eb07 Test: mute PluginManagerUnitTests.testSimplifiedNaming 2015-08-17 14:02:09 +02:00
Martijn van Groningen e649d96eb1 Merge pull request #12881 from martijnvg/allow_for_customable_query_cache
Allow a plugin to supply its own query cache implementation
2015-08-17 11:13:45 +02:00
Martijn van Groningen 5123167a99 test: added a unit test for #12261 2015-08-17 11:10:05 +02:00
Simon Willnauer e7d075f6ae Add elasticsearch version as a prefix for the staging URL
This is purely for maintainance reasons since it easier to see if we can drop
certain stageing urls if we have the version next to the hash.
I also removed the gpg passphrase from the example URL since it's better to get prompted?
2015-08-17 11:05:49 +02:00
Christoph Büscher 32dfd249e9 Merge branch 'master' into feature/query-refactoring
Conflicts:
	core/src/test/java/org/elasticsearch/plugins/PluginManagerIT.java
2015-08-17 10:16:55 +02:00
Simon Willnauer 59f390f5d0 Endless recovery loop with `indices.recovery.file_chunk_size=0Bytes`
This is caused by sending the same file to the chunk handler with offset
`0` which in-turn opens a new outputstream and waits for bytes. But the next round
will send 0 bytes again with offset 0. This commit adds some checks / validators that those
settings are positive byte values and fixes the RecoveryStatus to throw an IAE if the same file
is opened twice.
2015-08-17 09:58:17 +02:00
Martijn van Groningen 12c40fa58a Allow plugins to register custom `QueryCache` implementations. 2015-08-17 09:55:32 +02:00
Adrien Grand 7765b0497d Merge pull request #12497 from oyiadom/master
Update BulkProcessor.java
2015-08-17 09:43:41 +02:00
Adrien Grand 1f2345db34 Merge pull request #12913 from xuzha/xu-exception
Validate class before cast.
2015-08-17 09:38:59 +02:00
Harish Kayarohanam 3976854ada Improve error handling of ClassCastException in terms aggregations.
What is the problem we are trying to solve ?
===========================================

When we are doing aggregations against a field name as shown in
https://github.com/HarishAtGitHub/elasticsearch-tester/blob/master/12135.py#L37-L46

search = {
           "aggs": {
             "NAME": {
               "terms": {
                 "field": "ip_str",
                 "size": 10
               }
             }
           }
         }
and when the field "ip_str" has values of different types in different indices
. say one is of type StringTerms type and other is of IP(LongTerms type) then
the aggregation fails as the types do not match(incompatible).
The failure throws a class cast exception as follows:
{
   "error": {
      "root_cause": [],
      "type": "reduce_search_phase_exception",
      "reason": "[reduce] ",
      "phase": "query",
      "grouped": true,
      "failed_shards": [],
      "caused_by": {
         "type": "class_cast_exception",
         "reason": "org.elasticsearch.search.aggregations.bucket.terms.LongTerms$Bucket cannot be cast to org.elasticsearch.search.aggregations.bucket.terms.StringTerms$Bucket"
      }
   },
   "status": 503
}

which is hard to understand . User cannot infer anything about the cause of the problem and what he should do from seeing the
class cast exception.

What can be the possible solution ?
===================================

Make the exception more readable by showing him the root cause of the problem so that he can
understand which area actually caused the problem, so that he can take necessary steps further.

Code Analysis
=============

Debugging code shows that:
the query /{indices}/_search?search_type=count involves two phases
1) search phase
***************
     searchService.sendExecuteQuery(...) [Ref: TransportSearchCountAction]

     what happens here ?
        the phase 1, which is the search phase goes without error.
        In this phase the shards for the given indexes are collected and the search is done on all asynchronously
        and finally collected in the variable "firstResults" and given to meger phase.

        [Flow: .... -> TransportSearchTypeAction -> method performFirstPhase]

2) merge phase
**************
     searchPhaseController.merge(...firstResults...) [Ref: TransportSearchCountAction]

     what happens here ?
        the "firstresults" QuerySearchResults are now to be aggregated and combined.

        [Flow: SearchPhaseController.merge(...) -> ..... -> InternalTerms.doReduce(...)]

the phase 1, which is the search phase goes without error.
The problem comes in phase 2, which is merge phase.
Now the individual term buckets are available.
As per the test case , there are two indices cast and cast2, so by default 10 shards.
cast has ip_str of type StringTerms
cast2 has ip_str of type ip which is actually LongTerms

so here two types of Buckets exist. StringTerms_Bucket and LongTerms_Bucket.
Now the aggregation is to be put inside the BucketPriorityQueue(size 2: as out of 10, 2 has hits) finally.
(docs of PriorityQueue: https://lucene.apache.org/core/4_4_0/core/org/apache/lucene/util/PriorityQueue.html#insertWithOverflow(T))

Now first the LongTerms$Bucket is put inside.
then the StringTerms$Bucket is to be put in.
This is the area where exception is thrown. What happens is when adding the StringTerms$Bucket now it has to
goes through the code "lessThan(element, heap[1])"
which finally calls

---------------------------------------------------------------------------------------------
|      StringTerms$Bucket.compareTerms(other)  <---------------- Area of exception          |
|                                                                                           |
--------------------------------------------------------------------------------------------

where when comparing one to other a type cast is done and it fails as StringTerms$Bucket and LongTerms$Bucket are
incompatible.

Approach to solve:
==================

The best way is to make user understand that the problem is when reducing/merging/aggregating the buckets which came as a result of
querying different shards, so that this will make them infer that the problem is because the values of the fields are of different types.
The message is also user friendly and much better than the indecipherable classcastexception.

The only place to infer correctly that the aggregation has failed is in the place where aggregations take place.
so

at InternalTerms.java -> (BucketPriorityQueue)ordered.insertWithOverflow(b);

so here I can throw AggregationExecutionException saying it is because the buckets are of different
types.

But when can I infer at this point that the failure is due to mismatch of types of buckets ???
it can be possible only if at this point it is informed that the problem which occurred deep inside
is due to buckets that were incomparable.
so from just a classCastException we cannot make such a pointed exact inference, because
as class cast exception can be due to a number of scenarios and at a number of places.

so unless we inform the exact problem to InternalTerms it will not be able to infer properly.
so infer the classCastException at the compareTerms function itself that it is a IncomparableTermBucktesTypeException.
This is the best place to infer classCastException as this the place which generated the exception.
Best inference of exceptions can be done only at the source/origin of the exception.

so IncomparableTermBucktesTypeException to InternalTerms-> will make it infer and conclude on why
aggregation failed and give best information to user.

Close #12821
2015-08-17 09:35:32 +02:00
xuzha 7624ae4f96 There is no longer a network section in the nodes info and stats
requests. Remove _node/network and _node/stats/network

closes #12889
2015-08-16 19:34:04 -07:00
Ryan Ernst 2450e3ccc8 Internal: Flatten IndicesModule and add tests
The IndicesModule was made up of two submodules, one which
handled registering queries, and the other for registering
hunspell dictionaries. This change moves those into
IndicesModule. It also adds a new extension point type,
InstanceMap. This is simply a Map<K,V>, where K and V are
actual objects, not classes like most other extension points.
I also added a test method to help testing instance map extensions.
This was particularly painful because of how guice binds the key
and value as separate bindings, and then reconstitutes them
into a Map at injection time. In order to gain access to the
object which links the key and value, I had to tweak our
guice copy to not use an anonymous inner class for the Provider.

Note that I also renamed the existing extension point types, since
they were very redundant. For example, ExtensionPoint.MapExtensionPoint
is now ExtensionPoint.ClassMap.

See #12783.
2015-08-16 17:56:35 -07:00
Martijn van Groningen 532806af1a inner hits: Use provided StreamContext instead of fetching a new one.
Closes #12905
2015-08-16 23:39:31 +02:00
Ryan Ernst 9974b79c8a Merge pull request #12916 from rjernst/module_culling
Flatten ClusterModule and add more tests
2015-08-16 10:08:40 -07:00
Simon Willnauer 5ab0833990 Don't swallow cause if Store stats can't be build 2015-08-16 16:36:22 +02:00
Simon Willnauer 5699492575 Merge pull request #12917 from HarishAtGitHub/refactorprocessFirstPhase
Refactor - shard variable dependency from processFirstPhaseResults as shard is no more needed
2015-08-16 16:21:13 +02:00
Simon Willnauer 606f5b368b mute the entire InnerHitsIT - Relates to #12905 2015-08-16 16:08:32 +02:00
Harish Kayarohanam 8122243ee7 this is a small commit to remove the
shard variable dependency from processFirstPhaseResults as shard is no more
needed here . it only deals with the results obtained from the synchronous search on each shard.
2015-08-16 17:10:25 +05:30
Ryan Ernst 008dc8ec31 Internal: Flatten ClusterModule and add more tests
The ClusterModule contained a couple submodules. This moves the
functionality from those modules into ClusterModule. Two of those
had to do with DynamicSettings. This change also cleans up
how DynamicSettings are built, and enforces they are added, with
validators, in ClusterModule.

See #12783.
2015-08-16 01:23:05 -07:00
xuzha 062e038360 Throw IllegalArgumentException instead of ClassCastException,
Let stats aggregation returns 400 error when performed over an invalid field

closes #12842
2015-08-15 15:42:03 -07:00
Andrey Fadeyev fc418de46e Format plugin info a bit more user-friendly to output it in a terminal
Before:

PluginInfo{name='cloud-aws', description='The Amazon Web Service (AWS) Cloud plugin allows to use AWS API for the unicast discovery mechanism and add S3 repositories.', site=false, jvm=true, classname=org.elasticsearch.plugin.cloud.aws.CloudAwsPlugin, isolated=true, version='2.1.0-SNAPSHOT'}

After:

- Plugin information:
Name: cloud-aws
Description: The Amazon Web Service (AWS) Cloud plugin allows to use AWS API for the unicast discovery mechanism and add S3 repositories.
Site: false
Version: 2.1.0-SNAPSHOT
JVM: true
 * Classname: org.elasticsearch.plugin.cloud.aws.CloudAwsPlugin
 * Isolated: true
2015-08-15 14:42:29 +01:00
Andrey Fadeyev 7d6acde538 Show PluginInfo information after install only in verbose mode 2015-08-15 14:40:03 +01:00
Simon Willnauer 20f6b41337 Mute InnerHitsIT Relates to #12905 2015-08-15 08:41:28 +02:00
Simon Willnauer b447e2ae99 Move master to [2.1.0-SNAPSHOT] 2015-08-14 23:44:06 +02:00
Jason Tedor 6292bc07f9 Merge pull request #12892 from jasontedor/fix/12865
Validate settings specified in index templates at template creation time
2015-08-14 15:43:16 -06:00
Jason Tedor b88d2f6255 Validate settings specified in index templates at template creation time
Previously settings specified in index templates were not validated upon
template creation. Creating an index from an index template with invalid
settings could lead to cluster stability issues because creation of such
indexes would bypass index settings validation.

This commit adds validation of settings specified in index templates at
template creation time. This works by routing the index template
settings through the index settings validation mechanism.

Closes #12865
2015-08-14 15:23:10 -06:00
Ryan Ernst 33690b990a Merge pull request #12872 from rjernst/resolve_your_own_config
Remove Environment.resolveConfig
2015-08-14 13:56:32 -07:00
Simon Willnauer 03ceabb1da Merge pull request #12894 from s1monw/fix_repro_line
Fix reproduction line to include project filters
2015-08-14 22:52:55 +02:00
Lee Hinman 6f5a25d98e [DOC] Use 127.0.0.1 instead of localhost in READMEs
Users with IPv6 preferred over IPv4 may have `localhost` resolve to
`::1` instead of `127.0.0.1`, so we should be explicit so they don't run
into issues.
2015-08-14 14:47:58 -06:00
Martijn van Groningen 942d040f45 inner hits: Reset the `ShardTargetType` after serializing inner hits.
This fixes a bug where only the first top level search hit has a shard target and any subsequent search hits don't.
2015-08-14 22:38:12 +02:00
Lee Hinman 1b9877bb65 Use Java 7 version of Files.readAllLines instead of Java 8 version 2015-08-14 14:36:17 -06:00
Lee Hinman d35a3a37eb Also catch NoSuchFileException 2015-08-14 13:54:14 -06:00
Lee Hinman 33f118e9c8 Print out the name of the sum that failed 2015-08-14 13:51:47 -06:00
Simon Willnauer bba34de6b3 Fix reproduction line to include project filters
Today on a failure the reproduce line printed out by the test framework
will build all projects and might fail if the test class is not present.
This commit adds a reactor filter to the reproduction line to ensure
unrelated projects are skipped.

Closes #12838
2015-08-14 21:36:49 +02:00
Ryan Ernst 867f056cf6 Simplify random name index and move method to its only user 2015-08-14 11:22:20 -07:00
Christoph Büscher 407781e76a Adding comments to test 2015-08-14 18:42:45 +02:00
Lee Hinman 41d8b552e6 Validate checksums for plugins if available
When a plugin is downloaded, this change additionally tries to download
`${pluginurl}.sha1` and verify the SHA1 checksum for the file. If no
.sha1 file is found, it tries `${pluginurl}.md5`.

Note that if neither checksum file is found, a notice is printed but the
plugin can still be installed. If the checksum check fails, the plugin
install is aborted.

Example output if no checksums are available:

```
bin/plugin install elasticsearch/elasticsearch-analysis-icu/2.6.0-SNAPSHOT
-> Installing elasticsearch/elasticsearch-analysis-icu/2.6.0-SNAPSHOT...
Trying http://download.elastic.co/elasticsearch/elasticsearch-analysis-icu/elasticsearch-analysis-icu-2.6.0-SNAPSHOT.zip ...
Trying http://search.maven.org/remotecontent?filepath=elasticsearch/elasticsearch-analysis-icu/2.6.0-SNAPSHOT/elasticsearch-analysis-icu-2.6.0-SNAPSHOT.zip ...
Trying https://oss.sonatype.org/service/local/repositories/releases/content/elasticsearch/elasticsearch-analysis-icu/2.6.0-SNAPSHOT/elasticsearch-analysis-icu-2.6.0-SNAPSHOT.zip ...
Trying https://github.com/elasticsearch/elasticsearch-analysis-icu/archive/2.6.0-SNAPSHOT.zip ...
Trying https://github.com/elasticsearch/elasticsearch-analysis-icu/archive/master.zip ...
Downloading .....................................DONE
Verifying https://github.com/elasticsearch/elasticsearch-analysis-icu/archive/master.zip checksums if available ...
NOTE: Unable to verify checksum for downloaded plugin (unable to find .sha1 or .md5 file to verify)
```

Example output if checksums are available:

```
bin/plugin install elasticsearch/elasticsearch-analysis-icu/2.6.0-SNAPSHOT
-> Installing elasticsearch/elasticsearch-analysis-icu/2.6.0-SNAPSHOT...
Trying http://download.elastic.co/elasticsearch/elasticsearch-analysis-icu/elasticsearch-analysis-icu-2.6.0-SNAPSHOT.zip ...
Trying http://search.maven.org/remotecontent?filepath=elasticsearch/elasticsearch-analysis-icu/2.6.0-SNAPSHOT/elasticsearch-analysis-icu-2.6.0-SNAPSHOT.zip ...
Trying https://oss.sonatype.org/service/local/repositories/releases/content/elasticsearch/elasticsearch-analysis-icu/2.6.0-SNAPSHOT/elasticsearch-analysis-icu-2.6.0-SNAPSHOT.zip ...
Trying https://github.com/elasticsearch/elasticsearch-analysis-icu/archive/2.6.0-SNAPSHOT.zip ...
Trying https://github.com/elasticsearch/elasticsearch-analysis-icu/archive/master.zip ...
Downloading .....................................DONE
Verifying https://github.com/elasticsearch/elasticsearch-analysis-icu/archive/master.zip checksums if available ...
Downloading .DONE
```

Example output if checksums fail:

```
bin/plugin install elasticsearch/elasticsearch-analysis-kuromoji/2.5.0 -url http://localhost:8000/elasticsearch-analysis-kuromoji-2.5.0.zip
-> Installing elasticsearch/elasticsearch-analysis-kuromoji/2.5.0...
Trying http://localhost:8000/elasticsearch-analysis-kuromoji-2.5.0.zip ...
Downloading .............................................DONE
Verifying http://localhost:8000/elasticsearch-analysis-kuromoji-2.5.0.zip checksums if available ...
Downloading .DONE
ERROR: incorrect hash, file hash: [dbdc9c2cd32782054497a21fbdcae3ca1ff23c80], expected: [dbdc9c2cd32782054497a21fbdcae3ca1ff23c80-bad]
```

Resolves #12750
2015-08-14 10:40:40 -06:00
Alexander Reelsen 0240b581e7 PluginManager: Fix automatically generated URLs for official plugins
In order to match the paths of official plugins, we need to fix
the broken test by removing the elasticsearch prefix from the official
plugin names before testing.
2015-08-14 18:28:54 +02:00
Igor Motov e44991d2ef Mute test SharedClusterSnapshotRestoreIT#renameOnRestoreTest
Working on the fix
2015-08-14 12:12:53 -04:00
Christoph Büscher e01c0927a6 Aggregations: Make ValueParser.DateMath aware of timezone setting
This PR adds a timezone field to ValueParser.DateMath that is
set to UTC by default but can be set using the existing constructors.
This makes it possible for extended bounds setting in DateHistogram
to also use date math expressions that e.g. round by day and apply
this rounding in the time zone specified in the date histogram
aggregation request.

Closes #12278
2015-08-14 18:11:39 +02:00
Simon Willnauer 4c1ef2c943 Fix test to also include the mapper-size 2015-08-14 18:11:27 +02:00
Simon Willnauer 30692fe917 Add mapper-size to plugin list 2015-08-14 18:04:45 +02:00
Martijn van Groningen 0688dddebf Also delegate rewrite the the wrapped IndexSearcher.
In the the AssertingIndexSearcher is used we then also have extra validation when rewritting the query.
2015-08-14 16:34:04 +02:00
Adrien Grand 2fecc7e5c9 Merge pull request #12875 from jpountz/fix/dumber_context_indexsearcher
Simplify ContextIndexSearcher.
2015-08-14 15:42:23 +02:00
Alexander Reelsen cedbd20f2c PluginManager: Change staging URL to reflect S3 bucket
In order to create releases without actually changing the version
as part of a commit, we also need to reflect the path of the potentially
changing S3 repo.
2015-08-14 15:23:31 +02:00
Adrien Grand b3e7146b22 Simplify ContextIndexSearcher.
In particular this commit moves collector wrapping logic from
ContextIndexSearcher to QueryPhase.
2015-08-14 15:03:38 +02:00
javanna 1cfe580cb7 Modified AbstractQueryBuilder default hashcode impl
Closes #12145
2015-08-14 13:50:57 +02:00
javanna 3e828924d4 [TEST] add test for equals and hashcode to BaseQueryTestCase 2015-08-14 13:50:57 +02:00
Ryan Ernst f3d63095db Merge branch 'master' into resolve_your_own_config
Conflicts:
	core/src/main/java/org/elasticsearch/env/Environment.java
	core/src/test/java/org/elasticsearch/index/analysis/commongrams/CommonGramsTokenFilterFactoryTests.java
	core/src/test/java/org/elasticsearch/index/analysis/synonyms/SynonymsAnalysisTest.java
	plugins/analysis-kuromoji/src/test/java/org/elasticsearch/index/analysis/KuromojiAnalysisTests.java
2015-08-14 03:34:26 -07:00
Ryan Ernst be638fb6ef Internal: Remove Environment.resolveConfig
This method has multiple modes of resolving config files by
first looking in the config directory, then on the classpath,
and finally by prefixing with "config/" on the classpath.

Most of the places taking advantage of this were tests, so they
did not have to setup a real home dir with config. The only place
that was really relying on it was the code which loads names.txt
to randomly choose a node name.

This change fixes test to setup fake home dirs with their config
files. It also makes the logic for finding names.txt explicit:
look in config dir, and if it doesn't exist, load /config/names.txt
from the classpath.
2015-08-14 03:03:47 -07:00
javanna f8a90edab2 Merge branch 'master' into feature/query-refactoring
Conflicts:
	core/src/main/java/org/elasticsearch/index/query/TermsLookupQueryBuilder.java
	core/src/main/java/org/elasticsearch/index/query/TermsQueryBuilder.java
2015-08-14 11:40:49 +02:00
javanna 4010e7e9a7 Java api: restore support for minimumShouldMatch and disableCoord in TermsQueryBuilder
TermsQueryParser still parses those values although deprecated. These need to be present in the java api as well to get ready for the query refactoring, where the builders are the intermediate query format that we parse our json queries into. Whatever the parser supports need to be supported by the builder as well.

Closes #12870
2015-08-14 11:17:19 +02:00
javanna 2e0b548b06 Java api: remove support for lookup cache in TermsLooukpBuilder
TermsQueryParser doesn't support the cache field anymore, so if it gets set through java api, the subsequent parsing of that query will throw error

Relates to #12870
2015-08-14 11:16:55 +02:00
Ryan Ernst 470f5370b9 Merge pull request #12868 from rjernst/bye_bye_classloaders
Remove ClassLoader from Settings
2015-08-14 02:15:07 -07:00
Alexander Reelsen 0f3ada159e Release: Create pre release script
Refactored a part out of the release script, so the user can
change the version locally as well as move the documentation
and change the Version.java

The background of this change is to have a very simple release
process that puts stuff into a staging environment, so the beta
release can be tested, before it is officially released.

This means the build_release script can be removed soon.
2015-08-14 11:04:32 +02:00
Ryan Ernst 6dcfda99e8 Internal: Remove ClassLoader from Settings
Settings currently has a classloader member, which any user (plugin
or core ES code) can access to load classes/resources. This is extremely
error prone as setting the classloder on the Settings instance is a
public method. Furthermore, it is not really necessary. Classes that
need resources should load resources using normal means
(getClass().getResourceAsStream). Those that need classes
should use Class.forName, which will load the class with the
same classloader as the calling class. This means, in the few
places where classes are loaded by string name, they will use
the appropriate loader: either the default classloader which loads
core ES code, or a child classloader for each plugin.

This change removes the classloader member from Settings, as
well as other classloader related uses (except for a handful
of cases which must use a classloader, at least for now).
2015-08-13 23:55:27 -07:00
Ryan Ernst dcf3f4679f Fourth time, for real, last mock -> test jar 2015-08-13 19:40:31 -07:00
Ryan Ernst c16772b0fc Undo accidental commit of crap
This reverts commit 589eecf55d.
2015-08-13 19:39:55 -07:00
Ryan Ernst 589eecf55d Fourth time's a charm, one more mock class to add to test jar 2015-08-13 19:36:53 -07:00
Ryan Ernst d7da6e673f One more mock rule needed for test jar 2015-08-13 16:28:38 -07:00
Ryan Ernst a89ea15b41 Add more mock classes to test jar 2015-08-13 15:32:03 -07:00
Ryan Ernst c4f8c333ec Fix test jar to contain Mock classes that were moved in 71a3bdb 2015-08-13 15:17:22 -07:00
Ryan Ernst dc1a2d2e5a Merge branch 'master' into fix/12784 2015-08-13 14:35:48 -07:00
Ryan Ernst 9f64c75391 Remove leftover classes 2015-08-13 14:33:11 -07:00
Ryan Ernst ccef0551ef Tests: Refactor classes only plugged in by tests to use package private extension points
We previous used something like Class.forName to load mock classes,
where tests would set a setting that was *supposed* to only be used by
tests. This change make these impls package private so that only tests
can change out these implementations, through test plugins.

closes #12784
2015-08-13 13:52:55 -07:00
Simon Willnauer 1f41a8c682 Don't cache percolator query on loading percolators
No need to load catch this query since it's cheap and not reused.
If we cache it, it can cause assertions to be tripped since this
method is executed during postRecovery phase and might still run while
nodes are shutdown in tests.
2015-08-13 22:27:14 +02:00
javanna 8ac439fc1b Query refactoring: resolve guice injection issue after last commit 2015-08-13 19:09:07 +02:00
javanna c797445457 Query refactoring: simplified generic types in IndicesQueriesRegistry and IndexQueryParseService 2015-08-13 18:22:50 +02:00
javanna babfea0ab7 Query refactoring: fixed some leftover generics warnings
Leftover after adding the query builder type to QueryParser. getBuilderPrototype is better typed now and doesn't require unchecked cast anymore. Fixed also some Tuple usage without types.
2015-08-13 17:51:21 +02:00
javanna 135aab0092 [TEST] minor change in BaseQueryTestCase#toQuery
Simplified if when we decide what to do based on the query and return value of supportsBoostAndQueryName
2015-08-13 16:12:54 +02:00
David Pilato 4e23fe2657 [build] revert maven 3.3.x fix
We need to revert #12803 as it creates some troubles in IntelliJ and also might fail when using older elasticsearch versions.
2015-08-13 15:21:06 +02:00
Adrien Grand 75d84631da Merge pull request #12833 from saschamarkus/saschamarkus_issue_11510_rebase
Throw Exception for missing settings file
2015-08-13 14:26:03 +02:00
Simon Willnauer 74f18d8c16 Introduce a formal ExtensionPoint class to stream line extensions
This commit tries to add some infrastructure to streamline how extension
points should be strucutred. It's a simple approache with 4 implementations
for `highlighter`, `suggester`, `allocation_decider` and `shards_allocator`.
It simplifies adding new extension points and forces to register classes instead
of strings.
2015-08-13 13:47:11 +02:00
Christoph Büscher 14ce6c36c6 Query DSL: Remove duplicate code in AndQueryParser
When parsing inner array of filters, AndQueryParser seems to
check for correct "filters" field name but then does the same
kind of operation in the `else` branch of the stament. This
seems like it can be removed.
2015-08-13 12:29:44 +02:00
David Pilato fbd8f69273 Build of QA: Smoke Test Shaded Jar fails under maven 3.3.x
Build fails with maven 3.3.1 and 3.3.3. To reproduce, install one of the 3.3.x versions of maven and run `mvn clean verify` in the root directory of the project. The build will fail in the QA: Smoke Test Shaded Jar module with the following error:

```
Started J0 PID(99979@flea.local).
Suite: org.elasticsearch.shaded.test.ShadedIT
  2> NOTE: reproduce with: ant test  -Dtestcase=ShadedIT -Dtests.method=testJodaIsNotOnTheCP -Dtests.seed=2F4D23A7462CF921 -Dtests.locale= -Dtests.timezone=Asia/Baku -Dtests.asserts=true -Dtests.file.encoding=UTF-8
FAILURE 0.06s | ShadedIT.testJodaIsNotOnTheCP <<<
  > Throwable #1: junit.framework.AssertionFailedError: Expected an exception but the test passed: java.lang.ClassNotFoundException
  > at __randomizedtesting.SeedInfo.seed([2F4D23A7462CF921:3A9404F1F69FD80]:0)
  > at junit.framework.Assert.fail(Assert.java:57)
  > at java.lang.Thread.run(Thread.java:745)
  2> NOTE: reproduce with: ant test  -Dtestcase=ShadedIT -Dtests.method=testGuavaIsNotOnTheCP -Dtests.seed=2F4D23A7462CF921 -Dtests.locale= -Dtests.timezone=Asia/Baku -Dtests.asserts=true -Dtests.file.encoding=UTF-8
FAILURE 0.01s | ShadedIT.testGuavaIsNotOnTheCP <<<
  > Throwable #1: junit.framework.AssertionFailedError: Expected an exception but the test passed: java.lang.ClassNotFoundException
  > at __randomizedtesting.SeedInfo.seed([2F4D23A7462CF921:C2502FD54D83433D]:0)
  > at junit.framework.Assert.fail(Assert.java:57)
  > at java.lang.Thread.run(Thread.java:745)
  2> NOTE: reproduce with: ant test  -Dtestcase=ShadedIT -Dtests.method=testjsr166eIsNotOnTheCP -Dtests.seed=2F4D23A7462CF921 -Dtests.locale= -Dtests.timezone=Asia/Baku -Dtests.asserts=true -Dtests.file.encoding=UTF-8
FAILURE 0.01s | ShadedIT.testjsr166eIsNotOnTheCP <<<
  > Throwable #1: junit.framework.AssertionFailedError: Expected an exception but the test passed: java.lang.ClassNotFoundException
  > at __randomizedtesting.SeedInfo.seed([2F4D23A7462CF921:35593286F4269392]:0)
  > at junit.framework.Assert.fail(Assert.java:57)
  > at java.lang.Thread.run(Thread.java:745)
  2> NOTE: leaving temporary files on disk at: /Users/Shared/Jenkins/Home/workspace/elasticsearch-master/qa/smoke-test-shaded/target/J0/temp/org.elasticsearch.shaded.test.ShadedIT_2F4D23A7462CF921-001
  2> NOTE: test params are: codec=CheapBastard, sim=DefaultSimilarity, locale=, timezone=Asia/Baku
  2> NOTE: Mac OS X 10.10.4 x86_64/Oracle Corporation 1.8.0_25 (64-bit)/cpus=8,threads=1,free=482137936,total=514850816
  2> NOTE: All tests run in this JVM: [ShadedIT]
Completed [1/1] in 6.61s, 5 tests, 3 failures <<< FAILURES!

Tests with failures:
  - org.elasticsearch.shaded.test.ShadedIT.testJodaIsNotOnTheCP
  - org.elasticsearch.shaded.test.ShadedIT.testGuavaIsNotOnTheCP
  - org.elasticsearch.shaded.test.ShadedIT.testjsr166eIsNotOnTheCP
```
Please note that build doesn't fail with maven 3.2.x and it doesn't fail if mvn command is executed inside the qa/smoke-test-shaded directory. Only when the build is started from the root directory the error above can be observed.

The reason is because of the shaded version which depends on elasticsearch core.
When Maven build the module only, then elasticsearch core is not added to the dependency tree.

```sh
mvn dependency:tree -pl :smoke-test-shaded
```

```
[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ smoke-test-shaded ---
[INFO] org.elasticsearch.qa:smoke-test-shaded:jar:2.0.0-beta1-SNAPSHOT
[INFO] +- org.elasticsearch.distribution.shaded:elasticsearch:jar:2.0.0-beta1-SNAPSHOT:compile
[INFO] |  +- org.apache.lucene:lucene-core:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-backward-codecs:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-analyzers-common:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-queries:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-memory:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-highlighter:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-queryparser:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-sandbox:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-suggest:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-misc:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-join:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-grouping:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-spatial:jar:5.2.1:compile
[INFO] |  \- com.spatial4j:spatial4j:jar:0.4.1:compile
[INFO] +- org.hamcrest:hamcrest-all:jar:1.3:test
[INFO] \- org.apache.lucene:lucene-test-framework:jar:5.2.1:test
[INFO]    +- org.apache.lucene:lucene-codecs:jar:5.2.1:test
[INFO]    +- com.carrotsearch.randomizedtesting:randomizedtesting-runner:jar:2.1.16:test
[INFO]    +- junit:junit:jar:4.11:test
[INFO]    \- org.apache.ant🐜jar:1.8.2:test
```

But if shaded plugin is involved during the build, it modifies the `projectArtifactMap`:

```sh
mvn dependency:tree -pl org.elasticsearch.distribution.shaded:elasticsearch,:smoke-test-shaded
```

```
[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ smoke-test-shaded ---
[INFO] org.elasticsearch.qa:smoke-test-shaded:jar:2.0.0-beta1-SNAPSHOT
[INFO] +- org.elasticsearch.distribution.shaded:elasticsearch:jar:2.0.0-beta1-SNAPSHOT:compile
[INFO] |  \- org.elasticsearch:elasticsearch:jar:2.0.0-beta1-SNAPSHOT:compile
[INFO] |     +- org.apache.lucene:lucene-backward-codecs:jar:5.2.1:compile
[INFO] |     +- org.apache.lucene:lucene-analyzers-common:jar:5.2.1:compile
[INFO] |     +- org.apache.lucene:lucene-queries:jar:5.2.1:compile
[INFO] |     +- org.apache.lucene:lucene-memory:jar:5.2.1:compile
[INFO] |     +- org.apache.lucene:lucene-highlighter:jar:5.2.1:compile
[INFO] |     +- org.apache.lucene:lucene-queryparser:jar:5.2.1:compile
[INFO] |     |  \- org.apache.lucene:lucene-sandbox:jar:5.2.1:compile
[INFO] |     +- org.apache.lucene:lucene-suggest:jar:5.2.1:compile
[INFO] |     |  \- org.apache.lucene:lucene-misc:jar:5.2.1:compile
[INFO] |     +- org.apache.lucene:lucene-join:jar:5.2.1:compile
[INFO] |     |  \- org.apache.lucene:lucene-grouping:jar:5.2.1:compile
[INFO] |     +- org.apache.lucene:lucene-spatial:jar:5.2.1:compile
[INFO] |     |  \- com.spatial4j:spatial4j:jar:0.4.1:compile
[INFO] |     +- com.google.guava:guava:jar:18.0:compile
[INFO] |     +- com.carrotsearch:hppc:jar:0.7.1:compile
[INFO] |     +- joda-time:joda-time:jar:2.8:compile
[INFO] |     +- org.joda:joda-convert:jar:1.2:compile
[INFO] |     +- com.fasterxml.jackson.core:jackson-core:jar:2.5.3:compile
[INFO] |     +- com.fasterxml.jackson.dataformat:jackson-dataformat-smile:jar:2.5.3:compile
[INFO] |     +- com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:jar:2.5.3:compile
[INFO] |     |  \- org.yaml:snakeyaml:jar:1.12:compile
[INFO] |     +- com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:jar:2.5.3:compile
[INFO] |     +- io.netty:netty:jar:3.10.3.Final:compile
[INFO] |     +- com.ning:compress-lzf:jar:1.0.2:compile
[INFO] |     +- com.tdunning:t-digest:jar:3.0:compile
[INFO] |     +- org.hdrhistogram:HdrHistogram:jar:2.1.6:compile
[INFO] |     +- org.apache.commons:commons-lang3:jar:3.3.2:compile
[INFO] |     +- commons-cli:commons-cli:jar:1.3.1:compile
[INFO] |     \- com.twitter:jsr166e:jar:1.1.0:compile
[INFO] +- org.hamcrest:hamcrest-all:jar:1.3:test
[INFO] \- org.apache.lucene:lucene-test-framework:jar:5.2.1:test
[INFO]    +- org.apache.lucene:lucene-codecs:jar:5.2.1:test
[INFO]    +- org.apache.lucene:lucene-core:jar:5.2.1:compile
[INFO]    +- com.carrotsearch.randomizedtesting:randomizedtesting-runner:jar:2.1.16:test
[INFO]    +- junit:junit:jar:4.11:test
[INFO]    \- org.apache.ant🐜jar:1.8.2:test
```

A fix could consist of fixing something on Maven side. Probably something changed in a recent version and introduced this "issue" but it might be not really an issue. More a fix.

There are two workarounds:

1) exclude manually elasticsearch core from shaded version in smoke-test-shaded module and add manually each lucene lib needed by elasticsearch

2) add a new `elasticsearch-lucene` (lucene) POM module which simply declares all needed lucene libs in subprojects (such as the smoke tester one).

I choose the later.

Closes #12791.
2015-08-13 12:15:19 +02:00
Colin Goodheart-Smithe a9a4903a0a Aggregations: Removed unused factor parameter in DateHistogramBuilder
DateHistogramParser does not recognise this parameter and therefore setting it would throw an exception as noted in https://github.com/elastic/elasticsearch/issues/6490. It is also not documented.
2015-08-13 09:47:05 +01:00
Lee Hinman ae7a73dc1c [TEST] Skip counting ExtraFS files when asserting path is cleared 2015-08-12 16:47:27 -06:00
Lee Hinman b58970a3fe [TEST] Remove custom data path tests, replace with better tests
Replace with a better, "unit-y" tests that couple the shared data path
with the correctly set `path.shared_data` configuration setting
2015-08-12 16:05:38 -06:00
Christoph Büscher 397a4305ad Merge branch 'master' into feature/query-refactoring 2015-08-12 23:52:15 +02:00
Christoph Büscher e3fc6a0f7e Fix SimpleQueryStringBuilderTest, allow part of only with mapping 2015-08-12 23:50:20 +02:00
Lee Hinman 0cb3e21a3e [TEST] mark tests in IndicesCustomDataPathIT with AwaitsFix 2015-08-12 15:44:19 -06:00
Christoph Büscher 901f5d6e23 Adapted BaseQueryTestCase to changes on master 2015-08-12 22:20:21 +02:00
Christoph Büscher b7bfcdc5f8 Merge branch 'master' into feature/query-refactoring 2015-08-12 21:56:03 +02:00
Simon Willnauer 605253a39f Cut over master to 2.0.0-SNAPSHOT 2015-08-12 21:16:08 +02:00
Lee Hinman 79d1568b58 Remove the `node.enable_custom_paths` setting
This setting is useless now that we have the `path.shared_data` setting.

Resolves #12776
2015-08-12 12:23:05 -06:00
Lee Hinman ff5ad39c7a Add `path.shared_data`
This allows `path.shared_data` to be added to the security manager while
still allowing a custom `data_path` for indices using shadow replicas.

For example, configuring `path.shared_data: /tmp/foo`, then created an
index with:

```
POST /myindex
{
  "index": {
    "number_of_shards": 1,
    "number_of_replicas": 1,
    "data_path": "/tmp/foo/bar/baz",
    "shadow_replicas": true
  }
}
```

The index will then reside in `/tmp/foo/bar/baz`.

`path.shared_data` defaults to `null` if not specified.

Resolves #12714
Relates to #11065
2015-08-12 10:51:46 -06:00
Simon Willnauer fe3179d9cc Return `408 REQUEST_TIMEOUT` if `_cluster/health` times out
Today we return `200 OK` which is misleading since we really didn't
produce a valid response / didn't wait long enough.
2015-08-12 17:51:20 +02:00
Martijn van Groningen 777921dfb0 test: simplified percolator recovery tests 2015-08-12 16:51:27 +02:00
Simon Willnauer ed9eef92dd [TEST] Fix off-by-one bug in corruptFile util function
The upper bound must be 0-based since we are corrupting an offset into
the file but it can be a 1-based length of the file which results in an
uncorrupted file.
2015-08-12 16:43:31 +02:00
Nik Everett e4e60774c1 Merge pull request #12787 from nik9000/configured_plugin
QA: Add configuration to the jvm-example plugin
2015-08-12 07:36:34 -07:00
saschamarkus e59bbf8e31 Issue 11510: Throw Exception for missing settings file
An Exception will be thrown when the givven YAML-file doesn't exist.
2015-08-12 15:13:03 +02:00
Tanguy Leroux 03c327ff12 Expose ClassloadingMXBean in Node Stats
Closes #12738
2015-08-12 14:29:13 +02:00
Konrad Beiske 44c667dac6 Merge pull request #12828 from s1monw/allow_master_elect_extension
Expose zen ElectMasterService as a Discovery extension point
2015-08-12 13:40:13 +02:00
Simon Willnauer fcb0427ed1 Expose zen ElectMasterService as a Discovery extension point
Some users need to override ElectMasterService from zen to add functionality
inside their infrastructure. This commit allows to extend it.
2015-08-12 13:00:51 +02:00
Colin Goodheart-Smithe 73c5c8fe79 Search: Split SearchModule.configure() into separate methods
This change splits SearchModule.configure() contents into separate methods so that sub-class of it (e.g. tests) can use only the parts they require
2015-08-12 12:00:14 +01:00
Adrien Grand 88da485015 Tests: Allow REST tests to run over https.
This adds new settings to the REST client, so that tests can be run over SSL
and optionally use an alternate truststore.
2015-08-12 11:46:57 +02:00
Simon Willnauer 6316718ae0 Flatten SearchService and clean up build-in registration
Relates to #12783
2015-08-12 09:01:09 +02:00
Jason Tedor 3b2f28fbe1 Reflect renaming of base test class 2015-08-11 23:58:03 -07:00
Jason Tedor d56dc7899d Merge pull request #12561 from jasontedor/feature/12560
Disallow type names to start with dots for new indices except for .percolator
2015-08-11 23:40:11 -07:00
Jason Tedor f9da6036a2 Disallow type names to start with dots for new indices except for .percolator
This commit will disallow type names to start with dots for version 2 and later indices except for .percolator.

Closes #12560
2015-08-11 23:34:41 -07:00
Ryan Ernst 1c5b49403b Fix type checking to work with java 7 2015-08-11 23:11:54 -07:00
Ryan Ernst 41af2ca75f Make module testcase abstract to appease test naming convetion checks 2015-08-11 22:17:38 -07:00
Ryan Ernst a4a43ea9ae Fix missing license headers 2015-08-11 22:03:47 -07:00
Ryan Ernst 50ba3bcdd6 Internal: Flatten Allocation modules and add back ability to plugin ShardsAllocators
There were two submodules of AllocationModule. This combines them into a
single module, adds a base test case for module testing, and adds back
the ability for plugins to provide custom ShardsAllocators.

closes #12781
2015-08-11 20:49:07 -07:00
Nik Everett 79d9f5b775 Logging: Log less source in slowlog
Instead of logging the entire `_source` in the indexing slowlog we log by
default just the first 1000 characters - this is controlled by the
`index.indexing.slowlog.source` settings and can be set to `true` to log the
whole `_source`, `false` to log none of it, and a number to log at most that
many characters.

Closes #4485
2015-08-11 13:16:04 -07:00
Ryan Ernst 57549a8133 Mappings: Validate parsed document does not have trailing garbage that is invalid json
This remerges PR #11414. The added validation for trailing garbage is
now only done when the parse method created the parser.
2015-08-11 10:42:18 -07:00
Ryan Ernst 80cca9e7ad Merge pull request #12779 from rjernst/fix/12753
Fix field type compatiblity check to work when only one previous type exists
2015-08-11 10:24:47 -07:00
Adrien Grand 99f4f09b84 Merge pull request #12756 from keety/terms_query_disable_coord
Add support for `disable_coord` param to `terms` query
2015-08-11 16:13:58 +02:00
javanna a1be2d63f9 Query refactoring: better naming consistency for getters
In the query refactoring branch we've been introducing getter methods for every bit that you can set to each query. The naming is not every consistent at the moment. The applied naming convention are the following:

- `innerQuery()` for any inner query, when there's only one of them
- when there's more than one inner query, use a prefix that identifies which query it is, and the `query` suffix (e.g. `positiveQuery` or `littleQuery`)
- `fieldName()` for the name of the field to be queried
- `value()` for the actual query

These changes don't break bw comp given that these getters were all introduced with the query refactoring which hasn't been released yet. Also we are modifying getters that don't have a corresponding setter, as the fields are final, hence we are not breaking consistency between getter and setter.

Closes #12800
2015-08-11 15:55:18 +02:00
javanna ecc5516d32 add type safety for QueryParser
QueryParser now explicitly declares the type of QueryBuilder that it creates.

Closes #12773
2015-08-11 15:48:44 +02:00
javanna d088d5a3b4 delete SimpleQueryStringParser.NAME constant
Previously restored this constant due to a bad merge, fixed now
2015-08-11 13:45:28 +02:00
javanna e2181990c4 Merge branch 'master' into feature/query-refactoring
Conflicts:
	core/src/main/java/org/elasticsearch/index/query/SimpleQueryStringParser.java
2015-08-11 13:24:17 +02:00
Simon Willnauer 9c6ed8c238 Merge pull request #12260 from s1monw/issues/12239
Include stacktrace in rendered exceptions
2015-08-11 12:55:33 +02:00
javanna db7219901d simple query string: remove (not working) support for alternate formats
Removed attempt of parsing of `field` rather than `fields` and attempted support of the following syntax:

```
{
  "simple_query_string": {
    "body" : {
      "query": "foo bar"
    }
  }
}
```

Both these two syntaxes were undocumented, untested and not working.

Added test for case when `fields` is not specified, then the default field is queried.

Closes #12794
Closes #12798
2015-08-11 12:54:26 +02:00
Christoph Büscher 6071db89ea Fix BaseQueryTestCase after merge with master 2015-08-11 12:19:39 +02:00
Simon Willnauer ca20282e84 Include stacktrace in rendered exceptions
This commit includes the stacktrace into the structured exception rendering
to ensure we can find the reason / cause for certain things quicker. This
is enabled by default and is very verbose. Users can disable it via `rest.exception.stacktrace.skip = true|false`

Closes #12239
2015-08-11 11:55:59 +02:00
Christoph Büscher 097511601c Merge branch 'master' into feature/query-refactoring 2015-08-11 11:33:03 +02:00
Simon Willnauer f9487f7451 Apply additional plugin settings only if settings are not explicit
We have a way to allow a plugin to specify additional settings. These
settings should only be applied if they are not already existing in the
node settings.
2015-08-11 10:06:55 +02:00
Ryan Ernst f4827e55e3 Test: More attempts at forcing netty test to always use netty 2015-08-11 00:43:29 -07:00
Ryan Ernst b81f5f6142 Revert "Merge pull request #11414 from rjernst/fix/2315"
This reverts commit 937732b17d, reversing
changes made to 6fea6c542c.
2015-08-10 23:13:19 -07:00
Ryan Ernst 8a587ead89 Mappings: Validate parsed document does not have trailing garbage that is invalid json
See #2315
2015-08-10 21:57:14 -07:00
Ryan Ernst 6fea6c542c Test: Fix netty integ test to ensure local mode is not used.
closes #12788
2015-08-10 20:57:55 -07:00
Igor Motov 3354a816fc Changes in unassigned info and version might not be transferred as part of cluster state diffs
The equalTo logic of ShardRouting doesn't take version and unassignedInfo into the account when compares shard routings.  Since cluster state diff relies on equal to detect the changes that needs to be sent to other cluster, this omission might lead to changes not being properly propagated to other nodes in the cluster.

Closes #12387
2015-08-10 19:58:36 -04:00
Ryan Ernst 1b3cd504a5 test: quiet netty integ test 2015-08-10 16:57:02 -07:00
Nik Everett dfb6d4d08f QA: Create a test plugin with configuration
In order to test the way plugins with configuration are installed and removed
we need a plugin with configuration in the repository. The simplest way to
get one is to make an "example" plugin.

In the process of making this example it became aparent that cat actions were
difficult to create outside of the org.elasticsearch.rest.action.cat package
because key methods in AbstractCatAction were package private. This makes them
protected and uses them to create the example configured plugin.

Relates to #12717 but is only one step of many to close it.
2015-08-10 14:51:49 -07:00
Ryan Ernst 40f119d85a This method on settings loaded a class, based on a setting value, using
the default classloader. It had all kinds of leniency in how the
classname was found, and simply cannot work with plugins having isolated
classloaders.

This change removes that method. Some of the uses of it were for custom
extension points, like custom repository or discovery types. A lot were
just there to plugin mock implementations for tests. For the settings
that were legitimate, all now support plugins adding the given setting
via onModule. For those that were specific to tests for mocks, they now
use Classes.loadClass (a helper around Class.forName). This is a
temporary measure until (in a future PR) tests can change the
implementation via package private statics.

I also removed a number of unnecessary intermediate modules, added a
"jvm-example" plugin that can be filled in in the future as a smoke test
for breaking plugins, and gave some documentation to "spawn" modules
interface.

closes #12643
closes #12656
2015-08-10 14:04:45 -07:00
Ryan Ernst b47f5976a5 Mappings: Fix field type compatiblity check to work when only one previous type exists.
This was a straight up bug found in #12753. If only one type existed,
the compatibility check for a new type was not strict, so changes to
an updateable setting like search_analyzer got through (but only
partially). This change fixes the check and adds tests (which were
previously a TODO).

This also fixes a bug in dynamic field creation which woudln't copy
fielddata settings when duplicating a pre-existing field with the
same name.

closes #12753
2015-08-10 12:41:36 -07:00
Christoph Büscher fcad98c142 Use Locale.ROOT in Fuzziness class to toUpperCase() 2015-08-10 18:02:27 +02:00
Alex Ksikes 626b1abde4 Refactoring of FuzzyQuery
Relates to #10217

This PR is against the query-refactoring branch.

Closes #11865
2015-08-10 17:23:59 +02:00
Christoph Büscher 0d55bfd481 Removing @AwaitsFix in PluginMangerIT after merging in fix from master 2015-08-10 15:58:19 +02:00
Christoph Büscher d90552e2fb Merge branch 'master' into feature/query-refactoring 2015-08-10 15:56:42 +02:00
Christoph Büscher a269b96799 Merge pull request #12581 from cbuescher/fix/12531
Fix setting timezone on default DateTime formatter
2015-08-10 15:05:39 +02:00
Christoph Büscher cdb39cf5dc Aggregations: Fix setting timezone on default DateTime formatter
This PR prevents setting timezone on ValueFormatter.DateTime. Instead
the timezone information needed when printing buckets key-as-string
information is provided at constrution time of the ValueFormatter, making
sure we don't overwrite any constants. This, however, made it necessary to
be able to access the timezone information when resolving the format
in ValueSourseParser, so the `time_zone` parameter is now parsed alongside
the `format` parameter in ValueSourceParser rather than in DateHistogramParser.

Closes #12531
2015-08-10 15:04:46 +02:00
Simon Willnauer 9a7257c55a Remove lenient store type parsing
Today we accept store types like `nio_fs`, `nioFs`, `niofs` etc.
this commit removes the leniency and only accepts plain values without underscore.
Yet, this commit also has a BWC layer that upgrades existing indices to the new version.
Affected enums are:
 * `nio_fs`
 * `mmap_fs`
 * `simple_fs`
2015-08-10 13:39:08 +02:00
Alexander Reelsen e24b57e613 PluginManager: Do not try other URs if specific URL was configured
If an URL specified with --url on the command line cannot be reached,
the plugin manager tries URLs at download.elastic.co automatically.

This can lead to download wrong plugins and also exposes potentially
the name of an internal plugin to the download service.

This fix ensures, that the plugin manager simply aborts, if the specified
URL cannot be downloaded.
2015-08-10 13:09:02 +02:00
Christoph Büscher a3c294d4e9 Move flag to track filter context to QueryShardContext
Currently there is a flag in the QueryParseContext that keeps track
of whether an inner query sits inside a filter and should therefore
produce an unscored lucene query. This is done in the
parseInnerFilter...() methods that are called in the fromXContent()
methods or the parse() methods we haven't refactored yet. This needs
to move to the toQuery() method in the refactored builders, since the
query builders themselves have no information about the parent query
they might be nested in.

This PR moves the isFilter flag from the QueryParseContext to the re-
cently introduces QueryShardContext. The parseInnerFilter... methods
need to stay in the QueryParseContext for now, but they already delegate
to the flag that is kept in QueryShardContext. For refactored queries
(like BoolQueryBuilder) references to isFilter() are moved from the
parser to the corresponding builder. Builders where the inner query
was previously parsed using parseInnerFilter...() now use a newly
introduces toFilter(shardContext) method that produces the nested lucene
query with the filter context flag switched on.

Closes #12731
2015-08-10 12:19:56 +02:00
Christoph Büscher 32e98aa377 Added @AwaitsFix for failing PluginManagerIT test after talking to spinscale 2015-08-10 11:45:50 +02:00
Christoph Büscher 4e936d1964 Merge branch 'master' into feature/query-refactoring 2015-08-10 11:41:09 +02:00
keety 9a385de077 fix to support disable_coord param in terms query 2015-08-09 18:50:34 -04:00
Clinton Gormley 205262c737 Enable the license checker over distribution/* and plugins/*
Moved the license checker config into the parent pom, and overrede
the license dir/target-to-check in distributions/pom.

Disabled the license checker explicitly for projects which run integration
tests but have no licenses dir:

* core
* distribution
* qa
* plugins/delete-by-query
* plugins/mapper-size
* plugins/site-example

Closes #12752
Closes #12754
2015-08-09 18:18:59 +02:00
Robert Muir 368c41666c WE CANT EVEN GET BIN/ELASTICSEARCH WORKING!!!! 2015-08-09 08:09:44 -04:00
Jason Tedor f8415ae7ca Avoid use of forbidden method in test 2015-08-07 19:09:52 -04:00
Jason Tedor 23b348040e Add script compilation stats
This commit adds basic support to track the number of times scripts are
compiled and compiled scripts are evicted from the script cache. These
statistics are tracked at the node level.

Closes #12673
2015-08-07 18:25:55 -04:00
javanna d920c9d6bf Merge branch 'master' into feature/query-refactoring 2015-08-07 16:45:36 +02:00
javanna 56c0b46f96 [TEST] remove duplicated tests from SearchQueryIT
After merging filters and queries, tests against indices filter and indices query became exactly the same. we can remove one copy then.
2015-08-07 16:04:51 +02:00