Commit Graph

270 Commits

Author SHA1 Message Date
Shay Banon bc5bdbc5de Remove jsr166y now that we on Java 7, cleanup jsr166e to classes we use 2014-04-15 13:17:28 +02:00
Simon Willnauer 8bede7024f Use TransportBulkAction for internal request from IndicesTTLService
This prevents executing bulks internal autocreate indices logic
and ensures that this internal request never creates an index
automaticall.

This fixes a bug where the TTL purger thread ran after the actual
index it was purging was already closed / deleted and that re-created
that index.

Closes #5766
2014-04-15 12:40:25 +02:00
Simon Willnauer 754eb16835 Upgrade to Lucene 4.7.2 2014-04-14 18:30:03 +02:00
Bill Hwang dcc6a6e138 [BUILD] Remove site dependencies generation
Removed dependencies library generation.  It stalls Jenkins static analysis job
2014-04-07 16:01:02 -07:00
javanna 1ec4f8f04b [TEST] Replaced RestTestSuiteRunner with parametrized test that uses RandomizedRunner directly
ElasticsearchRestTests extends now ElasticsearchIntegrationTest and makes use of our ordinary test infrastructure, in particular all randomized aspects now come for free instead of having to maintain a separate (custom) tests runner

We previously parsed only the tests that needed to be run given the version of the cluster the tests are running against. This doesn't happen anymore as it didn't buy much and it would be harder to support as the tests get now parsed before the test cluster gets started. Thus all the tests are now parsed regardless of their skip sections, afterwards the ones that don't need to be run will be skipped through assume directives.

Fixed REST tests that rely on a specific number of shards as this change introduces also random number of shards and replicas (through randomIndexTemplate)

Closes #5654
2014-04-07 17:08:05 +02:00
Simon Willnauer 6f5b7fa086 [BUILD] Set -Dtests.jvms=auto by default to make use of multiple JVMs 2014-04-03 13:01:01 +02:00
Simon Willnauer 42b20d601f Upgrade to Lucene 4.7.1
* Removed XTermsFilter fixed in LUCENE-5502
 * Switched back to automaton queries that caused failures due to LUCENE-5532
 * Fixed Highlight test that has different results due to LUCENE-5538
2014-04-01 23:50:55 +02:00
javanna 806c4e87fb [TEST] Removed last occurences of cluster_seed, no longer used
Relates to #5233
2014-04-01 17:57:57 +02:00
Simon Willnauer 71de2bc414 [BUILD] Allow to set tests memory via the commandline 2014-04-01 14:12:52 +02:00
javanna 38dd501ab5 [TEST] added ExternalTestCluster that allows to run tests against an external cluster
All the ordinary test operations happen based on the ImmutableTestCluster base class and are executed via transport client. Will be used especially for the REST tests once migrated to the standard randomized runner.

Added new httpAddresses method to ImmutableTestCluster to be able to retrieve the http addresses to connect to for the REST tests. Both versions will look inside the cluster to figure out which nodes are available for http calls and their addresses.

The external cluster is used as global cluster if the tests.cluster system property is available. The property needs to contain a comma separated list of available elasticsearch nodes that will be used to connect to the cluster (e.g. localhost:9300,localhost:9301).

Only a subset of the integration tests can currently be run successfully against the external cluster, for more precision the ones that don't modify the cluster layout (don't require cluster() functionalities but rely only on immutableCluster()). Also at least two data nodes are required otherwise the ensureGreen calls cannot succeed.

Closes #5630
2014-04-01 11:45:35 +02:00
Kevin e78bbbf3ec add CBOR data format support 2014-03-28 20:30:39 +11:00
Simon Willnauer f1b32c4636 [Build] use the same execution hint file across the pom file 2014-03-27 16:28:22 +01:00
Simon Willnauer 11b51b1780 [Build] use branch version in execution hit file name 2014-03-27 15:32:53 +01:00
Adrien Grand b5b82626e7 Forbid Math.abs(int/long).
We have had a couple of bugs because of the use of these methods without paying
attention that it might return a negative value when provided with MIN_VALUE.
There is one common and legitimate usage of this method in order to perform
a modulo operation which would always return a positive number. This use-case
has been extracted to MathUtils.mod.

Close #5562
2014-03-27 14:50:43 +01:00
Robert Muir 5babf59813 Update to forbidden-apis 1.4.1
Closes #5492
2014-03-22 15:00:19 -04:00
Simon Willnauer 876b2592ac [Build] Skip topN hints if tests are skipped 2014-03-20 12:45:35 +01:00
Simon Willnauer 7c494461f3 [BUILD] Add top execution hints to test phase
Added a ant task that prints the Top N most expensive tests after
each test run.
2014-03-20 12:29:48 +01:00
David Smiley 644fdfc4aa Upgrade to Spatial4j 0.4.1 and JTS 1.13
Fixes #5279
2014-03-20 11:17:51 +01:00
Bill Hwang fe487373e6 Revert "Findbug warning supression"
This reverts commit 744eabad03.
2014-03-17 13:55:39 -07:00
Bill Hwang 744eabad03 Findbug warning supression
Added logic to enable findbug warnings supression via annotations
2014-03-17 13:35:37 -07:00
Dridi Boukelmoune 9500dddad3 Move systemd files from /etc to /usr/lib
As documented in systemd's manual pages tmpfiles.d(5) and systemd.unit(5),
a package should install its default configuration in /usr/lib, which can
be overriden by system administrators in /etc.

New locations in the rpm:
/usr/lib/systemd/system/elasticsearch.service
/usr/lib/tmpfiles.d/elasticsearch.conf
2014-03-17 14:06:34 +01:00
David Pilato c6915ef4d6 Enforce java version 1.7
When building elasticsearch, we now require to use java 1.7.

Maven will check that before compiling any class. If Java version is incorrect, you will get the following message:

```
[WARNING] Rule 0: org.apache.maven.plugins.enforcer.RequireJavaVersion failed with message:
Detected JDK Version: 1.6.0-65 is not in the allowed range [1.7,).
```

Closes #5428.
2014-03-17 08:43:33 +01:00
Simon Willnauer 821173b5cf Enforce query instance checking before it wrapper as a filter
We have the default QueryWrapperFilter as well as our custom one while
our wrapper is explicitly marked as no_cache such that it will never
be included in a cache. This was not consistenly used and caused several
problems during tests where p/c related queries were used as filters
and ended up in the cache. This commit adds the QueryWrapperFilter
ctor to the forbidden APIs to enforce the query instance checks.
2014-03-14 20:18:01 +01:00
javanna d80dd00424 upgrade randomized-testing to 2.1.1
Note that the standard `atLeast` implementation has now Integer.MAX_VALUE as upper bound, thus it behaves differently from what we expect in our tests, as we never expect the upper bound to be that high.
Added our own `atLeast` to `AbstractRandomizedTest` so that it has the expected behaviour with a reasonable upper bound.
See https://github.com/carrotsearch/randomizedtesting/issues/131
2014-03-14 11:47:00 +01:00
Martijn van Groningen 6f8f773f8c Disabled query size estimation in percolator, because this is too expensive cpu wise.
Lucene's RamUsageEstimator.sizeOf(Object) is to expensive.
Query size estimation will be enabled when a cheaper way of query size estimation can be found.

Closes #5372
Relates to #5339
2014-03-14 15:29:24 +07:00
Bill Hwang 2e56253293 Added static analysis profile to pom.xml
Added pmd, findbug as well as site generation logic to top pom.xml file
Created customized pmd ruleset
2014-03-13 12:23:07 -07:00
Adrien Grand e3b87926bf [Build] Remove XReferenceManager and XSearcherManager from forbidden-apis' exclude list.
These classes have been removed on the upgrade to Lucene 4.7.
2014-03-12 15:06:39 +01:00
Igor Motov 39d2377be6 Use patched version of TermsFilter to prevent using wrong cached results
See LUCENE-5502

Closes #5363
2014-03-11 20:48:22 -04:00
Shay Banon 8cfff9d796 jackson: upgrade to 2.3.2 2014-03-09 23:40:43 +01:00
Shay Banon 3ea746c45e mvel: upgrade to 2.1.9 2014-03-09 22:56:09 +01:00
Shay Banon 992747a159 Force merges to not happen when indexing a doc / flush
Today, even though our merge policy doesn't return new merge specs on SEGMENT_FLUSH, merge on the scheduler is still called on flush time, and can cause merges to stall indexing during merges. Both for the concurrent merge scheduler (the default) and the serial merge scheduler. This behavior become worse when throttling kicks in (today at 20mb per sec).

In order to solve it (outside of Lucene for now), we wrap the merge scheduler with an EnableMergeScheduler, where, on the thread level, using a thread local, the call to merge can be enabled/disabled.

A Merges helper class is added where all explicit merges operations should go through. If the scheduler is the enabled one, it will enable merges before calling the relevant explicit method call. In order to make sure Merges is the only class that calls the explicit merge calls, the IW variant of them is added to the forbidden APIs list.

closes #5319
2014-03-05 12:26:26 +00:00
Zachary Tong 7b16c5857d Percentiles aggregation.
A new metric aggregation that can compute approximate values of arbitrary
percentiles.

Close #5323
2014-03-03 18:06:14 +01:00
Simon Willnauer 8ceb98752d Move master to Java 1.7
Closes #5267
2014-02-27 15:12:02 +01:00
Simon Willnauer 30d7b8de2f Upgrade to Lucene 4.7
Closes #5104
Closes #5129
Closes #3757
2014-02-26 22:21:10 +01:00
Bill Hwang db57f7ed0e Add thrid party license generation profile 2014-02-20 10:05:10 -08:00
Isabel Drost-Fromm 48004ff8a5 Add mustache templating to query execution.
Adds support for storing mustache based query templates that can later be filled
with query parameter values at execution time. Templates may be both quoted,
non-quoted and referencing templates stored in config/scripts/*.mustache by file
name.

See docs/reference/query-dsl/queries/template-query.asciidoc for templating
examples.

Implementation detail: mustache itself is being shaded as it depends directly on
guava - so having it marked optional but included in the final distribution
raises chances of version conflicts downstream.

Fixes #4879
2014-02-20 12:21:59 +01:00
mrsolo dffc7cd06d code coverage hookup 2014-02-14 10:05:40 -08:00
Simon Willnauer ab97bf0fd9 s/\t/ / 2014-02-07 13:35:58 +01:00
Simon Willnauer 0fb8d982be Use patched version of ReferenceManager to prevent infinite loop in
RefrenceManager#accquire()

See LUCENE-5436
2014-02-06 21:45:38 +01:00
Simon Willnauer 9cf8251a0d Add RamUsageEstimator#sizeOf(Object) to forbidden APIs
This method can be a performance trap since it traverse the
entire object tree that is referenced by the provided object.
See LUCENE-5373
2014-01-31 21:43:20 +01:00
Simon Willnauer 91acca7836 Upgrade to Lucene 4.6.1
This upgrade includes a fix for RAM estimation on IndexReader
that allows to expose the amount of used bytes per segment now
as a setting in Elasticsearch. (LUCENE-5373)

Additionally this bugfix release contained a small fix for highlighting
that was already ported to Elasticsearch when reported (LUCENE-5361)

Closes #4897
2014-01-28 10:35:39 +01:00
Simon Willnauer 25b49dd50b Install SecurityManager inside ElasticsearchTestCase for easier randomization
We currently run always with SecurityManager installed. To make sure we
work also without we should randomly swap it out ie. run without the
security manager.
2014-01-24 22:18:05 +01:00
Simon Willnauer 55e8df40cd Move provided lucene-expression jar below the test-framework
Having a dep that refrences lucene-core before the test framework
confuses some runners.
2014-01-23 16:10:57 +01:00
Simon Willnauer 416e328cea Mark 'lucene-expression' as 'provided' in pom.xml
We currently pull in the lucene-expression module that is referenced
by lucene-suggest. Yet, we don't make use of this dependency at all
and it pulls in a bunch of unshaded libs like `antlr` and `asm` which
are pretty common in other projects. We should exclude this
dependency since we don't use it at all and it causes problems
when Elasticsearch is used as a node client. (see #4858)

If we mark the dependency as provided it won't be included in the
distribution.

Closes #4859
Closes #4858
2014-01-23 14:23:41 +01:00
Simon Willnauer 53192919c6 Move to [2.0] snap 2014-01-21 17:07:39 +01:00
Shay Banon f1174eac3a upgrade to guava 16.0
also fixes #4830
2014-01-21 15:26:03 +01:00
Simon Willnauer 7f51fbc5ab Add SecurityManger / policy when running tests.
This commit adds a security manager to the test JVMs
that prevents mainly writing files outside of the JVMs
current test directory.
2014-01-17 15:15:10 +01:00
Simon Willnauer 85ca6c6762 move to [1.0.0] SNAP 2014-01-16 10:28:38 +01:00
Alexander Reelsen c6155c5142 release [1.0.0.RC1] 2014-01-15 17:02:22 +00:00
David Pilato c386155a73 add MockPageCacheRecycler in test jar
MockPageCacheRecycler is missing in test jar which makes failing tests when using
test jar in plugins:

```
1> [2014-01-11 10:51:30,531][ERROR][test                     ] FAILURE  : testWikipediaRiver(org.elasticsearch.river.wikipedia.WikipediaRiverTest)
  1> REPRODUCE WITH  : mvn test -Dtests.seed=5DAFD4FBAE587363 -Dtests.class=org.elasticsearch.river.wikipedia.WikipediaRiverTest -Dtests.method=testWikipediaRiver -Dtests.prefix=tests -Dtests.network=true -Dfile.encoding=MacRoman -Duser.timezone=Europe/Paris -Des.logger.level=INFO -Des.node.local=true -Dtests.cluster_seed=134842C2D806FFC0
  1> Throwable:
  1> java.lang.NoClassDefFoundError: org/elasticsearch/cache/recycler/MockPageCacheRecycler
  1>     org.elasticsearch.test.cache.recycler.MockPageCacheRecyclerModule.configure(MockPageCacheRecyclerModule.java:30)
  1>     org.elasticsearch.common.inject.AbstractModule.configure(AbstractModule.java:60)
```

Closes #4694.
2014-01-13 09:06:59 +01:00