Commit Graph

5204 Commits

Author SHA1 Message Date
Chris Earle de6b4f35b1 Remove inaccurate Javadoc on Setting constructor
The 'Setting' constructor has some outdated Javadoc that suggested that it would automatically apply 'Property.NodeScope' if no scope is supplied, but no scope is added in that case.
2016-05-25 09:28:34 -05:00
Simon Willnauer eab3113204 Drop 1.x BWC and cut over to Writeable for Translog.Operation (#18565)
We still maintain BWC for the translog operations back to 1.1 which is not
supported in the current version anyway. This commit drops the bwc and moves
the operations to the Writeable interface enforcing immutability.
2016-05-25 11:51:28 +02:00
Britta Weber 6862c48791 Merge pull request #18495 from brwe/geo-query-highlight-II
skip all geo point queries in plain highlighter
2016-05-25 11:35:59 +02:00
Yannick Welsch dee34c916c Expand wildcards to closed indices in /_cat/indices (#18545)
Closed indices are already displayed when no indices are explicitly selected. This commit ensures that closed indices are also shown when wildcard filtering is used. It also addresses another issue that is caused by the fact that the cat action is based internally on 3 different cluster states (one when we query the cluster state to get all indices, one when we query cluster health, and one when we query indices stats). We currently fail the cat request when the user specifies a concrete index as parameter that does not exist. The implementation works as intended in that regard. It checks this not only for the first cluster state request, but also the subsequent indices stats one. This means that if the index is deleted before the cat action has queried the indices stats, it rightfully fails. In case the user provides wildcards (or no parameter at all), however, we fail the indices stats as we pass the resolved concrete indices to the indices stats request and fail to distinguish whether these indices have been resolved by wildcards or explicitly requested by the user. This means that if an index has been deleted before the indices stats request gets to execute, we fail the overall cat request. The fix is to let the indices stats request do the resolving again and not pass the concrete indices.

Closes #16419
Closes #17395
2016-05-25 10:02:14 +02:00
Tal Levy 0fa67e1538 Expose underlying processor to blame for thrown exception within CompoundProcessor (#18342)
Fixes #17823
2016-05-24 14:25:40 -07:00
Jason Tedor 568c26a76c Log OS and JVM on startup
This commit adds the OS and JVM to the initial logline on startup.

Relates #18557
2016-05-24 16:55:55 -04:00
Ali Beyad 105aee08b3 Removes multiple toXContent entry points for SnapshotInfo
SnapshotInfo had a toXContent and an externalToXContent, the former for
writing snapshot info to the snapshot blob and the latter for writing the
snapshot info to the APIs. This commit unifies writing x-content to one
method, toXContent, which distinguishes which format to write the
snapshot info in based on the Params parameter.  In addition, it makes
use of the already existing snapshot specific params found in the
BlobStoreFormat.

Closes #18494
2016-05-24 15:44:22 -04:00
Britta Weber b3a8c54928 Test for deadlock when relocating and indexing concurrently
see #18553

Reproduces with

gradle :core:integTest -Dtests.class=org.elasticsearch.recovery.RelocationIT -Dtests.method="testIndexAndRelocateConcurrently"  -Dtests.iters=20 -Dtests.failfast=true -Dtests.logger.level=DEBUG -Dtests.seed=9BE7064B819064FA:C44B70F31707D081 -Dtests.timeoutSuite=60000!
2016-05-24 20:06:21 +02:00
Nik Everett a93f578bf6 Move parsing of allocation commands into REST
Port them to the ObjectParser.

Don't let plugins register custom allocation commands
2016-05-24 11:59:05 -04:00
Jason Tedor f210605af8 Add test for parsing fractional seconds
This commit adds a test that ensures that strings containing a
fractional number of seconds are correctly parsed into milliseconds.

Relates #18548
2016-05-24 11:26:36 -04:00
Clinton Gormley 9c9bea9258 Set version to 5.0.0-alpha3 (#18550)
* Set version to 5.0.0-alpha3

* Updated version in qa/backwards tests too
2016-05-24 16:46:05 +02:00
Tanguy Leroux 1f011f9dea Remove Delete-By-Query plugin
closes #18469
2016-05-24 13:28:20 +02:00
Martijn van Groningen 27cc2fe4dc Moved the percolator from core to its own module
Significant changes:
* AbstractQueryTestCase has moved to the test framework module, in order for query builder tests in modules and plugins
* Added support to AbstractQueryTestCase to register plugins
* Lift the restriction that only one percolator could be added per index. This validation existed in MapperService, but because the percolator moved to a module it could no longer exist there. Instead of bringing it back it was removed. This validation existed since the percolator cache only supported one percolator query per document, since the percolator cache has been removed this restriction could removed as well.
* While moving percolator tests to the new module, also removed a couple of tests for the deprecated percolate and mpercolate api. These APIs are now sugar  APIs for bwc and rediect to the searvh and msearvh APIs. Some tests were still testing as if percolate and mpercolate API did the percolation, but this no longer the case and these tests could be removed.
2016-05-24 11:01:57 +02:00
Ryan Ernst ef435d0099 Remove unnecessary boxing and use of deprecated Double ctors 2016-05-23 13:44:59 -07:00
Lee Hinman bfce901edf Merge remote-tracking branch 'dakrone/explain-add-fetch-in-progress' 2016-05-23 09:43:16 -06:00
Lee Hinman 8040ed0c16 Add whether the shard state fetch is pending to the allocation explain API
If the shard state fetch is still pending, this will now return a
message like:

```json
{
  "shard" : {
    "index" : "i",
    "index_uuid" : "de1W1374T4qgvUP4a9Ieaw",
    "id" : 0,
    "primary" : false
  },
  "assigned" : false,
  "shard_state_fetch_pending": true,
  "unassigned_info" : {
    "reason" : "INDEX_CREATED",
    "at" : "2016-04-26T16:34:53.227Z"
  },
  "allocation_delay_ms" : 0,
  "remaining_delay_ms" : 0,
  "nodes" : {
    "z-CbkiELT-SoWT91HIszLA" : {
      "node_name" : "Brain Cell",
      "node_attributes" : {
        "testattr" : "test"
      },
      "store" : {
        "shard_copy" : "NONE"
      },
      "final_decision" : "NO",
      "final_explanation" : "the shard state fetch is pending",
      "weight" : 5.0,
      "decisions" : [ ]
    }
  }
}
```

Adds the `shard_state_fetch_pending` field and uses the state to
influence the final decision and final explanation.

Relates to #17372
2016-05-23 09:42:57 -06:00
Adrien Grand 31e4c16ec3 Merge pull request #18509 from terradatum/epoch
Support full range of Java Long for epoch DateTime
2016-05-23 12:27:38 +02:00
Adrien Grand 459916f5dd Remove custom Base64 implementation. #18413
This replaces o.e.common.Base64 with java.util.Base64.
2016-05-23 11:32:42 +02:00
Tanguy Leroux e7eb664c78 Change BlobPath.buildAsString() method 2016-05-23 10:50:40 +02:00
Adrien Grand c5a9edf1c7 Add `Character.MODIFIER_SYMBOL` to the list of symbol categories. #18402
Closes #18388
2016-05-23 10:11:35 +02:00
Jim Ferenczi 238d390637 Fixes for _only_nodes preferences:
* Handle multiple attributes/name (coma separated).
  * Shuffle the nodes that match the preferences.

Fix #12546
Fix #12700
2016-05-23 09:44:52 +02:00
Adrien Grand cb2cfdd9c0 Speed up named queries. #18470
Named queries have a performance bug when they are used with expensive queries
that need to perform a lot of work up-front like fuzzy or range queries
(including with points). The reason is that they currently re-create the weight
and scorer for every hit. Instead we should create weights exactly once and
use a single Scorer for all documents that are on the same segment.
2016-05-23 08:56:40 +02:00
Martijn van Groningen c1a0929123 percolator: Add support dor MatchNoDocsQuery in query terms extract service
Before the query extraction would have been aborted and the percolator query would be marked as unknown.
This resulted in a situation that these queries always need to be evaluated by the memory index at search time.
By adding support for this query many more percolator query candidate hits can skip the expensive memory index verification step. For example the `match` query parser returns a MatchNoDocsQuery if the query terms are removed by text analysis (lets query text only contained stop words).
2016-05-22 22:42:19 +02:00
G. Richard Bellamy cf54903580 Support full range of Java Long for epoch DateTime
Remove the arbitrary limit on epoch_millis and epoch_seconds of 13 and 10
characters, respectively. Instead allow any character combination that can
be converted to a Java Long.

Update the docs to reflect this change.
2016-05-22 13:08:20 -07:00
Ryan Ernst 37d36f2f4c Merge branch 'master' into java9 2016-05-21 14:19:58 -07:00
Ryan Ernst f01f15d3b8 Document the hack 2016-05-21 14:14:12 -07:00
Jason Tedor ba14aca218 Refactor property placeholder use of env. vars
This commit is a slight refactoring of the use of environment variables
in replacing property placeholders. In commit
115f983827 the constructor for
Settings.Builder was made package visible to provide a hook for tests to
mock obtaining environment variables. But we do not need to go that far
and can instead provide a small hook for this for tests without opening
up the constructor. Thus, in this commit we refactor
Settings.Builder#replacePropertyPlaceholders to a package-visible method
that accepts a function providing environment variables by names. The
public-visible method just delegates to this method passing in
System::getenv and tests can use the package-visible method to mock the
behavior they need without relying on external environment variables.
2016-05-21 17:06:56 -04:00
Ryan Ernst 41a5c0cfa1 Force java9 log4j hack in testing 2016-05-21 13:41:38 -07:00
Ryan Ernst b26f5711d6 Fix log4j buggy java version detection 2016-05-21 13:18:12 -07:00
Ryan Ernst 1d40c4bbc1 Make java9 work again
This change makes ES compile with java9 again, build 118.
* There are a handful of changes due to failure to determine types during compile.
* The attachment plugins which use tika needed to have tika upgraded in order to pickup fixes there for java 9.
* azure discovery and s3 repository indirectly depend on jaxb, which is no longer in the default modules. They now add a jaxb dependency externally, and make JarHell allow for this package.
2016-05-21 09:41:51 -07:00
Lee Hinman 1d8441b681 Merge remote-tracking branch 'dakrone/remove-script-mode' 2016-05-20 15:22:14 -06:00
Jason Tedor 115f983827 Fix env. var placeholder test so it's reproducible
This commit modifies the settings test for environment variables
placeholders so that it is reproducible. The underlying issue is that
the set of environment variables from system to system can vary, and
this means that's we can never be sure that a failing test will be
reproducible. This commit simplifies this test to not rely on external
forces that could influence reproducibility.

Relates #18501
2016-05-20 17:11:37 -04:00
Lee Hinman fdfd2a2f18 Remove ScriptMode class in favor of boolean true/false
This removes the ScriptMode class entirely, which was an enum with two
options (ON and OFF) which essentially boiled down to true and false.
Now the boolean values are used instead.
2016-05-20 15:01:30 -06:00
Jason Tedor 4c7993ea71 Netty request/response tracer should wait for send
We write to Netty channels in an async fashion, but notify listeners via
a transport service adapter before we are certain that the channel write
succeeded. In particular, the tracer logs are implemented via a
transport service adapter and this means that we can write tracer logs
before a write was successful and in some cases the write might fail
leading to misleading logs. This commit attaches the transport service
adapters to channel writes as a listener so that the notification occurs
only after a successful write.

Relates #18500
2016-05-20 16:26:46 -04:00
Ali Beyad 923d90d434 Remove use of a Fields class in snapshot responses that contains
x-content keys, in favor of declaring/using the keys directly.

Closes #18497
2016-05-20 15:07:41 -04:00
Simon Willnauer 35e705877b Limit retries of failed allocations per index (#18467)
Today if a shard fails during initialization phase due to misconfiguration, broken disks,
missing analyzers, not installed plugins etc. elasticsaerch keeps on trying to initialize
or rather allocate that shard. Yet, in the worst case scenario this ends in an endless
allocation loop. To prevent this loop and all it's sideeffects like spamming log files over
and over again this commit adds an allocation decider that stops allocating a shard that
failed more than N times in a row to allocate. The number or retries can be configured via
`index.allocation.max_retry` and it's default is set to `5`. Once the setting is updated
shards with less failures than the number set per index will be allowed to allocate again.

Internally we maintain a counter on the UnassignedInfo that is reset to `0` once the shards
has been started.

Relates to #18417
2016-05-20 20:37:45 +02:00
Britta Weber b493f0defe skip all geo point queries in plain highlighter
Geo queries and plain highlighter do not seem to work well
together (https://issues.apache.org/jira/browse/LUCENE-7293)
so we need to skip all geo related queries when we highlight.

closes #17537
2016-05-20 20:07:10 +02:00
Jason Tedor 61f40156d3 Do not decode path when sending error
Today when sending a REST error to a client, we send the decoded
path. But decoding that path can already be the cause of the error in
which case decoding it again will just throw an exception leading to us
never sending an error back to the client. It would be better to send
the entire raw path to the client and that is what we do in this commit.

Relates #18477
2016-05-20 12:15:30 -04:00
Igor Motov ce88d7f9ab Fix race condition in snapshot initialization
When a snapshot initialization fails, the create snapshot method may return before the snapshot metadata in the cluster state is removed. This can cause follow up snapshot-API related calls to fail due to a snapshot still running. This is causing CI failures when we try to delete indices that were participating in failed snapshot to a read-only repository.

Closes #18121
2016-05-20 10:52:08 -04:00
Daniel Mitterdorfer 8b962bb234 Increase log level for NettyHttpRequestSizeLimitIT
This test fails spuriosly in CI and is not reproducible locally.

With this commit we temporarily increase the log level in a few
packages that are suspected to reveal the cause.
2016-05-20 15:12:52 +02:00
Jason Tedor 140f9dfe5f Fix scaling thread pool test bug
This commit fixes a test bug in the scaling thread pool configuration
test. In particular, the test randomization could select min and max for
a thread pool configuration where both are equal to zero. This is a
violation of the requirements of the ThreadPoolExecutor. With this
commit, we now ensure that the max is bounded below by one.
2016-05-20 08:58:08 -04:00
Martijn van Groningen 80fee8666f percolator: Removed percolator cache
Before 5.0 for it was required that the percolator queries were cached in jvm heap as Lucene queries for two reasons:
1) Performance. The percolator evaluated all percolator queries all the time. There was no pre-selecting queries that are likely to match like we have today.
2) Updates made to percolator queries were visible in realtime, Today these changes are visible in near realtime. So updating no longer requires the percolator to have the queries in jvm heap.

So having the percolator queries in jvm heap via the percolator cache is now less attractive. Especially when there are many percolator queries then these queries can consume many GBs of jvm heap.
Removing the percolator cache does make the percolate query slower compared to how the execution time in 5.0.0-alpha1 and alpha2, but it is still faster compared to 2.x and before.
2016-05-20 14:52:16 +02:00
Christoph Büscher d3fe22c990 Improve adding clauses to `span_near` and `span_or` query
Currently the query builders expose the clauses of the span
query as a modifiable list. Instead we should make the that
getter return an unmodifiable list. Also renaming the method
used to add a clause from `clause(spanQuery)` to
`addClause(spanQuery)`.
2016-05-20 13:36:55 +02:00
Boaz Leskes 34ef5306d2 Snapshotting and sync could cause a dead lock TranslogWriter (#18481)
#18360 introduced an extra lock in order to allow writes while syncing the translog. This caused a potential deadlock with snapshotting code where we first acquire the instance lock, followed by a sync (which acquires the syncLock). However, the sync logic acquires the syncLock first, followed by the instance lock.

I considered solving this by not syncing the translog on snapshot - I think we can get away with just flushing it. That however will create subtleties around snapshoting and whether operations in them are persisted. I opted instead to have slightly uglier code with nest synchronized, where the scope of the change is contained to the TranslogWriter class alone.
2016-05-20 12:56:24 +02:00
Jason Tedor c257e2c51f Remove settings and system properties entanglement
Today when parsing settings during bootstrap, we add a system property
for every Elasticsearch setting. Additionally, settings can be set via
system properties. This commit simplifies this situation.
 - settings are no longer propogated to system properties
 - system properties can not be used to set settings
 - the "es." prefix on settings is no longer required (nor permitted)
 - test logging has a dedicated system property (tests.logger.level)

Relates #18198
2016-05-19 14:08:08 -04:00
Clinton Gormley dc33a83231 Remove the preserve_original option from the FingerprintAnalyzer (#18471)
The preserve_original option to the ASCIIFoldingFilter doesn't
play well with the FingerprintFilter, as it ends up producing
fingerprints like:

    "and consistent godel gödel is said sentence this yes"

The goal of the OpenRefine algorithm is to product a small normalized
ASCII fingerprint. There's no need to expose preserve_original.
2016-05-19 19:37:13 +02:00
Christoph Büscher d2515727d0 Improve random DateTimeZone creation in tests
We often require a random joda DateTimeZone in our tests. Currently
there are a few options for generating such a random DateTimeZone
from the set of available ids. Currently most random picks are not
really reproducable across different jvms because they rely on order
in the ids set implementation. The helper in DateProcessorFactoryTests
thus performs a sort on the set of ids before random picking from
the result, so I moved this to ESTestCase to make it publicly
available and changed all other tests to use that method.
2016-05-19 18:12:48 +02:00
Boaz Leskes 4d6887075f Log IndexShard.refresh logs under trace (#18435)
We log them every second...
2016-05-19 17:12:37 +02:00
Christoph Büscher 757ccf00b2 Enforce MatchQueryBuilder#maxExpansions() to be strictly positive 2016-05-19 16:59:37 +02:00
Jeff Evans 3cf4214255 Add better error message when analyzer created without tokenizer or analyzer type (#18455)
Closes #15492
2016-05-19 15:47:07 +02:00