Commit Graph

5030 Commits

Author SHA1 Message Date
mikemccand d2ac95d93c Core: DistributorDirectory shouldn't search for directory when reading existing file
This was causing too much work e.g. when pulling node stats or when
opening a new reader, because the least_used distributor would
unnecessarily check free disk space on all path.data entires every
time we try to open a file for reading or check its length.

Closes #7306

Closes #7323
2014-08-19 08:56:06 -04:00
javanna 8458138b8c Internal: get request while percolating existing documents to keep around headers and context of the original percolate request
Closes #7333
2014-08-19 14:32:42 +02:00
javanna b1f532eb85 Internal: changed index templates api category from indices to cluster 2014-08-19 13:43:36 +02:00
Martijn van Groningen 383e64bd5c Aggregations: Add `children` bucket aggregator that is able to map buckets between parent types and child types using the already builtin parent/child support.
Closes #6936
2014-08-19 12:40:51 +02:00
David Pilato 122c2b7a12 Cli: parsing multiple times throws `AlreadySelectedException`
This issue has been fixed in commons-cli:1.3 project which sadly has not been released yet.
See https://issues.apache.org/jira/browse/CLI-183

This patch builds another list of options with no selected groups by default.

When commons-cli:1.3 will be released, we need to remove this patch.

Closes #7282.
2014-08-19 10:55:41 +02:00
mikemccand 81ced48f99 Core: switch to fixed thread pool by default for management threads
Switch management threads to a fixed thread pool with up to 5 threads, and queue size of 100 by default, after which excess incoming requests are rejected.

Closes #7318

Closes #7320
2014-08-18 15:46:29 -04:00
Martijn van Groningen 95a4e699bf [TEST] Don't invoke RoutingNodes.assertShardStats() via asserts many times in tests, just once on each reroute call. 2014-08-18 19:56:04 +02:00
Robert Muir ae326c4232 Tests: tone down random compression tests
Closes #7314
2014-08-18 10:37:45 -04:00
Britta Weber 9addac8300 function_score: remove explanation of query score from functions
The score is explained already, it should not be again explained per function.

Also, remove explanation from parameter list of ScoreFunction#explainScore()
and leave only the score.

This also removes ExplainableSearchScript which is not used anywhere and
was the only reason to have the Explanation in the parameter anyway.

closes #7245
2014-08-18 16:07:00 +02:00
Colin Goodheart-Smithe b2286915cd Geo: Adds support for GeoJSON GeometryCollection
Closes #2796
2014-08-18 13:55:39 +01:00
Colin Goodheart-Smithe 2c69dc06d1 Mapping: Fixes using nested doc array with strict mapping
Closes #7304
2014-08-18 10:19:29 +01:00
Adrien Grand 436e37cd76 Aggregations: Make the list of buckets for terms and histogram returned as a java.util.List.
The terms and histogram aggregations always have an order. So it would make the
response easier to consume to return the buckets as a list instead of a
collection in order to make it easier to do things like getting the first/last
buckets.

Close #7275
2014-08-18 09:25:32 +02:00
Igor Motov 0156bcbf32 Fix failed engine exception logging 2014-08-16 22:09:13 -04:00
uboness 40c63ad07a Fixed a request headers bug in transport client
Where the configured request headers were not sent with sniffing requests (both node/info & cluster state sniffing)
2014-08-16 03:01:14 +02:00
Igor Motov f4115b84a7 Add more logging for testSnapshotAndRestore backward compatibility test 2014-08-15 19:23:19 -04:00
javanna d0e2d768b8 Java api: fixed inconsistencies in PercolateRequest, added return type for method chaining where missing
Closes #7294
2014-08-15 15:57:14 +02:00
Robert Muir 5c7cefa292 Analysis: Add keep_types for filtering by token type 2014-08-15 09:28:12 -04:00
Colin Goodheart-Smithe f4d75f0212 REST API: Allows all options for expand_wildcards parameter
This change means that the default settings for expand_wildcards are only applied if the expand_wildcards parameter is not specified rather than being set upfront. It also adds the none and all options to the parameter to allow the user to specify no expansion and expansion to all indexes (equivalent to 'open,closed')

Closes #7258
2014-08-15 12:50:11 +01:00
Britta Weber d6312178ad [TEST] remove unneeded explain(true)
closes #7248
2014-08-14 15:59:21 +02:00
Britta Weber 9bebccba0c function_score: fix explanation, [ was missing 2014-08-14 15:59:19 +02:00
Adrien Grand feefc41127 [TESTS] Make DisabledFieldDataFormatTests less evil. 2014-08-14 15:26:15 +02:00
Adrien Grand a4f974dcaa Internal: Add some @Nullable annotations and fix related compilation warnings.
Added @Nullable to:
 - IndicesService.indexService
 - IndexService.shard
 - IndexService.shardInjector

This change doesn't try to do anything smart but just makes sure that a
*MissingException is thrown instead of a NullPointerException when the requested
object doesn't exist.

Close #7251
2014-08-14 14:55:52 +02:00
javanna 084793fca7 Internal: fixed wrong version check s/onOrBefore/before in TransportShardSingleOperationAction
relates to #7223
2014-08-14 14:19:37 +02:00
uboness f4a7793f89 Introduced a new elasticsearch exception family that can hold headers
- These heades will be copied as response header on the rest response
2014-08-14 12:34:38 +02:00
Colin Goodheart-Smithe 7602b13a58 Geo: Improved error handling in geo_distance
geo_distance filter now throws a parse exception if no distance parameter is supplied

Close #7260
2014-08-14 10:25:01 +01:00
Alexander Reelsen 6023a3a1a1 Plugins: Add executable flag to every file in bin/ after install
The PluginManager does not preserve permissions on install. This patch
sets the executable flag on every file in bin/ on plugin install.

Closes #7177
2014-08-14 10:53:08 +02:00
javanna 4d05d1d7b0 Internal: adjusted BroadcastShardOperationResponse subclasses visibility
Also replaced int,String pair with ShardId that holds the same info and serializes it the same way.
Replaced shardId and index getters in BroadcastOperationRequest with a single ShardId getter.

Closes #7255
2014-08-13 17:38:18 +02:00
Ryan Ernst 98063ba244 Add [1.3.3] and [1.2.5] version constants. 2014-08-13 08:03:52 -07:00
Lee Hinman fe86eddecb Forbid index names over 100 characters in length
Fixes #4417
2014-08-13 14:51:21 +02:00
Lee Hinman 4dc060527c Add GroovyCollections to the sandbox whitelist
Also clarify in the docs that changing the whitelist/blacklist settings
replace the list, they don't add to it.

Fixes #7089
Fixes #7088
2014-08-13 14:47:49 +02:00
Thomas Peuss 089658a36f A content decompressor that throws a human readable message when
compression is disabled and the user sends compressed content.
2014-08-13 12:25:11 +02:00
javanna ba8df3b5ba Update api & Indices stats: fixed version checks for no-op updates which got in after 1.3 was released
Also added basic bw comp test for indices stats api.

Relates to #6822
2014-08-13 11:45:09 +02:00
David Pilato 90dfb350e0 Tests: move plugin dir to plugins dir
We should be consistent in our naming for classes and resources.
2014-08-13 11:42:05 +02:00
David Pilato 02a90f3684 Fix: VerboseProgress(PrintWriter) does not set the writer 2014-08-13 11:37:05 +02:00
David Pilato 655282a2c6 Remove `numeric_range` filter
As done with #4034, `numeric_range` filter has been deprecated since 1.0.0.

Closes #7108.
2014-08-13 10:19:45 +02:00
javanna 270b109e65 Internal: adjusted visibility to package private for BroadcastShardOperationRequest subclasses and their constructors
Also replaced the String,int pair for index and shard_id with ShardId object that holds the same info and serialized it the same way too.

Closes #7235
2014-08-13 09:51:51 +02:00
Colin Goodheart-Smithe cd4aea841a Geo: fixes computation of geohash neighbours
The geohash grid it 8 cells wide and 4 cells tall. GeoHashUtils.neighbor(String,int,int.int) set the limit of the number of cells in y to < 3 rather than <= 3 resulting in it either not finding all neighbours or incorrectly searching for a neighbour in a different parent cell.

Closes #7226
2014-08-13 08:38:44 +01:00
Colin Goodheart-Smithe 2906d3e6dc Core: Store index creation time in index metadata
This change stores the index creation time in the index metadata when an index is created.  The creation time cannot be changed but can be set as part of the create index request to allow for correct creation times for historical data.

Closes #7119
2014-08-12 21:34:50 +01:00
mikemccand 8e35e921f7 Java client API: CreateIndexRequestBuilder.addMapping throws IllegalStateException if you add same type more than once
Previously, it would silently overwrite the previous mapping, which was trappy.

Closes #7231

Closes #7243
2014-08-12 14:32:38 -04:00
Alexander Reelsen e689a0ad71 Test: Allow CliTool to write out stacktraces
In order to have the possibility of debugging on the command line, the user
now can either set the es.cli.debug system property
which results in stack traces being written to to the terminal.

Closes #7222
2014-08-12 17:43:36 +02:00
David Pilato 9e6868733c Query DSL: Cache range filter on date field by default
A range filter on a date field with a numeric `from`/`to` value is **not** cached by default:

    DELETE /test

    PUT /test/t/1
    {
      "date": "2014-01-01"
    }

    GET /_validate/query?explain
    {
      "query": {
        "filtered": {
          "filter": {
            "range": {
              "date": {
                "from": 0
              }
            }
          }
        }
      }
    }

Returns:

    "explanation": "ConstantScore(no_cache(date:[0 TO *]))"

This patch fixes as well not caching `from`/`to` when using `now` value not rounded.
Previously, a query like:

    GET /_validate/query?explain
    {
      "query": {
        "filtered": {
          "filter": {
            "range": {
              "date": {
                "from": "now"
                "to": "now/d+1"
              }
            }
          }
        }
      }
    }

was cached.

Also, this patch does not cache anymore `now` even if the user asked for caching it.
As it won't be cached at all by definition.

Added as well tests for all possible combinations.

Closes #7114.
2014-08-12 15:27:02 +02:00
David Pilato b72f44b93a [Test] fix plugins: `bin` and `config` only plugins do not install correctly
Related to #7152.
2014-08-12 14:59:22 +02:00
David Pilato 14a028d62c plugins: `bin` and `config` only plugins do not install correctly
When installing a bin only plugin, it is identified as a site plugin.

A current workaround would be to create in the zip file another empty dir. So if you have:

* `bin/myfile.sh`
* `empty/empty.txt`

the `bin` content will be extracted as expected.

Closes #7152.
2014-08-12 14:40:51 +02:00
javanna 0ec7aa4492 [TEST] don't use multiple names (e.g. aliases) pointing to the same concrete index when using indexRandom
indexRandom will try to delete bogus documents multiple times since they get tracked by indexOrAlias/id, and after the actual deletion any other attempt throws error and fails the test
2014-08-12 14:34:11 +02:00
javanna 5d987ad5e2 Internal: changed every single index operation to not replace the index within the original request
An anti-pattern that we have in our code, noticeable for java API users, is that we modify incoming requests by replacing the index or alias with the concrete index. This way not only the request has changed, but all following communications that use that request will lose the information on whether the original request was performed against an alias or an index.

Refactored the following base classes: `TransportShardReplicationOperationAction`, `TransportShardSingleOperationAction`, `TransportSingleCustomOperationAction`, `TransportInstanceSingleOperationAction` and all subclasses by introduced an InternalRequest object that contains the original request plus additional info (e.g. the concrete index). This internal request doesn't get sent over the transport but rebuilt on each node on demand (not different to what currently happens anyway, as concrete index gets set on each node). When the request becomes a shard level request, instead of using the only int shardId we serialize the ShardId that contains both concrete index name (which might then differ ffrom the original one within the request) and shard id.

Using this pattern we can move get, multi_get, explain, analyze, term_vector, multi_term_vector, index, delete, update, bulk to not replace the index name with the concrete one within the request. The index name within the original request will stay the same.

Made it also clearer within the different transport actions when the index needs to be resolved and when that's not needed (e.g. shard level request), by exposing `resolveIndex` method. Moved check block methods to parent classes as their content was always the same on every subclass.

Improved existing tests by randomly introducing the use of an alias, and verifying that the responses always contain the concrete index name and not the original one, as that's the expected behaviour.

Added backwards compatibility tests to make sure that the change is applied in a backwards compatible manner.

Closes #7223
2014-08-12 13:25:23 +02:00
Colin Goodheart-Smithe 371d6021e7 Fix for failing BasePolygonBuilder 2014-08-12 11:04:07 +01:00
Colin Goodheart-Smithe 128b83e4a5 Geo: Better error for invalid multipolygon
Closes #7126
2014-08-12 10:30:26 +01:00
Colin Goodheart-Smithe 7c5a954b93 Geo: fixes geo_shapes which intersect dateline
If a geo_shape had edges which either ran vertically along the dateline or touched the date line but did not cross it they would fail to parse.  This is because the code which splits a polygon along the dateline did not take into account the case where the polygon touched but did not cross the dateline.  This PR fixes those issues and provides tests for them.

Close #7016
2014-08-12 09:54:16 +01:00
javanna 98fa8f9ba4 Internal: adjusted TermVectorRequest serialization to not serialize and de-serialize the index twice
Closes #7221
2014-08-12 10:33:09 +02:00
mikemccand 983c14b41a Test: don't need to wait for concrete mappings since we map both fields when we create the index 2014-08-12 04:24:10 -04:00
Martijn van Groningen a40cb169b6 Mappings: Make sure that multi fields are serialized in alphabetic order to ensure that the source is always the same.
Closes #7215
2014-08-11 18:45:02 +02:00
Martijn van Groningen 565dd90860 Core: Avoid null references that may be returned due to concurrent changes or inconsistent cluster state
Closes #7181
2014-08-11 18:32:26 +02:00
javanna ca5a17e4ba [DOCS] fixed DeleteIndexedScriptResponse javadocs 2014-08-11 17:55:31 +02:00
javanna c2594c0d3b Internal: adjusted visibility of GetRequest members (from protected to private) and resolved warning 2014-08-11 17:55:30 +02:00
Martijn van Groningen 2801d06aee Core: Pass down the types from the delete mapping request to the delete by query request.
The `.percolator` type is a hidden type and therefor the types from the delete mapping request should passed down to the delete by query request, otherwise the percolator type gets ignored and the percolator queries don't get deleted from disk (only unregistered).

Closes #7087
2014-08-11 17:19:03 +02:00
Ryan Ernst c1b6e53cbb Internal: Fix a very rare case of corruption in compression used for
internal cluster communication.

See CorruptedCompressorTests for details on how this bug can be hit.
This change also removes the ability to use the unsafe variant of
ChunkedEncoder, removing support for the compress.lzf.decoder setting.
2014-08-11 07:26:09 -07:00
Adrien Grand bc41190dba [Build] @Nullable annotation is not applied to primitive types. 2014-08-11 15:28:41 +02:00
Colin Goodheart-Smithe 5349ecdb77 Mapping: fixes dynamic mapping of geo_point fields
If a dynamic mapping for a geo_point field is defined and the first document specifies the value of the field as a geo_point array, the dynamic mapping throws an error as the array is broken into individual number before consulting the dynamic mapping configuration.  This change adds a check of the dynamic mapping before the array is split into individual numbers.

Closes #6939
2014-08-11 13:08:32 +01:00
Adrien Grand 1210b08ecb [TEST] Fix SimpleQueryTests.testRangeQuery assumptions.
This test assumed that the `num` field was mapped as an integer on all shards
and thus that all of them should fail when providing a timezone. However, since
it used dynamic mappings, some shards might have this field not mapped, as a
consequence they didn't fail.
2014-08-11 14:00:49 +02:00
javanna a03860970b Internal: refactored TransportSingleCustomOperationAction, subclasses and requests
TransportSingleCustomOperationAction is subclassed by two similar, yet different transport action: TransportAnalyzeAction and TransportGetFieldMappingsAction. Made their difference and similarities more explicit by sharing common code and moving specific code to subclasses:
- moved index field to the parent SingleCustomOperationAction class
- moved the common check blocks code to the parent transport action class
- moved the main transport handler to the TransportAnalyzeAction subclass as it is only used to receive external requests through clients. In the case of the TransportGetFieldMappingsIndexAction instead, the action is internal and executed only locally as part of the user facing TransportGetFieldMappingsAction. The corresponding request gets sent over the transport though as part of the related shard request
- removed the get field mappings index action from the action names mapping as it is not a transport handler anymore. It was before although never used.

Closes #7214
2014-08-11 11:08:38 +02:00
olivier bourgain ac40eae3e3 Core: Improve XContentBuilder API. 2014-08-11 10:40:10 +02:00
javanna c7a9b3da5b Internal: removed needless serialization code from TransportIndexReplicationAction and corresponding request object
TransportIndexReplicationAction is always executed locally, as an internal action that is part of either delete by query or delete (when routing is required but not specified). Only the corresponding shard level requests get sent over the transport, hence no transport endpoint is needed for the index version, nor the index request itself is supposed to be sent over the transport.

Moved classes from org.elasticsearch.action.delete.index to org.elasticsearch.action.delete and adjusted visibility so that internal requests are not public anymore.

Also removed serialization code from IndexDeleteResponse as it never gets sent over transport either.

Closes #7211
2014-08-11 10:02:04 +02:00
Alexander Reelsen fbd337921f Test: Improved CLI testing infrastructure
Added a CaptureOutputTerminal class to the infrastructure, which can be used
in tests, to make sure that CLI commands write out the right data
2014-08-11 09:52:55 +02:00
javanna d01e79429a [TEST] moved testUpdateRequest unit test method to a proper unit test class 2014-08-08 23:26:06 +02:00
Areek Zillur 99ae3066de Suggester: add suggestRequest to Requests and fix broken javadocs in client
closes #7206
2014-08-08 12:09:28 -04:00
mikemccand 5335e3d22d Test: make sure mappings are on all shards to prevent false test failures 2014-08-08 10:51:23 -04:00
Robert Muir 1a09e7180a vary offset/length (i hope) in compressed stream test 2014-08-08 09:54:46 -04:00
javanna 6d3bcc4451 Java API: add index, type and id to ExplainResponse
Index, type and id were returned as part of the REST explain api response, but not through java api. That info was read out of the request, relying on the fact that the index would get overridden with the concrete one within that same request.

Closes #7201
2014-08-08 12:52:03 +02:00
javanna 7f2b18864c Java API: fixed warnings in TermVectorRequest, also called super.validate for index not null check 2014-08-08 12:05:29 +02:00
Robert Muir 5377d03173 Fix BytesStreamInput(BytesReference) ctor with nonzero offset 2014-08-08 03:22:34 -04:00
Ryan Ernst 1386232e1f Test: Add test with mix of ints/longs/strings/bytes to compress. 2014-08-08 00:21:41 -07:00
Alexander Reelsen 724b14cef6 Mapping API: Improve IP address validation
Until now, IP addresses were only checked for four dots, which
allowed invalid values like 127.0.0.111111

This adds an additional check for validation.

Closes #7131
2014-08-08 08:48:38 +02:00
Robert Muir 4e16258648 add multithreaded compressed streams tests 2014-08-08 01:49:00 -04:00
Robert Muir 3f13967c36 remove dead unused code 2014-08-07 20:13:10 -04:00
Robert Muir e8e71e323a move unused code to src/test 2014-08-07 19:36:58 -04:00
Robert Muir 87670c613f ensure we exceed 65k in size sometimes in test 2014-08-07 18:40:47 -04:00
Robert Muir 59aabf30f8 add more evilness to compressed streams test 2014-08-07 18:30:27 -04:00
Martijn van Groningen e4d35c458a [TEST] Associate type with percolate query 2014-08-07 23:51:18 +02:00
Robert Muir d341274457 test some linedocs data with compressed streams 2014-08-07 17:20:41 -04:00
Robert Muir 975327775a test some random bytes with compressed streams 2014-08-07 16:47:36 -04:00
Ryan Ernst 90d2cb7dd5 Internal: Remove (mostly) unused 'failure' member from ShardSearchFailure.
closes #6837
closes #6861
2014-08-07 13:12:54 -07:00
Robert Muir 197ef0b205 test random strings in CompressedStringTests 2014-08-07 15:45:41 -04:00
uboness 1f9bceb5c5 Cleaned up TransportMessage and added transient context to it
- The context enables setting arbitrary transient data on the message (this data is not serialized with the request)
 - Changed header accessors/mutators so header manipulation will be done directly on the request (to void NPE with transport message headers when dealing with maps that can potentially be null)
2014-08-07 17:00:53 +02:00
javanna dbb513f5b0 Java api: remove operationThreaded setter from ExplainRequestBuilder, already available in base class SingleShardOperationRequestBuilder
It also didn't follow the setter convention that we adopted for request builders.
Fixed also javadocs warning caused byt missing descriptions for tag.

Closes #7186
2014-08-07 13:41:57 +02:00
javanna b73b37a8f0 Internal: fixed typo in NodeService s/disovery/discovery 2014-08-06 13:31:11 +02:00
javanna feb3839f0f [TEST] fixed unicast bw comp test configuration
The second internal node, when present, wasn't able to join the existing cluster due ti misconfigured unicast hosts, thus it would form its own cluster.
2014-08-06 13:31:11 +02:00
Adrien Grand 584d2c32dd [TEST] Assert that put mappings requests are acked in GeoMappingTests. 2014-08-06 11:56:20 +02:00
javanna 6f09eb1b06 [TEST] simplified unicast bw comp test
also fixed unchecked warning in CompositeTestCluster
2014-08-06 09:54:53 +02:00
uboness 6f73c93692 Added an option to add arbitrary headers to the client requests
The headers are key/value pairs defined in the settings under the `request.headers` namespace.
2014-08-06 03:33:08 +02:00
mikemccand 06709faff2 Fix false assert trip 2014-08-05 15:47:21 -04:00
Shay Banon f216dc4ab8 [TEST] make sure all shards have docs
we need that in order for refresh to be effective and actually refresh in the second round of indexing, otherwise, it caches a 0 docs shard and a refresh won't expire anything there
2014-08-05 20:28:47 +02:00
Shay Banon e6e2781ee7 [Query Cache] Add a request level flag to control query cache
A request level flag, defaults to be unset, to control the query cache. When not set, it defaults to the index level settings, when explicitly set, will override the index level setting
closes #7167
2014-08-05 18:28:49 +02:00
markharwood e6b459cb9f Update API enhancement - add support for scripted upserts.
In the case of inserts the UpdateHelper class will now allow the script used to apply updates to run on the upsert doc provided by clients. This allows the logic for managing the internal state of the data item to be managed by the script and is not reliant on clients performing the initialisation of data structures managed by the script.

Closes #7143
2014-08-05 16:52:44 +01:00
Shay Banon 418ce50ec4 Query Cache: Support shard level query response caching
The query cache allow to cache the (binary serialized) response of the shard level query phase execution based on the actual request as the key. The cache is fully coherent with the semantics of NRT, with a refresh (that actually ended up refreshing) causing previous cached entries on the relevant shard to be invalidated and eventually evicted.

This change enables query caching as an opt in index level setting, called `index.cache.query.enable` and defaults to `false`. The setting can be changed dynamically on an index. The cache is only enabled for search requests with search_type count.

The indices query cache is a node level query cache. The `indices.cache.query.size` controls what is the size (bytes wise) the cache will take, and defaults to `1%` of the heap. Note, this cache is very effective with small values in it already. There is also the advanced option to set `indices.cache.query.expire` that allow to control after a certain time of inaccessibility the cache will be evicted.

Note, the request takes the search "body" as is (bytes), and uses it as the key. This means same JSON but with different key order will constitute different cache entries.

This change includes basic stats (shard level, index/indices level, and node level) for the query cache, showing how much is used and eviction rates.

While this is a good first step, and the goal is to get it in, there are a few things that would be great additions to this work, but they can be done as additional pull requests:

- More stats, specifically cache hit and cache miss, per shard.
- Request level flag, defaults to "not set" (inheriting what the setting is).
- Allowing to change the cache size using the cluster update settings API
- Consider enabling the cache to query phase also when asking hits are involved, note, this will only include the "top docs", not the actual hits.
- See if there is a performant manner to solve the "out of order" of keys in the JSON case.
- Maybe introduce a filter element, that is outside of the request, that is checked, and if it matches all docs in a shard, will not be used as part of the key. This will help with time based indices and moving windows for shards that fall "inside" the window to be more effective caching wise.
- Add a more infra level support in search context that allows for any element to mark the search as non deterministic (on top of the support for "now"), and use it to not cache search responses.

closes #7161
2014-08-05 17:45:42 +02:00
Alexander Reelsen 35e67c84fa CORS: Allowed to configure allow-credentials header to work via SSL
This adds support to return the "Access-Control-Allow-Credentials" header
if needed, so CORS will work flawlessly with authenticated applications.

Closes #6380
2014-08-05 17:33:06 +02:00
Shay Banon 1d01b2ac6a [TEST] increase ack timeout on large cluster
when we have a small machine running it with randmoized larger number of nodes (5), we need more time to process it
2014-08-05 16:52:46 +02:00
Lee Hinman 8124bcae1e Make "cluster.routing.allocation.allow_rebalance" a dynamic setting
Also makes it a static constant and changes all tests to use it instead
of a string.

Fixes #7092
2014-08-05 16:26:09 +02:00
Shay Banon 9d79848998 [TEST] add explicit options to no master tests 2014-08-05 13:27:19 +02:00
javanna c788a5e67b [TEST} make sure unicast bw comp test uses specifically set transport port and properly configure unicast hosts
Added also assertBusy block needed since we test the local cluster state on each node, not only on the master node.
2014-08-05 11:47:10 +02:00
Colin Goodheart-Smithe 9c89fcf5a2 Aggregations: key_as_string only shown when format specified in terms agg
The key_as_string field is now not shown in the terms aggregation for long and double fields unless the format parameter is specified

Closes #7125
2014-08-05 10:38:59 +01:00
uboness 0da5cecc3c Added custom transport client settings to test infra
It's now possible to define the additional customesettings for transport clients by extending `transportClientSettings` callback method on `ElasticsearchIntegrationTest`.
2014-08-04 23:58:30 +02:00
javanna 74aa35bdcd [TEST] Fixed GetTermVectorTests, added missing break statements in randomization switch 2014-08-04 17:16:10 +02:00
javanna 24a1a0f07f [TEST] Fixed action names bw comp tests, action not found is expected for newly added exists actions 2014-08-04 17:00:40 +02:00
David Pilato 873a45eaba Search: add time zone setting for relative date math in range filter/query
Filters and Queries now supports `time_zone` parameter which defines which time zone should be applied to the query or filter to convert it to UTC time based value.

When applied on `date` fields the `range` filter and queries accept also a `time_zone` parameter.

The `time_zone` parameter will be applied to your input lower and upper bounds and will move them to UTC time based date:

[source,js]
--------------------------------------------------
{
    "constant_score": {
        "filter": {
            "range" : {
                "born" : {
                    "gte": "2012-01-01",
                    "lte": "now",
                    "time_zone": "+1:00"
                }
            }
        }
    }
}

{
    "range" : {
        "born" : {
            "gte": "2012-01-01",
            "lte": "now",
            "time_zone": "+1:00"
        }
    }
}
--------------------------------------------------

In the above examples, `gte` will be actually moved to `2011-12-31T23:00:00` UTC date.

NOTE: if you give a date with a timezone explicitly defined and use the `time_zone` parameter, `time_zone` will be
ignored. For example, setting `from` to `2012-01-01T00:00:00+01:00` with `"time_zone":"+10:00"` will still use `+01:00` time zone.

Closes #3729.
2014-08-04 15:42:03 +02:00
javanna d2fea5378a Transport: better categorization for transport actions
Our transport relies on action names that tell what we need to do with each message received and sent on any node, together with the content of the request itself.
The action names could use a better categorization and more consistent naming though, the following are the categories introduced with this commit:

- indices: for all the apis that execute against indices
  - admin: for the apis that allow to perform administration tasks against indices
  - data: for the apis that are about data
    - read: apis that read data
    - write: apis that write data
    - benchmark: apis that run benchmarks

- cluster: for all the cluster apis
  - admin: for the cluster apis that allow to perform administration tasks
  - monitor: for the cluster apis that allow to monitor the system

- internal: for all the internal actions that are used from node to node but not directly exposed to users

The change is applied in a backwards compatible manner: we keep the mapping old-to-new action name around, and when receiving a message, depending on the version of the node we receive it from, we use the received action name or we convert it to the previous version (old to new if version < 1.4). When sending a message, depending on the version of the node we talk to, we use the updated action or we convert it to the previous version (new to old if version < 1.4).
For the cases where we don't know the version of the node we talk to, namely unicast ping, transport client nodes info and transport client sniff mode (which calls cluster state), we just use a lower bound for the version, thus we will always use the old action name, which can be understood by both old nodes and new nodes.

Added test that enforces known updated categories for transport action names and test that verifies all action names have a pre 1.4 version for bw compatibility

Added backwards compatibility tests for unicast and transport client in sniff mode, the one for the ordinary transport client (which calls nodes info) is implicit as it's used all the time in our bw comp tests.
Added also backwards comp test that sends an empty message to any of the registered transport handler exposed by older nodes and verifies that what gets back is not ActionNotFoundTransportException, which would mean that there is a problem in the actions mappings.

Added TestCluster#getClusterName abstract method and allow to retrieve externalTransportAddress and internalCluster from CompositeTestCluster.

Closes #7105
2014-08-04 15:24:16 +02:00
mikemccand a58d9a1dd0 Core: simultaneous create/delete against same id can cause silently inconsistent replica
If simultaneous create & delete operations arrive against the same id,
it's possible that primary and replica see those operations in
different orders, which may result in replica throwing
DocumentAlreadyExistsException when the primary didn't which would
lead to replica being inconsistent (missing a document that primary
had indexed).

This push fixes the issue, by never throwing DAEE from the replica on
create.

Closes #7146 #7142
2014-08-04 09:14:09 -04:00
javanna 8989d062cd MultiGet & MultiTermVector api: fail when using no routing and an alias to an index that has routing required (for that doc type)
Made sure that the routing required check is performed against the concrete index, added use of aliases to existing routing tests.

Taken the change to unify the failure message as well to this form: routing is required for [" + index + "]/[" + type + "]/[" + id + "]

Closes #7145
2014-08-04 14:19:19 +02:00
Shay Banon 5795e4fbd7 [TEST] better failure message when back location is missing 2014-08-04 12:50:37 +02:00
Martijn van Groningen c8cc59df57 Percolator should cache index field data instances.
Before the index reader used by the percolator didn't allow to register a CoreCloseListener, but now it does, making it safe to cache index field data cache entries.
Creating field data structures is relatively expensive and caching them can save a lot of noise if many queries are evaluated in a percolator call.

Closes #6806
Closes #7081
2014-08-04 10:23:34 +02:00
Britta Weber 5706858722 Add parameter to GET for checking if generated fields can be retrieved
Fields of type `token_count`, `murmur3`, `_all` and `_field_names` are generated only when indexing.
If a GET requests accesses the transaction log (because no refresh
between indexing and GET request) then these fields cannot be retrieved at all.
Before the behavior was so:

`_all, _field_names`: The field was siletly ignored
`murmur3, token_count`: `NumberFormatException` because GET tried to parse the values from the source.

In addition, if these fields were not stored, the same behavior occured if the fields were
retrieved with GET after a `refresh()` because here also the source was used to get the fields.

Now, GET accepts a parameter `ignore_errors_on_generated_fields` which has
the following effect:
- Throw exception with meaningful error message explaining the problem if set to false (default)
- Ignore the field if set to true
- Always ignore the field if it was not set to stored

This changes the behavior for `_all` and `_field_names` as now an Exception is thrown if a user
tries to GET them before a `refresh()`.

closes #6676
closes #6973
2014-08-04 08:15:34 +02:00
Britta Weber a3cefd919e significant terms: add google normalized distance, add chi square
closes #6858
2014-08-04 08:15:26 +02:00
uboness b667bcdedf fixed platform independent line separator in CliToolTests 2014-08-02 21:02:22 +02:00
Shay Banon 95762e8126 Support "default" for tcpNoDelay and tcpKeepAlive
Allow to set the value default to network.tcp.no_delay and network.tcp.keep_alive so they won't be set at all, since on solaris, setting tcpNoDelay can actually cause failure
relates to #7115
2014-08-02 17:32:41 +02:00
uboness 5ccc7beaf4 Added a cli infrastructure
CliTool is a base class for command-line interface tools (such as the plugin manager and potentially others). It supports the following:
  - single or multi command tool
  - help printing infrastructure (based on help files)
  - consistent mechanism of parsing arguments (based on commons-cli lib)
  - separation of argument parsing and command execution (for easier unit testing)
  - terminal abstraction (will use System.console() when available)
2014-08-02 17:16:27 +02:00
Shay Banon 2d31349ab0 Fix missing break statement causing reroute serialization failure
closes #7135
2014-08-02 16:52:52 +02:00
Areek Zillur b81b240924 [Fix] CompletionMapper throws misleading error on null value
closes #6399
2014-08-01 15:25:05 -04:00
uboness 3c9c9f33e2 Aggregations Added Filters aggregation
A multi-bucket aggregation where multiple filters can be defined (each filter defines a bucket). The buckets will collect all the documents that match their associated filter.

This aggregation can be very useful when one wants to compare analytics between different criterias. It can also be accomplished using multiple definitions of the single filter aggregation, but here, the user will only need to define the sub-aggregations only once.

Closes #6118
2014-08-01 16:01:08 +01:00
Adrien Grand d9d5b35be9 Sort: Make `ignore_unmapped` work for cross-index queries.
Close #2255
2014-08-01 15:30:17 +02:00
Lee Hinman db7b6097cc [TEST] check breaker reset after parent trip instead of trip count 2014-08-01 15:18:38 +02:00
javanna d5b6de3295 Removed support for aliases as part of index settings
Now that we have explicit support for aliases when creating indices and as part of index templates, we may remove support for aliases (only names) as part of index settings. This is partially breaking as the following calls:

curl -XPUT localhost:9200/index -d '{
  "settings" : {
    "aliases" : [ "alias1"]
  }
}

and

curl -XPUT localhost:9200/index -d '{
  "settings" : {
    "index.aliases" : [ "alias1"]
  }
}

were previously supported and will need to be replaced with

curl -XPUT localhost:9200/index -d '{
  "aliases" : {
    "alias1": {}
  }
}

Closes #5545
2014-08-01 13:49:43 +02:00
Skye Book 0040ed4f6f Function score query: Add missing whitespace when throwing exception
DecayFunctionParser throws a parse exception with a string containing "scaleand origin", this fixes the spacing issue.
2014-08-01 12:43:26 +02:00
Adrien Grand a9d5c03924 Aggregations: Fix infinite loop in the histogram reduce logic.
The histogram reduce method can run into an infinite loop if the
Rounding.nextRoundingValue value is buggy, which happened to be the case for
DayTimeZoneRoundingFloor.

DayTimeZoneRoundingFloor is fixed, and the histogram reduce method has been
changed to fail instead of running into an infinite loop in case of a buffy
nextRoundingValue impl.

Close #6965
2014-08-01 08:58:10 +02:00
Adrien Grand 99b32901d2 Mappings: Fix `copy_to` behavior on nested documents.
Today, `copy_to` always copies a field to the current document, which is often
wrong in the case of nested documents. For example, if you have a nested field
called `n` which has a sub-field `n.source` whose content should be copied to
`target`, then the latter field should be created in the root document instead
of the nested one, since it doesn't have `n.` as a prefix. On the contrary, if
you configure the destination field to be `n.target`, then it should go to the
nested document.

Close #6701
2014-08-01 08:57:33 +02:00
Areek Zillur 1d581e6286 Search Exists API: Checks if any matching documents exist for a given query
Implements a new Exists API allowing users to do fast exists check on any matched documents for a given query.
This API should be faster then using the Count API as it will:
 - early terminate the search execution once any document is found to exist
 - return the response as soon as the first shard reports matched documents

closes #6995
2014-07-31 15:42:30 -04:00
David Pilato 85eb0ea0e7 Generate timestamp when path is null
Index process fails when having `_timestamp` enabled and `path` option is set.
It fails with a `TimestampParsingException[failed to parse timestamp [null]]` message.

Reproduction:

```
DELETE test
PUT  test
{
    "mappings": {
        "test": {
            "_timestamp" : {
                "enabled" : "yes",
                "path" : "post_date"
            }
        }
    }
}
PUT test/test/1
{
  "foo": "bar"
}
```

You can define a default value for when timestamp is not provided
within the index request or in the `_source` document.

By default, the default value is `now` which means the date the document was processed by the indexing chain.

You can disable that default value by setting `default` to `null`. It means that `timestamp` is mandatory:

```
{
    "tweet" : {
        "_timestamp" : {
            "enabled" : true,
            "default" : null
        }
    }
}
```

If you don't provide any timestamp value, indexation will fail.

You can also set the default value to any date respecting timestamp format:

```
{
    "tweet" : {
        "_timestamp" : {
            "enabled" : true,
            "format" : "YYYY-MM-dd",
            "default" : "1970-01-01"
        }
    }
}
```

If you don't provide any timestamp value, indexation will fail.

Closes #4718.
Closes #7036.
2014-07-31 19:48:22 +02:00
Brian Murphy 6b39aa615e [TEST] Reduce number of updates and wait after 1st NoNodeException. 2014-07-31 18:01:30 +01:00
Britta Weber fe86c8bc88 _geo_distance sort: allow many to many geo point distance
Add computation of disyance to many geo points. Example request:

```
{
  "sort": [
    {
      "_geo_distance": {
        "location": [
          {
            "lat":1.2,
            "lon":3
          },
          {
             "lat":1.2,
            "lon":3
          }
        ],
        "order": "desc",
        "unit": "km",
        "sort_mode": "max"
      }
    }
  ]
}
```

closes #3926
2014-07-31 17:33:45 +02:00
Shay Banon 739e977aa7 cluster block with auto create index bulk action can cause bulk execution to not return
when there is a cluster block (like no master yet discovered), the bulk action doesn't properly catch the exception of inner execute to notify the listener, causing the bulk operation to hang
closes #7086
2014-07-31 16:57:49 +02:00
Martijn van Groningen 5ea6267883 [TEST] All shards should be allocated before snapshotting & restoring 2014-07-31 16:22:24 +02:00
Martijn van Groningen 1c59ae1b99 [TEST] Increased logging and make use of prepareCreate and assertAcked 2014-07-31 16:22:24 +02:00
Shay Banon 521f8b28b5 [TEST] Concurrent percolation more randomized + safe use of rand 2014-07-31 11:56:36 +02:00
Alex Ksikes e3b3b6c055 Term Vectors API: adds support for wildcards in selected fields
This could useful to generate all term vectors or a chosen set of them.

Closes #7061
2014-07-30 17:44:37 +02:00
Alexander Reelsen 2077d4be48 Packaging: Dont remove ancestors on deb removal
The used -p option could result in accidentally deleting more directories
than /var/lib/elasticsearch - so this option was removed

Note: This only happens if the directories are empty, but still isnt needed.

Relates #5770
2014-07-30 15:01:06 +02:00
Lee Hinman f9f8459c79 [TEST] randomize byte size and thread count in breaker tests 2014-07-30 12:02:16 +02:00
mikemccand dbcc4e9255 Test: finish all merges before test returns to side step 'Delete Index failed - not acked' failure 2014-07-29 11:54:45 -04:00
Simon Willnauer 873b491f4e [TEST] Make default flush opertion blocking 2014-07-29 13:48:50 +02:00
Simon Willnauer af5b2ae28a [TEST] wait for yellow after index was created in BWC tests 2014-07-29 13:48:42 +02:00
Adrien Grand d67e013e08 [TESTS] Don't create an unclosed threadpool in BigArraysTests. 2014-07-29 13:26:53 +02:00
Lee Hinman 5aa2d0cf61 Add support for the `_name` parameter to the simple_query_string query 2014-07-29 12:41:41 +02:00
Lee Hinman 36cf595367 [TESTS] spin in a loop checking request breaker, because multiple clusters could be running 2014-07-29 11:26:34 +02:00
javanna 91c4824a0f Transport client: don't add listed nodes to connected nodes list in sniff mode
This commit effectively reverts e1aa91d , as it is not needed anymore to add the original listed nodes. The cluster state local call made will in fact always return at least the local node (see #6811).

There were a couple of downsides caused by putting the original listed nodes among the connected nodes:
1) in the following retries, they weren't seen as listed nodes anymore, thus the light connect wasn't used
2) among the connected nodes some were "bad" duplicates as they are already there and don't contain all needed info for each node. This was causing serialization problems for instance given that the node version was missing on the `DiscoveryNode` object.

Closes #7067
2014-07-28 21:48:03 +02:00
javanna fcf4d5a38d Transport Client: fixed the node retry mechanism which could fail without trying all the connected nodes
The RetryListener was notified twice for each single failure, which caused some additional retries, but more importantly was making the client reach the maximum number of retries (number of connected nodes) too quickly, meanwhile ongoing retries which could succeed were not completed yet.

The TransportService used to throw ConnectTransportException due to throwConnectException set to true, and also notify the listener of any exception received from a separate thread through the request holder.

Simplified exception handling by just removing the throwConnectException option from the TransportService, used only in the transport client. The transport client now relies solely on the request holder to notify of failures and eventually retry.

Closes #6829
2014-07-28 20:34:46 +02:00
Simon Willnauer eecbf8a559 Add [1.3.2] version constant 2014-07-28 17:22:18 +02:00
Lee Hinman a93ee599d3 [TESTS] fix circuit breaker tests for remote clusters and bwc
Adds additional version checks in NodeStats for older versions

When using an external cluster (backwards compatibility tests), the act
of checking the request breaker requires a network buffer, which
increments the breaker. This change only checks the request breaker in
InternalTestCluster and uses Guice to retrieve it instead of
a (possible) network request.

Also removed the now unused InternalCircuitBreakerService class
2014-07-28 17:18:24 +02:00
javanna 4e5ad568bb Rest: fixed filters execution order to be from lowest to highest rather than the other way around
Closes #7019
2014-07-28 16:54:42 +02:00
javanna 0e9594e02d Internal: use AtomicInteger instead of volatile int for the current action filter position
Also improved filter chain tests to not rely on execution time, and made filter chain tests look more similar to what happens in reality by removing multiple threads creation in testTooManyContinueProcessing (something we don't support anyway, makes little sense to test it).

Closes #7021
2014-07-28 16:54:42 +02:00
David Pilato 264d59c3e2 Plugin Lucene version checker: use `Lucene.parseVersionLenient`
With commit 07c632a2d4dbefe44e8f25dc4ded6cf143d60e41, we now have a new Lucene.parseVersionLenient(String, Version) method which tries to find an existing Lucene version based on the two first digits X.Y of X.Y.Z String.
2014-07-28 16:38:44 +02:00
Colin Goodheart-Smithe 162200f6ed Aggregations: Stops direct subclassing of InternalNumericMetricsAggregation
Must subclass either InternalNumericMetricsAggregation.SingleValue or InternalNumericMetricsAggregation.MultiValue
2014-07-28 14:13:23 +01:00
Itamar Syn-Hershko dd0b42838d [QUERY] Separate parsing impl from setter in SearchParseElement
This commit makes it easier to reuse the inner highlighting, fetch
and rescore parsing logic by plugins or other internal parts.

Closes #3602
2014-07-28 14:53:04 +02:00
Simon Willnauer d403e68f43 add missing import 2014-07-28 14:33:51 +02:00
Simon Willnauer bf7f97d22f [CORE] Support alpha/beta releases in version parsing too
Pull Request #7055 fixed Version parsing for bugfix releases
causing problems with minor version in segments files. Even though
we never release anything with lucene in alpha / beta status this
commit fixes lenient parsing for these cases.

Relates to #7055
2014-07-28 14:04:39 +02:00
Simon Willnauer d2493ea48a [CORE] Support parsing lucene minor version strings
We parse the version that is shipped with the Lucene segments in order
to find the version of lucene that wrote a particular segment. Yet, some lucene
version ie:
 * 4.3.1 (Elasticsearch 0.90.2)
 * 4.5.1 (Elasticsearch 0.90.7)
 * 3.6.1 (pre Elasticsearch 0.90.0)

wrote illegal strings containing the minor version which causes IAE exceptions
being thrown from lucenes parsing method.

Closes #7055
2014-07-28 13:02:00 +02:00
Lee Hinman 07c9b5b08d Change logging level for circuit breaking to warn 2014-07-28 12:10:13 +02:00
Lee Hinman 6abe4c951d Add HierarchyCircuitBreakerService
Adds a breaker for request BigArrays, which are used for parent/child
queries as well as some aggregations. Certain operations like Netty HTTP
responses and transport responses increment the breaker, but will not
trip.

This also changes the output of the nodes' stats endpoint to show the
parent breaker as well as the fielddata and request breakers.

There are a number of new settings for breakers now:

`indices.breaker.total.limit`: starting limit for all memory-use breaker,
defaults to 70%

`indices.breaker.fielddata.limit`: starting limit for fielddata breaker,
defaults to 60%
`indices.breaker.fielddata.overhead`: overhead for fielddata breaker
estimations, defaults to 1.03

(the fielddata breaker settings also use the backwards-compatible
setting `indices.fielddata.breaker.limit` and
`indices.fielddata.breaker.overhead`)

`indices.breaker.request.limit`: starting limit for request breaker,
defaults to 40%
`indices.breaker.request.overhead`: request breaker estimation overhead,
defaults to 1.0

The breaker service infrastructure is now generic and opens the path to
adding additional circuit breakers in the future.

Fixes #6129

Conflicts:
	src/main/java/org/elasticsearch/index/fielddata/IndexFieldData.java
	src/main/java/org/elasticsearch/index/fielddata/IndexFieldDataService.java
	src/main/java/org/elasticsearch/index/fielddata/RamAccountingTermsEnum.java
	src/main/java/org/elasticsearch/index/fielddata/ordinals/GlobalOrdinalsBuilder.java
	src/main/java/org/elasticsearch/index/fielddata/ordinals/InternalGlobalOrdinalsBuilder.java
	src/main/java/org/elasticsearch/index/fielddata/plain/AbstractIndexOrdinalsFieldData.java
	src/main/java/org/elasticsearch/index/fielddata/plain/DisabledIndexFieldData.java
	src/main/java/org/elasticsearch/index/fielddata/plain/IndexIndexFieldData.java
	src/main/java/org/elasticsearch/index/fielddata/plain/NonEstimatingEstimator.java
	src/main/java/org/elasticsearch/index/fielddata/plain/PackedArrayIndexFieldData.java
	src/main/java/org/elasticsearch/index/fielddata/plain/ParentChildIndexFieldData.java
	src/main/java/org/elasticsearch/index/fielddata/plain/SortedSetDVOrdinalsIndexFieldData.java
	src/main/java/org/elasticsearch/node/internal/InternalNode.java
	src/test/java/org/elasticsearch/index/aliases/IndexAliasesServiceTests.java
	src/test/java/org/elasticsearch/index/codec/CodecTests.java
	src/test/java/org/elasticsearch/index/fielddata/AbstractFieldDataTests.java
	src/test/java/org/elasticsearch/index/fielddata/IndexFieldDataServiceTests.java
	src/test/java/org/elasticsearch/index/mapper/MapperTestUtils.java
	src/test/java/org/elasticsearch/index/query/IndexQueryParserFilterCachingTests.java
	src/test/java/org/elasticsearch/index/query/SimpleIndexQueryParserTests.java
	src/test/java/org/elasticsearch/index/query/guice/IndexQueryParserModuleTests.java
	src/test/java/org/elasticsearch/index/search/FieldDataTermsFilterTests.java
	src/test/java/org/elasticsearch/index/search/child/ChildrenConstantScoreQueryTests.java
	src/test/java/org/elasticsearch/index/similarity/SimilarityTests.java
2014-07-28 11:27:33 +02:00
Martijn van Groningen 5631bbb02b [TEST] All shards should be allocated before snapshotting. 2014-07-28 10:48:35 +02:00
Martijn van Groningen 86c0d693c3 [TEST] Ignore Lucene40 codec 2014-07-28 10:40:25 +02:00
Colin Goodheart-Smithe f7b7f67522 Aggregations: fixed value count so it can be used in terms order
Closes #7050
2014-07-28 09:19:01 +01:00
Martijn van Groningen 2e9ee5c937 The `nested` aggregator should also resolve and use the parentFilter of the closest `reverse_nested` aggregator.
Closes #6994
Closes #7048
2014-07-28 10:07:57 +02:00
mikemccand e42b73c6d4 Test: more verbosity for this test on failure 2014-07-26 04:42:26 -04:00
Adrien Grand f682461b2f Mappings: Enforce non-null settings.
No that we are using the index created version to make index-time decisions,
assuming that the version is the current version when settings are null is
very error-prone. Instead we should ensure that settings are always non-null
and contain the version when the index was created.

Close #7032
2014-07-25 21:01:44 +02:00
David Pilato 11eced01da Add multi_field support for Mapper externalValue (plugins)
In context of mapper attachment and other mapper plugins, when dealing with multi fields, sub fields never get the `externalValue` although it was set.

Here is a full script which reproduce the issue when used with mapper attachment plugin:

```
DELETE /test

PUT /test
{
    "mappings": {
        "test": {
            "properties": {
                "f": {
                    "type": "attachment",
                    "fields": {
                        "f": {
                            "analyzer": "english",
                            "fields": {
                                "no_stemming": {
                                    "type": "string",
                                    "store": "yes",
                                    "analyzer": "standard"
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}

PUT /test/test/1
{
    "f": "VGhlIHF1aWNrIGJyb3duIGZveGVz"
}

GET /test/_search
{
    "query": {
        "match": {
           "f": "quick"
        }
    }
}

GET /test/_search
{
    "query": {
        "match": {
           "f.no_stemming": "quick"
        }
    }
}

GET /test/test/1?fields=f.no_stemming
```

Related to https://github.com/elasticsearch/elasticsearch-mapper-attachments/issues/57

Closes #5402.
2014-07-25 16:59:42 +02:00
Colin Goodheart-Smithe 655157c83a Aggregations: Added an option to show the upper bound of the error for the terms aggregation.
This is only applicable when the order is set to _count.  The upper bound of the error in the doc count is calculated by summing the doc count of the last term on each shard which did not return the term.  The implementation calculates the error by summing the doc count for the last term on each shard for which the term IS returned and then subtracts this value from the sum of the doc counts for the last term from ALL shards.

Closes #6696
2014-07-25 14:24:24 +01:00
Alexander Reelsen a1e335b1e9 CORS: Support regular expressions for origin to match against
This commit adds regular expression support for the allow-origin
header depending on the value of the request `Origin` header.

The existing HttpRequestBuilder is also extended to support the
OPTIONS HTTP method.

Relates #5601
Closes #6891
2014-07-25 10:51:22 +02:00
Alexander Reelsen 35e562343f Tests: Remove HttpClient to only use one Http client
The HTTP client implementation used by the Elasticsearch REST tests is
backed by apache http client instead of a self written helper class,
that uses HttpUrlConnection. This commit removes the old simple HttpClient
class and uses the more powerful and reliable one for all tests.

It also fixes a minor bug, that when sending a 301 redirect, a Location
header needs to be added as well, which was uncovered by the switching
to the new client.

Closes #7003
2014-07-25 10:26:52 +02:00
Adrien Grand 51fd2f513c [TESTS] Fix NPE in FreqTermsEnumTests. 2014-07-25 09:12:01 +02:00
Martijn van Groningen a0e5684d7b [TEST] more logging 2014-07-25 01:16:32 +02:00
Adrien Grand a3d8022dc5 Fielddata: Fix thread safety issue with field data on the `_index` field. 2014-07-24 19:04:22 +02:00
Lee Hinman 89e03910f4 Add a periodic cleanup thread for IndexFieldCache caches
Fixes #7010
2014-07-24 17:23:52 +02:00
Martijn van Groningen 297a97cd23 Core: Use the provided cluster state instead of fetching a new cluster state from cluster service.
Close #7013
2014-07-24 16:23:42 +02:00
Colin Goodheart-Smithe 5483c62de6 Geo: Fixes parse error with complex shapes
The bug reproduces when the point under test for the placement of the hole of the polygon has an x coordinate which only intersects with the ends of edges in the main polygon. The previous code threw out these cases as not relevant but an intersect at 1.0 of the distance from the start to the end of an edge is just as valid as an intersect at any other point along the edge.  The fix corrects this and adds a test.

Closes #5773
2014-07-24 15:17:55 +01:00
Simon Willnauer bd51d7a07f Add `wait_if_ongoing` option to _flush requests
This commit adds the ability to force blocking on the flush operaition
to make sure all files have been written and synced to disk. Without
this option a flush might be executing at the same time causing the
current flush to fail and return before all files being synced.

Closes #6996
2014-07-24 15:34:53 +02:00
Colin Goodheart-Smithe 127649d174 Aggregations: Added pre and post offset to histogram aggregation
Added preOffset and postOffset parameters to the API for the histogram aggregation which work in the same way as in the date histogram

Closes #6605
2014-07-24 14:32:33 +01:00
Adrien Grand f5d1e0a37d [TESTS] Ensure yellow in SimpleFacetsTests.testFilterFacetWithFacetFilterPostMode. 2014-07-24 15:21:20 +02:00
Shay Banon eb37a5992b remove use of recycled set in filters eviction
closes #7012
2014-07-24 15:00:30 +02:00
javanna d9ff42f88a Internal: expose the indices names every action relates to if applicable
Added two new interfaces:
1) IndicesRequest that allows to retrieve the indices the request relates to in a generic manner, together with the indices options that tell how they are going to get resolved and expanded
2) CompositeIndicesRequest for compound requests that hold multiple indices request like MultiSearchRequest, MultiGetRequest, MultiTermVectorsRequest, BulkRequest, BenchmarkRequest, PercolateRequest, MultiPercolateRequest and MoreLikeThisRequest

Taken the chance to streamline the indices options and add them to every request where it makes sense (although they can't be changed from the outside), rather than leaving them implicit in the related TransportAction when indices get expanded (tipycally MetaData#concreteIndices or MetaData#concreteSingleIndex). Added IndicesOptions parameter to MetaData#concreteSingleIndex to make sure it is taken from the request, where the information belongs, instead of hardcoded within MetaData. The concreteSingleIndex method remains but it's just a utility method that returns a single index instead of an array and complains otherwise.

Also made sure NPE is never thrown when setting indices(null) to IndicesAliasesRequest, similar to what SearchRequest does.

Closes #6933
2014-07-24 14:42:40 +02:00
Adrien Grand 6f31b1135a [Benchmark] Make TermsAggregationSearchBenchmark fairer to uninverted field data.
The benchmark indexes 200 unique full-width longs. For uninverted field data
we try to use the most memory-efficient storage, and in that case it would use
two arrays: one for the doc->ordinals mapping and one for the ordinal->value
mapping. Which is slower than what doc values do by storing directly the
mapping from docs to values.
2014-07-24 14:35:47 +02:00
Colin Goodheart-Smithe fdf2bb9371 Aggregations: Better JSON output scoping
Before this change each aggregation had to output an object field with its name and write its JSON inside that object.  This allowed for badly behaved aggregations which could write JSON content in the root of the 'aggs' object.  this change move the writing of the aggregation name to a level above the aggregation itself, ensuring that aggregations can only write within there own scope in the JSON output.

Closes #7004
2014-07-24 12:02:40 +01:00
Robert Muir d8cd755445 Speed up string sort with custom missing value
Today if the user supplies a custom missing value for a string sort,
we do it in an extremely slow way, not using ordinals but dereferencing
bytes for every document. Ordinals are only used if the missing value
is _first or _last.

Instead, use ordinals with custom missing values too.

Closes #7005
2014-07-24 06:27:59 -04:00
Simon Willnauer f130d60b72 [TEST] Don't randomize preference PRIMARY it might not try replicas depending on the clusterstate 2014-07-24 11:36:31 +02:00
Martijn van Groningen 73f7f426de Made `_source` parsing in `top_hits` aggregation consistent with regular `_source` parsing in search api.
Closes #6997
2014-07-24 11:23:59 +02:00
Adrien Grand 8cb4471cca [TESTS] Add more assertions to SimpleFacetsTests. 2014-07-24 11:13:53 +02:00
Brian Murphy ce864d4016 [REFACTOR] TransportActions
Get rid of boilerplate code for handling transport actions.
Make these transport actions extend HandledTransportAction where this code
now lives.
2014-07-24 11:05:29 +01:00
javanna 3e30fa2089 Internal: streamline use of IndexClosedException when executing operation on closed indices
Single index operations to use the newly added IndexClosedException introduced with #6475. This way we can also fail faster when we are trying to execute operations on closed indices and their use is not allowed (depending on indices options). Indices blocks are still checked but we can already throw error while resolving indices (MetaData#concreteIndices).

Effectively this change also affects what we return when using one of the following apis: analyze, bulk, index, update, delete, explain, get, multi_get, mlt, term vector, multi_term vector. We now return `{"error":"IndexClosedException[[test] closed]","status":403}` instead of `{"error":"ClusterBlockException[blocked by: [FORBIDDEN/4/index closed];]","status":403}`.

Closes #6988
2014-07-24 10:33:58 +02:00
Colin Goodheart-Smithe dc9e9cb4cc Aggregations: change to default shard_size in terms aggregation
The default shard size in the terms aggregation now uses BucketUtils.suggestShardSideQueueSize() to set the shard size if the user does not specify it as a parameter.

Closes #6857
2014-07-24 07:55:09 +01:00
Areek Zillur 5487c56c70 Search & Count: Add option to early terminate doc collection
Allow users to control document collection termination, if a specified terminate_after number is
set. Upon setting the newly added parameter, the response will include a boolean terminated_early
flag, indicating if the document collection for any shard terminated early.

closes #6876
2014-07-23 15:10:15 -04:00
Robert Muir 66825ac851 Change numeric data types to use SORTED_NUMERIC docvalues type
instead of a custom encoding in BINARY.

In low level benchmarks this is 2x to 5x faster: its also optimized
for the common case where fields actually only contain at most one
value for each document.

Additionally SORTED_NUMERIC doesn't lose values if they appear more
than once, so mathematical computations such as averages are correct.

Closes #6967
2014-07-23 14:55:03 -04:00
Adrien Grand ff2903d2c6 [TEST] Don't recycle in facets.
The recycling happening in facets is done manually and arrays are sometimes not
released. Aggregations do it in a less error-prone way by registering on to the
SearchContext.
2014-07-23 20:20:16 +02:00
Adrien Grand 629f91ae57 Fielddata: goodbye comparators.
This commit removes custom comparators in favor of the ones that are in Lucene.

The major change is for nested documents: instead of having a comparator wrapper
that deals with nested documents, this is done at the fielddata level by having
a selector that returns the value to use for comparison.

Sorting with custom missing string values might be slower since it is using
TermValComparator since Lucene's TermOrdValComparator only supports sorting
missing values first or last. But other than this particular case, this change
will allow us to benefit from improvements on comparators from the Lucene side.

Close #5980
2014-07-23 20:08:36 +02:00
Adrien Grand 76511158b5 Fielddata: Fix the ordinals impl for sparse fields.
Caused by #6908
2014-07-23 17:39:43 +02:00
Simon Willnauer b51bd3a645 Add version 1.2.4 and 1.3.1 to the version table 2014-07-23 16:26:48 +02:00
Simon Willnauer be96f57c11 [TEST] Fix SimpleThreadPoolTests to exclude test infra threads 2014-07-23 14:44:08 +02:00
mikemccand cc4d7c6272 Core: don't load bloom filters by default
This change just changes the default for index.codec.bloom.load to
false: with recent performance improvements to ID lookup, such as
#6298, bloom filters don't give much of a performance gain anymore,
and they can consume non-trivial RAM when there are many tiny
documents.

For now, we still index the bloom filters, so if a given app wants
them back, it can just update the index.codec.bloom.load to true.

Closes #6959
2014-07-23 05:58:41 -04:00
Lee Hinman 15ccd787a5 [TEST] Maven reproductions should always include 'clean' target 2014-07-23 11:47:58 +02:00
mikemccand 5ccd44519a Test: make test less evil 2014-07-23 05:35:52 -04:00
mikemccand 55986907e8 Test: add more verbosity when this test fails 2014-07-23 05:23:15 -04:00
Lee Hinman c38a9d73e7 [TEST] Add test for _score and doc[] access in Groovy scripts 2014-07-23 09:58:38 +02:00
Britta Weber 734e656a91 Make _all field accessible with GET
GET only returned null even when stored if requested with GET like this:

`curl -XGET "http://localhost:9200/test/test/1?fields=_all"`

Instead, it should simply behave like a String field and return the
concatenated fields as String.

closes #6924
2014-07-23 09:16:35 +02:00
Adrien Grand 08f8731b6f Core: Drop UnsafeUtils.
This class potentially does unaligned memory access and does not bring much
now that we switched to global ords for terms aggregations.

Close #6962
2014-07-23 08:41:11 +02:00
Areek Zillur f39d4e1f89 PhraseSuggester: Collate option should allow returning phrases with no matching docs
A new option `prune` has been added to allow users to control phrase suggestion pruning when `collate`
is set. If the new option is set, the phrase suggestion option will contain a boolean `collate_match`
indicating whether the respective result had hits in collation.

CLoses #6927
2014-07-22 17:17:15 -04:00
Simon Willnauer 0faffcf372 [TEST] Add simple sort assertions for bwc tests
Today we only do count searches to ensure sane results are returned
after upgrading etc. This change adds sorting to the picture asserting
on simple numeric sorting that uses field data etc. after upgrading.

Relates to #6967
2014-07-22 22:22:09 +02:00
Shay Banon 50ececbbcf Unicast discovery: only disconnect from temporary connected nodes
In unicast discovery, we try to reuse existing discovery nodes based on the node address they have. If we find an existing node based on its address, and for some reason its not connected, don't add it to the list of nodes to disconnect from, as that (full) connection is useful down the road
closes #6966
2014-07-22 21:29:57 +02:00
Shay Banon 88f3afe4b5 Fix connect concurrency, can cause connection nodes to close
Looking at the connect code, if 2 threads at the same time try and connect to a node, and both enter sequentially the connectLock code block, the second one would try and put the connection in the map, and close the replaced channels, which will cause the existing connection to close as well (since it removes the node from the connectedNodes map)
To fix this, simply make sure we properly check the existence of the connection within the connectionLock block, so there won't be concurrent connections going on.
While doing this, also went over all the mutation code that handles disconnections, and made sure they are properly done only within a connection lock.
closes #6964
2014-07-22 19:48:47 +02:00
mikemccand 72b3d6ef75 Test: make sure randomizer doesn't swap in SerialMergeScheduler on us 2014-07-22 13:06:04 -04:00
mikemccand 1e92f0f4ff Core: allow index.merge.scheduler.max_thread_count to be updated dynamically
Lucene allows the max_thread_count to be updated, but this wasn't
fully exposed in Elasticsearch.

Closes #6925
2014-07-22 11:23:46 -04:00
Clinton Gormley f14af3599a Fixed typo in AbstractFieldMapper
similariry -> similarity
2014-07-22 15:54:09 +02:00
Brian Murphy 7d9b012ca1 [FIX] Fix update parser to accept script_id 2014-07-22 14:22:57 +01:00
Adrien Grand 3c142e550d Fielddata: Switch to Lucene DV APIs.
This commits removes BytesValues/LongValues/DoubleValues/... and tries to use
Lucene's APIs such as NumericDocValues or RandomAccessOrds instead whenever
possible.

The next step would be to take advantage of the fact that APIs are the same in
Lucene and Elasticsearch in order to remove our custom comparators and use
Lucene's.

There are a few side-effects to this change:
 - GeoDistanceComparator has been removed, DoubleValuesComparator is used instead
   on top of dynamically computed values (was easier than migrating
   GeoDistanceComparator).
 - SortedNumericDocValues doesn't guarantee uniqueness so long/double terms
   aggregators have been updated to make sure a document cannot fall twice in
   the same bucket.
 - Sorting by maximum value of a field or running a `max` aggregation is
   potentially significantly faster thanks to the random-access API.

Our aggs and p/c aggregations benchmarks don't report differences with this
change on uninverted field data. However the fact that doc values don't need
to be wrapped anymore seems to help a lot. For example
TermsAggregationSearchBenchmark reports ~30% faster terms aggregations on doc
values on string fields with this change, which are now only ~18% slower than
uninverted field data although stored on disk.

Close #6908
2014-07-22 15:16:24 +02:00
Brian Murphy 0de30e1798 [TEST] Fix indexed script test. 2014-07-22 13:58:07 +01:00
Nik Everett 79433d23e3 Update: Detect noop updates sent with doc_as_upsert
This should help prevent spurious updates that just cause extra writing
and cache invalidation for no real reason.

Close #6822
2014-07-22 14:55:34 +02:00
Brian Murphy ebcc1e0bf5 [FIX] Indexed scripts should always use script_id. 2014-07-22 13:49:31 +01:00
Simon Willnauer 55df94d91e Revert "Fix InternalSearchHits serialization to be deterministic"
This reverts commit 8db7b2b20b.
2014-07-22 14:43:31 +02:00
Simon Willnauer 8db7b2b20b Fix InternalSearchHits serialization to be deterministic
The assertion on binary equality for streamable serialization
sometimes fails due to the usage of identify hashmaps inside
the InternalSearchHits serialization. This only happens if
the number of shards the result set is composed of is very high.
This commit makes the serialziation deterministic and removes
the need to serialize the ordinal due to in-order serialization.
2014-07-22 14:14:31 +02:00
Brian Murphy 924f91588b [TEST] Reduce the number of threads in StressUpdateDelete test. 2014-07-22 11:16:32 +01:00
Boaz Leskes a344fe6590 IndicesStore fix logging format 2014-07-22 07:47:40 +03:00
Boaz Leskes fee22fc0b2 [Discovery] fix MembershipAction.sendJoinRequestBlocking should now expect an empty response
The change introduced in #6949 (do not serialize the cluster state) also means master now responds with an empty response rather then a JoinResponse. However, sendJoinRequestBlocking still expected a JoinRequest.
2014-07-22 00:43:01 +03:00
Boaz Leskes 690820dae3 [Discovery] remove unneeded cluster state serialization during cluster join process
At the moment we serialize the cluster state in JoinResponse and ValidateJoinRequest. However this state is not used anywhere and can be removed to save on network overhead

Closes #6949
2014-07-21 23:30:07 +03:00
Brian Murphy c74552e006 [TEST] Stress update/delete
Reduce top end of scaling parameters to avoid OOMs on nightly runs.
2014-07-21 21:16:41 +01:00
Shay Banon 8dca4b7e07 More lenient type parsing in histo/cardinality aggs
closes #6948
closes #6893
2014-07-21 21:31:37 +02:00
Brian Murphy b0ce090876 [TEST] Stress Update/Delete
Create the test index with merging disabled so deletes don't get merged away
this was breaking the version calculation.
2014-07-21 20:11:05 +01:00
Brian Murphy 6d641ea40d [TEST] Remove TransportUpdateActionTest
This test has been made obselete by the UpdateTests.
2014-07-21 17:55:51 +01:00
Brian Murphy cbb4edf2ae [TEST] Plugin Manager
Undo removal of change.
2014-07-21 16:25:31 +01:00
Brian Murphy 8b169617a2 [TEST] Stress test for update and delete concurrency.
This test deletes and updates using upserts documents over several threads in a
tight loop. It counts the number of responses and verifies that the versions at
the end are correct.
2014-07-21 16:04:21 +01:00
Simon Willnauer e730c76be7 [RESTORE] Fail restore if snapshot is corrupted
today if a snapshot is corrupted the restore operation
never terminates. Yet, if the snapshot is corrupted there
is no way to restore it anyway. If such a snapshot is restored
today the only way to cancle it is to delete the entire index which
might cause dataloss. This commit also fixes an issue in InternalEngine
where a deadlock can occur if a corruption is detected during flush
since the InternalEngine#snapshotIndex aqcuires a topLevel read lock
which prevents closing the engine.

Closes #6938
2014-07-21 16:15:53 +02:00
Simon Willnauer d65a9d63a2 [Engine] `index.fail_on_corruption` is not updateable
The `index.fail_on_corruption` was not updateable via the index settings
API. This commit also fixed the setting prefix to be consistent with other
setting on the engine. Yet, this feature is unreleased so this won't break anything.

Closes #6941
2014-07-21 14:59:44 +02:00
Martijn van Groningen f3114fe774 [TEST] top_hits aggs with breath_first collect mode only fails if scoring is used (score based sort or track_scores) 2014-07-21 11:21:50 +02:00
Martijn van Groningen ba1042e7d1 Aggregations: Track scores should be applied properly for `top_hits` aggregation.
Closes #6934
2014-07-21 10:03:42 +02:00
Martijn van Groningen 2d2088b259 [TEST] Temporarily ignore transport update tests. 2014-07-21 09:57:03 +02:00
Simon Willnauer 6e81cb5cd6 [TEST] Add SuppressSysoutChecks to DistributorDirectoryTest 2014-07-20 19:20:18 +02:00
Shay Banon 2c07588a1a [TEST] an active shard might also be relocating 2014-07-20 10:44:24 +02:00
Alexander Reelsen 1816951b6b Netty: Refactoring to make MessageChannelHandler extensible
Small refactorings to make the MessageChannelHandler more extensible.
Also allowed access to the different netty pipelines

This is the fix after the first version had problems with the HTTP
transport due to wrong reusing channel handlers, which is the reason
why tests failed.

Relates #6889
Closes #6915
2014-07-18 16:30:31 +02:00
javanna bbf7e6be92 Plugins: Introduced pluggable filter chain to be able to filter transport actions execution
It's now possible to inject action filters from plugins via `ActionModule#registerFilter` through the following code:

```
    public void onModule(ActionModule actionModule) {
          actionModule.registerFilter(MyFilter.class);
    }
```

Also made `TransportAction#execute` methods final to enforce the execution of the filter chain. By default the chain is empty though.

Note that the action filter chain is executed right after the request validation, as the filters might rely on a valid request to do their work.

Closes #6921
2014-07-18 16:18:50 +02:00
Adrien Grand 97f4247880 [TESTS] Switch more tests to ElasticsearchSingleNodeTest.
This avoids loading modules manually all the time.
2014-07-18 14:35:00 +02:00
Shay Banon 972fa10945 Better message for invalid internal transport message format
closes ##6916
2014-07-18 13:49:38 +02:00
Simon Willnauer a302f3a9dd [TEST] Randomize Preference in search requests
Closes #6863
2014-07-18 12:30:53 +02:00
Simon Willnauer 6b519633ab [TEST] Randomize FS Translog implementation
Closes #6658
2014-07-18 11:51:42 +02:00
Simon Willnauer d79717c341 [CLIENT] Unknown node version should be a lower bound
Today when we start a `TransportClient` we use the given transport
addresses and create a `DiscoveryNode` from it without knowing the
actual nodes version. We just use the `Version.CURRENT` which is an
upper bound. Yet, the other node might be a version less than the
currently running and serialisation of the nodes info might break. We
should rather use a lower bound here which is the version of the first
release with the same major version as `Version.CURRENT` since this is
what we officially support.

This commit moves to use the minimum major version or an RC / Snapshot
if the current version is a snapshot.

Closes #6894
2014-07-18 11:31:15 +02:00
Alex Ksikes f22f3db30f Term Vectors API: Computes term vectors on the fly if not stored in the index.
Adds the ability to the Term Vector API to generate term vectors for some
chosen fields, even though they haven't been explicitely stored in the index.

Relates to #5184
Closes #6567
2014-07-17 23:29:05 +02:00
Simon Willnauer 6fc5acd760 Return null info if service is not started
Closes #6906
2014-07-17 16:47:46 +02:00
Simon Willnauer 2bec59ec3e [Bulk] Release semaphore again after close
The bulk processor tries to acquire all leases for the semaphore to wait
for all pending requests. Yet, we should release them afterwards again to
ensure we don't ever deadlock if there is a bug in the processor.
This commit also adds a testcase for this method
2014-07-17 16:24:43 +02:00
matt-preston 42b71a004a [Bulk] Add blocking close method to BulkProcessor
Blocks until all bulk requests have completed.

Closes #4158
Closes #6314
2014-07-17 15:57:45 +02:00
uboness cc8f7ddb9a Added three frequency levels for resource watching
It's now possible to register watchers along with a specified check frequency. There are three frequencies: low, medium, high. Each one is associated with a check interval that determines how frequent the watchers will check for changes and notify listeners if needed. By default, the intervals are 5s, 30s and 60s respectively, but they can also be customized in the settings. also:

  - Added the WatcherHandle construct by which one can stop it (remove it) and resume it (re add it). Also provices access to the watchers itself and the frequency by which it's checked
  - Change the default frequency to 30 seconds interval (used to be 60 seconds). The only watcher that is currently effected by this is the script watcher (now auto-loading scripts will auto-load every 30 seconds if changed)
2014-07-17 14:57:53 +02:00
Boaz Leskes bdbe86dd2d [Store] delete unallocated shards under a cluster state task
This is to prevent a rare racing condition where the very same shard gets allocated to the node after our sanity check that the cluster state didn't check and the actual deletion of the files.

Closes #6902
2014-07-17 14:51:07 +02:00
Alexander Reelsen bb421d7ea3 Revert "Netty: Refactoring to make MessageChannelHandler extensible"
This results in unstable tests, most likely due to Channels being mixed
up by wrongly creating the pipelines. Needs investigation and a test.

This reverts commit db7f0d36af.
2014-07-17 14:49:15 +02:00
Boaz Leskes 38d8e3ccc2 [Infra] remove indicesLifecycle.Listener from IndexingMemoryController
The IndexingMemoryController determines the amount of indexing buffer size and translog buffer size each shard should have. It takes memory from inactive shards (indexing wise) and assigns it to other shards. To do so it needs to know about the addition and closing of shards. The current implementation hooks into the indicesService.indicesLifecycle() mechanism to receive call backs, such shard entered the POST_RECOVERY state. Those call backs are typically run on the thread that actually made the change. A mutex was used to synchronize those callbacks with IndexingMemoryController's background thread, which updates the internal engines memory usage on a regular interval. This introduced a dependency between those threads and the locks of the internal engines hosted on the node. In a *very* rare situation (two tests runs locally) this can cause recovery time outs where two nodes are recovering replicas from each other.

 This commit introduces a a lock free approach that updates the internal data structures during iterations in the background thread.

Closes #6892
2014-07-17 14:31:38 +02:00
Adrien Grand 9714dd55c2 Refactoring: Rename FieldMapper.termsFilter to fieldDataTermsFilter.
FieldMapper has two methods
`Filter termsFilter(List values, @Nullable QueryParseContext)` which is supposed
to work on the inverted index and
`Filter termsFilter(QueryParseContext, List, QueryParseContext)` which is
supposed to work on field data. Let's rename the second one to
`fieldDataTermsFilter` and remove the unused `QueryParseContext`.

Close #6888
2014-07-17 12:42:18 +02:00
Martijn van Groningen 9037365adf [TEST] Use array* helper assert methods instead of verifying the array directly 2014-07-17 09:39:51 +02:00
David Pilato 26bac39e0e bin/plugin removes itself
If you call `bin/plugin --remove es-plugin` the plugin got removed but the file `bin/plugin` itself was also deleted.

We now don't allow the following plugin names:

* elasticsearch
* plugin
* elasticsearch.bat
* plugin.bat
* elasticsearch.in.sh
* service.bat

Closes #6745
2014-07-17 08:57:01 +02:00
Alexander Reelsen db7f0d36af Netty: Refactoring to make MessageChannelHandler extensible
Small refactorings to make the MessageChannelHandler more extensible.
Also allowed access to the different netty pipelines

Closes #6889
2014-07-17 08:28:55 +02:00
Boaz Leskes 6fe1d9860e [Scripts] GroovyScriptEngineService don't create ScoreAccess if lookup is null 2014-07-17 08:05:10 +02:00
Ryan Ernst 7139f44241 Scripts: Make ScoreAccessor utility available for other script engines.
With the removal of setNextScore in #6864, script engines must use
the Scorer to find the score of a document.  The DocLookup is updated
appropriately to do this, but most script engines require a Number to be
bound for numeric variables.  Groovy already had an encapsulation for
this funtionality, and this moves it out to be shared with other script
engines.

closes #6898
2014-07-16 16:33:04 -07:00
Ryan Ernst 770447ce1a Scripting: Remove setNextScore in SearchScript.
While it would be nice to do this all the way up the chain (into
score functions), this at least removes the weird dual
setNextScore/setScorer for SearchScripts.

closes #6864
2014-07-16 15:10:01 -07:00
Martijn van Groningen ca7fa4f9ec Query: When parent doc is found stop iterating over segments and continue with the next top matching child doc.
The parent doc has been found and there is no need to check subsequent segments.

Closes #6895
Closes #1843
2014-07-17 00:01:06 +02:00
Martijn van Groningen 473d1717be Search: Added support for named filters/queries inside nested filters and queries.
Closes #6293
2014-07-16 23:43:18 +02:00
Alexander Reelsen 70bad405c3 Serialization: Fix bwc issue by falling back to old threadpool serialization
This fixes an issue introduced by the serialization changes in #6486
which are not needed at all. Node that the serialization itself is not broken
but the TransportClient uses its own version on initial connect and getting
the NodeInfos.
2014-07-16 17:58:32 +02:00
Simon Willnauer 90ea4610c8 [FIELDDATA] Use KeyedLock in IndexFieldDataService
Today we synchronize when updating the IndexFieldDataService
datastructures. This might unnecessarily block progress if multiple
request need different fielddata instance for different fields.

This commit also fixes clear calls to actually consistently clear
the caches in the case of an exception.

Closes #6855
2014-07-16 15:49:17 +02:00
Simon Willnauer e30176cc69 Add read/write lock semantics to KeyedLock 2014-07-16 15:49:00 +02:00
Alexander Reelsen b0c0ff8ac0 Threadpool Info: Allow to serialize negative thread pool sizes
As a SizeValue is used for serializing the thread pool size, a negative number
resulted in throwing an exception when deserializing (using -ea an assertionerror
was thrown).

This fixes a check for changing the serialization logic, so that negative numbers are read correctly, by adding an internal UNBOUNDED value.

Closes #6325
Closes #5357
2014-07-16 15:25:53 +02:00
Martijn van Groningen 62da59f76f [TEST] Verify if clear cache request went to all shards. 2014-07-16 13:48:39 +02:00
Colin Goodheart-Smithe 8260138e59 Aggregations: Fixed Histogram key_as_string bug
The key as string field in the response for the histogram aggregation will now only show if format is specified on the request.

Closes #6655
2014-07-16 11:36:48 +01:00
Brian Murphy cbd2a97abd [DOCS] : Indexed scripts/templates
These are the docs for the indexed scripts/templates feature.
Also moved the namespace for the REST endpoints.

Closes #6851
2014-07-16 10:49:02 +01:00
Adrien Grand 7e44fe6017 [TESTS] Stabilize DisabledFieldDataFormatTests by setting number_of_replicas to 0. 2014-07-16 10:37:25 +02:00
Boaz Leskes 84ad29ddfa [Recovery] don't start a gateway recovery if source node is not found
Due to change introduced in #6825, we now start a local gateway recovery for replicas, if the source node can not be found. The recovery then fails because we never recover replicas from disk.

Closes #6879
2014-07-16 10:01:01 +02:00
Boaz Leskes d869163b66 [Infra] re-send failed shard messages when receiving a cluster state that still refers to them
In rare cases we may fail to send a shard failure event to the master, or there is no known master when the shard has failed (ex. a couple of node leave the cluster canceling recoveries and causing a master to step down at the same time). When that happens and a cluster state arrives from the (new) master we should resend the shard failure in order for the master to remove the shard from this node.

Closes #6881
2014-07-16 09:59:49 +02:00
Martijn van Groningen f1c2cdb9c8 Store: Only send shard exists requests if shards exist locally on disk and are not allocated on that node according to the cluster state.
Closes #6870
2014-07-16 08:23:32 +01:00
Ryan Ernst 35037e31b2 [AGGS] Pass current docid being processed to scripts.
Scripts may internally cache based on docid (as expressions do). This
change makes numeric aggregations using scripts pass the docid when
it changes.
2014-07-15 14:40:12 -07:00
Shay Banon 323210729e Releasable XContentBuilder
make the builder releasable (auto closeable), and use it in shards state
also make XContentParser releasable (AutoCloseable) and not closeable since it doesn't throw an IOException
closes #6869
2014-07-16 04:28:03 +09:00
Shay Banon 9345194a65 Share numeric data analyzer instances between mappings
use similar mechanism that shares numeric analyzers for long/double/... for dates as well. This has nice memory save properties with many date fields mapping case, as well as analysis saves (thread local resources)
closes #6843
2014-07-16 04:24:52 +09:00
Ryan Ernst e764c5f08a Fix transient testScore failure by making DF consistent for query. 2014-07-15 11:18:13 -07:00
Brian Murphy 86dfad24fe [FIX] normalize serialization of ScriptType in UpdateRequest 2014-07-15 18:14:48 +01:00
Nik Everett da5fb34163 Mappings: Add transform to document before index.
Closes #6566
2014-07-15 18:40:46 +02:00
Adrien Grand 56cfa8e47e [TEST] Enforce that the node from ElasticsearchSingleNodeTest is alone in its cluster.
In particular, this removes the ability to create new node and adds the ability
to get a reference to the single node that the cluster is made of.
2014-07-15 17:26:52 +02:00
mikemccand ba443a4819 Test: make test less evil so it doesn't take so long 2014-07-15 11:19:18 -04:00
Simon Willnauer 05d92b9b68 [TEST] Enforce provided settings in test base classes
Enviorment variables might override the tests settings even if
they are explicitly set. Other base classes like InternalTestCluster
also specify `config.ignore_system_properties: true` to ensure `what
we set is what we get`
2014-07-15 17:06:20 +02:00
Ryan Ernst 64ab22816c Scripting: Add script engine for lucene expressions.
These are javascript expressions, which can only access numeric
fielddata, parameters, and _score. They can only be used for searches (not document updates).

closes #6818
2014-07-15 07:49:01 -07:00
Simon Willnauer 1464bea00f Added [1.4.0] version 2014-07-15 16:16:50 +02:00
Simon Willnauer 3900d61254 [TESTS] Add ElasticsearchSingleNodeTest.
This test makes it easy to create a lightweight node (no http, indices stored
in RAM, ...) whose main purpose is to get an instance of the Guice injector
for unit tests.

This should help not have to update lots of unit tests when we add a new
Guice dependency.
2014-07-15 15:42:58 +02:00
Patrick Peschlow 9742d08f53 [UPDATE] Handle document missing exception on retry correctly.
Throwables thrown on update retries are now caught and handled via
the provided callback. This commit also contains an integration test
demonstrating the bug and validating the fix.

Closes #6355
Closes #6724
2014-07-15 14:52:29 +02:00
javanna 3c54eb9b8f Internal: make transport action name available in TransportAction base class
Each transport action is associated with at least an action name, which is the action name that gets serialized together with the request and identifies what to do with the request itself. Also, the action name is the name of the registered transport handler that handles incoming request for the transport action.

This commit makes the action name available in a generic manner in the TransportAction base class, so that it can be used when needed by subclasses, or in the base class for instance for action filtering.

Closes #6860
2014-07-15 14:35:35 +02:00
mikemccand 4194ab31c8 Core: don't close/reopen IndexWriter when changing RAM buffer size
Today we close/reopen IW when we change the RAM buffer but that's
costly because it means the next NRT reader is a full reopen.  The RAM
buffer size setting is a live one in IndexWriter, even if there are no
buffered docs in RAM when you call it.

Separately it would be nice if Lucene let you manage a "reader pool"
that could outlive individual IW instances ...

Closes #6856
2014-07-15 08:32:30 -04:00
Simon Willnauer 557b634f4a [SEARCH] Wrap filter only once in ApplyAcceptedDocsFilter
We potentially wrap the given filter multiple times when iterating the
subreaders. We only need to do this once.

Closes #6873
2014-07-15 12:44:35 +02:00
Simon Willnauer 7de9d3d2cb [TEST] Fix test to use keyword analyzer since it relies on exact terms without processing 2014-07-15 12:26:51 +02:00
Simon Willnauer 82cc227da3 Added missing version based serialization 2014-07-15 12:13:45 +02:00
Brian Murphy 73d93e5fd8 Indexed Scripts/Templates: cleanup
This commit cleans up some code around the indexed script/templates feature.
Remove dead code in ScriptService.
Remove setXScript methods for UpdateRequestBuilder and use setScript(script,type) instead
2014-07-15 10:49:39 +01:00
Simon Willnauer 2bac455cd0 Add version check before reading script type in UpdateRequest 2014-07-15 10:16:08 +02:00
Simon Willnauer 70bd24ed14 [TEST] use higher timeout to wait for balanced cluster
CorruptFileTest sometimes hits conditions where lots of rebalancing
happens. In such a case the default timeout is just not enough - this
timeout just makes sure that the cluster has enough time to balance
itself.
2014-07-15 09:56:41 +02:00
Areek Zillur 0bd774423d [TEST] fix random preference string tests to accomodate for the new more strict preference type 2014-07-14 20:15:00 -04:00
Areek Zillur d0d1b98d23 Stats: Expose IndexWriter and VersionMap RAM usage to ShardStats and _cat endpoint
This commit adds the RAM usage of IndexWriter and VersionMap

Closes #6483
2014-07-14 19:46:12 -04:00
Areek Zillur 4fb79fe787 [TEST] fix random preference string test to accomodate for the new more strict preference type 2014-07-14 19:01:04 -04:00
Areek Zillur 0173298f29 [TEST] fix PhraseInIndex test bug for shard failure 2014-07-14 17:12:51 -04:00
Areek Zillur 76343899ea Phrase Suggester: Add collate option to PhraseSuggester
The newly added collate option will let the user provide a template query/filter which will be executed for every phrase suggestions generated to ensure that the suggestion matches at least one document for the filter/query.
The user can also add routing preference `preference` to route the collate query/filter and additional `params` to inject into the collate template.

Closes #3482
2014-07-14 16:07:52 -04:00
Simon Willnauer f8be82f54a [TEST] close IW before listing directory content 2014-07-14 22:00:03 +02:00
uboness 04b412b597 Added more utility methods to Settings
- names() to return the direct settings names
- getAsSettings(String) to return the settings mapped to the given name (like getByPrefix(...) except no need to provide a tailing '.')
2014-07-14 20:25:35 +02:00
mikemccand f392a99bd9 Test: add diagnostics to get some details next time this test fails 2014-07-14 13:48:56 -04:00
Brian Murphy 9c0b25dcce [TEST] Fix test issues with branches still using mvel as default scripting language 2014-07-14 16:51:17 +01:00
Simon Willnauer c91ec7c730 [TEST] fix compile problems on Java 8 2014-07-14 17:17:24 +02:00
Brian Murphy e79b7086de Indexed Scripts/Templates: Add support for storing/deleting/getting scripts/template from an index.
This change allow elasticsearch users to store scripts and templates in an index for use at search time.
Scripts/Templates are stored in the .scripts index. The type of the events is set to the script language.
Templates use the mustache language so their type is be "mustache".
Adds the concept of a script type to calls to the ScriptService types are INDEXED,INLINE,FILE.
If a script type of INDEXED is supplied the script will be attempted to be loaded from the indexed, FILE will
look in the file cache and INLINE will treat the supplied script argument as the literal script.
REST endpoints are provided to do CRUD operations as is a java client library.
All query dsl points have been upgraded to allow passing in of explicit script ids and script file names.
Backwards compatible behavior has been preserved so this shouldn't break any existing querys that expect to
pass in a filename as the script/template name. The ScriptService will check the disk cache before parsing the
script.

Closes #5921 #5637 #5484
2014-07-14 14:37:55 +01:00
Boaz Leskes dcb2107a1d [Test] testSingleNodeWithFlush should wait for yellow
O.w. it can restart a node before all primaries are started , leading to a red state down the road...
2014-07-14 15:13:52 +02:00
mikemccand 80774877ff Test: add test case verifying updating merge IO throttle settings works
Closes #6842
2014-07-14 08:37:29 -04:00
Martijn van Groningen b275393e01 [TEST] For connection rules make TransportAddress the identity instead of DiscoveryNode.
For unicast ping the DiscoveryNode identity is based on its id, which in that stage is a dummy value, this breaks any rule in the mock tran
However the TransportAddress is a valid value in unicast ping and all other places, so that is a better alternative.

Closes #6836
2014-07-14 12:52:45 +02:00
Clinton Gormley 6e70edb0a4 Analysis: Improve Hunspell error messages
The Hunspell service would throw a confusing error message if more than
one affix file was present.  This commit distinguishes between the two
error cases: where there are no affix files and when there are too many
affix files.

Also implements lazy dictionary loading, which was used in the tests
but not implemented.

Closes #6850
2014-07-14 12:13:32 +02:00
Britta Weber 74927adced significant terms: infrastructure for changing easily the significance heuristic
This commit adds the infrastructure to allow pluging in different
measures for computing the significance of a term.
Significance measures can be provided externally by overriding

- SignificanceHeuristic
- SignificanceHeuristicBuilder
- SignificanceHeuristicParser

closes #6561
2014-07-14 11:00:50 +02:00
Boaz Leskes 8865e60e93 [Transport] possible NPE during shutdown for requests using timeouts
Closes #6849
2014-07-14 10:52:29 +02:00
Simon Willnauer 86bc79202d [ENGINE] Mark store as corrupted before sending failed shard
We have to mark a shard as corrupted if necessary before the
shard failed event is fired ie. before we call the corresponding
listener in the engine. Otherwise the shard might be re-allocated
on the same node and just started up without being marked as corrupted.

Relates to #5924
2014-07-14 10:14:58 +02:00
Simon Willnauer e8ff007852 [RECOVERY] Increment Store refcount on RecoveryTarget
We should make sure we have incremented the store refcount
before we start the recovery on the recovyer target.

Closes #6844
2014-07-14 09:18:25 +02:00
Boaz Leskes ab11c6821d [Test] one more tweak to testLocalNodeMasterListenerCallbacks 2014-07-13 17:59:45 +02:00
Boaz Leskes c3e842e363 [Test] renamed testListenerCallbacks to testLocalNodeMasterListenerCallbacks
Also clean up internal variable namings and fixed usage of wrong setting causing last node to not use the min_master_node settings.
2014-07-13 17:51:01 +02:00
Martijn van Groningen af38b9f7ba Core: Added missing return statements.
Closes #6841
2014-07-13 15:53:05 +02:00
Igor Motov 60b317caa4 Snapshot/Restore: Add ability to restore indices without their aliases
Closes #6457
2014-07-13 17:52:41 +09:00
Shay Banon f7a88fdd3e [TEST] wait for green before deleting mapping 2014-07-13 17:21:26 +09:00
Shay Banon fb6d847aac [TEST] wait for green before deleting mapping 2014-07-13 17:17:14 +09:00
Boaz Leskes 5e3742762a [Test] testHostOnMessages - only decrease latch after setting transport addresses 2014-07-12 09:11:27 +02:00
Martijn van Groningen 05ca763b10 [TEST] Ensure that one node is part of the cluster. 2014-07-11 17:51:35 +02:00
Martijn van Groningen 6547ff3eb0 Print trace log if not enough master nodes could be found. 2014-07-11 17:42:11 +02:00
uboness 25a21c6a01 Cleanup of the transport request/response messages
Now both TransportRequest and TransportResponse inherit from a base TransportMessage that holds the message headers and also now added the remote transport address (where this message came from).
2014-07-11 16:41:01 +02:00
Boaz Leskes c4c0270c52 [Tests] Enhance ZenUnicastDiscoveryTest
This started out as a simple correction to a missing setting problem, but go bigger into more general work on the ZenUnicastDiscoveryTets suite. It now works with both network and local mode. I also merge the different ZenUnicast test suites into a single place.

Closes #6835
2014-07-11 16:37:52 +02:00
Britta Weber 6d8fff65dc Throw exception if function in function score query is null
closes #6292 #6784
2014-07-11 13:57:11 +02:00
Shay Banon 43a5cbe9be Only use IndexShard instance to lookup recovery status
make sure we use the instance itself to look it up, and not the shard id, as we might get another instance
leftover from #6825
2014-07-11 11:38:36 +02:00
Boaz Leskes f480969503 [Gateway] set a default of 5m to `recover_after_time` when any to the `expected*Nodes` is set
The `recovery_after_time` tells the gateway to wait before starting recovery from disk. The goal here is to allow for more nodes to join the cluster and thus not start potentially unneeded replications. The `expectedNodes` setting (and friends) tells the gateway when it can start recovering even if the `recover_after_time` has not yet elapsed. However, `expectedNodes` is useless if one doesn't set `recovery_after_time`. This commit changes that by setting a sensible default of 5m for `recover_after_time` *if* a `expectedNodes` setting is present.

Closes #6742
2014-07-11 11:28:45 +02:00
Alex Ksikes af4eee594c More Like This: ensures selection of best terms is indeed O(n)
Previously the size of the priority queue was wrongly set to the total number
of terms. Instead, it should be set to 'maxQueryTerms'. This makes the
selection of best terms O(n), instead of O(n*log(n)).

Jira patch: https://issues.apache.org/jira/browse/LUCENE-5795

Closes #6657
2014-07-11 11:14:31 +02:00
Shay Banon 01ca81e2a3 Improve handling of failed primary replica handling
Out of #6808, we improved the handling of a primary failing to make sure replicas that are initializing are properly failed as well. After double checking it, it has 2 problems, the first, if the same shard routing is failed again, there is no protection that we don't apply the failure (which we do in failed shard cases), and the other was that we already tried to handle it (wrongly) in the elect primary method.
This change fixes the handling to work correctly in the elect primary method, and adds unit tests to verify the behavior
The change also expose a problem in our handling of replica shards that stay initializing during primary failure and electing another replica shard as primary, where we need to cancel its ongoing recovery to make sure it re-starts from the new elected primary
closes #6825
2014-07-11 10:51:59 +02:00
Simon Willnauer a84777e990 [TEST] Fix CorruptedFileTest to always corrupt the latest delete generation if a .del file is picked 2014-07-11 10:22:11 +02:00
Simon Willnauer 35a52cd04a [TEST] Temporarily don't corrupt .del files since they are generational and we might pick the wrong one 2014-07-11 08:41:38 +02:00
Boaz Leskes 8f0a4ed390 [Test] testCorruptionOnNetworkLayer had a typo in test name. 2014-07-11 08:30:46 +02:00
Simon Willnauer bb964e7817 Revert "Improve handling of failed primary replica handling"
This reverts commit 75ed24f6b6.
2014-07-10 21:30:15 +02:00
Lee Hinman 107534c062 Do not ignore ConnectTransportException for shard replication operations
A ConnectTransportException should fail the replica shard

Closes #6183
2014-07-10 18:49:05 +02:00
Shay Banon 75ed24f6b6 Improve handling of failed primary replica handling
Out of #6808, we improved the handling of a primary failing to make sure replicas that are initializing are properly failed as well. After double checking it, it has 2 problems, the first, if the same shard routing is failed again, there is no protection that we don't apply the failure (which we do in failed shard cases), and the other was that we already tried to handle it (wrongly) in the elect primary method.
This change fixes the handling to work correctly in the elect primary method, and adds unit tests to verify the behavior
closes #6816
2014-07-10 18:30:18 +02:00
Simon Willnauer 4f131dfffb [TEST] Fold SuggestActionTest into SuggestSearchTests
Instead of running the tests twice this commit just randomizes the API
that we use to return the suggestions.
2014-07-10 18:02:10 +02:00
Colin Goodheart-Smithe 0e5f9898d1 Aggregations: DateHistogramBuilder accepts String preOffset and postOffset
This is what DateHistogramParser expects so will enable the builder to build valid requests using these variables.
Also added tests for preOffset and postOffset since these tests did not exist

Closes #5586
2014-07-10 16:38:09 +01:00
Simon Willnauer 0a988ad8f7 [STORE] Treat reading past EOF as an corrupted index when we fail to read segment infos 2014-07-10 17:25:47 +02:00
Simon Willnauer 81e86eba6e [TEST] Wait for longer on slow nodes until replicating has kicked in 2014-07-10 16:52:51 +02:00
javanna eddb378bae [TEST] added ability to provide settings for external nodes in backwards compatibility tests
Closes #6809
2014-07-10 16:45:16 +02:00
Simon Willnauer e7c67bf03b [TEST] Do RollingUpgrade in BWC tests 2014-07-10 16:24:01 +02:00
Simon Willnauer 62002e8192 [TEST] Close TransportClient after it's used in BulkProcessorTests otherwise it will leave threads behind 2014-07-10 16:06:38 +02:00
Simon Willnauer da148ca8b8 [TEST] Subclass ElasticsearchTestCase in LoggingConfigurationTests 2014-07-10 15:28:38 +02:00
Simon Willnauer 72e6150bc1 [STORE]: Make use of Lucene build-in checksums
Since Lucene version 4.8 each file has a checksum written as it's
footer. We used to calculate the checksums for all files transparently
on the filesystem layer (Directory / Store) which is now not necessary
anymore. This commit makes use of the new checksums in a backwards
compatible way such that files written with the old checksum mechanism
are still compared against the corresponding Alder32 checksum while
newer files are compared against the Lucene build in CRC32 checksum.

Since now every written file is checksummed by default this commit
also verifies the checksum for files during recovery and restore if
applicable.

Closes #5924

This commit also has a fix for #6808 since the added tests in
`CorruptedFileTest.java` exposed the issue.

Closes #6808
2014-07-10 15:04:00 +02:00
Shay Banon 9ca5e6e3e1 Add local node to cluster state
Today, the tribe node needs the local node so it adds it when it starts, but other APIs would benefit from adding the local node, also, adding the local node should be done in a cleaner manner, where it belongs, which is right after the discovery service starts in the cluster service
closes #6811
2014-07-10 14:49:52 +02:00
Simon Willnauer c9266e8b6b [TEST] Wait for primary allocations before restart
This commit ensures that all primaries are allocated before we
restart the node. If one primary is in post recovery when we
restart it will not be allocated otherwise.
2014-07-10 11:54:24 +02:00
Simon Willnauer fcadab869d [TEST] SuppressSysoutChecks on ElasticsearchTokenStreamTestCase 2014-07-10 07:48:12 +02:00
uboness c324103cbb added a fix to the PluginManagerTests to create config & bin dirs if they don't exist 2014-07-10 00:50:11 +02:00
Guillaume Nodet 263819c674 [ENV] Release node env if initialization fails
If the node initialisation fails, make sure the
node environment is closed correctly and thus
all locks (on data directories) being properly released.

Closes #6715
2014-07-10 00:14:52 +02:00
Simon Willnauer d82a434d10 [STORE] Make a hybrid directory default using `mmapfs` and `niofs`
`mmapfs` is really good for random access but can have sideeffects if
memory maps are large depending on the operating system etc. A hybrid
solution where only selected files are actually memory mapped but others
mostly consumed sequentially brings the best of both worlds and
minimizes the memory map impact.
This commit mmaps only the `dvd` and `tim` file for fast random access
on docvalues and term dictionaries.

Closes #6636
2014-07-10 00:01:43 +02:00
Simon Willnauer b69fa52588 [TEST] Mute PluginManagerTests#testLocalPluginInstallWithBinAndConfig 2014-07-10 00:00:41 +02:00
Simon Willnauer 9e4d738d7e [TEST] SuppressSysoutChecks on ElasticsearchPostingsFormatTest 2014-07-09 23:24:31 +02:00
uboness 6dae32b09a Added a check on moving bin & config plugin dirs
Plugins can contain bin & config sub-dirs that are copied to es's bin & config directories. If moving these directories fails we now throw an error.
2014-07-09 23:05:12 +02:00
Shay Banon 808c52706a [TEST] relax size test, to not run into OOM 2014-07-09 23:03:06 +02:00
Shay Banon 94539979c7 [TEST] make completion tests more resilient to delayed mapping
especially the tests that check for update of mapping, we need to make sure that the cluster is green so mappings won't get override, also, put mapping during index creation when possible
2014-07-09 22:19:06 +02:00
uboness 610900b781 Enables plugins to define default logging configuration for their needs.
- adds support for multiple logging configurations under the config dir (will pick up any logging.xxx in the config folder tree)
- plugins can now define a top level config directory that will be copied under es config dir and will be renamed after the plugin name (same as the support we have the plugin "bin" dirs)

 Closes #6802
2014-07-09 21:46:21 +02:00
Shay Banon 3e202b4343 Improve large bytes request handling by detecting content composite buffer
There is a special type of request that tries to not allocate another buffer when sending bytes request (used by the public cluster state action). With the new pages bytes reference support, the content can already be a composite channel buffer, take that into account when building the actual composite buffer that will be sent over the network
closes #6756
2014-07-09 21:40:15 +02:00
Shay Banon 8910e09beb Disable JSONP by default
By default, disable the option to use JSONP in our REST layer
closes #6795
2014-07-09 21:17:17 +02:00
Shay Banon b301132d7b [TEST] move more tests to assertBusy from awaitBusy 2014-07-09 21:16:23 +02:00