Commit Graph

10081 Commits

Author SHA1 Message Date
Ryan Doyle a2c93304f8 Docs: Add Wireshark protocol dissection support
Hi guys, I've finished my Wireshark dissector for Elasticsearch. It is now merged into their master. See https://code.wireshark.org/review/#/c/4948/

Closes #8495
2014-11-17 13:07:01 +01:00
Martijn van Groningen 28f3ea1b8d Test: Let the random parent/child tests use the query and filter parsers instead of creating the queries and filters in a custom way.
By using the query and filter parsers we increase the test coverage and make the random parent/child tests simpler.
2014-11-17 10:05:46 +01:00
Boaz Leskes 461c20049f Test: CorruptedFileTest.testCorruptionOnNetworkLayer used node settings when creating an index
Test used `indices.recovery.concurrent_streams` when creating an index but this is a node setting. Moved it to the node settings and added similar settings to speed up concurrent recoveries.

Also fixed a misleading log message in ShardRecoveryHandler when logging a remove corruption
2014-11-16 23:54:36 +01:00
Simon Willnauer b0b7c917c3 Prevent double wrapping directories in MockDirectoryWrapper 2014-11-16 20:54:49 +01:00
Simon Willnauer e6908de04a [TEST] Close node env otherwise windows can't delete tmp files 2014-11-16 17:44:38 +01:00
Simon Willnauer 176b4eb1ed Fix synchronization - leftover from refactoring 2014-11-16 15:58:10 +01:00
Simon Willnauer 1c64a113de [CORE] Intorduce shards level locks to prevent concurrent shard modifications
Today it's possible that the data directory for a single shard is used by more than on
IndexShard->Store instances. While one shard is already closed but has a concurrent recovery
running and a new shard is creating it's engine files can conflict and data can potentially
be lost. We also remove shards data without checking if there are still users of the files
or if files are still open which can cause pending writes / flushes or the delete operation
to fail. If the latter is the case the index might be treated as a dangeling index and is brought
back to life at a later point in time.

This commit introduces a shard level lock that prevents modifications to the shard data
while it's still in use. Locks are created per shard and maintined in NodeEnvironment.java.
In contrast to most java concurrency primitives those locks are not reentrant.

This commit also adds infrastructure that checks if all shard locks are released after tests.
2014-11-16 14:24:29 +01:00
Boaz Leskes 37661aed60 Logging: BroadcastOperationAction - added trace logging for successful shard-level responses
In order to be able to trace the exact shards that participated in the operation.
2014-11-15 18:33:56 +01:00
Martijn van Groningen 983a108776 Percolator: Remove `index.percolator.allow_unmapped_fields` setting.
There should be no option to opt out from strict field resolution for percolator query parsing.

Closes #8439
2014-11-14 22:39:07 +01:00
Martijn van Groningen 5714b0a7ad Aggregations: Changed child filters to not require a random access based bitset in `nested` agg.
Also the nested agg now requires docs to be consumed / scored in order.

Closes #8454
2014-11-14 21:40:44 +01:00
Martijn van Groningen 284491d874 Core: In the bitset cache only eagerly load bitsets for parent nested object fields.
Don't eagerly cache parent type filters in bitset cache or nested object fields that are leafs.
Also let parent/child queries not rely on FixedBitSetFilter, but rather on regular Filter

Closes #8440
2014-11-14 21:00:14 +01:00
Ken Wu 31fa4dc58b Configuration: Tab characters in YAML should throw an exception.
Throw an exception if there is a 'tab' character in the elasticsearch.yml file

Close #8259
2014-11-14 16:24:52 +01:00
nwarz e77f9720d2 Mapping: Fixes Merging of default analyzer
Fixed behaviour where two representations of the default index analyzer weren't being treated as equivalent. Added REST test to confirm fix.

Closes #2716
2014-11-14 09:58:26 +00:00
Olivier Favre 4d68d3d053 Provide more context variables in update scripts
In addition to `_source`, the following variables are available through
the `ctx` map: `_index`, `_type`, `_id`, `_version`, `_routing`,
`_parent`, `_timestamp`, `_ttl`.

Some of these fields are more useful still within the context of an
Update By Query, see #1607, #2230, #2231.
2014-11-14 10:14:39 +01:00
Alex Ksikes 936b4c63fc Term Vectors: Fix NPE with dfs and no tvs
Fixes a bug with dfs option for when term vectors are not stored and not generated.
2014-11-14 09:11:13 +01:00
Alexander Reelsen 9956e7721d Tests: Improve netty test behaviour
Based on some test failures, this commit fixes two minor things

* Bind ports only on so called ephemeral ports to prevent try to
  bind to ports where elasticsearch already runs on
* Remove @Network annotation as it was used in a wrong scope
2014-11-13 15:48:50 +01:00
Colin Goodheart-Smithe f56976c2b3 Removed bwc for GetIndexRequest serialisation 2014-11-13 14:20:42 +00:00
Vidar Kongsli 9e2f6bb5e5 Added quotes to allow spaces in installation path 2014-11-13 09:14:24 -05:00
Clinton Gormley 32fc657d71 Docs: Fixed a bad ref to docs-bulk-udp which no longer exists in master 2014-11-13 14:34:49 +01:00
Colin Goodheart-Smithe 353574d6af Indices API: Fix GET index API always running all features
Previous to this change all features (_alias,_mapping,_settings,_warmer) are run regardless of which features are actually requested. This change fixes the request object to resolve this bug
2014-11-13 13:22:46 +00:00
Clinton Gormley 6b05b229af Docs: Changed breaking docs in master to correspond with 1.x for easier merging 2014-11-13 13:50:57 +01:00
Colin Goodheart-Smithe 34b37ab7f0 [DOCS] Added documentation for log4j-extras dependency 2014-11-13 12:40:14 +00:00
Philip McMahon 4194a699c0 Logging: Add log4j-extras dependency
Close #7927
2014-11-13 12:39:30 +00:00
javanna c1428b5964 [DOCS] Expand logging documentation
Updated log4j link so it doesn't point to log4j 2.0 but version 1.2. Clarified which formats are supported and briefly explained what loggers and appenders are, plus added a link to the log4j docs.

Closes #5305
Closes #8455
2014-11-13 11:08:10 +01:00
Joel Taddei 7e72800c83 [DOCS] Corrected syntax error in search curl cmd
Closes #8447
2014-11-12 17:21:19 +01:00
Colin Goodheart-Smithe 972afe61a0 Mappers: Better validation of mapping JSON
Closes #7205
2014-11-12 14:32:25 +00:00
Mark Walkom bfd1bcd30a Updated threadpool documentation to elaborate/clarify what the pools are for and their values
Closes #8446
2014-11-12 22:33:38 +11:00
Martijn van Groningen 0b15859e33 Discovery: Removed unnecessary DiscoveryService reference from LocalDiscover, which causes circular Guice dependencies errors. 2014-11-12 11:25:41 +01:00
Israel Tsadok 7590629531 Docs: note about confusing disk threshold settings 2014-11-12 09:24:03 +01:00
Michael McCandless 7a22bfba3c Core: still don't load bloom filters, even when Directory instance doesn't have a codecService 2014-11-11 16:56:20 -05:00
Michael McCandless a783d342d2 Test: dump all threads when delete index fails during test cleanup 2014-11-11 10:49:26 -05:00
Martijn van Groningen 94c1a7dabe Docs: Fix incorrect documentation for the `index.query.parse.allow_unmapped_fields` setting.
The `index.query.parse.allow_unmapped_fields` setting can't influence whether unmapped fields are allowed
in alias filters and percolator queries.
2014-11-11 15:13:55 +00:00
Simon Willnauer 16cb0dc7a6 [TEST] Disable compression in BWC test for version < 1.3.2
The compression bug fixed in #7210 can still strike us since we are
running BWC test against these version. This commit disables compression
forcefully if the compatibility version is < 1.3.2 to prevent debugging
already known issues.
2014-11-11 14:12:34 +01:00
Lee Hinman f7d0f4d4e2 Refactor term analysis for simple_query_string prefix queries
Also fixes an issue where the token stream could be non-closed if an
exception occured.
2014-11-11 12:00:27 +01:00
Jörg Prante 8aa64c6b76 Query: add option for analyze wildcard/prefix also to simple_query_string query
The query_string query has an option for analyzing wildcard/prefix (#787) by a best effort approach.

This adds `analyze_wildcard` option also to simple_query_string.

The default is set to `false` so the existing behavior of simple_query_string is unchanged.
2014-11-11 10:12:17 +01:00
Michael McCandless 85fba3636a Test: restore logging to prior state 2014-11-10 18:09:12 -05:00
Michael McCandless 91bef2e40f Test: switch to TRACE logging for some components 2014-11-10 16:30:40 -05:00
Boaz Leskes 59117125b4 Internal: extend refresh-mapping logic to the _default_ type
When data nodes receive mapping updates from the master, the parse it and merge it into their own in memory representation (if there). If this results in different bytes then the master sent, the nodes will send a refresh-mapping command to indicate to the master that it's byte level storage of the mapping should be refreshed via the document mappers. This comes handy when the mapping format has changed, in a backwards compatible manner, and we want to make sure we can still rely on the bytes to identify changes.  An example of such a change can be seen at #4760.

This commit extends the logic to include the `_default_` type, which was never refreshed before. In some unlucky scenarios, this caused the _default_ mapping to be parsed with every cluster state update.

Closes #8413
2014-11-10 20:40:52 +01:00
Michael McCandless 87f6d6bc40 remove wrong repository 2014-11-10 14:14:11 -05:00
Michael McCandless 8aebb9656b Core: add max_determinized_states to query_string and regexp query/filter
This prevents too-difficult regular expressions from consuming
excessive RAM/CPU; the default max_determinized_states is 10,000 (same
as Lucene) but query_string and regepx query/filter can override
per-request.

The also upgrades to a new Lucene 5.0.0 snapshot.

Closes #8386

Closes #8357
2014-11-10 13:43:48 -05:00
Clinton Gormley f9810e591e Rest API: indices.delete should have "index" as a required param 2014-11-10 17:06:43 +01:00
Ryan Ernst ba621ee325 Scripting: Add score() back to AbstractSearchScript
See #8377
closes #8416
closes #8417
2014-11-10 07:46:46 -08:00
Clinton Gormley cff544dcc2 Docs: Removed old coming/added tags 2014-11-10 14:41:24 +01:00
Britta Weber c5a4c1d6b4 [docs] add 2d vis for decay functions and parameters
closes #8420
2014-11-10 10:56:41 +01:00
Colin Goodheart-Smithe d0da605a39 [TEST] added Get Index bwc test 2014-11-10 09:14:39 +00:00
Adrien Grand 144813629a Internal: Inverse DocIdSets' heuristic to find out fast DocIdSets.
DocIdSets.isFast(DocIdSet) has two issues:
 - it works on the DocIdSet interface while some doc sets can generate either
   slow or fast sets depending on their options (eg. whether an OrDocIdSet is
   fast or not depends on the wrapped clauses).
 - it only works because the result of this method is only taken into account
   when a DocIdSet has non-null `bits()`.

This commit changes this method to work on top of a DocIdSetIterator and to use
a black-list rather than a white list: slow iterators should really be the
exception rather than the rule.

Close #8380
2014-11-10 09:40:44 +01:00
Boaz Leskes a1d5bcaa35 Test: MinimumMasterNodesTests.testCanNotBringClusterDown should always set minimum master nodes. 2014-11-10 09:29:20 +01:00
Martijn van Groningen 696beb6e13 Core: Remove NestedDocsFilter, because it isn't used and also don't eagerly load it in bitset filter cache.
Closes #8414
2014-11-09 22:51:56 +00:00
Simon Willnauer 1368229075 [PING] Don't bubble up exception if we are shutting down and ping schedules get rejected 2014-11-09 19:27:45 +01:00
Simon Willnauer 0ff44d4d27 [STORE] Synchronize operations that modify file mappings on DistributorDirectory
The rename(String, String) method doesn't allow this implementation to use a simple
concurrent map. There is a race during a rename operation where files are not fully
renamed but already visible via #listAll(). This inconsistency can lead to problems
when opening commit points since the pending_segments_N as well as segments_N are visible
but not yet atomically renamed.

Yet, non of the methods that are synced are long running such that adding sychronization
doesn't introduce bottlenecks here. The Direcotry#sync(...) method is not synchronized since
it doesn't change any mapping nor does it depend on the mapping.
2014-11-09 19:05:33 +01:00