Commit Graph

1197 Commits

Author SHA1 Message Date
Simon Willnauer f5daa165f1 Remove ability to plug-in TransportService (#20505)
TransportService is such a central part of the core server, replacing
it's implementation is risky and can cause serious issues. This change removes the ability to
plug in TransportService but allows registering a TransportInterceptor that enables
plugins to intercept requests on both the sender and the receiver ends. This is a commonly used
and overwritten functionality but encapsulates the custom code in a contained manner.
2016-09-16 09:47:53 +02:00
Boaz Leskes 577dcb3237 Add current cluster state version to zen pings and use them in master election (#20384)
During a networking partition, cluster states updates (like mapping changes or shard assignments)
are committed if a majority of the masters node received the update correctly. This means that the current master has access to enough nodes in the cluster to continue to operate correctly. When the network partition heals, the isolated nodes catch up with the current state and get the changes they couldn't receive before. However, if a second partition happens while the cluster
is still recovering from the previous one *and* the old master is put in the minority side, it may be that a new master is elected which did not yet catch up. If that happens, cluster state updates can be lost.

This commit fixed 95% of this rare problem by adding the current cluster state version to `PingResponse` and use them when deciding which master to join (and thus casting the node's vote). 

Note: this doesn't fully mitigate the problem as a cluster state update which is issued concurrently with a network partition can be lost if the partition prevents the commit message (part of the two phased commit of cluster state updates) from reaching any single node in the majority side *and* the partition does allow for the master to acknowledge the change. We are working on a more comprehensive fix but that requires considerate work  and is targeted at 6.0.
2016-09-15 23:39:11 +02:00
Nik Everett d0be96df7b Clean up snapshots after each REST test
The only repository we can be sure is safe to clean is `fs` so we clean
any snapshots in those repositories after each test. Other repositories
like url and azure tend to throw exceptions rather than let us fetch
their contents during the REST test. So we clean what we can....

Closes #18159
2016-09-15 14:49:11 -04:00
Boaz Leskes 8469c98e34 Fix LongGCDisruption to be aware of log4j2 (#20348)
LongGCDisruption simulates a Long GC by suspending all threads belonging to a node. That's fine, unless those threads hold shared locks that can prevent other nodes from running. Concretely the logging infrastructure, which is shared between the nodes, can cause some deadlocks. LongGCDisruption has protection for this, but it needs to be updated to point at log4j2 classes, introduced in #20235

This commit also fixes improper handling of retry logic in LongGCDisruption and adds a protection against deadlocking the test code which activates the disruption (and uses logging too! :)).

On top of that we have some new, evil and nasty tests.
2016-09-15 08:50:18 +02:00
Ali Beyad 3f79874042 Prevent the rolling upgrades rest tests from cleaning up indices
after finishing if a the tests.rest.preserve_indices system property
is set
2016-09-14 23:34:19 -04:00
Simon Willnauer 17ddee7011 Remove TransportService#registerRequestHandler leniency (#20469)
`TransportService#registerRequestHandler` allowed to register
handlers more than once and issues an annoying warn log message when
this happens. This change simple throws an exception to prevent regsitering
the same handler more than once. This commit also removes the ability
to remove request handlers.

Relates to #20468
2016-09-14 20:32:29 +02:00
Luca Cavanna 14e17f44a1 Replace usage of LuceneTestCase#getBaseTempDirForTestClass (#20484)
LuceneTestCase#getBaseTempDirForTestClass is deprecated, we should not use it.

Closes #15845
2016-09-14 19:35:20 +02:00
Simon Willnauer 89640965d2 Unguice SearchModule (#20456)
After this change SearchModule doesn't subclass AbstractModule anymore and all wiring
happens in `Node.java`. As a side-effect several tests don't need a guice injector anymore.
2016-09-14 10:07:53 +02:00
Jason Tedor 7560101ec7 Complete Elasticsearch logger names
This commit modifies the logger names within Elasticsearch to be the
fully-qualified class name as opposed removing the org.elasticsearch
prefix and dropping the class name. This change separates the root
logger from the Elasticsearch loggers (they were equated from the
removal of the org.elasticsearch prefix) and enables log levels to be
set at the class level (instead of the package level).

Relates #20457
2016-09-13 22:46:54 -04:00
Jason Tedor fbe27664a6 Fix prefix logging
Today we add a prefix when logging within Elasticsearch. This prefix
contains the node name, and index and shard-level components if
appropriate.

Due to some implementation details with Log4j 2 , this does not work for
integration tests; instead what we see is the node name for the last
node to startup. The implementation detail here is that Log4j 2 there is
only one logger for a name, message factory pair, and the key derived
from the message factory is the class name of the message factory. So,
when the last node starts up and starts setting prefixes on its message
factories, it will impact the loggers for the other nodes.

Additionally, the prefixes are lost when logging an exception. This is
due to another implementation detail in Log4j 2. Namely, since we log
exceptions using a parameterized message, Log4j 2 decides that that
means that we do not want to use the message factory that we have
provided (the prefix message factory) and so logs the exception without
the prefix.

This commit fixes both of these issues.

Relates #20429
2016-09-13 14:46:34 -04:00
Nicholas Knize 1a60e1c3d2 Update docs for LatLonPoint cut over
This commit removes documentation for:

* geohash cell query
* lat_lon parameter
* geohash parameter
* geohash_precision parameter
* geohash_prefix parameter

It also updates failing tests that reference these parameters for backcompat.
2016-09-13 12:18:21 -05:00
Nicholas Knize ef926894f4 Cut over geo_point field and queries to new LatLonPoint type
This commit cuts over geo_point fields to use Lucene's new point-based LatLonPoint type for indexes created in 5.0. Indexes created prior to 5.0 continue to use their respective encoding type. Below is a description of the changes made to support the new encoding type:

* New indexes use a new LatLonPointFieldMapper which provides a parse method for the new type
* The new LatLonPoint parse method removes support for lat_lon and geohash parameters
* Backcompat testing for deprecated lat_lon and geohash parameters is added to all unit and integration tests
* LatLonPointFieldMapper provides DocValues support (enabled by default) which uses Lucene's new LatLonDocValuesField type
* New LatLonPoint field data classes are added for aggregation support (wraps LatLonPoint's Numeric Doc Values)
* MultiFields use the geohash as the string value instead of the lat,lon string making it easier to perform geo string queries on the geohash instead of a lat,lon comma delimited string.

Removed Features:

* With the removal of geohash indexing, GeoHashCellQuery support is removed for all new indexes (still supported on existing indexes)
* LatLonPoint does not support a Distance Range query because it is super inefficient. Instead, the geo_distance_range query should be accomplished using either the geo_distance aggregation, sorting by descending distance on a geo_distance query, or a boolean must not of the excluded distance (which is what the distance_range query did anyway).

TODO:

* fix/finish yaml changes for plugin and rest integration tests
* update documentation
2016-09-13 12:17:36 -05:00
Jason Tedor 013e3f6fcc Remove unused import from BootstrapForTesting
This commit removes an unused import for o.e.c.l.LogConfigurator from
o.e.b.BootstrapForTesting.
2016-09-13 09:49:15 -04:00
Tanguy Leroux 6090c51fc5 Add quiet option to disable console logging (#20422)
This commit adds a -q/--quiet option to Elasticsearch so that it does not log anything in the console and closes stdout & stderr streams. This is useful for SystemD to avoid duplicate logs in both journalctl and /var/log/elasticsearch/elasticsearch.log while still allows the JVM to print error messages in stdout/stderr if needed.

closes #17220
2016-09-13 14:08:24 +02:00
Lee Hinman 44278db1bc Merge pull request #20433 from dakrone/remove-cluster-name-folder-fallback
No longer allow cluster name in data path
2016-09-12 17:01:49 -05:00
Lee Hinman 94625d74e4 No longer allow cluster name in data path
In 5.x we allowed this with a deprecation warning. This removes the code
added for that deprecation, requiring the cluster name to not be in the
data path.

Resolves #20391
2016-09-12 15:47:01 -06:00
Simon Willnauer 686994ae2d Deguice SearchService and friends (#20423)
This change removes the guice dependency handling for SearchService and
several related classes like SearchTransportController and SearchPhaseController.
The latter two now have package private constructors and dependencies like FetchPhase
are now created by calling their constructors explicitly. This also cleans up several users
of the DefaultSearchContext and centralized it's creation inside SearchService.
2016-09-12 22:42:55 +02:00
Ali Beyad b1e87aa13c Split allocator decision making from decision application (#20347)
Splits the PrimaryShardAllocator and ReplicaShardAllocator's decision
making for a shard from the implementation of that decision on the
routing table. This is a step toward making it easier to use the same
logic for the cluster allocation explain APIs.
2016-09-12 16:21:39 -04:00
Boaz Leskes b08352047d Introduce IndexShardTestCase (#20411)
Introduce a base class for unit tests that are based on real `IndexShard`s. The base class takes care of all the little details needed to create and recover shards. 

This commit also moves `IndexShardTests` and `ESIndexLevelReplicationTestCase` to use the new base class. All tests in `IndexShardTests` that required a full node environment were moved to a new `IndexShardIT` suite.
2016-09-12 18:20:25 +02:00
Ali Beyad f39f9b9760 Update discovery nodes after cluster state is published (#20409)
Before, when there was a new cluster state to publish,
zen discovery would first update the set of nodes to
ping based on the new cluster state, then publish the new
cluster state. This is problematic because if the cluster
state failed to publish, then the set of nodes to ping
should not have been updated.

This commit fixes the issue by updating the set of
nodes to ping for fault detection only *after* the new
cluster state has been published.
2016-09-12 12:07:51 -04:00
Luca Cavanna 4b00cc37a1 Merge pull request #20382 from javanna/enhancement/cleanup_parse_elements
Cleanup sub fetch phase extension point
2016-09-09 22:47:15 +02:00
Tal Levy dda32545bb add ignore_missing option to relevant processors (#20194) 2016-09-09 12:20:18 -07:00
javanna 90ab460fcc move parsing of search ext sections to the coordinating node 2016-09-09 19:10:42 +02:00
javanna 65c7f61ad9 decouple registration of SearchExtParsers from sub fetch phases
Search section supports an ext section that is used to provide additional config needed from plugins. It is now tied to sub fetch phases because it is the only section that may need additional config, but there is no reason for the two to be tightly coupled.

It is now possible to register a searchExtParser independently from a sub fetch phase. All a search ext parser does is parsing some ext section of a search request, whose parsed resulting object is stored in the search context for later retrieval.
2016-09-09 18:05:49 +02:00
javanna f9530dfe8f remove FetchSubPhaseContext in favour of generic fetch sub phase builder of type object
The context was an object where the parsed info are stored. That is more of what we call the builder since after the search refactoring. No need for generics in FetchSubPhaseParser then. Also the previous setHitsExecutionNeeded wasn't useful, it can be removed as well, given that once there is a parsed ext section, it will become a builder that can be retrieved by the sub fetch phase. The sub fetch phase is responsible for doing nothing in case the builder is not set, meaning that the fetch sub phase is plugged in but the request didn't have the corresponding section.
2016-09-09 18:05:49 +02:00
javanna dc2ba90f48 clarify that SearchParseElement is only used for custom fetch sub phases and clean up extension point
SearchParseElement is renamed to FetchSubPhaseParser and moved to the search.fetch package. Its parse method doesn't get the SearchContext as argument anymore, only the XContentParser, and the return type is what gets parsed (the fetch sub phase context which we may as well rename later).

It is the parser that initializes the FetchSubPhaseContext then. SearchService retrieves the parser by name, calls parse against it and stores the result of parsing by name. No need for FetchSubPhase.ContextFactory anymore, which can be removed.
2016-09-09 18:05:49 +02:00
javanna a33ca70ff5 make docValueFields similar to other standard sub fetch phases
Given that doc value fields is our own fetch sub phase, it doesn't need to be implemented like if it was plugged in from the outside. It doesn't need its own fetch sub phase context, but it can just be an instance member in SearchContext
2016-09-09 18:05:49 +02:00
Jason Tedor d8475488b8 Disable console logging
Previously we would disable console logging in certain circumstances
(for example, if Elasticsearch is not in the foreground, or if
Elasticsearch is in the foreground but an exception was thrown during
bootstrap). This commit makes this handling work with Log4j 2. This will
prevent users from seeing double bootstrap check failure messages.

Relates #20387
2016-09-09 09:15:35 -04:00
Jason Tedor de43565abc Do not log full bootstrap checks exception
By default, when an exception causes the JVM to terminate, the stack
trace is printed. In the case of failing bootstrap checks, this stack
trace is useless to the user, and might even distract them from seeing
that the bootstrap checks failed for reasons under their control. With
this commit, we cause the stack trace for a failing bootstrap check to
be truncated.

We also modify some methods to not declare that they throw the top level
checked exception type Exception, but instead explicitly declare the
exceptions that they throw. These exceptions are caught and wrapped in a
BootstrapException so that we can percolate only two exception types out
of Bootstrap#init as checked exception, BootstrapException and
NodeValidationException.

Relates #19989
2016-09-08 10:56:11 -04:00
Tanguy Leroux 4fb7ac8254 Clean up XContentBuilder
This commit cleans most of the methods of XContentBuilder so that:
- Jackson's convenience methods are used instead of our custom ones (ie field(String,long) now uses Jackson's writeNumberField(String, long) instead of calling writeField(String) then writeNumber(long))
- null checks are added for all field names and values
- methods are grouped by type in the class source
- methods have the same parameters names
- duplicated methods like field(String, String...) and array(String, String...) are removed
- varargs methods now have the "array" name to reflect that it builds arrays
- unused methods like field(String,BigDecimal) are removed
- all methods now follow the execution path: field(String,?) -> field(String) then value(?), and value(?) -> writeSomething() method. Methods to build arrays also follow the same execution path.
2016-09-08 15:09:09 +02:00
Alexander Lin f825e8f4cb Exposing lucene 6.x minhash filter. (#20206)
Exposing lucene 6.x minhash tokenfilter

Generate min hash tokens from an incoming stream of tokens that can
be used to estimate document similarity.

Closes #20149
2016-09-07 09:38:12 +02:00
Simon Willnauer 11f2da5f14 Skip loading of jansi from log4j2 (#20334)
Jython shades `jansi` into it's classpath without changing it's package or
anything like that. This causes attempts to load native code on windows which
blows up tests. This change adds `log4j.skipJansi=true` system property to our
tests as well as to the JVM properties we set.
2016-09-06 05:53:00 -04:00
Simon Willnauer 5c2d9fa158 Improve error reporting for tests with BackgroundIndexer (#20324)
The BackgroundIndexer now uses auto-generated IDs randomly. This causes some problems
for tests that still rely on the fact that the IDs are increasing integers. This change
exposes all IDs via a Set<String> to iterate over for tests.
2016-09-05 16:28:49 +02:00
Nik Everett 549ca3178b Rename method in OldIndexUtils
loadIndexList -> loadDataFilesList. The new method name is more accurate.
2016-09-02 10:16:30 -04:00
javanna 7c03f65c36 [TEST] adjusted EsTestCase#randomPositiveLong 2016-09-02 10:23:49 +02:00
javanna 536d13ff11 ProcessInfo to implement Writeable rather than Streamable 2016-09-02 10:23:05 +02:00
Simon Willnauer 825b80f2a6 [TEST] fix possible NPE in ClientYamlTestExecutionContext 2016-09-02 10:07:58 +02:00
Jason Tedor 1e80adbfbe Configure test logging with Log4j 2
This commit configures test logging for Log4j 2. The default logger
configuration uses the console appender but at the error level, so most
tests are missing logging. Instead, this commit provides a configuration
for tests which is picked up from the classpath by Log4j 2 when it
initializes. However, this now means that we can no longer initialize
Log4j with a bare-bones configuration when tests run as doing so will
prevent Log4j 2 from attempting to configure logging via the
classpath. Consequently, we move this needed initialization (as
commented, to avoid a message about a status logger not being configured
when we are preparing to configure Log4j from properties files in the
config directory) to only run when we are explicitly configuring Log4j
from properties files.

Relates #20284
2016-09-01 14:00:47 -04:00
Simon Willnauer a0becd26b1 Optimize indexing for the autogenerated ID append-only case (#20211)
If elasticsearch controls the ID values as well as the documents
version we can optimize the code that adds / appends the documents
to the index. Essentially we an skip the version lookup for all
documents unless the same document is delivered more than once.

On the lucene level we can simply call IndexWriter#addDocument instead
of #updateDocument but on the Engine level we need to ensure that we deoptimize
the case once we see the same document more than once.

This is done as follows:

1. Mark every request with a timestamp. This is done once on the first node that
receives a request and is fixed for this request. This can be even the
machine local time (see why later). The important part is that retry
requests will have the same value as the original one.

2. In the engine we make sure we keep the highest seen time stamp of "retry" requests.
This is updated while the retry request has its doc id lock. Call this `maxUnsafeAutoIdTimestamp`

3. When the engine runs an "optimized" request comes, it compares it's timestamp with the
current `maxUnsafeAutoIdTimestamp` (but doesn't update it). If the the request
timestamp is higher it is safe to execute it as optimized (no retry request with the same
timestamp has been run before). If not we fall back to "non-optimzed" mode and run the request as a retry one
and update the `maxUnsafeAutoIdTimestamp` unless it's been updated already to a higher value

Relates to #19813
2016-09-01 10:39:40 +02:00
Simon Willnauer 419627c460 Ensure ESTestCase is initialized before we run tests 2016-09-01 09:39:44 +02:00
Jason Tedor 76ab02e002 Merge branch 'master' into log4j2
* master:
  Avoid NPE in LoggingListener
  Randomly use Netty 3 plugin in some tests
  Skip smoke test client on JDK 9
  Revert "Don't allow XContentBuilder#writeValue(TimeValue)"
  [docs] Remove coming in 2.0.0
  Don't allow XContentBuilder#writeValue(TimeValue)
  [doc] Remove leftover from CONSOLE conversion
  Parameter improvements to Cluster Health API wait for shards (#20223)
  Add 2.4.0 to packaging tests list
  Docs: clarify scale is applied at origin+offest (#20242)
2016-08-31 16:37:55 -04:00
Stian Lindhom c2eddaf2c9 Avoid NPE in LoggingListener
This commit avoids an NPE that could arise when implementing an
ESTestCase for test classes placed in the default package.

Relates #20269
2016-08-31 16:11:12 -04:00
Ali Beyad 4641254ea6 Parameter improvements to Cluster Health API wait for shards (#20223)
* Params improvements to Cluster Health API wait for shards

Previously, the cluster health API used a strictly numeric value
for `wait_for_active_shards`. However, with the introduction of
ActiveShardCount and the removal of write consistency level for
replication operations, `wait_for_active_shards` is used for
write operations to represent values for ActiveShardCount. This
commit moves the cluster health API's usage of `wait_for_active_shards`
to be consistent with its usage in the write operation APIs.

This commit also changes `wait_for_relocating_shards` from a
numeric value to a simple boolean value `wait_for_no_relocating_shards`
to set whether the cluster health operation should wait for
all relocating shards to complete relocation.

* Addresses code review comments

* Don't be lenient if `wait_for_relocating_shards` is set
2016-08-31 11:58:19 -04:00
Jason Tedor e166459bbe Merge branch 'master' into log4j2
* master:
  Increase visibility of deprecation logger
  Skip transport client plugin installed on JDK 9
  Explicitly disable Netty key set replacement
  percolator: Fail indexing percolator queries containing either a has_child or has_parent query.
  Make it possible for Ingest Processors to access AnalysisRegistry
  Allow RestClient to send array-based headers
  Silence rest util tests until the bogusness can be simplified
  Remove unknown HttpContext-based test as it fails unpredictably on different JVMs
  Tests: Improve rest suite names and generated test names for docs tests
  Add support for a RestClient base path
2016-08-31 10:59:27 -04:00
Jason Tedor 21dbc5ba84 Add empty test to ESLoggerUsageTests
This commit adds an empty test to ESLoggerUsageTests to avoid the test
suite from failing for having no tests after the existing tests were
marked as awaits fix in 1d197eddcc.
2016-08-31 04:41:07 -04:00
Jason Tedor 1d197eddcc Mark ESLoggerUsageTests as awaits fix
This commit marks the ESLoggerUsageTests as awaits fix as these tests
are not currently compatible with the Log4j 2 API.
2016-08-30 22:30:14 -04:00
Jason Tedor 4e69ac0272 Add link to open logger usage issue
This commit adds comments linking to an open issue regarding updating
the logger usage check for the Log4j 2 API.
2016-08-30 21:13:17 -04:00
Jason Tedor abf8a1a3f0 Avoid allocating log parameterized messages
This commit modifies the call sites that allocate a parameterized
message to use a supplier so that allocations are avoided unless the log
level is fine enough to emit the corresponding log message.
2016-08-30 18:17:09 -04:00
Ryan Ernst 2a7a187bf8 Silence rest util tests until the bogusness can be simplified 2016-08-30 14:58:44 -07:00
Ryan Ernst e19f2b6348 Tests: Improve rest suite names and generated test names for docs tests
Rest test suites are currently only the directory above the yaml test
file. That is confusing when there are more than one directory level
which contain yaml tests, as there are in generated docs tests. This
change makes rest tests use the full relative path to the rest test root
as the suite name, and also makes the test names for docs tests a little
clearer (that they are testing an example from a specific line number,
instead of just the line number as an opaque test name).
2016-08-30 13:55:44 -07:00
Jason Tedor 7da0cdec42 Introduce Log4j 2
This commit introduces Log4j 2 to the stack.
2016-08-30 13:31:24 -04:00
Jason Tedor 5a8f2d7fb3 Disable logger usage checks
This commit disables the logger usage checks as they will not be
compatible with Log4j 2. This disabling is temporary, they will return.
2016-08-30 13:28:07 -04:00
javanna 61145bfb2f [TEST] minor cleanups to AbstractQueryTestCase
Removed null check for token, if we are outside the null it already means it is null.

Fixed typo in comment and remove leftover assignment to unused local variable.
2016-08-29 16:52:11 +02:00
Yannick Welsch f070c8727b [TEST] Add additional logging to testStaleMasterNotHijackingMajority
This test is periodically failing. As I suspect that the GCDisruption scheme is somehow making the wrong node block on
its cluster state update thread, I've added some more logging and a thread dump once the given assertion triggers
again.
2016-08-29 13:42:13 +02:00
Yannick Welsch 1b75cb63a2 Add recovery source to ShardRouting (#19516)
Adds an explicit recoverySource field to ShardRouting that characterizes the type of recovery to perform:

- fresh empty shard copy
- existing local shard copy
- recover from peer (primary)
- recover from snapshot
- recover from other local shards on same node (shrink index action)
2016-08-27 16:11:10 +02:00
Tanguy Leroux 68b943dc53 Fix MoreLikeThisQueryBuilderTests.testUnknownObjectException()
Objects hierarchy must be tracked when entering/leaving an object so that it better knows if the "newField" has been inserted into an arbitrary holding object.

Can be reproduced with gradle :core:test -Dtests.seed=760F8BD0F7E46D45 -Dtests.class=org.elasticsearch.index.query.MoreLikeThisQueryBuilderTests -Dtests.method="testUnknownObjectException" -Dtests.security.manager=true -Dtests.locale=ko -Dtests.timezone=Etc/Zulu
2016-08-25 20:54:06 +02:00
Tanguy Leroux fbcfddbb77 Fix AbstractQueryTestCase.testUnknownObjectException()
When need to check the whole hierarchy of objects to know if the newly inserted "newField" object is part of an arbitrary holding object or not.

Reproduced with `gradle :modules:percolator:test -Dtests.seed=736B0B67DA7A3632 -Dtests.class=org.elasticsearch.percolator.PercolateQueryBuilderTests -Dtests.method="testUnknownObjectException" -Dtests.security.manager=true -Dtests.locale=es-ES -Dtests.timezone=ART`
2016-08-25 16:24:22 +02:00
Michael McCandless 1fe3e36934 Merge pull request #20147 from mikemccand/lucene_620_upgrade
Upgrade to Lucene 6.2.0
2016-08-25 06:03:34 -04:00
Tanguy Leroux 20719f9b2f Improve AbstractQueryTestCase#unknownObjectExceptionTest()
This method fails when a randomized string value contains a double-quote. This commit changes the method so that it is not based on string concatenation anymore. It now use XContentGenerator & XContentParser to mutate the valid queries.

Related #19864
2016-08-25 10:57:30 +02:00
Mike McCandless 5eb66e3378 Mark Scandinavian analysis components as multi term aware 2016-08-24 19:50:25 -04:00
Mike McCandless 7492300544 Remove now unused Store.renameFile, and obsolete commented out code 2016-08-24 18:20:30 -04:00
Mike McCandless 0ccfe69789 Upgrade to Lucene 6.2.0 2016-08-24 17:26:28 -04:00
Jim Ferenczi 4682fc34ae Add the ability to disable the retrieval of the stored fields entirely
This change adds a special field named _none_ that allows to disable the retrieval of the stored fields in a search request or in a TopHitsAggregation.

To completely disable stored fields retrieval (including disabling metadata fields retrieval such as _id or _type) use _none_ like this:

````
POST _search
{
   "stored_fields": "_none_"
}
````
2016-08-24 16:40:08 +02:00
Nicholas Knize 28ed0e7abf Deprecate optimize_bbox on geodistance queries
Deprecates the optimize_bbox parameter on geodistance queries. This has no longer been needed since version 2.2 because lucene geo distance queries (postings and LatLonPoint) already optimize by bounding box.
2016-08-23 09:14:54 -05:00
Yannick Welsch 771668f380 Use routingResult method to update cluster state after reroute
This ensures that the routing table as well as the metadata (with the primary terms and in-sync allocation ids) is updated.
2016-08-19 17:15:02 +02:00
Ryan Ernst 8c60455ed6 Fix checkstyle line length violations in allocation tests 2016-08-17 16:28:31 -07:00
Ryan Ernst 1ff348ed7f Plugins: Make custom allocation deciders use pull based extensions
This change converts AllocationDecider registration from push based on
ClusterModule to implementing with a new ClusterPlugin interface.
AllocationDecider instances are allowed to use only Settings and
ClusterSettings.
2016-08-17 15:55:31 -07:00
Ryan Ernst 2ea50bc162 Merge pull request #20018 from rjernst/split_disk_threshold
Internal: Split disk threshold monitoring from decider
2016-08-17 07:57:50 -07:00
Yannick Welsch 27a760f9c1 Add routing changes API to RoutingAllocation (#19992)
Adds a class that records changes made to RoutingAllocation, so that at the end of the allocation round other values can be more easily derived based on these changes. Most notably, it:

- replaces the explicit boolean flag that is passed around everywhere to denote changes to the routing table. The boolean flag is automatically updated now when changes actually occur, preventing issues where it got out of sync with actual changes to the routing table.
- records actual changes made to RoutingNodes so that primary term and in-sync allocation ids, which are part of index metadata, can be efficiently updated just by looking at the shards that were actually changed.
2016-08-17 10:46:59 +02:00
Ryan Ernst b2c0f2d08f Internal: Split disk threshold monitoring from decider
In addition to be an allocation decider, DiskThresholdDecider also
monitors the used disk in order to trigger a reroute when the thresholds
are crossed. This change splits out the settings for disk thresholds
into DiskThresholdSettings, and moves the monitoring to a new
DiskThresholdMonitor.  DiskThresholdDecider is then in line with other
allocation deciders, needing only Settings and ClusterSettings for
construction, which will allow deguicing allocation deciders.
2016-08-17 00:22:16 -07:00
Lee Hinman 1825d8060c Merge remote-tracking branch 'dakrone/lockobtainfailed-replacement' 2016-08-16 14:41:27 -06:00
Lee Hinman 1de3388fa3 Switching LockObtainFailedException over to ShardLockObtainFailedException
`LobObtainFailedException` should be reserved for on-disk locks that
Lucene attempts (like `write.lock`). This switches our in-memory
semaphore locks for shards to use a different exception. Additionally,
ShardLockObtainFailedException no longer subclasses IOException, since
no IO is being done is this case.

Resolves #19978
2016-08-16 14:37:36 -06:00
Nik Everett 46bf8baf2e Switch aggregation registration for push to pull
Adds `getAggregations` to `SearchPlugin` which can be used to register
aggregations.

Fixup MockNode which wasn't createing MockBigArrays.
2016-08-16 09:08:36 -04:00
Nik Everett cf6e1a4362 Move all FetchSubPhases to `o.e.search.fetch.subphase`
As the most complicated `FetchSubPhase` highlighting gets its own package
(`o.e.seach.fetch.subphase.highlight`. No other `FetchSubPhase`s get their
own package. Instead they all reside together in `o.e.search.fetch.subphase`.

Add package descriptions to `o.e.search.fetch` and subpackages.
2016-08-12 18:21:15 -04:00
Jason Tedor 1f0673c9bd Default max local storage nodes to one
This commit defaults the max local storage nodes to one. The motivation
for this change is that a default value greather than one is dangerous
as users sometimes end up unknowingly starting a second node and start
thinking that they have encountered data loss.

Relates #19964
2016-08-12 09:26:20 -04:00
Nik Everett 9f8f2ea54b Remove ESIntegTestCase#pluginList
It was a useful method in 1.7 when javac's type inference wasn't as
good, but now we can just replace it with `Arrays.asList`.
2016-08-11 15:44:02 -04:00
Yannick Welsch 522b137097 Make NetworkPartition disruption scheme configurable (#19534)
This commit separates the description of the links in the network that are to be disrupted from the failure that is to be applied to the links (disconnect/unresponsive/delay). Previously we had subclasses for the various kind of network disruption schemes combining on one hand failure mode (disconnect/unresponsive/delay) as well as the network links to cut (two partitions / bridge partitioning) into a single class.
2016-08-11 14:55:06 +02:00
Adrien Grand 0d6ac57acf Collapse o.e.index.mapper packages. #19921
I also reduced the visibility of a couple classes and renamed/consolidated some
test classes for consistency, eg. removing the `Simple` prefix or using the
`<Type>FieldMapperTests` convention for testing field mappers.
2016-08-10 17:51:11 +02:00
javanna 7d4a6499e1 [TEST] add inline comments to AbstractQueryTestCase#unknownObjectExceptionTest 2016-08-10 12:21:25 +02:00
javanna 8391e6de37 [TEST] enable testUnknownObjectException for alternate query versions too 2016-08-10 12:21:25 +02:00
javanna 0a98b5e56e [TEST] make AbstractQueryTestCase#testUnknownObjectException more accurate
testUnknownObjectException used to generate malformed json objects in some cases, due to the existence of arrays as it was not closing the injected object correctly. That is why the test was catching JsonParseException among the exception that are expected to be thrown. That is fixed by tracking where the new object is placed and placing its end object marker to the right level rather than always at the end.

Also introduced a mechanism to explicitly declare objects that won't cause any exception when they get additional objects injected, so that there is no need to override the method anymore as that caused copy pasting of the whole test method. This also makes sure that changes are reflected in tests, as those inner objects are not skipped but we actually check that what is declared is true (no exceptions get thrown when an additional object is added within them.
2016-08-10 11:48:51 +02:00
Lee Hinman 5849c488b5 Merge remote-tracking branch 'dakrone/compliation-breaker' 2016-08-09 11:57:26 -06:00
Lee Hinman 2be52eff09 Circuit break the number of inline scripts compiled per minute
When compiling many dynamically changing scripts, parameterized
scripts (<https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-scripting-using.html#prefer-params>)
should be preferred. This enforces a limit to the number of scripts that
can be compiled within a minute. A new dynamic setting is added -
`script.max_compilations_per_minute`, which defaults to 15.

If more dynamic scripts are sent, a user will get the following
exception:

```json
{
  "error" : {
    "root_cause" : [
      {
        "type" : "circuit_breaking_exception",
        "reason" : "[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead",
        "bytes_wanted" : 0,
        "bytes_limit" : 0
      }
    ],
    "type" : "search_phase_execution_exception",
    "reason" : "all shards failed",
    "phase" : "query",
    "grouped" : true,
    "failed_shards" : [
      {
        "shard" : 0,
        "index" : "i",
        "node" : "a5V1eXcZRYiIk8lecjZ4Jw",
        "reason" : {
          "type" : "general_script_exception",
          "reason" : "Failed to compile inline script [\"aaaaaaaaaaaaaaaa\"] using lang [painless]",
          "caused_by" : {
            "type" : "circuit_breaking_exception",
            "reason" : "[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead",
            "bytes_wanted" : 0,
            "bytes_limit" : 0
          }
        }
      }
    ],
    "caused_by" : {
      "type" : "general_script_exception",
      "reason" : "Failed to compile inline script [\"aaaaaaaaaaaaaaaa\"] using lang [painless]",
      "caused_by" : {
        "type" : "circuit_breaking_exception",
        "reason" : "[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead",
        "bytes_wanted" : 0,
        "bytes_limit" : 0
      }
    }
  },
  "status" : 500
}
```

This also fixes a bug in `ScriptService` where requests being executed
concurrently on a single node could cause a script to be compiled
multiple times (many in the case of a powerful node with many shards)
due to no synchronization between checking the cache and compiling the
script. There is now synchronization so that a script being compiled
will only be compiled once regardless of the number of concurrent
searches on a node.

Relates to #19396
2016-08-09 10:26:27 -06:00
javanna 329eaaea65 [TEST] expand AbstractQueryTestCase#testQueryWrappedInArray to run against query alternate versions 2016-08-08 19:09:43 +02:00
javanna 2437226802 [TEST] restore tests repeatability in AbstractQueryTestCase
Some random operations were conditionally performed in the before test, which made tests not repeatable. For instance take the seed chain to repeat a specific iteration and try to reproduce it, this conditional code would get executed in both cases when trying to isolate the failure, but not among the different iterations (as only the first method/iteration executes it), hence the failure will not reproduce.

Moved the random operations to beforeClass and left the non random part in the before method, which is needed as it depends on some method that can be overridden by subclasses.
2016-08-05 22:38:31 +02:00
Luca Cavanna 4c1a3b9a53 Merge pull request #19791 from javanna/fix/multiple_fields_queries
Query parsers to throw exception when multiple field names are provided
2016-08-05 15:53:35 +02:00
Ali Beyad f59ca9083b Snapshot repository cleans up empty index folders (#19751)
This commit cleans up indices in a snapshot repository when all
snapshots containing the index are all deleted. Previously, empty
indices folders would lay around after all snapshots containing
them were deleted.
2016-08-05 09:39:02 -04:00
javanna 7f0bd56094 [TEST] use expectThrows wherever possible in query builder unit tests 2016-08-05 13:55:18 +02:00
Nik Everett 1e587406d8 Fail yaml tests and docs snippets that get unexpected warnings
Adds `warnings` syntax to the yaml test that allows you to expect
a `Warning` header that looks like:
```
    - do:
        warnings:
            - '[index] is deprecated'
            - quotes are not required because yaml
            - but this argument is always a list, never a single string
            - no matter how many warnings you expect
        get:
            index:    test
            type:    test
            id:        1
```

These are accessible from the docs with:
```
// TEST[warning:some warning]
```

This should help to force you to update the docs if you deprecate
something. You *must* add the warnings marker to the docs or the build
will fail. While you are there you *should* update the docs to add
deprecation warnings visible in the rendered results.
2016-08-04 15:23:05 -04:00
Daniel Mitterdorfer 4598c36027 Fix various concurrency issues in transport (#19675)
Due to various issues (most notably a missing happens-before edge
between socket accept and channel close in MockTcpTransport),
MockTcpTransportTests sometimes did not terminate.

With this commit we fix various concurrency issues that led to
this hanging test.

Failing example build: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-os-compatibility/os=oraclelinux/835/console
2016-08-04 21:00:59 +02:00
javanna cd9388ce66 [TEST] parse query alternate versions in strict mode
AbstractQueryTestCase parses the main version of the query in strict mode, meaning that it will fail if any deprecated syntax is used. It should do the same for alternate versions (e.g. short versions). This is the way it is because the two alternate versions for ids query are both deprecated. Moved testing for those to a specific test method that isolates the deprecations and actually tests that the two are deprecated.
2016-08-04 19:49:43 +02:00
javanna 146f02183d [TEST] remove unused methods and fix some warnings in AbstractQueryTestCase
Also fix line length issues
2016-08-04 10:06:25 +02:00
Luca Cavanna c5a9427293 Merge pull request #19750 from javanna/fix/npe_parse_field_array
Throw ParsingException if a query is wrapped in an array
2016-08-03 18:21:39 +02:00
javanna 4805250ecf Throw ParsingException if a query is wrapped in an array
Our parsing code accepted up until now queries in the following form (note that the query starts with `[`:

```
{
    "bool" : [
        {
          "must" : []
        }
    ]
}
```

This would lead to a null pointer exception as most parsers assume that the field name ("must" in this example) is the first thing that can be found in a query if its json is valid, hence always non null while parsing. Truth is that the additional array layer doesn't make the json invalid, hence the following code fragment would cause NPE within ParseField, because null gets passed to `parseContext.isDeprecatedSetting`:

```
if (token == XContentParser.Token.FIELD_NAME) {
    currentFieldName = parser.currentName();
} else if (parseContext.isDeprecatedSetting(currentFieldName)) {
    // skip
} else if (token == XContentParser.Token.START_OBJECT) {
```

We could add null checks in each of our parsers in lots of places, but we rely on `currentFieldName` being non null in all of our parsers, and we should consider it a bug when these unexpected situations are not caught explicitly. It would be best to find a way to prevent such queries altogether without changing all of our parsers.

The reason why such a query goes through is that we've been allowing a query to start with either `[` or `{`. The only reason I found is that we accept `match_all : []`. This seems like an undocumented corner case that we could drop support for. Then we can be stricter and accept only `{` as start token of a query. That way the only next token that the parser can encounter if the json is valid (otherwise the json parser would barf earlier) is actually a field_name, hence the assumption that all our parser makes hold.

The downside of this is simply dropping support for `match_all : []`

Relates to #12887
2016-08-03 17:05:14 +02:00
Nik Everett ca8f666c66 Add line number to yaml test failures
Old:
```
   > Throwable #1: java.lang.AssertionError: expected [2xx] status code but api [reindex] returned [400 Bad Request] [{"error":{"root_cause":[{"type":"parsing_exception","reason":"[reindex] failed to parse field [dest]","line":1,"col":25}],"type":"parsing_exception","reason":"[reindex] failed to parse field [dest]","line":1,"col":25,"caused_by":{"type":"illegal_argument_exception","reason":"[dest] unknown field [asdfadf], parser not found"}},"status":400}]
   >    at __randomizedtesting.SeedInfo.seed([9325F8C5C6F227DD:1B71C71F680E4A25]:0)
   >    at org.elasticsearch.test.rest.yaml.section.DoSection.execute(DoSection.java:119)
   >    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:309)
   >    at java.lang.Thread.run(Thread.java:745)
```

New:
```
   > Throwable #1: java.lang.AssertionError: Failure at [reindex/10_basic:12]: expected [2xx] status code but api [reindex] returned [400 Bad Request] [{"error":{"root_cause":[{"type":"parsing_exception","reason":"[reindex] failed to parse field [dest]","line":1,"col":25}],"type":"parsing_exception","reason":"[reindex] failed to parse field [dest]","line":1,"col":25,"caused_by":{"type":"illegal_argument_exception","reason":"[dest] unknown field [asdfadf], parser not found"}},"status":400}]
   >    at __randomizedtesting.SeedInfo.seed([444DEEAF47322306:CC19D175E9CE4EFE]:0)
   >    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:329)
   >    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:309)
   >    at java.lang.Thread.run(Thread.java:745)
   > Caused by: java.lang.AssertionError: expected [2xx] status code but api [reindex] returned [400 Bad Request] [{"error":{"root_cause":[{"type":"parsing_exception","reason":"[reindex] failed to parse field [dest]","line":1,"col":25}],"type":"parsing_exception","reason":"[reindex] failed to parse field [dest]","line":1,"col":25,"caused_by":{"type":"illegal_argument_exception","reason":"[dest] unknown field [asdfadf], parser not found"}},"status":400}]
   >    at org.elasticsearch.test.rest.yaml.section.DoSection.execute(DoSection.java:119)
   >    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:325)
   >    ... 37 more
```

Sorry for the longer stack trace, but I wanted to be sure I didn't throw
anything away by accident.
2016-08-03 10:59:57 -04:00
Britta Weber abcb4c8a97 [Test] move methods from bwc test to test package for use in plugins (#19738)
* [Test] move methods from bwc test to test package for use in other plugins
2016-08-03 11:41:46 +02:00
Ryan Ernst df8dc64e9b Plugins: Make NamedWriteableRegistry immutable and add extenion point for named writeables
Currently any code that wants to added NamedWriteables to the
NamedWriteableRegistry can do so via guice injection of the registry,
and registering at construction time. However, this makes the registry
complex: it has both get and register methods synchronized, and there is
likely contention on the read side from multiple threads.  The
registration has mostly already been contained to guice modules at node
construction time.

This change makes the registry immutable, taking all of the
NamedWriteable readers at construction time. It also allows plugins to
added arbitrary named writables that it may use in its own transport
actions.
2016-08-02 15:56:25 -07:00
Ali Beyad c4ae23f5d8 Enables implementations of the BlobContainer interface to (#19749)
conform with the requirements of the writeBlob method by
throwing a FileAlreadyExistsException if attempting to write
to a blob that already exists. This change means implementations
of BlobContainer should never overwrite blobs - to overwrite a
blob, it must first be deleted and then can be written again.

Closes #15579
2016-08-02 09:48:21 -04:00
Ali Beyad 456ea56527 Cleans up the BlobContainer interface by removing the (#19727)
writeBlob method takes a BytesReference in favor of just
the writeBlob method that takes an InputStream.

Closes #18528
2016-08-02 09:21:43 -04:00
Ali Beyad 25d8eca62d Removes the notion of write consistency level across all APIs in
favor of waiting for active shard copy count (wait_for_active_shards).
2016-08-01 13:35:29 -04:00
Ali Beyad 9f88a8194a Merge pull request #19706 from elastic/enhancement/snapshot-blob-handling
More resilient blob handling in snapshot repositories
2016-08-01 12:03:53 -04:00
Tanguy Leroux 386902903e [TEST] Kill remaining lang-groovy messy tests
After #13834 many tests that used Groovy scripts (for good or bad reason) in their tests have been moved in the lang-groovy module and the issue #13837 has been created to track these messy tests in order to clean them up.

The work started with #19280, #19302 and #19336 and this PR moves the remaining messy tests back in core, removes the dependency on Groovy, changes the scripts in order to use the mocked script engine, and change the tests to integration tests.

It also moves IndexLookupIT test back (even if it has good chance to be removed soon) and fixes its tests.

It also changes AbstractQueryTestCase to use custom script plugins in tests.

closes #13837
2016-08-01 16:59:47 +02:00
Alexander Lin 9ac6389e43 Rename operation to result and reworking responses
* Rename operation to result and reworking responses
* Rename DocWriteResponse.Operation enum to DocWriteResponse.Result

These are just easier to interpret names.

Closes #19664
2016-08-01 10:42:58 -04:00
Alexander Lin 119026b4fb Remove isCreated and isFound from the Java API
This is cleanup work from #19566, where @nik9000 suggested trying to nuke the isCreated and isFound methods. I've combined nuking the two methods with removing UpdateHelper.Operation in favor of DocWriteResponse.Operation here.

Closes #19631.
2016-07-29 14:21:43 -04:00
Nik Everett 2e7336dc10 Add package-info to o.e.test.rest
This removes two packages, consolidating them into their parent package
and adds `package-info.java` files to describe all of the packages under
`org.elasticsearch.test.rest`.
2016-07-28 16:07:44 -04:00
David Pilato 0d2ccf0989 Merge branch 'pr/15724-gce-network-host-master' 2016-07-28 16:59:18 +02:00
Nik Everett fb45f6a8a8 Add authentication to reindex-from-remote
The tests for authentication extend ESIntegTestCase and use a mock
authentication plugin. This way the clients don't have to worry about
running it. Sadly, that means we don't really have good coverage on the
REST portion of the authentication.

This also adds ElasticsearchStatusException, and exception on which you
can set an explicit status. The nice thing about it is that you can
set the RestStatus that it returns to whatever arbitrary status you like
based on the status that comes back from the remote system.
reindex-from-remote then uses it to wrap all remote failures, preserving
the status from the remote Elasticsearch or whatever proxy is between us
and the remove Elasticsearch.
2016-07-27 14:17:41 -04:00
David Pilato e9339a1960 Merge branch 'master' into pr/15724-gce-network-host-master 2016-07-27 11:24:53 +02:00
Boaz Leskes 6f76740a58 await fix testConcurrentSendRespondAndDisconnect 2016-07-26 23:42:10 +02:00
Nik Everett 9270e8b22b Rename client yaml test infrastructure
This makes it obvious that these tests are for running the client yaml
suites. Now that there are other ways of running tests using the REST
client against a running cluster we can't go on calling the shared
client yaml tests "REST tests". They are rest tests, but they aren't
**the** rest tests.
2016-07-26 13:53:44 -04:00
David Pilato 0d3edee928 Merge branch 'master' into pr/15724-gce-network-host-master 2016-07-26 18:51:01 +02:00
David Pilato fde15ae470 Move custom name resolvers to NetworkService CTOR
Instead of using NetworkModule we can directly inject them in NetworkService CTOR.

See https://github.com/elastic/elasticsearch/pull/15765#issuecomment-235307974
2016-07-26 18:26:30 +02:00
Boaz Leskes fabfd425f0 remove socket timeout from MockTcpTransport
added in b208a7dbae
2016-07-26 18:04:05 +02:00
Boaz Leskes dbdb6341a5 increase logging information in testConcurrentSendRespondAndDisconnect 2016-07-26 18:02:22 +02:00
Daniel Mitterdorfer b208a7dbae Add socket timeout in MockTcpTransport
With this commit we set an explicit socket timeout in
MockTcpTransport to avoid hanging tests in case of disconnections.
2016-07-26 16:04:51 +02:00
Nik Everett a95d4f4ee7 Add Location header and improve REST testing
This adds a header that looks like `Location: /test/test/1` to the
response for the index/create/update API. The requirement for the header
comes from https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

https://tools.ietf.org/html/rfc7231#section-7.1.2 claims that relative
URIs are OK. So we use an absolute path which should resolve to the
appropriate location.

Closes #19079

This makes large changes to our rest test infrastructure, allowing us
to write junit tests that test a running cluster via the rest client.
It does this by splitting ESRestTestCase into two classes:
* ESRestTestCase is the superclass of all tests that use the rest client
to interact with a running cluster.
* ESClientYamlSuiteTestCase is the superclass of all tests that use the
rest client to run the yaml tests. These tests are shared across all
official clients, thus the `ClientYamlSuite` part of the name.
2016-07-25 17:02:40 -04:00
Boaz Leskes b90dff7292 increase log level to debug in testConcurrentSendRespondAndDisconnect 2016-07-25 22:01:09 +02:00
Ali Beyad 2f831c3abb BytesArray tests fix: offsets don't matter on a zero bytes array
Closes #19582
2016-07-25 15:22:08 -04:00
Tanguy Leroux f745c96949 Clean up more messy tests
After #13834 many tests that used Groovy scripts (for good or bad reason) in their tests have been moved in the lang-groovy module and the issue #13837 has been created to track these messy tests in order to clean them up.

This commit moves more tests back in core, removes the dependency on Groovy, changes the scripts in order to use the mocked script engine, and change the tests  to integration tests.
2016-07-25 17:02:49 +02:00
Boaz Leskes cd596772ee Persistent Node Names (#19456)
With #19140 we started persisting the node ID across node restarts. Now that we have a "stable" anchor, we can use it to generate a stable default node name and make it easier to track nodes over a restarts. Sadly, this means we will not have those random fun Marvel characters but we feel this is the right tradeoff.

On the implementation side, this requires a bit of juggling because we now need to read the node id from disk before we can log as the node node is part of each log message. The PR move the initialization of NodeEnvironment as high up in the starting sequence as possible, with only one logging message before it to indicate we are initializing. Things look now like this:

```
[2016-07-15 19:38:39,742][INFO ][node                     ] [_unset_] initializing ...
[2016-07-15 19:38:39,826][INFO ][node                     ] [aAmiW40] node name set to [aAmiW40] by default. set the [node.name] settings to change it
[2016-07-15 19:38:39,829][INFO ][env                      ] [aAmiW40] using [1] data paths, mounts [[ /(/dev/disk1)]], net usable_space [5.5gb], net total_space [232.6gb], spins? [unknown], types [hfs]
[2016-07-15 19:38:39,830][INFO ][env                      ] [aAmiW40] heap size [1.9gb], compressed ordinary object pointers [true]
[2016-07-15 19:38:39,837][INFO ][node                     ] [aAmiW40] version[5.0.0-alpha5-SNAPSHOT], pid[46048], build[473d3c0/2016-07-15T17:38:06.771Z], OS[Mac OS X/10.11.5/x86_64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_51/25.51-b03]
[2016-07-15 19:38:40,980][INFO ][plugins                  ] [aAmiW40] modules [percolator, lang-mustache, lang-painless, reindex, aggs-matrix-stats, lang-expression, ingest-common, lang-groovy, transport-netty], plugins []
[2016-07-15 19:38:43,218][INFO ][node                     ] [aAmiW40] initialized
```

Needless to say, settings `node.name` explicitly still works as before.

The commit also contains some clean ups to the relationship between Environment, Settings and Plugins. The previous code suggested the path related settings could be changed after the initial Environment was changed. This did not have any effect as the security manager already locked things down.
2016-07-23 22:46:48 +02:00
Jason Tedor 2d1b0587dd Introduce Netty 4
This commit adds transport-netty4, a transport and HTTP implementation
based on Netty 4.

Relates #19526
2016-07-22 22:26:35 -04:00
Ali Beyad a0a4d67eae All snapshot metadata files use UUID for the blob ID 2016-07-22 13:52:13 -04:00
gfyoung d98fd36dad Added deleteBlob IOException test 2016-07-22 13:48:45 -04:00
javanna db8beeba3b Merge branch 'master' into feature/async_rest_client 2016-07-22 15:51:03 +02:00
Boaz Leskes bd574d92ae Verify lower level transport exceptions don't bubble up on disconnects (#19518)
#19096 introduced a generic TCPTransport base class so we can have multiple TCP based transport implementation. These implementations can vary in how they respond internally to situations where we concurrently send, receive and handle disconnects and can have different exceptions. However, disconnects are important events for the rest of the code base and should be distinguished from other errors (for example, it signals TransportMasterAction that it needs to retry and wait for the a (new) master to come back).  Therefore, we should make sure that all the implementations do the proper translation from their internal exceptions into ConnectTransportException which is used externally. 

Similarly we should make sure that the transport implementation properly recognize errors that were caused by a disconnect as such and deal with them correctly. This was, for example, the source of a build failure at https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-intake/1080 , where a concurrency issue cause SocketException to bubble out of MockTcpTransport.

This PR adds a tests which concurrently simulates connects, disconnects, sending and receiving and makes sure the above holds. It also fixes anything (not much!) that was found it.
2016-07-22 14:35:47 +02:00
Tal Levy f7cd86ef6d rethrow script compilation exceptions into ingest configuration exceptions (#19318)
* rethrow script compilation exceptions into ingest configuration exceptions
* update readProcessor to rethrow any exception as an ElasticsearchException
2016-07-20 10:37:56 -07:00
javanna a9b5c5adbe restore throws IOException clause on all performRequest sync methods
We throw IOException, which is the exception that is going to be thrown in 99% of the cases. A more generic exception can happen, and if it is a runtime one we just let it bubble up as is, otherwise we wrap it into runtime one so that we don't require to catch Exception everywhere, which seems odd.

Also adjusted javadocs for all performRequest methods
2016-07-19 15:18:05 +02:00
javanna 1bb33cf572 Remove RestClient#JSON_CONTENT_TYPE constant, already available in ContentType class 2016-07-19 15:17:12 +02:00
javanna e742d65e02 [TEST] Make sure the last response body is always available in our REST tests
With the introduction of the async client, ResponseException doesn't eagerly read the response body anymore into a string. That is better, but raised a problem in our REST tests infra: we were reading the response body twice, while it can only be consumed once. Introduced a RestTestResponseException that wraps a ResponseException and exposes the body which now gets read only once.
2016-07-19 15:16:45 +02:00
javanna 41e97a7cb1 RestClient: take builder out to its own class
The RestClient class is getting bigger and bigger, its builder can definitely be taken out to its own top level class: RestClientBuilder
2016-07-19 15:16:45 +02:00
javanna 1fbec71243 Rest client: introduce async performRequest method and use async client under the hood for sync requests too
The new method accepts the usual parameters (method, endpoint, params, entity and headers) plus a response listener and an async response consumer. Shortcut methods are also added that don't require params, entity and the async response consumer optional.

There are a few relevant api changes as a consequence of the move to async client that affect sync methods:
- Response doesn't implement Closeable anymore, responses don't need to be closed
- performRequest throws Exception rather than just IOException, as that is the the exception that we get from the FutureCallback#failed method in the async http client
- ssl configuration is a bit simpler, one only needs to call setSSLStrategy from a custom HttpClientConfigCallback, that doesn't end up overridng any other default around connection pooling (it used to happen with the sync client and make ssl configuration more complex)

Relates to #19055
2016-07-19 15:15:58 +02:00
Nik Everett a2a7ea1f17 Make ExtendedBounds immutable
We used to mutate it as part of building the aggregation. That
caused assertVersionSerializable to fail because it assumes that
requests aren't mutated after they are sent.

Closes #19481
2016-07-19 08:48:14 -04:00
Simon Willnauer 8394544548 Add a dedicated client/transport project for transport-client (#19435)
The `client/transport` project adds a new jar build project that
pulls in all dependencies and configures all required modules.

Preinstalled modules are:
 * transport-netty
 * lang-mustache
 * reindex
 * percolator

The `TransportClient` classes are still in core
while `TransportClient.Builder` has only a protected construcutor
such that users are redirected to use the new `TransportClientBuilder`
from the new jar.

Closes #19412
2016-07-18 15:42:24 +02:00
Martijn van Groningen e0ebf5da1c Template cleanup:
* Removed `Template` class and unified script & template parsing logic. Templates are scripts, so they should be defined as a script. Unless there will be separate template infrastructure, templates should share as much code as possible with scripts.
* Removed ScriptParseException in favour for ElasticsearchParseException
* Moved TemplateQueryBuilder to lang-mustache module because this query is hard coded to work with mustache only
2016-07-18 10:16:01 +02:00
Ali Beyad 687e2e12b3 Merge pull request #19450 from elastic/feature/friendly-index-creation
Makes index creation more friendly
2016-07-15 11:48:21 -04:00
Ali Beyad d78f40fb1e Index creation waits for active shard copies before returning (#18985)
Before returning, index creation now waits for the configured number
of shard copies to be started. In the past, a client would create an
index and then potentially have to check the cluster health to wait
to execute write operations. With the cluster health semantics changing
so that index creation does not cause the cluster health to go RED,
this change enables waiting for the desired number of active shards
to be active before returning from index creation.

Relates #9126
2016-07-15 11:19:27 -04:00
Martijn van Groningen d0069f0fbb Provide access to ThreadContext in ingest plugins
Also introduced a `Processor.Parameters` class that is holder for several services processors rely on,
the  IngestPlugin#getProcessors(...) method has been changed to accept `Processor.Parameters` instead
of each service seperately.
2016-07-15 08:16:15 +02:00
Jason Tedor 31c648eee8 Rename transport-netty to transport-netty3
This commit renames the Netty 3 transport module from transport-netty to
transport-netty3. This is to make room for a Netty 4 transport module,
transport-netty4.

Relates #19439
2016-07-14 22:03:14 -04:00
Jason Tedor 575fa4e00a Fix line-length in o/e/t/r/s/Features.java
This commit fixes a line-length checkstyle violation in
o/e/t/r/s/Features.java.
2016-07-14 18:10:35 -04:00
Honza Král e21b1e8066 [TEST] add 'yaml' feature for the test runner (#19436)
Also renamed 30_yaml.yaml to 30_json.yaml since it tests json, not yaml
2016-07-14 17:30:32 +02:00
Simon Willnauer 5616251f22 Remove `node.mode` and `node.local` settings (#19428)
Today `node.mode` and `node.local` serve almost the same purpose, they
are a shortcut for `discovery.type` and `transport.type`. If `node.local: true`
or `node.mode: local` is set elasticsearch will start in _local_ mode which means
only nodes within the same JVM are discovered and a non-network based transport
is used. The _local_ mode it only really used in tests or if nodes are embedded.
For both, embedding and tests explicit configuration via `discovery.type` and `transport.type`
should be preferred.

This change removes all the usage of these settings and by-default doesn't
configure a default transport implemenation since netty is now a module. Yet, to make
the user expericence flawless, plugins or modules can set a `http.type.default` and
`transport.type.default`. Plugins set this via `PluginService#additionalSettings()`
which enforces _set-once_ which prevents node startup if set multiple times. This means
that our distributions will just startup with netty transport since it's packaged as a
module unless `transport.type` or `http.transport.type` is explicitly set.

This change also found a bunch of bugs since several NamedWriteables were not registered if a
transport client is used. Now that we don't rely on the `node.mode` leniency which is inherited
instead of using explicit settings, `TransportClient` uses `AssertingLocalTransport` which detects these problems since it serializes all messages.

Closes #16234
2016-07-14 13:21:10 +02:00
Simon Willnauer 29fd0f1bd8 [TEST] Remove wrong transportName from MockTcpTransport#ctor 2016-07-13 12:50:52 +02:00
Simon Willnauer 067ca1f996 [TEST] Use a semaphore to block unitl all in-flight requests are released 2016-07-13 10:31:05 +02:00
Simon Willnauer 814c7224f9 Merge pull request #19392 from elastic/modularize_netty
This moves all netty related code into modules/transport-netty the module is build as a zip file as well as a JAR to serve as a dependency for transport client. For the time being this is required otherwise we have no network based impl. for transport client users. This might be subject to change given that we move forward http client.
2016-07-13 09:52:03 +02:00
Simon Willnauer eba69ffade [TEST] First decrement in-flight requests before releasing the latch 2016-07-12 22:58:03 +02:00
Simon Willnauer ec55f9fff7 [TEST] Make AbstractSimpleTransportTestCase#testTimeoutSendExceptionWithDelayedResponse more robust and wait for in-flight request 2016-07-12 20:41:37 +02:00
Simon Willnauer 4fb79707bd Fix remaining tests that either need access to the netty module or require explict configuration
Some tests still start http implicitly or miss configuring the transport clients correctly.
This commit fixes all remaining tests and adds a depdenceny to `transport-netty` from
`qa/smoke-test-http` and `modules/reindex` since they need an http server running on the nodes.

This also moves all required permissions for netty into it's module and out of core.
2016-07-12 16:29:57 +02:00
Luca Cavanna f6aec3fdb5 Merge pull request #19373 from javanna/enhancement/rest_client_builder_callback
Rest Client: add callback to customize http client settings
2016-07-12 13:30:27 +02:00
javanna 512b8be791 RestClient: simplify ssl configuration and make http config callback functional friendly 2016-07-12 13:25:55 +02:00
Boaz Leskes 081d04afac Make NotMasterException a first class citizen (#19385)
That exception is currently serialized as its current base class IllegalStateException which confuses code supposed to deal with the stepping down of a master. This is an important exception and we should be able to serialize it correctly. This commit fixes it by moving the exception to inherit from ElasticsearchException and properly register it.

As a bonus I adapted CapturingTransport to properly simulate serialized exceptions.
2016-07-12 12:44:40 +02:00
javanna fa0b354e66 Rest Client: add callback to customize http client settings
The callback replaces the ability to fully replace the http client instance. By doing that, one used to lose any default that the RestClient had set for the underlying http client. Given that you'd usually override one or two things only, like a couple of timeout values, the ssl factory or the default credentials providers, it is not uder friendly if by doing that users end up replacing the whole http client instance and lose any default set by us.
2016-07-12 12:31:28 +02:00
Simon Willnauer 199a5a1f04 Fix TcpTransport#sendRequest to raise NotConnectedExcepiton if we get disconnected while sending
This also fixes a race in AbstractSimpleTransportTestCase where we never wait long enough
for all response to finish causing expected failures.
2016-07-12 10:56:20 +02:00
Ryan Ernst 93aebbef0f Merge branch 'master' into modularize_netty 2016-07-11 23:49:00 -07:00
Ryan Ernst 7195d1e0ff Fix plugins service to not double bind plugin components 2016-07-11 17:05:56 -07:00
Nik Everett 8263873783 Switch search extension from push to pull
Switches most search behavior extensions from push (`onModule(SearchModule)`)
to pull (`implements SearchPlugin`). This effort in general gives plugin
authors a much cleaner view of how to extend Elasticsearch and starts to
set up portions of Elasticsearch as "the plugin API". This commit in
particular does that for search-time behavior like customized suggesters,
highlighters, score functions, and significance heuristics.

It also switches most such customization to being done at search module
construction time which is much, much easier to reason about from a testing
perspective. It also helps significantly in the process of de-guice-ing
Elasticsearch's startup.

There are at least two major search time extensions that aren't covered in
this commit that will simply have to wait for the next commit on the topic
because this one has already grown large: custom aggregations and custom
queries. These will likely live in the same SearchPlugin interface as well.
2016-07-11 18:49:05 -04:00
Ryan Ernst 99ac65931a Plugins: Add components creator as bridge between guice and new plugin init world
This change adds a createComponents() method to Plugin implementations
which they can use to return already constructed componenents/services.
Eventually this should be just services ("components" don't really do
anything), but for now it allows any object so that preconstructed
instances by plugins can still be bound to guice. Over time we should
add basic services as arguments to this method, but for now I have left
it empty so as to not presume what is a necessary service.
2016-07-11 14:14:06 -07:00
Simon Willnauer 048e4416e7 Move netty transport and http into a module
This moves all netty code and it's dependency into a module.
2016-07-11 22:21:29 +02:00
Ali Beyad 0faf638710 Blocked allocations on primary causes RED health
If the allocation decision for a primary shard was NO, this should
cause the cluster health for the shard to go RED, even if the shard
belongs to a newly created index or is part of cluster recovery.

Relates #9126
2016-07-11 15:32:13 -04:00
Ali Beyad 417bd0cd63 Index creation does not cause the cluster health to go RED
Previously, index creation would momentarily cause the cluster health to
go RED, because the primaries were still being assigned and activated.
This commit ensures that when an index is created or an index is being
recovered during cluster recovery and it does not have any active
allocation ids, then the cluster health status will not go RED, but
instead be YELLOW.

Relates #9126
2016-07-11 15:30:47 -04:00
Simon Willnauer 47bd2f9ca5 More cleanups aroung tests that require HTTP to be enalbed. (#19363)
this commit moves the most of the http related integ tests out into it's own 
`qa/smoke-test-http` project where most of the test can run against the external cluster.
2016-07-11 20:44:57 +02:00
Nik Everett 4b171b84cb Fix modifier order
checkstyle
2016-07-11 12:59:45 -04:00
Christoph Büscher 0d428b6ba8 Add test for GeoHashUtils#bbox() 2016-07-11 10:46:31 -05:00
Simon Willnauer ee193f7697 [TEST] Catch RejectedOperationException when disconnecting from node in MockTcpTransport 2016-07-11 16:36:26 +02:00
Simon Willnauer 07260d4351 [TEST] Use AbstractRunnable when forking off threads on an executor 2016-07-11 16:27:07 +02:00
Simon Willnauer 3f3c93ec65 Add blocking socket based MockTcpTransport (#19332)
Today we have a bunch of tests that use netty transport for several reasons
these tests use it because they need to run some tcp based transport. Yet, this
couples our tests tightly to the netty implementation which should be tested on it's own.
This change adds a plain socket based blocking TcpTransport implementation that is used by
default in tests if local transport is suppressed or if network is selected.
It also adds another tcp network implementation as a showcase how the interface works.
2016-07-11 12:17:52 +02:00
javanna 942e342662 Rest Client: use short performRequest methods when possible 2016-07-11 10:36:26 +02:00
Jason Tedor e86aa29f67 Die with dignity
Today when a thread encounters a fatal unrecoverable error that
threatens the stability of the JVM, Elasticsearch marches on. This
includes out of memory errors, stack overflow errors and other errors
that leave the JVM in a questionable state. Instead, the Elasticsearch
JVM should die when these errors are encountered. This commit causes
this to be the case.

Relates #19272
2016-07-07 14:44:03 -04:00
Tanguy Leroux b58f2eb5c2 Move back some messy tests from Groovy plugin to core
This commit moves back some messy tests that have been placed in lang-groovy module in https://github.com/elastic/elasticsearch/pull/13834. It removes the dependency on Groovy plugin as well as change back the tests to integration tests (IT suffix).

It also changes the current MockScriptEngine and MockScriptPlugin to make it easier to use.
2016-07-07 15:26:36 +02:00
Alexander Reelsen 71b48fb16c Dependencies: Update to jopt-5.0 (#19278)
The new version of jopt allows us to remove a couple of TODOs in the code.

Closes #12368
2016-07-07 08:50:10 +02:00
Ryan Ernst e7818f75e1 Fix checkstyle for TestProcessor 2016-07-05 22:33:08 -07:00
Ryan Ernst 2fc41adeb5 Merge branch 'master' into ingest_plugin_api 2016-07-05 20:53:03 -07:00
Jason Tedor d0765d0761 Merge branch 'master' into feature/seq_no
* master: (192 commits)
  [TEST] Fix rare OBOE in AbstractBytesReferenceTestCase
  Reindex from remote
  Rename writeThrowable to writeException
  Start transport client round-robin randomly
  Reword Refresh API reference (#19270)
  Update fielddata.asciidoc
  Fix stored_fields message
  Add missing footer notes in mapper size docs
  Remote BucketStreams
  Add doc values support to the _size field in the mapper-size plugin
  Bump version to 5.0.0-alpha5.
  Update refresh.asciidoc
  Update shrink-index.asciidoc
  Change Debian repository for Vagrant debian-8 box
  [TEST] fix test to account for internal empyt reference optimization
  Upgrade to netty 3.10.6.Final (#19235)
  [TEST] fix histogram test when extended bounds overlaps data
  Remove redundant modifier
  Simplify TcpTransport interface by reducing send code to a single send method (#19223)
  Fix style violation in InstallPluginCommand.java
  ...
2016-07-05 22:01:07 -04:00
Nik Everett b3c015e2bb Reindex from remote
This adds a remote option to reindex that looks like

```
curl -POST 'localhost:9200/_reindex?pretty' -d'{
  "source": {
    "remote": {
      "host": "http://otherhost:9200"
    },
    "index": "target",
    "query": {
      "match": {
        "foo": "bar"
      }
    }
  },
  "dest": {
    "index": "target"
  }
}'
```

This reindex has all of the features of local reindex:
* Using queries to filter what is copied
* Retry on rejection
* Throttle/rethottle
The big advantage of this version is that it goes over the HTTP API
which can be made backwards compatible.

Some things are different:

The query field is sent directly to the other node rather than parsed
on the coordinating node. This should allow it to support constructs
that are invalid on the coordinating node but are valid on the target
node. Mostly, that means old syntax.
2016-07-05 16:13:17 -04:00
Jason Tedor 96f283c195 Rename writeThrowable to writeException
This commit renames writeThrowable to writeException. The situation here
stems from the fact that the StreamOutput method for serializing
Exceptions needs to accept Throwables too as Throwables can be the cause
of serialized Exceptions. Yet, we do not serialize Throwables in the
Error sub-hierarchy in a way that they can be deserialized into their
initial type. This leads to an asymmetry in the StreamOutput method for
serializing Exceptions and the StreamInput method for writing
Excpetions. Namely, the former will accept Throwables but the latter
will only return Exceptions. A goal with the stream methods has always
been symmetry in the method names so that serialization/deserialization
routines appear symmetrical in code. It is this asymmetry on the
input/output types for Exceptions on StreamOutput/StreamInput that
clashes with the desired symmetry of naming. Despite this, we should
favor symmetry in the naming of the methods. This commit renames
StreamOutput#writeThrowable to StreamOutput#writeException which leaves
us with Exception StreamInput#readException and void
StreamOutput#writeException(Throwable).
2016-07-05 14:37:01 -04:00
Boaz Leskes 6861d3571e Persistent Node Ids (#19140)
Node IDs are currently randomly generated during node startup. That means they change every time the node is restarted. While this doesn't matter for ES proper, it makes it hard for external services to track nodes. Another, more minor, side effect is that indexing the output of, say, the node stats API results in creating new fields due to node ID being used as keys.

The first approach I considered was to use the node's published address as the base for the id. We already [treat nodes with the same address as the same](https://github.com/elastic/elasticsearch/blob/master/core/src/main/java/org/elasticsearch/discovery/zen/NodeJoinController.java#L387) so this is a simple change (see [here](https://github.com/elastic/elasticsearch/compare/master...bleskes:node_persistent_id_based_on_address)). While this is simple and it works for probably most cases, it is not perfect. For example, if after a node restart, the node is not able to bind to the same port (because it's not yet freed by the OS), it will cause the node to still change identity. Also in environments where the host IP can change due to a host restart, identity will not be the same. 

Due to those limitation, I opted to go with a different approach where the node id will be persisted in the node's data folder. This has the upside of connecting the id to the nodes data. It also means that the host can be adapted in any way (replace network cards, attach storage to a new VM). I

It does however also have downsides - we now run the risk of two nodes having the same id, if someone copies clones a data folder from one node to another. To mitigate this I changed the semantics of the protection against multiple nodes with the same address to be stricter - it will now reject the incoming join if a node exists with the same id but a different address. Note that if the existing node doesn't respond to pings (i.e., it's not alive) it will be removed and the new node will be accepted when it tries another join.

Last, and most importantly, this change requires that *all* nodes persist data to disk. This is a change from current behavior where only data & master nodes store local files. This is the main reason for marking this PR as breaking.

Other less important notes:
- DummyTransportAddress is removed as we need a unique network address per node. Use `LocalTransportAddress.buildUnique()` instead.
- I renamed `node.add_lid_to_custom_path` to `node.add_lock_id_to_custom_path` to avoid confusion with the node ID which is now part of the `NodeEnvironment` logic.
- I removed the `version` paramater from `MetaDataStateFormat#write` , it wasn't really used and was just in the way :)
- TribeNodes are special in the sense that they do start multiple sub-nodes (previously known as client nodes). Those sub-nodes do not store local files but derive their ID from the parent node id, so they are generated consistently.
2016-07-04 21:09:25 +02:00
Tanguy Leroux 0e7faf1005 Enable Checkstyle RedundantModifier 2016-07-04 15:22:12 +02:00
Jason Tedor 3343ceeae4 Do not catch throwable
Today throughout the codebase, catch throwable is used with reckless
abandon. This is dangerous because the throwable could be a fatal
virtual machine error resulting from an internal error in the JVM, or an
out of memory error or a stack overflow error that leaves the virtual
machine in an unstable and unpredictable state. This commit removes
catch throwable from the codebase and removes the temptation to use it
by modifying listener APIs to receive instances of Exception instead of
the top-level Throwable.

Relates #19231
2016-07-04 08:41:06 -04:00
Ryan Ernst 5a66c08ae9 Merge branch 'master' into ingest_plugin_api 2016-07-01 16:27:52 -07:00
Ryan Ernst 822c995367 Internal: Remove generics from LifecycleComponent
The only reason for LifecycleComponent taking a generic type was so that
it could return that type on its start and stop methods. However, this
chaining has no practical necessity. Instead, start and stop can be
void, and a whole bunch of confusing generics disappear.
2016-07-01 16:17:42 -07:00
Ryan Ernst e5caadc4f3 Merge branch 'master' into ingest_plugin_api 2016-07-01 12:35:26 -07:00
Nik Everett f30a70c51f Fix comment
I forgot a word....
2016-07-01 14:48:08 -04:00
Nik Everett ff42d7cfc6 Add embedded stash key support to rest tests
This allowes embedding stash keys in string like `t${key}est`. This
allows simple string concatenation like acitons.

The test for this is in `ObjectPathTests` because `Stash` doesn't seem
to have a test on its own and it is simple enough to test embedded
stashes this way. And this is a way I expect them to be used eventually.
2016-07-01 14:11:11 -04:00
Ryan Ernst 65c9b0b588 Merge branch 'master' into ingest_plugin_api 2016-07-01 09:26:17 -07:00
Tanguy Leroux 8c40b2b54e Fix order of modifiers 2016-07-01 16:57:14 +02:00
Simon Willnauer 5c8164a561 Clean up BytesReference (#19196)
BytesReference should be a really simple interface, yet it has a gazillion
ways to achieve the same this. Methods like `#hasArray`, `#toBytesArray`, `#copyBytesArray`
`#toBytesRef` `#bytes` are all really duplicates. This change simplifies the interface
dramatically and makes implementations of it much simpler. All array access has been removed
and is streamlined through a single `#toBytesRef` method. Utility methods to materialize a
compact byte array has been added too for convenience.
2016-07-01 16:09:31 +02:00
javanna dd781d410a fix line length problems in all classes under o.e.test.rest package 2016-07-01 11:13:10 +02:00
javanna 0b5a549305 [TEST] remove special treatment for stashed $body in REST tests, instead always evaluate the stash through ObjectPath
When we introduced docs testing we added a special case for $body in Stash, so that the last stashed body could be evaluated, and expressions like "$body.took" could be extracted out of it. We can instead do that for any object in the stash, by simply wrapping the internal map in an ObjectPath instance. We can then drop the special stashResponse method and go back to using the ordinary stashValue too.

The downside of this change is that it adds a feature that may not be supported by other REST test runners, namely the evaluation of compouned paths from the stash. If we have "object" stashed as an object, it is now possible to extract directly each subobject of it as well e.g. "object.subobject.field1". None of the current REST tests rely on this, but our docs snippets tests do.
2016-07-01 11:13:10 +02:00
javanna 43b82ce244 [TEST] remove feature yaml from REST tests
The only runner that supported it was the java runner, we can use json format instead given that the default one with cat apis is text
2016-07-01 11:13:10 +02:00
javanna 60bafa5d78 [TEST] parse yaml responses too through ObjectPath rather than only json responses
No need to match against yaml responses via regexes in REST tests, yaml responses can be properly parsed via ObjectPath instead. Few REST tests need to be updated accordingly.
2016-07-01 11:13:10 +02:00
javanna 34f5c50a7f [TEST] eagerly parse response body at ObjectPath initialization and read content type from response headers
We are going to parse the body anyways whenever it's in json format as it is going to be stashed. It is not useful to lazily parse it anymore. Also this allows us to not rely on automatic detection of the xcontent type based on the content of the response, but rather read the content type from the response headers.
2016-07-01 11:13:10 +02:00
javanna d5df738538 [TEST] ObjectPath to support parsing yaml or json that have an array as root object
ObjectPath used a Map up until now for the internal representation of its navigable object. That works in most of the cases, but there could also be an array as root object, in which case a List needs to be used instead of a Map. This commit changes the internal representation of the object to Object which can either be a List or a Map. The change is minimal as ObjectPath already had the checks in place to verify the type of the object in the current position and navigate through it.

  Note: The new test added to ObjectPathTest uses yaml format explicitly as auto-detection of json format works only for a json object that starts with '{', not if the root object is actually an array and starts with '['.
2016-07-01 11:13:10 +02:00
javanna bbaa23bdfd [TEST] extend ObjectPathTests to support also yaml format 2016-07-01 11:13:10 +02:00
javanna 44dc801e90 [TEST] make JsonPath independent of data format, rename to ObjectPath
The internal representation of the object that JsonPath gives access to is a map. That is independent of the initial input format, which is json but could also be yaml etc.
This commit renames JsonPath to ObjectPath and adds a static method to create an ObjectPath from an XContent
2016-07-01 11:13:10 +02:00
javanna 76199ce497 [TEST] rename REST tests Stash methods to distinguish between retrieving a value and replacing values within a map
Stash#unstashMap -> replaceStashedValues
Stash#unstashValue -> getValue
2016-07-01 11:13:10 +02:00
javanna 62462f5d9b [TEST] replace ResponseBodyAssertion with existing MatchAssertion
We introduced a special response_body assertion to test our docs snippets. The match assertion does the same job though and can be reused and adapted where needed. ResponseBodyAssertion contains provides much better and accurate errors though, which can be now utilized in MatchAssertion so that many more REST tests can benefit from readable error messages.

 Each response body gets always stashed and can be retrieved for later evaluations already. Instead of providing the response body as strings that get parsed to json objects separately, then converted to maps as ResponseBodyAssertion did, we parse everything once, the json is part of the yaml test, which is supported. The only downside is that json comments cannot be used, rather yaml comments should be used (// C style vs # ). There were only two docs tests that were using comments in ingest-node.asciidoc where I went ahead and remove the comments which didn't seem that useful anyways.
2016-07-01 11:13:10 +02:00
javanna 598c36128e Revert "Raised IOException on deleteBlob (#18815)"
This reverts commit d24cc65cad as it seems to be causing test failures.
2016-07-01 11:00:32 +02:00
gfyoung d24cc65cad Raised IOException on deleteBlob (#18815)
Raise IOException on deleteBlob if the blob doesn't exist

This commit raises an IOException on BlobContainer#deleteBlob
if the blob does not exist, in conformance with the BlobContainer
interface contract.  Each implementation of BlobContainer now
conforms to this contract (file system, S3, Azure, HDFS).  This 
commit also contains blob container tests for each of the 
repository implementations.

Closes #18530
2016-06-30 23:00:10 -04:00
Nik Everett f5a269b029 Start migration away from aggregation streams
We'll migrate to NamedWriteable so we can share code with the rest
of the system. So we can work on this in multiple pull requests without
breaking Elasticsearch in between the commits this change supports
*both* old style `InternalAggregations.stream` serialization and
`NamedWriteable` style serialization. As such it creates about a
half dozen `// NORELEASE` comments that will have to be removed
once the migration is complete.

This also introduces a boolean `transportClient` flag to `SearchModule`
which is used to skip inappropriate registrations for for the
transport client while still registering the things it needs. In
this case that means that the `InternalAggregation` subclasses are
registered with the `NamedWriteableRegistry` but the `AggregationBuilder`
subclasses are not.

Finally, this moves aggregation registration from guice configuration
time to `SearchModule` construction time. This will make it simpler to
work with in the future as we further clean up Elasticsearch's
extension points.
2016-06-30 12:57:34 -04:00
Boaz Leskes 09ca6d6ed2 Add a BridgePartition to be used by testAckedIndexing (#19172)
We have long worked to capture different partitioning scenarios in our testing infra. This PR adds a new variant, inspired by the Jepsen blogs, which was forgotten far - namely a partition where one node can still see and be seen by all other nodes. It also updates the resiliency page to better reflect all the work that was done in this area.
2016-06-30 17:58:12 +02:00
jaymode 983a64c833 Add support for `teardown` section in REST tests
This commits adds support for a `teardown` section that can be defined in REST tests to
clean up any items that may have been created by the test and are not cleaned up by
deletion of indices and templates.
2016-06-30 11:33:29 -04:00
Ryan Ernst 0732004ae8 Merge pull request #19177 from rjernst/ingest_factory_generic
Remove generics from ingest Processor.Factory
2016-06-30 08:08:26 -07:00
Simon Willnauer 40ec639c89 Factor out abstract TCPTransport* classes to reduce the netty footprint (#19096)
Today we have a ton of logic inside the NettyTransport* codebase. The footprint
of the code that has a direct netty dependency is large and alternative implementations
are pretty hard today since they need to know all about our proticol etc.
This change moves most of the code into TCPTransport* baseclasses and moves all
the protocol send code together. The base classes now contain the majority of the logic
while NettyTransport* classes remain to implement the glue code, configuration and optimization.
2016-06-30 13:41:53 +02:00
Ryan Ernst e4f265eb3a Ingest: Remove generics from Processor.Factory
The factory for ingest processor is generic, but that is only for the
return type of the create mehtod. However, the actual consumer of the
factories only cares about Processor, so generics are not needed.

This change removes the generic type from the factory. It also removes
AbstractProcessorFactory which only existed in order pull the optional
tag from config. This functionality is moved to the caller of the
factories in ConfigurationUtil, and the create method now takes the tag.
This allows the covariant return of the implementation to work with
tests not needing casts.
2016-06-30 02:33:54 -07:00
Ryan Ernst 08b3b6264e Tests pass, started removing generics from processor factory 2016-06-30 01:49:22 -07:00
Ryan Ernst f1376262fe Merge branch 'master' into ingest_plugin_api 2016-06-29 14:16:16 -07:00
Simon Willnauer 872cdffc27 Factor out ChannelBuffer from BytesReference (#19129)
The ChannelBuffer interface today leaks into the BytesReference abstraction
which causes a hard dependency on Netty across the board. This chance moves
this dependency and all BytesReference -> ChannelBuffer conversion into
NettyUtlis and removes the abstraction leak on BytesReference.
This change also removes unused methods on the BytesReference interface
and simplifies access to internal pages.
2016-06-29 10:45:05 +02:00
Ryan Ernst 258c3e86ab Added IngestPlugin api, cutover common and geoip, changed ingest factory
api to take ProcessorsRegistry
2016-06-28 10:52:07 -07:00
Yannick Welsch 3cc2251e33 Fix number of arguments provided to logger calls 2016-06-28 17:38:56 +02:00
Yannick Welsch 98276111e1 Re-enable logger usage checks
It was inadvertently disabled after applying code review comments. This commit reenables the logger usage checker and makes it less naggy when encountering logging usages of the form  logger.info(someStringBuilder). Previously it would fail with the error message "First argument must be a string constant so that we can statically ensure proper place holder usage". Now it will only fail in case any arguments are provided as well, for example logger.info(someStringBuilder, 42).
2016-06-28 16:48:05 +02:00
Boaz Leskes 2512594d9e Testing infra - stablize data folder usage and clean up (#19111)
The plan for persistent node ids ( #17811 ) is to tie the node identity to a file stored in it's data folders. As such it becomes important that nodes in our testing infra have better affinity with their data folders and that their data folders are not cleaned underneath them. The first is important because we fix the random seed used for node id generation (for reproducibility) and allowing the same node to use two different data folders causes two separate nodes to have the same id, which prevents the cluster from forming. The second is important, for example, where a full cluster restart / single node restart need to maintain node identity and wiping the data folders at the wrong moment prevents this.

Concretely this commit does the following:
1) Remove previous attempts to have data folder per role using a prefix. This wasn't effective as it was using the data paths settings which are only used for part of the runs. An attempt to completely separate the paths via the home dir failed due to assumptions made by index custom path about node data folder ordinal uniqueness (see #19076)
2) Change full cluster restarts to start up nodes in the same order their were first created in, only randomly swapping nodes with the same roles.
3) Change test cluster reset methods to first shutdown the unneeded nodes and then re-start the shared nodes that were shut down, so they'll reclaim their data folders.
4) Improve data folder wiping logic and make sure it wipes only folders of "offline" nodes.
5) Add some very basic tests
2016-06-28 16:38:56 +02:00
Jason Tedor 2f638b5a23 Keep input time unit when parsing TimeValues
This commit modifies TimeValue parsing to keep the input time unit. This
enables round-trip parsing from instances of String to instances of
TimeValue and vice-versa. With this, this commit removes support for the
unit "w" representing weeks, and also removes support for fractional
values of units (e.g., 0.5s).

Relates #19102
2016-06-27 18:41:18 -04:00
Nik Everett 79fa778e33 Fix percolator tests
They need their plugin or they'll break!
2016-06-27 15:34:36 -04:00
Ryan Ernst 33ccc5aead Merge branch 'master' into mapper_plugin_api 2016-06-27 11:19:59 -07:00
Boaz Leskes cb0824e957 Make shard store fetch less dependent on the current cluster state, both on master and non data nodes (#19044)
#18938 has changed the timing in which we send out to nodes to fetch their shard stores. Instead of doing this after the cluster state resulting of the node's join was published, #18938 made it be sent concurrently to the publishing processes. This revealed a couple of points where the shard store fetching is dependent of the current state of affairs of the cluster state, both on the master and the data nodes. The problem discovered were already present without #18938 but required a failure/extreme situations to make them happen.This PR tries to remove as much as possible of these dependencies making shard store fetching simpler and make the way to re-introduce #18938 which was reverted.

These are the notable changes:
1) Allow TransportNodesAction (of which shard store fetching is derived) callers to supply concrete disco nodes, so it won't need the cluster state to resolve them. This was a problem because the cluster state containing the needed nodes was not yet made available through ClusterService. Note that long term we can expect the rest layer to resolve node ids to concrete nodes, making this mode the only one needed.
2) The data node relied on the cluster state to have the relevant index meta data so it can find data when custom paths are used. We now fall back to read the meta data from disk if needed.
3) The data node was relying on it's own IndexService state to indicate whether the data it has corresponds to an existing allocation. This is of course something it can not know until it got (and processed) the new cluster state from the master. This flag in the response is now removed. This is not a problem because we used that flag to protect against double assigning of a shard to the same node, but we are already protected from it by the allocation deciders.
4) I removed the redundant filterNodeIds method in TransportNodesAction - if people want to filter they can override resolveRequest.
2016-06-27 15:05:06 +02:00
Nik Everett 71b95fb63c Switch analysis from push to pull
Instead of plugins calling `registerTokenizer` to extend the analyzer
they now instead have to implement `AnalysisPlugin` and override
`getTokenizer`. This lines up extending plugins in with extending
scripts. This allows `AnalysisModule` to construct the `AnalysisRegistry`
immediately as part of its constructor which makes testing anslysis
much simpler.

This also moves the default analysis configuration into `AnalysisModule`
which is how search is setup.

Like `ScriptModule`, `AnalysisModule` no longer extends `AbstractModule`.
Instead it is only responsible for building `AnslysisRegistry`. We still
bind `AnalysisRegistry` but we only do so in `Node`. This is means it
is available at module construction time so we slowly remove the need to
bind it in guice.
2016-06-26 07:15:42 -04:00
Ryan Ernst 6995bde710 Merge branch 'master' into mapper_plugin_api 2016-06-24 11:15:06 -07:00
Jason Tedor 112669daed Merge branch 'master' into feature/seq_no
* master: (416 commits)
  docs: removed obsolete information, percolator queries are not longer loaded into jvm heap memory.
  Upgrade JNA to 4.2.2 and remove optionality
  [TEST] Increase timeouts for Rest test client (#19042)
  Update migrate_5_0.asciidoc
  Add ThreadLeakLingering option to Rest client tests
  Add a MultiTermAwareComponent marker interface to analysis factories. #19028
  Attempt at fixing IndexStatsIT.testFilterCacheStats.
  Fix docs build.
  Move templates out of the Search API, into lang-mustache module
  revert - Inline reroute with process of node join/master election (#18938)
  Build valid slices in SearchSourceBuilderTests
  Docs: Convert aggs/misc to CONSOLE
  Docs: migration notes for _timestamp and _ttl
  Group client projects under :client
  [TEST] Add client-test module and make client tests use randomized runner directly
  Move upgrade test to upgrade from version 2.3.3
  Tasks: Add completed to the mapping
  Fail to start if plugin tries broken onModule
  Remove duplicated read byte array methods
  Rename `fields` to `stored_fields` and add `docvalue_fields`
  ...
2016-06-23 11:52:11 -04:00
Yannick Welsch a5908a5da5 [TEST] Increase timeouts for Rest test client (#19042)
Some Rest / Doc tests were running into the default socket timeout of 10 seconds.
2016-06-23 14:05:56 +02:00
Adrien Grand 7ba5bceebe Add a MultiTermAwareComponent marker interface to analysis factories. #19028
This is the same as what Lucene does for its analysis factories, and we hawe
tests that make sure that the elasticsearch factories are in sync with
Lucene's. This is a first step to move forward on #9978 and #18064.
2016-06-23 10:19:24 +02:00
Tanguy Leroux 04da1bda0d Move templates out of the Search API, into lang-mustache module
This commit moves template support out of the Search API to its own dedicated Search Template API in the lang-mustache module. It provides a new SearchTemplateAction that can be used to render templates before it gets delegated to the usual Search API. The current REST endpoint are identical, but the Render Search Template endpoint now uses the same Search Template API with a new "simulate" option. When this option is enabled, the Search Template API only renders template and returns immediatly, without executing the search.

Closes #17906
2016-06-23 09:30:53 +02:00
Nik Everett 0bf447c697 Group client projects under :client
:client ---------> :client:rest
:client-sniffer -> :client:sniffer
:client-test ----> :client:test

This lines the client up with how we do things like modules and
plugins.
2016-06-22 14:26:41 -04:00
javanna 490d9c8cf7 Merge branch 'master' into feature/http_client 2016-06-22 09:50:07 +02:00
Adrien Grand db9af54ec0 Remove `_timestamp` and `_ttl` on 5.x indices. #18980
This removes the ability to use `_timestamp` and `_ttl` on indices created on
or after 5.0.

Closes #18280
2016-06-22 08:35:54 +02:00
Ryan Ernst e817b5daa3 Plugins: Remove guice from Mapper plugins
This changes adds a MapperPlugin interface which allows pull style
retrieval of mappers and metadata mappers added by plugins. For now, I
have kept the MapperRegistry, but this should be removed in the future
as it is just a silly container for 2 maps which could themselves be
passed around.
2016-06-21 22:50:39 -07:00
Nik Everett 8925400f67 Remove guice from ScriptService
Makes ScriptModule just a plain class that manages building the
ScriptSettings and ScriptService from plugins. When we *need*
to bind ScriptService with guice we bind it in a lambda.
2016-06-21 16:45:45 -04:00
Adrien Grand 8078c205f9 Revert "Remove `_timestamp` and `_ttl` on 5.x indices. #18980"
This reverts commit 969e953645.
Docs are failing because of the removed functionality. I will
fix the docs before pushing it again.
2016-06-21 19:19:49 +02:00
Adrien Grand 969e953645 Remove `_timestamp` and `_ttl` on 5.x indices. #18980
This removes the ability to use `_timestamp` and `_ttl` on indices created on
or after 5.0.

Closes #18280
2016-06-21 18:04:58 +02:00
javanna 886cb37efb Merge branch 'master' into feature/http_client 2016-06-21 15:53:37 +02:00
Nik Everett ba1d6907ab Quiet the logging of the docs tests
Significantly quiets the logging of the docs tests by:
1. Switching two log statements to debug level.
2. Only calling ESTestCase#afterIfFailed if the test failure wasn't
just assumptions being violated.
2016-06-21 08:31:09 -04:00
Martijn van Groningen 82f7bfad98 ingest: merged o.e.ingest.core with o.e.ingest and in ingest-common module added o.e.ingest.common package
and moved all code to that package.
2016-06-21 09:24:00 +02:00
Simon Willnauer 459665914b Detach BigArrays from Guice (#18973)
BigArrays can be fully constructed without Guice, this change cleans up
it's creation and the mocking in MockNode.
2016-06-20 13:18:19 +02:00
Simon Willnauer e50314bb6e Remove NodeClientModule and PluginsModule 2016-06-20 11:53:07 +02:00
Simon Willnauer 7fea5bd8e7 Remove obsolete Modules that can simply be inlined in node creation 2016-06-20 11:28:14 +02:00
Simon Willnauer 260f38fd76 Remove VersionModule and use Version#current consistently.
We pretended to be able to ackt like a different version node for so long it's
time to be honest and remove this ability. It's just confusing and where needed
and tested we should build dedicated extension points.
2016-06-20 10:55:52 +02:00
Tanguy Leroux 98951b1203 Compile each Groovy script in its own classloader
closes #18572
2016-06-20 08:17:09 +02:00
Boaz Leskes 14cd8a6794 Introduce Replication unit tests using real shards (#18930)
This commit introduce unit testing infrastructure to test replication operations using real index shards. This is infra is complementary to the full integration tests and unit testing of ReplicationOperation we already have. The new ESIndexLevelReplicationTestCase base makes it easier to test and simulate failure mode that require real shards and but do not need the full blow stack of a complete node.

The commit also add a simple "nothing is wrong" test plus a test that checks we don't drop docs during the various stages of recovery.

For now, only single doc indexing is supported but this can be easily extended in the future.
2016-06-18 18:53:47 +02:00
Areek Zillur 9356a6090f Merge branch 'master' into enhancement/rollover_api 2016-06-17 11:35:57 -04:00
Simon Willnauer bdb6dcea3a Cleanup ClusterService dependencies and detached from Guice (#18941)
This change removes some unnecessary dependencies from ClusterService
and cleans up ClusterName creation. ClusterService is now not created
by guice anymore.
2016-06-17 17:07:19 +02:00
Areek Zillur 545ffa7801 Merge branch 'master' into enhancement/rollover_api 2016-06-17 10:33:11 -04:00
javanna af93533a17 Merge branch 'master' into feature/http_client 2016-06-17 13:50:18 +02:00
Areek Zillur 6adffa6b7b Merge branch 'master' into enhancement/rollover_api 2016-06-16 17:27:32 -04:00
Ryan Ernst 8196cf01e3 Merge branch 'master' into plugin_name_api 2016-06-16 13:49:28 -07:00
Simon Willnauer b22c526b34 Cut over settings registration to a pull model (#18890)
Today we have a push model for registering basically anything. All our extension points
are defined on modules which we pass in to plugins. This is harder to maintain and adds
unnecessary dependencies on the modules itself. This change moves towards a pull model
where the plugin offers a getter kind of method to get the extensions. This will also
help in the future if we need to pass dependencies to the extension points which can
easily be defined on the method as arguments if a pull model is used.
2016-06-16 15:52:58 +02:00
Nik Everett 5aa4769b25 Move waitForTaskCompletion into TaskManager
This allows for listening for the waiting to start using
MockTaskManager. This allows us to work around a race condition
in the TasksIT.
2016-06-16 09:45:46 -04:00
Simon Willnauer 18ff051ad5 Simplify ScriptModule and script registration (#18903)
Registering a script engine or native scripts still uses Guice today
and is much more complicated than needed. This change moves to a pull
based model where script plugins have to implement a dedicated interface
`ScriptPlugin` and defines simple getter returning instances rather than
classes.
2016-06-16 09:35:13 +02:00
Ryan Ernst a4503c2aed Plugins: Remove name() and description() from api
In 2.0 we added plugin descriptors which require defining a name and
description for the plugin. However, we still have name() and
description() which must be overriden from the Plugin class. This still
exists for classpath plugins. But classpath plugins are mainly for
tests, and even then, referring to classpath plugins with their class is
a better idea. This change removes name() and description(), replacing
the name for classpath plugins with the full class name.
2016-06-15 17:12:22 -07:00
Tal Levy a26260fb72 new ScriptProcessor for Ingest (#18193)
add new ScriptProcessor for executing ES Scripts within pipelines
2016-06-15 14:57:18 -07:00
Daniel Mitterdorfer f32b700472 Exclude admin / diagnostic requests from HTTP request limiting
With this commit we exclude certain HTTP requests that are needed to inspect the cluster
from HTTP request limiting to ensure these commands are processed even in critical
memory conditions.

Relates #17951, relates #18145, closes #18833
2016-06-15 14:29:46 +02:00
javanna ace3a7b146 Merge branch 'master' into feature/http_client 2016-06-15 11:44:46 +02:00
Simon Willnauer 429dd3a876 Simplify FetchSubPhase registration and detach it from Guice (#18862)
this commit removes FetchSubPhrase registration by class to registration
by instance. No Guice binding needed anymore.
2016-06-15 09:13:02 +02:00
Nik Everett d0e4485d42 Move NamingConventionsCheck into buildSrc
This will let things that don't depend on :test:framework like the
client use it.

Also skip initializing the classes we check because we don't care
about their initialization behavior because we're not executing them.
This makes the naming conventions check pretty close to instant
from a "human eye" perspective.
2016-06-14 18:30:34 -04:00
Colin Goodheart-Smithe d7e3f9e4eb #18854 Remove size 0 options in aggregations
Remove size 0 options in aggregations
2016-06-14 15:32:42 +01:00
Simon Willnauer 4d78f280ed Remove dead code and dead parameters (#18855) 2016-06-14 15:25:44 +02:00
Colin Goodheart-Smithe cfd3356ee3 Remove size 0 options in aggregations
This removes the ability to set `size: 0` in the `terms`, `significant_terms` and `geohash_grid` aggregations for the reasons described in https://github.com/elastic/elasticsearch/issues/18838

Closes #18838
2016-06-14 13:07:02 +01:00
Adrien Grand 44c653f5a8 Upgrade to lucene-6.1.0-snapshot-3a57bea. 2016-06-10 16:18:12 +02:00
javanna cf6e713d77 Merge branch 'master' into feature/http_client 2016-06-09 17:43:45 +02:00
javanna 437c4f210b rename ElasticsearchResponse to Response and ElasticsearchResponseException to ResponseException 2016-06-09 14:38:32 +02:00
javanna 04d620da74 require hosts when creating RestClient.Builder
Also fix order of arguments when using assertEquals
2016-06-08 12:37:50 +02:00
Jason Tedor d896886973 Merge branch 'master' into feature/seq_no
* master: (51 commits)
  Switch QueryBuilders to new MatchPhraseQueryBuilder
  Added method to allow creation of new methods on-the-fly.
  more cleanups
  Remove cluster name from data path
  Remove explicit parallel new GC flag
  rehash the docvalues in DocValuesSliceQuery using BitMixer.mix instead of the naive Long.hashCode.
  switch FunctionRef over to methodhandles
  ingest: Move processors from core to ingest-common module.
  Fix some typos (#18746)
  Fix ut
  convert FunctionRef/Def usage to methodhandles.
  Add the ability to partition a scroll in multiple slices. API:
  use painless types in FunctionRef
  Update ingest-node.asciidoc
  compute functional interface stuff in Definition
  Use method name in bootstrap check might fork test
  Make checkstyle happy (add Lookup import, line length)
  Don't hide LambdaConversionException and behave like real javac compiled code when a conversion fails. This works anyways, because fallback is allowed to throw any Throwable
  Pass through the lookup given by invokedynamic to the LambdaMetaFactory. Without it real lambdas won't work, as their implementations are private to script class
  checkstyle have your upper L
  ...
2016-06-07 17:57:53 -04:00
Martijn van Groningen f611f1c99e ingest: Move processors from core to ingest-common module.
Folded grok processor into ingest-common module.

The rest tests have been moved to ingest-common module as well, because these tests don't run in the rest-api-spec module but in the distribution:integ-test-zip module
and adding a test plugin there felt just wrong to me. I think this is ok. I left a tiny ingest rest test behind in that tests with an empty pipeline.

Removed messy tests, these tests were already covered in the rest tests

Added ingest test plugin in test infra so that each module testing integration with ingest doesn't need write its own plugin

Moved reindex ingest tests to qa module

Closes #18490
2016-06-07 17:32:52 +02:00
Jason Tedor da74323141 Register thread pool settings
This commit refactors the handling of thread pool settings so that the
individual settings can be registered rather than registering the top
level group. With this refactoring, individual plugins must now register
their own settings for custom thread pools that they need, but a
dedicated API is provided for this in the thread pool module. This
commit also renames the prefix on the thread pool settings from
"threadpool" to "thread_pool". This enables a hard break on the settings
so that:
 - some of the settings can be given more sensible names (e.g., the max
   number of threads in a scaling thread pool is now named "max" instead
   of "size")
 - change the soft limit on the number of threads in the bulk and
   indexing thread pools to a hard limit
 - the settings names for custom plugins for thread pools can be
   prefixed (e.g., "xpack.watcher.thread_pool.size")
 - remove dynamic thread pool settings

Relates #18674
2016-06-06 22:09:12 -04:00
Areek Zillur d96fe20e3a add named writable registry glue 2016-06-06 16:11:46 -04:00
Jason Tedor a60b8948ba Merge branch 'master' into feature/seq_no
* master: (184 commits)
  Add back pending deletes (#18698)
  refactor matrix agg documentation from modules to main agg section
  Implement ctx.op = "delete" on _update_by_query and _reindex
  Close SearchContext if query rewrite failed
  Wrap lines at 140 characters (:qa projects)
  Remove log file
  painless: Add support for the new Java 9 MethodHandles#arrayLength() factory (see https://bugs.openjdk.java.net/browse/JDK-8156915)
  More complete exception message in settings tests
  Use java from path if JAVA_HOME is not set
  Fix uncaught checked exception in AzureTestUtils
  [TEST] wait for yellow after setup doc tests (#18726)
  Fix recovery throttling to properly handle relocating non-primary shards (#18701)
  Fix merge stats rendering in RestIndicesAction (#18720)
  [TEST] mute RandomAllocationDeciderTests.testRandomDecisions
  Reworked docs for index-shrink API (#18705)
  Improve painless compile-time exceptions
  Adds UUIDs to snapshots
  Add test rethrottle test case for delete-by-query
  Do not start scheduled pings until transport start
  Adressing review comments
  ...
2016-06-06 11:16:22 -04:00
Yannick Welsch 0a8afa2e72 Add back pending deletes (#18698)
Triggering the pending deletes logic was accidentally removed in the clean up PR #18602.
2016-06-06 15:14:09 +02:00
javanna a461dd84d2 Build: add hamcrest and securemock to version.properties 2016-06-06 15:02:52 +02:00
Boaz Leskes 4844325921 Introduced Global checkpoints for Sequence Numbers (#15485)
Global checkpoints are update by the primary and represent the common part of history across shard copies, as know at a given time. The primary is also in charge of periodically broadcast this information to the replicas. See #10708 for more details.
2016-06-06 12:53:04 +02:00
javanna 56e689e1b3 [TEST] remove unused method 2016-06-04 01:05:53 +02:00
javanna b15279b5ef Allow to pass socket facttry registry to createDefaultHttpClient method 2016-06-03 23:59:26 +02:00
javanna b891c46657 [TEST] remove status matcher and hasStatus assertion
All it does is checking the status code of a response, which can be done with a single line in each test
2016-06-03 23:25:17 +02:00
javanna f17f0f9247 rename ElasticsearchResponse#getFirstHeader to getHeader 2016-06-03 18:28:31 +02:00
javanna 23a94bb974 [TEST] create standard RestClient at first request and reuse it
A RestClient instance is now created whenever EsIntegTestCase#getRestClient is invoked for the first time. It is then kept until the cluster is cleared (depending on the cluster scope of the test).

Renamed other two restClient methods to createRestClient, as that instance needs to be closed and managed in the tests.
2016-06-03 18:00:54 +02:00
javanna e81aad972a remove usage of deprecated api 2016-06-03 16:01:07 +02:00
javanna eae914ae8e Replace rest test client with low level RestClient
We still have a wrapper called RestTestClient that is very specific to Rest tests, as well as RestTestResponse etc. but all the low level bits around http connections etc. are now handled by RestClient.
2016-06-03 16:01:07 +02:00
javanna 325b723930 [TEST] add rest client test dependency and replace usage of HttpRequestBuilder with RestClient in integration tests 2016-06-03 16:01:07 +02:00
Ali Beyad b720216395 Adds UUIDs to snapshots
This commit adds a UUID for each snapshot, in addition to the already
existing repository and snapshot name. The addition of UUIDs will enable
more robust handling of the deletion of previous snapshots and lingering
files from partially failed delete operations, on top of being able to
uniquely track each snapshot.

Closes #18228
Relates #18156
2016-06-02 17:01:48 -04:00
Christoph Büscher 9067407cdd Adressing review comments 2016-06-02 16:19:23 +02:00
Christoph Büscher e2b6dbc020 Add tests to check that toQuery() doesn't return null 2016-06-02 11:25:56 +02:00
Christoph Büscher 359f45988f Handle empty query bodies at parse time and remove EmptyQueryBuilder
Currently we support empty query clauses like the filter in

"constant_score" : {  "filter" : { } }

How these clauses are handled depends on the surrounding query.
They later are either ignored, converted to match all or no documents or
passed up further in the query hierarchy. During parsing these claues are
currently represented as EmptyQueryBuilders. When not handled anywhere else,
these special cases need to be checked for on the shard when building the
lucene query.

This is trappy, so this PR changes the parsing of compound queries. Instead
of returning QueryBuilder, the core query parsing method
QueryShardContext#parseInnerQueryBuilder() now return an Optional which can
be empty in the case of empty query clauses. This has the advantage of forcing
callers to deal with this sooner or later. When encountering empty Optionals,
compound query builders now have the choice to ignore them, pass them on or
rewrite to a different query, depending on context.
2016-06-02 11:25:56 +02:00
Yannick Welsch c20bf5d747 [TEST] Fix tests that rely on assumption that data dirs are removed after index deletion (#18681)
Relates to #18602
2016-06-01 17:02:09 +02:00
Simon Willnauer 88800e8e47 Move PageCacheRecycler into BigArrays (#18666)
PageCacheRecycler is really just an implementation detail of
BigArrays. There is no need to leak this class anywhere outside of it.
2016-06-01 09:43:11 +02:00
Ali Beyad 0efac76f01 Clarify the semantics of the BlobContainer interface
This commit clarifies the behavior that must be adhered to by any
implementors of the BlobContainer interface.  This is done through
expanded Javadocs.

Closes #18157
Closes #15580
2016-05-31 19:22:55 -04:00
Jason Tedor e21d8b31f1 Remove thread pool from page cache recycler
The page cache recycler has a dependency on thread pool that was there
for historical reasons but is no longer needed. This commit removes this
now unneeded dependency.

Relates #18664
2016-05-31 14:51:58 -04:00
Simon Willnauer 502a775a7c Add primitive to shrink an index into a single shard (#18270)
This adds a low level primitive operations to shrink an existing
index into a new index with a single shard. This primitive expects
all shards of the source index to allocated on a single node. Once the target index is initializing on the shrink node it takes a snapshot of the source index shards and copies all files into the target indices data folder. An [optimization](https://issues.apache.org/jira/browse/LUCENE-7300) coming in Lucene 6.1 will also allow for optional constant time copy if hard-links are supported by the filesystem. All mappings are merged into the new indexes metadata once the snapshots have been taken on the merge node.

To shrink an existing index all shards must be moved to a single node (one instance of each shard) and the index must be read-only:

```BASH
$ curl -XPUT 'http://localhost:9200/logs/_settings' -d '{
    "settings" : {
        "index.routing.allocation.require._name" : "shrink_node_name",
        "index.blocks.write" : true 
    }
}
```
once all shards are started on the shrink node. the new index can be created via:

```BASH
$ curl -XPUT 'http://localhost:9200/logs/_shrink/logs_single_shard' -d '{
    "settings" : {
        "index.codec" : "best_compression",
        "index.number_of_replicas" : 1
    }
}'
```

This API will perform all needed check before the new index is created and selects the shrink node based on the allocation of the source index. This call returns immediately, to monitor shrink progress the recovery API should be used since all copy operations are reflected in the recovery API with byte copy progress etc.

The shrink operation does not modify the source index, if a shrink operation should
be canceled or if the shrink failed, the target index can simply be deleted and
all resources are released.
2016-05-31 10:41:44 +02:00
Boaz Leskes 318a4e3ef6 Introduce dedicated master nodes in testing infrastructure (#18514)
This PR changes the InternalTestCluster to support dedicated master nodes. The creation of dedicated master nodes can be controlled using a new `supportsMasterNodes` parameter to the ClusterScope annotation. If set to true (the default), dedicated master nodes will randomly be used. If set to false,  no master nodes will be created and data nodes will also be allowed to become masters. If active, test runs will either have 1 or 3 masternodes
2016-05-27 08:44:20 +02:00
Yannick Welsch 31b0777c91 Simplify delayed shard allocation (#18351)
This commit simplifies the delayed shard allocation implementation by assigning clear responsibilities to the various components that are affected by delayed shard allocation:

- UnassignedInfo gets a boolean flag delayed which determines whether assignment of the shard should be delayed. The flag gets persisted in the cluster state and is thus available across nodes, i.e. each node knows whether a shard was delayed-unassigned in a specific cluster state. Before, nodes other than the current master were unaware of that information.
- This flag is initially set as true if the shard becomes unassigned due to a node leaving and the index setting index.unassigned.node_left.delayed_timeout being strictly positive. From then on, unassigned shards can only transition from delayed to non-delayed, never in the other direction.
- The reroute step is in charge of removing the delay marker (comparing timestamp when node left to current timestamp).
- A dedicated service DelayedAllocationService, reacting to cluster change events, has the responsibility to schedule reroutes to remove the delay marker.

Closes #18293
2016-05-26 13:39:55 +02:00
Adrien Grand cad959b980 Validate parameters of native sig score scripts so that we know which ones are not set. 2016-05-26 10:07:38 +02:00
Jason Tedor 9d39b05845 Remove deprecation suppression
Failing the build on deprecation warnings was removed in
19b3ec88af. This commit removes the
suppressed deprecation warnings so that their use is surfaced in the
build now.

Relates #18582
2016-05-25 17:15:36 -04:00
Nik Everett bef1c8511d s/tests.logger.level/tests.es.logger.level/
This is a leftover spot that wasn't changed. It was breaking
ClusterSettingsIT#ClusterSettingsIT because that test expected
the test's log level to default to the default logger level for
the nodes.
2016-05-24 13:25:16 -04:00
Martijn van Groningen 27cc2fe4dc Moved the percolator from core to its own module
Significant changes:
* AbstractQueryTestCase has moved to the test framework module, in order for query builder tests in modules and plugins
* Added support to AbstractQueryTestCase to register plugins
* Lift the restriction that only one percolator could be added per index. This validation existed in MapperService, but because the percolator moved to a module it could no longer exist there. Instead of bringing it back it was removed. This validation existed since the percolator cache only supported one percolator query per document, since the percolator cache has been removed this restriction could removed as well.
* While moving percolator tests to the new module, also removed a couple of tests for the deprecated percolate and mpercolate api. These APIs are now sugar  APIs for bwc and rediect to the searvh and msearvh APIs. Some tests were still testing as if percolate and mpercolate API did the percolation, but this no longer the case and these tests could be removed.
2016-05-24 11:01:57 +02:00
Ryan Ernst f6074d383b Merge pull request #18532 from rjernst/less_assert_busy
Tests: Remove unnecessary Callable variant of assertBusy
2016-05-23 17:11:54 -07:00
Chris Earle b49635539d Remove support for -Des.* system properties in integration tests
This now requires that system properties passed to Gradle must be in the form of "-Dtests.es.*" instead of
"-Des.*". It then chops off "tests.es." and passes that as a "-E" property to Elasticsearch.

Also changed system properties:

- `tests.logger.level` became `tests.es.logger.level`
- `node.mode` became `tests.es.node.mode`
- `node.local` became `tests.es.node.local`
2016-05-23 19:38:21 -04:00
Ryan Ernst c7b45b2cc7 Tests: Remove unnecessary Callable variant of assertBusy
The assertBusy method currently has both a Runnable and Callable
version. This has caused confusion with type inference and lambdas
sometimes, in particular with java 9. This change removes the callable
version as nothing was actually using it.
2016-05-23 16:17:43 -07:00
Jason Tedor f63d1255d1 Cleanup settings and system properties entanglement
This commit cleans up some additional places where system properties
were being used to pass settings to Elasticsearch.

Relates #18524
2016-05-23 14:47:22 -04:00
Luca Cavanna d2afe759a7 prevent registration of duplicated rest spec (#18504)
Rather than having one win against the other, reject duplicated apis. Also enforce the convention that see the api name have the same name as the name of the rest spec file that defines it.
2016-05-23 12:17:42 +02:00
Jason Tedor ad7229fe72 Merge branch 'master' into feature/seq_no
* master: (158 commits)
  Document the hack
  Refactor property placeholder use of env. vars
  Force java9 log4j hack in testing
  Fix log4j buggy java version detection
  Make java9 work again
  Don't mkdir directly in deb init script
  Fix env. var placeholder test so it's reproducible
  Remove ScriptMode class in favor of boolean true/false
  [rest api spec] fix doc urls
  Netty request/response tracer should wait for send
  Filter client/server VM options from jvm.options
  [rest api spec] fix url for reindex api docs
  Remove use of a Fields class in snapshot responses that contains x-content keys, in favor of declaring/using the keys directly.
  Limit retries of failed allocations per index (#18467)
  Proxy box method to use valueOf.
  Use the build-in valueOf method instead of the custom one.
  Fixed tests and added a comment to the box method.
  Fix boxing.
  Do not decode path when sending error
  Fix race condition in snapshot initialization
  ...
2016-05-21 21:04:43 -04:00
Ryan Ernst 37d36f2f4c Merge branch 'master' into java9 2016-05-21 14:19:58 -07:00
Ryan Ernst 41a5c0cfa1 Force java9 log4j hack in testing 2016-05-21 13:41:38 -07:00
Ryan Ernst 1d40c4bbc1 Make java9 work again
This change makes ES compile with java9 again, build 118.
* There are a handful of changes due to failure to determine types during compile.
* The attachment plugins which use tika needed to have tika upgraded in order to pickup fixes there for java 9.
* azure discovery and s3 repository indirectly depend on jaxb, which is no longer in the default modules. They now add a jaxb dependency externally, and make JarHell allow for this package.
2016-05-21 09:41:51 -07:00
Lee Hinman fdfd2a2f18 Remove ScriptMode class in favor of boolean true/false
This removes the ScriptMode class entirely, which was an enum with two
options (ON and OFF) which essentially boiled down to true and false.
Now the boolean values are used instead.
2016-05-20 15:01:30 -06:00
Martijn van Groningen 80fee8666f percolator: Removed percolator cache
Before 5.0 for it was required that the percolator queries were cached in jvm heap as Lucene queries for two reasons:
1) Performance. The percolator evaluated all percolator queries all the time. There was no pre-selecting queries that are likely to match like we have today.
2) Updates made to percolator queries were visible in realtime, Today these changes are visible in near realtime. So updating no longer requires the percolator to have the queries in jvm heap.

So having the percolator queries in jvm heap via the percolator cache is now less attractive. Especially when there are many percolator queries then these queries can consume many GBs of jvm heap.
Removing the percolator cache does make the percolate query slower compared to how the execution time in 5.0.0-alpha1 and alpha2, but it is still faster compared to 2.x and before.
2016-05-20 14:52:16 +02:00
Luca Cavanna fcee329332 update http client version to 4.5.2 and http-core 4.4.4 (#18399)
StrictHostnameVerifier can now be removed
2016-05-20 12:02:42 +02:00
Jason Tedor c257e2c51f Remove settings and system properties entanglement
Today when parsing settings during bootstrap, we add a system property
for every Elasticsearch setting. Additionally, settings can be set via
system properties. This commit simplifies this situation.
 - settings are no longer propogated to system properties
 - system properties can not be used to set settings
 - the "es." prefix on settings is no longer required (nor permitted)
 - test logging has a dedicated system property (tests.logger.level)

Relates #18198
2016-05-19 14:08:08 -04:00
Christoph Büscher d2515727d0 Improve random DateTimeZone creation in tests
We often require a random joda DateTimeZone in our tests. Currently
there are a few options for generating such a random DateTimeZone
from the set of available ids. Currently most random picks are not
really reproducable across different jvms because they rely on order
in the ids set implementation. The helper in DateProcessorFactoryTests
thus performs a sort on the set of ids before random picking from
the result, so I moved this to ESTestCase to make it publicly
available and changed all other tests to use that method.
2016-05-19 18:12:48 +02:00
Tanguy Leroux 35d3bdab84 Add Google Cloud Storage repository plugin
Closes #12880
2016-05-19 13:26:23 +02:00
Jason Tedor ecce53f0df Add I/O statistics on Linux
This commit adds a variety of real disk metrics for the block devices
that back Elasticsearch data paths. A collection of statistics are read
from /proc/diskstats and are used to report the raw metrics for
operations and read/write bytes.

Relates #15915
2016-05-17 16:16:39 -04:00
Adrien Grand 864ed04059 Lessen leniency of the query dsl. #18276
This change does the following:
 - Queries that are currently unsupported such as prefix queries on numeric
   fields or term queries on geo fields now throw an error rather than returning
   a query that does not match anything.
 - Fuzzy queries on numeric, date and ip fields are now unsupported: they used
   to create range queries, we now expect users to use range queries directly.
   Fuzzy, regexp and prefix queries are now only supported on text/keyword
   fields (including `_all`).
 - The `_uid` and `_id` fields do not support prefix or range queries anymore as
   it would prevent us to store them more efficiently in the future, eg. by
   using a binary encoding.

Note that it is still possible to ignore these errors by using the `lenient`
option of the `match` or `query_string` queries.
2016-05-16 17:37:00 +02:00
Jason Tedor 15d3d74444 Merge branch 'master' into feature/seq_no
* master: (904 commits)
  Removes unused methods in the o/e/common/Strings class.
  Add note regarding thread stack size on Windows
  painless: restore accidentally removed test
  Documented fuzzy_transpositions in match query
  Add not-null precondition check in BulkRequest
  Build: Make run task you full zip distribution
  Build: More pom generation improvements
  Add test for wrong array index
  Take return type from "after" field.
  painless: build descriptor of array and field load/store in code; fix array index to adapt type not DEF
  Build: Add developer info to generated pom files
  painless: improve exception stacktraces
  painless: Rename the dynamic call site factory to DefBootstrap and make the inner class very short (PIC = Polymorphic Inline Cache)
  Remove dead code.
  Avoid race while retiring executors
  Allow only a single extension for a scripting engine
  Adding REST tests to ensure key_as_string behavior stays consistent
  [test] Set logging to 11 on reindex test
  [TEST] increase logger level until we know what is going on
  Don't allow `fuzziness` for `multi_match` types cross_fields, phrase and phrase_prefix
  ...
2016-05-14 20:23:59 -04:00
Robert Muir 2028691e66 painless: improve exception stacktraces
closes #18319
2016-05-13 15:40:45 -04:00
Lee Hinman 9bcdafedda Allow only a single extension for a scripting engine
Previously multiple extensions could be provided, however, this can lead
to confusion with on-disk scripts (ie, "foo.js" and "foo.javascript")
having different content. Only a single extension is now supported.

The only language currently supporting multiple extensions was the
Javascript engine ("js" and "javascript"). It now only supports the
`.js` extension.

Relates to #10598
2016-05-13 09:54:31 -06:00
Lee Hinman efff3918d8 Remove support for mulitple languages per scripting engine 2016-05-13 09:24:31 -06:00
Lee Hinman a4060f7436 Remove vestiges of script engine sandboxing
This removes all the mentions of the sandbox from the script engine
services and permissions model. This means that the following settings
are no longer supported:

```yaml
script.inline: sandbox
script.stored: sandbox
```

Instead, only a `true` or `false` value can be specified.

Since this would otherwise break the default-allow parameter for
languages like expressions, painless, and mustache, all script engines
have been updated to have individual settings, for instance:

```yaml
script.engine.groovy.inline: true
```

Would enable all inline scripts for groovy. (they can still be
overridden on a per-operation basis).

Expressions, Painless, and Mustache all default to `true` for inline,
file, and stored scripts to preserve the old scripting behavior.

Resolves #17114
2016-05-13 09:24:31 -06:00
Yannick Welsch 7753420540 Make ShardRouting and UnassignedInfo immutable (#17821)
This makes defensive copying of ShardRouting objects obsolete whenever we do a reroute and trashes less objects.
2016-05-10 19:11:04 +02:00
Nik Everett ddc531e729 Build a plugin for testing docs
This makes it much easier to apply to other projects.

Fixes to doc tests infrastructure:
* Fix comparing lists. Was totally broken.
* Fix order of actual vs expected parameters.
* Allow multiple `// TESTRESPONSE` lines with substitutions to join
into one big list of subtitutions. This makes lets the docs look
tidier.
* Exclude build from snippet scanning
* Allow subclasses of ESRestTestCase access to the admin execution context
2016-05-09 14:07:27 -04:00
Nik Everett b7d02fbd1e Improve logging of raw rest actions on failure
Log the method and the path.
2016-05-09 13:04:33 -04:00
Nik Everett ef2e3a8c39 Rest tests: More defense around stashing body
Integration tests failed:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-intake/483/console

We'll see if the rest tests were hiding some other failure.
2016-05-09 09:52:23 -04:00
Ryan Ernst 3d1be071c9 Merge branch 'master' into pom_gen 2016-05-06 12:56:51 -07:00
Chris Earle 5be79ed02c Add Failure Details to every NodesResponse
Most of the current implementations of BaseNodesResponse (plural Nodes) ignore FailedNodeExceptions.

- This adds a helper function to do the grouping to TransportNodesAction
- Requires a non-null array of FailedNodeExceptions within the BaseNodesResponse constructor
- Reads/writes the array to output
- Also adds StreamInput and StreamOutput methods for generically reading and writing arrays
2016-05-06 14:59:43 -04:00
Christoph Büscher 7d14728960 Add xContent shuffling to some more tests
This adds some random shuffling of xContent to some more test cases.

Relates to #5831
2016-05-06 10:46:39 +02:00
Adrien Grand de8354dd7f Allow binary sort values. #17959
The `ip` field uses a binary representation internally. This breaks when
rendering sort values in search responses since elasticsearch tries to write a
binary byte[] as an utf8 json string. This commit extends the `DocValueFormat`
API in order to give fields a chance to choose how to render values.

Closes #6077
2016-05-06 09:27:02 +02:00
Ryan Ernst e16af604bf Build: Add pom generation to assemble task
In preparation for a unified release process, we need to be able to
generate the pom files independently of trying to actually publish. This
change adds back the maven-publish plugin just for that purpose. The
nexus plugin still exists for now, so that we do not break snapshots,
but that can be removed at a later time once snapshots are happenign
through the unified tools. Note I also changed the dir jars are written
into so that all our artifacts are under build/distributions.
2016-05-05 17:57:44 -07:00
Nik Everett 4b1c116461 Generate and run tests from the docs
Adds infrastructure so `gradle :docs:check` will extract tests from
snippets in the documentation and execute the tests. This is included
in `gradle check` so it should happen on CI and during a normal build.

By default each `// AUTOSENSE` snippet creates a unique REST test. These
tests are executed in a random order and the cluster is wiped between
each one. If multiple snippets chain together into a test you can annotate
all snippets after the first with `// TEST[continued]` to have the
generated tests for both snippets joined.

Snippets marked as `// TESTRESPONSE` are checked against the response
of the last action.

See docs/README.asciidoc for lots more.

Closes #12583. That issue is about catching bugs in the docs during build.
This catches *some* bugs in the docs during build which is a good start.
2016-05-05 13:58:03 -04:00
Jason Tedor 784c9e5fb9 Introduce node handshake
This commit introduces a handshake when initiating a light
connection. During this handshake, node information, cluster name, and
version are received from the target node of the connection. This
information can be used to immediately validate that the target node is
a member of the same cluster, and used to set the version on the
stream. This will allow us to extend APIs that are used during initial
cluster recovery without a major version change.

Relates #15971
2016-05-04 20:06:47 -04:00
Jason Tedor 78d615f320 Merge pull request #18110 from jasontedor/strings-split-as-array
Remove Strings#splitStringToArray

Remove arbitrary separator/wildcard from PathTrie
2016-05-04 09:38:47 -04:00
Jason Tedor 2dea449949 Remove Strings#splitStringToArray
This commit removes the method Strings#splitStringToArray and replaces
the call sites with invocations to String#split. There are only two
explanations for the existence of this method. The first is that
String#split is slightly tricky in that it accepts a regular expression
rather than a character to split on. This means that if s is a string,
s.split(".")  does not split on the character '.', but rather splits on
the regular expression '.' which splits on every character (of course,
this is easily fixed by invoking s.split("\\.") instead). The second
possible explanation is that (again) String#split accepts a regular
expression. This means that there could be a performance concern
compared to just splitting on a single character. However, it turns out
that String#split has a fast path for the case of splitting on a single
character and microbenchmarks show that String#split has 1.5x--2x the
throughput of Strings#splitStringToArray. There is a slight behavior
difference between Strings#splitStringToArray and String#split: namely,
the former would return an empty array in cases when the input string
was null or empty but String#split will just NPE at the call site on
null and return a one-element array containing the empty string when the
input string is empty. There was only one place relying on this behavior
and the call site has been modified accordingly.
2016-05-04 08:12:41 -04:00
Isabel Drost-Fromm a8bf75983f Merge branch 'master' into tests/switch_to_random_value_other_than_for_sort 2016-05-04 10:24:46 +02:00
Daniel Mitterdorfer 0a6f40c7f5 Enable HTTP compression by default with compression level 3
With this commit we compress HTTP responses provided the client
supports it (as indicated by the HTTP header 'Accept-Encoding').

We're also able to process compressed HTTP requests if needed.

The default compression level is lowered from 6 to 3 as benchmarks
have indicated that this reduces query latency with a negligible
increase in network traffic.

Closes #7309
2016-05-03 08:53:15 +02:00
Isabel Drost-Fromm 372eceb854 Switch to using predicate for testing existing value 2016-05-02 15:41:05 +02:00
Isabel Drost-Fromm 47fefdd273 Switch from separate sort_mode to more general randomValueOtherThan
... for sort tests only ...
2016-04-28 14:45:56 +02:00
Jason Tedor efeec4d096 Merge pull request #17017 from jasontedor/generic-thread-pool
Actually bound the generic thread pool
2016-04-26 08:27:48 -04:00
Alexander Reelsen 486c783f08 Testing: Remove unused junit rule (#17947)
This rule was used to repeat failed tests due to binding on an already bound port. The test has been fixed
so we can get rid of this rule as well.
2016-04-26 09:53:49 +02:00
Adrien Grand 31a9845bc2 Remove the `SearchType` setter on `SearchContext`. #17955
It was not used.
2016-04-26 09:08:37 +02:00
Ali Beyad d39eb2d691 Adds tombstones to cluster state for index deletions
Previously, we would determine index deletes in the cluster state by
comparing the index metadatas between the current cluster state and the
previous cluster state and decipher which ones were missing (the missing
ones are deleted indices).  This led to a situation where a node that
went offline and rejoined the cluster could potentially cause dangling
indices to be imported which should have been deleted, because when a node
rejoins, its previous cluster state does not contain reliable state.

This commit introduces the notion of index tombstones in the cluster
state, where we are explicit about which indices have been deleted.
In the case where the previous cluster state is not useful for index
metadata comparisons, a node now determines which indices are to be
deleted based on these tombstones in the cluster state.  There is also
functionality to purge the tombstones after exceeding a certain amount.

Closes #17265
Closes #16358
Closes #17435
2016-04-25 15:43:20 -04:00
Jason Tedor 5608fa7ac1 Actually bound the generic thread pool
This commit actually bounds the size of the generic thread pool. The
generic thread pool was of type cached, a thread pool with an unbounded
number of workers and an unbounded work queue. With this commit, the
generic thread pool is now of type scaling. As such, the cached thread
pool type has been removed. By default, the generic thread pool is
constructed with a core pool size of four, a max pool size of 128 and
idle workers can be reaped after a keep-alive time of thirty seconds
expires. The work queue for this thread pool remains unbounded.
2016-04-25 06:47:26 -04:00
Martijn van Groningen c5ad2e2865 Changed indexed scripts to be stored in the cluster state instead of the `.scripts` index.
Also added max script size soft limit for stored scripts.

Closes #16651
2016-04-22 13:42:55 +02:00
Nik Everett 65f6f6bc8d Normalize registration for SignificanceHeuristics
When I pulled on the thread that is "Remove PROTOTYPEs from
SignificanceHeuristics" I ended up removing SignificanceHeuristicStreams
and replacing it with readNamedWriteable. That seems like a lot at once
but it made sense at the time. And it is what we want in the end, I think.

Anyway, this also converts registration of SignificanceHeuristics to
use ParseFieldRegistry to make them consistent with Queries, Aggregations
and lots of other stuff.

Adds a new and wonderous hack to support serialization checking of
NamedWriteables registered by plugins!

Related to #17085
2016-04-19 09:47:37 -04:00
Daniel Mitterdorfer 3688629e11 Adjust line-length of transport related classes to coding standard 2016-04-15 10:12:24 +02:00
Ali Beyad b87fd54ba9 Improvements to the IndicesService class
This commit contains the following improvements/fixes:
  1. Renaming method names and variables to better reflect the purpose
of the method and the semantics of the variable.
  2. For deleting indexes, replace the closed parameter passed to the
delete index/store methods with obtaining the index's state from the
IndexSettings that is already passed in.
  3. Added tests to the IndexWithShadowReplicaIT suite, some of which
show issues in the shadow replica delete process that are captured in
Github issue 17695.

Closes #17638
2016-04-14 11:14:02 -04:00
Nik Everett 64f5a4f848 Stop map collisions on FiltersTests
Adds randomUnique to generate unique things and uses it to make unique
keys.

The offending seed was 81AE616FEAD10F17.
2016-04-13 08:35:45 -04:00
Daniel Mitterdorfer 117bc68af3 Limit request size on HTTP level
With this commit we limit the size of all in-flight requests on
HTTP level. The size is guarded by the same circuit breaker that
is also used on transport level. Similarly, the size that is used
is HTTP content length.

Relates #16011
2016-04-13 09:58:08 +02:00
Daniel Mitterdorfer 52b2016447 Limit request size on transport level
With this commit we limit the size of all in-flight requests on
transport level. The size is guarded by a circuit breaker and is
based on the content size of each request.

By default we use 100% of available heap meaning that the parent
circuit breaker will limit the maximum available size. This value
can be changed by adjusting the setting

network.breaker.inflight_requests.limit

Relates #16011
2016-04-13 09:54:59 +02:00
Adrien Grand 226644ea2c Do not assume term queries use the inverted index. #17532
We have a couple places in the code base that assume that search is always done
on the inverted index. However with the new points API in Lucene 6, this is not
true anymore. This commit makes MappedFieldType.indexedValueForSearch protected
and fixes call sites to keep working for field types that use the inverted
index and either work differently ar throw an exception otherwise. For instance,
it will still be possible to run cross_fields multi match queries on numeric
fields, but the score contributions will not be blended as well as before, and
significant terms aggregations on long terms will not be possible anymore since
points do not record document frequencies.
2016-04-12 09:47:20 +02:00
Adrien Grand 0eb1a816c8 Allow the query cache to be disabled. #16268
This replaces the internal `index.queries.cache.type` setting with
a new `index.queries.cache.enabled` setting, which is documented.

Closes #15802
2016-04-11 18:06:16 +02:00
Alexander Reelsen da19ddf3e6 Ingest Attachment: Allow to prevent base64 conversions by using raw bytes (#16601)
CBOR is natively supported in Elasticsearch and allows for byte arrays.
This means, that by using CBOR the user can prevent base64 conversions
for the data being sent back and forth.

This PR adds support to extract data from a byte array in addition to
a string. This also required to add a ByteArrayValueSource class.
2016-04-11 14:14:56 +02:00
David Pilato 1e346d1ac1 Merge branch 'fix/17625-close-ingest-factory' 2016-04-11 10:00:19 +02:00
Nik Everett 525ce40d1c Give SearchContext a toString
and move the string capturing to capture time.
2016-04-10 20:55:31 -04:00
Nik Everett ac94e5f287 Provide more information about open contexts
Sometimes we get a test failure caused by search contexts left open.
The tests include a stack trace of the call that opened the context
but nothing else about the context. This adds more information about
the context that has been left open like what query it was running,
what shard it targeted, and whether or not it was a scroll.

Relates to #17582
2016-04-10 20:55:31 -04:00
David Pilato 24f48b86b5 Update after review and add a Test 2016-04-09 13:14:25 +02:00
Adrien Grand 42526ac28e Remove Settings.settingsBuilder.
We have both `Settings.settingsBuilder` and `Settings.builder` that do exactly
the same thing, so we should keep only one. I kept `Settings.builder` since it
has my preference but also it is the one that we use in examples of the Java API.
2016-04-08 18:10:02 +02:00
Chris Earle d97d5ebb8b Remove hostname from NetworkAddress.format
This removes the inconsistent output of IP addresses. The format was parsing-unfriendly and it makes it hard
to reason about API responses, such as to _nodes.

With this change in place, it will never print the hostname as part of the default format, which has the
added benefit that it can be used consistently for URIs, which was not the case when the hostname might
appear at the front with "hostname/ip:port".
2016-04-07 17:27:59 -04:00
Adrien Grand c33300c543 Make MappedFieldType responsible for providing a parser/formatter. #17546
Aggregations need to perform instanceof calls on MappedFieldType instances in
order to know how they should be parsed or formatted. Instead, we should let
the field types provide a formatter/parser that can can be used.
2016-04-07 16:57:50 +02:00
jaymode f9d1e8a5f3 Root rest api delegates to a transport action
This change makes the root (/) rest api delegate to a transport action to get the
data for the response. This aligns this rest api with all of the other apis, which
delegate to one or more actions.

In doing this, unit tests were added to provide coverage of the RestMainAction
and the associated classes.
2016-04-07 10:03:49 -04:00
Jason Tedor 0a69985153 Merge pull request #17038 from jasontedor/enable_acked
Prepare for enabling acked indexing
2016-04-06 18:13:28 -04:00
Jimmy Jones f157dae053 Disallow unquoted field names, fix testcases using unquoted JSON 2016-04-06 14:37:15 -06:00
Clinton Gormley cbbf80ca35 v2.3.0 has been released and no longer needs to be hardcoded as -SNAPSHOT 2016-04-04 19:03:43 +02:00
Jason Tedor c7c8b1d825 Merge branch 'master' into enable_acked
* master: (156 commits)
  Make JNA calls optional
  Added RPM metadata
  Remove PROTOTYPE from MLT.Item
  Remove PROTOTYPE from VersionType
  Fix mistake in TopHits change
  Remove PROTOTYPEs from highlighting
  Clean up some log messages
  Command line arguments with comma must be quoted on windows
  Cluster Health should run on applied states, even if waitFor=0 #17440
  ingest: make concrete processor impl final, like all other processor concrete impls.
  Improve some test method comments.
  Document task id's as string in the rest spec
  Replace FieldStatsProvider with a method on MappedFieldType. #17334
  cleanup test
  Remove MathUtils. #17454
  Addressing review comments
  fix javadocs
  Make TranslogConfig immutable and pass TranslogGeneration as a ctor arg to Translog
  [reindex] Don't get rejected
  Remove redundant commit - #openTranslog() already commits in that case
  ...
2016-04-02 13:56:00 -04:00
Christoph Büscher 9d68a515b8 Merge pull request #17453 from cbuescher/add-xcontent-randomization
Add randomization of XContentBuilder output to query tests
2016-04-01 15:02:01 +02:00
Christoph Büscher 7a1b06ce0b Improve some test method comments. 2016-04-01 11:04:56 +02:00
Christoph Büscher 1a697a1ae6 Addressing review comments 2016-03-31 21:46:17 +02:00
Simon Willnauer baa2d51e59 Merge pull request #17422 from s1monw/recovery_mem_buffer_access
Move translog recover outside of the engine

We changed the way we manage engine memory buffers to an
open model where each shard can essentially has infinite memory.
The indexing memory controller is responsible for moving memory to disk
when it's needed. Yet, this doesn't work today when we recover from store/translog
since the engine is not fully initialized such that IMC has no access to the engine,
neither to it's memory buffer nor can it move data to disk.

The biggest issue here is that translog recovery happends inside the Engine constructor
which is problematic by itself since it might take minutes and uses a not yet fully
initialzied engine to perform write operations on.

This change detaches the translog recovery and makes it the responsibility of the caller
to run it once the engine is fully constructed or skip it if not necessary.
2016-03-31 21:03:00 +02:00
Christoph Büscher bbb6d91147 Add randomization of XContentBuilder output to query tests
Currently our testing of parsing query builders is limited to the
default order of the parameters that each builders toXContent()
method produces. To better test real queries where the order of
parameters can be different, this change adds a helper
method to ESTestCase that takes a XContentBuilder and randomly
shuffles the order of the fields inside an object. This is
used in AbstractQueryTestCase, but it can be used in other similar
places in the future.
2016-03-31 18:17:39 +02:00
Simon Willnauer 1e06139584 Move translog recover outside of the engine
We changed the way we manage engine memory buffers to an
open model where each shard can essentially has infinite memory.
The indexing memory controller is responsible for moving memory to disk
when it's needed. Yet, this doesn't work today when we recover from store/translog
since the engine is not fully initialized such that IMC has no access to the engine,
neither to it's memory buffer nor can it move data to disk.

The biggest issue here is that translog recovery happends inside the Engine constructor
which is problematic by itself since it might take minutes and uses a not yet fully
initialzied engine to perform write operations on.

This change detaches the translog recovery and makes it the responsibility of the caller
to run it once the engine is fully constructed or skip it if not necessary.
2016-03-30 23:24:24 +02:00
javanna b9f9b2e3ee Merge branch 'master' into enhancement/discovery_node_one_getter 2016-03-30 17:22:40 +02:00
javanna 62ac7d219f Remove DiscoveryNodes#masterNode in favour of existing DiscoveryNodes#getMasterNode 2016-03-30 15:28:32 +02:00
javanna f8b5d1f5b0 Remove DiscoveryNodes#masterNodeId in favour of existing DiscoveryNodes#getMasterNodeId 2016-03-30 15:28:06 +02:00
javanna 2dbba45f2c Rename static DiscoveryNode#localNode(Settings) to DiscoveryNode#isLocalNode(Settings) 2016-03-30 15:27:26 +02:00
javanna 49e952e272 Rename static DiscoveryNode#dataNode(Settings) to isDataNode 2016-03-30 15:26:41 +02:00
javanna 2230fec9ea Rename static DiscoveryNode#masterNode(Settings) to isMasterNode 2016-03-30 15:26:10 +02:00
javanna a8bbdff3bc Remove DiscoveryNode#name in favour of existing DiscoveryNode#getName 2016-03-30 14:47:36 +02:00
javanna 9889f10e5e Remove DiscoveryNode#id in favour of existing DiscoveryNode#getId 2016-03-30 14:42:15 +02:00
Camilo Diaz Repka 7be11a36cd Refactor: replace all ocurrences of ESTestCase.getRandom() for random().
Remove getRandom().
2016-03-29 23:18:05 -04:00
javanna 061f09d9a4 Merge branch 'master' into enhancement/remove_node_client_setting 2016-03-29 20:19:33 +02:00
Jason Tedor c4324f9964 Merge branch 'master' into enable_acked
* master: (25 commits)
  Replication operation that try to perform the primary phase on a replica should be retried
  split long line in ConvertProcessorTests
  add  type conversion support to ConvertProcessor
  percolator: Make explain use the two phase iterator
  test: make sure we don't flush during indexing the percolator queries
  Added experimental annotation to the update-by-query and reindex docs
  Fixed bad YAML in reindex REST test: 50_routing.yaml
  Update-by-query rest tests: fixed bad yaml and deleted a client-dependent test
  Prevents exception being raised when ordering by an aggregation which wasn't collected
  The reindex body is now required, which changes the exception thrown by the REST test
  Docs: Included Nodes Task API and tidied reindex/update-by-query
  Rename update-by-query REST tests to update_by_query
  REST: The body is required in the reindex API
  The source parameter should not be defined in the delete-by-query REST spec
  Renamed update-by-query REST spec to update_by_query
  Fix test bug in TypeQueryBuilderTests.
  Add comment why it is safe to check the number of nested fields in MapperService.merge.
  Automatically add a sub keyword field to string dynamic mappings. #17188
  Type filters should not have a performance impact when there is a single type. #17350
  Add API to explain why a shard is or isn't assigned
  ...
2016-03-29 11:42:34 -04:00
Colin Goodheart-Smithe ff3fd99074 Prevents exception being raised when ordering by an aggregation which wasn't collected
If a terms aggregation was ordered by a metric nested in a single bucket aggregator which did not collect any documents (e.g. a filters aggregation which did not match in that term bucket) an ArrayOutOfBoundsException would be thrown when the ordering code tried to retrieve the value for the metric. This fix fixes all numeric metric aggregators so they return their default value when a bucket ordinal is requested which was not collected.

Closes #17225
2016-03-29 13:28:03 +01:00
javanna de5cbda8e7 Merge branch 'master' into enhancement/remove_node_client_setting 2016-03-29 10:48:47 +02:00
Lee Hinman 80ab366de4 Add API to explain why a shard is or isn't assigned
This adds a new `/_cluster/allocation/explain` API that explains why a
shard can or cannot be allocated to nodes in the cluster. Additionally,
it will show where the master *desires* to put the shard, according to
the `ShardsAllocator`.

It looks like this:

```
GET /_cluster/allocation/explain?pretty
{
  "index": "only-foo",
  "shard": 0,
  "primary": false
}
```

Though, you can optionally send an empty body, which means "explain the
allocation for the first unassigned shard you find".

The output when a shard is unassigned looks like this:

```
{
  "shard" : {
    "index" : "only-foo",
    "index_uuid" : "KnW0-zELRs6PK84l0r38ZA",
    "id" : 0,
    "primary" : false
  },
  "assigned" : false,
  "unassigned_info" : {
    "reason" : "INDEX_CREATED",
    "at" : "2016-03-22T20:04:23.620Z"
  },
  "nodes" : {
    "V-Spi0AyRZ6ZvKbaI3691w" : {
      "node_name" : "Susan Storm",
      "node_attributes" : {
        "bar" : "baz"
      },
      "final_decision" : "NO",
      "weight" : 0.06666675,
      "decisions" : [ {
        "decider" : "filter",
        "decision" : "NO",
        "explanation" : "node does not match index include filters [foo:\"bar\"]"
      } ]
    },
    "Qc6VL8c5RWaw1qXZ0Rg57g" : {
      "node_name" : "Slipstream",
      "node_attributes" : {
        "bar" : "baz",
        "foo" : "bar"
      },
      "final_decision" : "NO",
      "weight" : -1.3833332,
      "decisions" : [ {
        "decider" : "same_shard",
        "decision" : "NO",
        "explanation" : "the shard cannot be allocated on the same node id [Qc6VL8c5RWaw1qXZ0Rg57g] on which it already exists"
      } ]
    },
    "PzdyMZGXQdGhqTJHF_hGgA" : {
      "node_name" : "The Symbiote",
      "node_attributes" : { },
      "final_decision" : "NO",
      "weight" : 2.3166666,
      "decisions" : [ {
        "decider" : "filter",
        "decision" : "NO",
        "explanation" : "node does not match index include filters [foo:\"bar\"]"
      } ]
    }
  }
}
```

And when the shard *is* assigned, the output looks like:

```
{
  "shard" : {
    "index" : "only-foo",
    "index_uuid" : "KnW0-zELRs6PK84l0r38ZA",
    "id" : 0,
    "primary" : true
  },
  "assigned" : true,
  "assigned_node_id" : "Qc6VL8c5RWaw1qXZ0Rg57g",
  "nodes" : {
    "V-Spi0AyRZ6ZvKbaI3691w" : {
      "node_name" : "Susan Storm",
      "node_attributes" : {
        "bar" : "baz"
      },
      "final_decision" : "NO",
      "weight" : 1.4499999,
      "decisions" : [ {
        "decider" : "filter",
        "decision" : "NO",
        "explanation" : "node does not match index include filters [foo:\"bar\"]"
      } ]
    },
    "Qc6VL8c5RWaw1qXZ0Rg57g" : {
      "node_name" : "Slipstream",
      "node_attributes" : {
        "bar" : "baz",
        "foo" : "bar"
      },
      "final_decision" : "CURRENTLY_ASSIGNED",
      "weight" : 0.0,
      "decisions" : [ {
        "decider" : "same_shard",
        "decision" : "NO",
        "explanation" : "the shard cannot be allocated on the same node id [Qc6VL8c5RWaw1qXZ0Rg57g] on which it already exists"
      } ]
    },
    "PzdyMZGXQdGhqTJHF_hGgA" : {
      "node_name" : "The Symbiote",
      "node_attributes" : { },
      "final_decision" : "NO",
      "weight" : 3.6999998,
      "decisions" : [ {
        "decider" : "filter",
        "decision" : "NO",
        "explanation" : "node does not match index include filters [foo:\"bar\"]"
      } ]
    }
  }
}
```

Only "NO" decisions are returned by default, but all decisions can be
shown by specifying the `?include_yes_decisions=true` parameter in the
request.

Resolves #14593
2016-03-28 15:21:02 -06:00
Jason Tedor 4793630eb8 Merge branch 'master' into enable_acked
* master: (419 commits)
  Remove PROTOTYPE from ShapeBuilders
  Take filterNodeIds into consideration while sending tasks actions requests to nodes
  test: cleanup imports and method rename
  Remove PROTOTYPE from SortBuilders
  percolator: Add query extract support for the blended term query and the common terms query.
  Don't iterate over shard routing if it's null
  [TEST] Reduce size of random shapes
  Add some debug logging to testPrimaryRelocationWhileIndexing
  Order methods in IndicesClusterStateService according to execution
  Tidied up percolator doc annotations
  In cat.snapshots, repository is required
  Do not retrieve all indices stats when checking for cache resets
  Enforce `discovery.zen.minimum_master_nodes` is set when bound to a public ip #17288
  Port Primary Terms to master #17044
  Revert "Add debug logging for Vagrant upgrade test"
  Ownership for data, logs, and configs for packages
  add on_failure exception metadata to ingest document for verbose simulate
  Revert "Merge pull request #16843 from xuzha/s3-encryption"
  Update Format, add new settings into the setting test
  Update and rebase the init implementation.
  ...
2016-03-28 12:29:53 -04:00
javanna a9f4982c40 Merge branch 'master' into enhancement/remove_node_client_setting 2016-03-25 20:16:40 +01:00
javanna 93ce36a198 separated attributes from node roles in DiscoveryNode
Node roles are now serialized as well, they are not part of the node attributes anymore. DiscoveryNodeService takes care of dividing settings into attributes and roles. DiscoveryNode always requires to pass in attributes and roles separately.
2016-03-25 20:14:27 +01:00
Boaz Leskes dcd2642dad Merge branch 'master' into feature/seq_no 2016-03-25 17:26:14 +01:00
Boaz Leskes 91021e3019 merge from master 2016-03-25 15:50:48 +01:00
Tanguy Leroux a22529cceb Do not retrieve all indices stats when checking for cache resets 2016-03-25 13:16:12 +01:00
Boaz Leskes fe43eef1b5 Port Primary Terms to master #17044
Primary terms is a way to make sure that operations replicated from stale primary are rejected by shards following a newly elected primary.

Original PRs adding this to the seq# feature branch #14062 , #14651 . Unlike those PR, here we take a different approach (based on newer code in master) where the primary terms are stored in the meta data only (and not in `ShardRouting` objects).

Relates to #17038

Closes #17044
2016-03-25 12:01:00 +01:00
javanna 27d4994aff Merge branch 'master' into enhancement/remove_node_client_setting 2016-03-24 18:10:11 +01:00
Alexander Reelsen b2573858b6 Version: Set version to 5.0.0-alpha1
Change version, required a minor fix in the RPM building.
In case of a alpha/beta version, the release will contain alpha/beta
as the RPM version cannot contains dashes/tildes.
2016-03-24 08:36:08 +01:00
Honza Král b139f4e0bf [TEST] Move yaml test requiring yaml, add skip:yaml
Clients don't ship with yaml (de)serializer by default so this test must
be optionally skipped
2016-03-23 14:50:23 +01:00
javanna 030453d320 Merge branch 'master' into enhancement/remove_node_client_setting 2016-03-23 11:25:34 +01:00
Adrien Grand e50eeeaffb Refactor fielddata mappings. #17148
The fielddata settings in mappings have been refatored so that:
 - text and string have a `fielddata` (boolean) setting that tells whether it
   is ok to load in-memory fielddata. It is true by default for now but the
   plan is to make it default to false for text fields.
 - text and string have a `fielddata_frequency_filter` which contains the same
   thing as `fielddata.filter.frequency` used to (but validated at parsing time
   instead of being unchecked settings)
 - regex fielddata filtering is not supported anymore and will be dropped from
   mappings automatically on upgrade.
 - text, string and _parent fields have an `eager_global_ordinals` (boolean)
   setting that tells whether to load global ordinals eagerly on refresh.
 - in-memory fielddata is not supported on keyword fields anymore at all.
 - the `fielddata` setting is not supported on other fields that text and string
   and will be dropped when upgrading if specified.
2016-03-23 09:48:13 +01:00
Boaz Leskes 7c8cdf4a71 merged from master 2016-03-22 19:21:28 +01:00
Simon Willnauer 1988b8b387 [TEST] Reuse EsTestCase#createAnalysisService in KuromojiAnalysisTests 2016-03-22 13:45:20 +01:00
Boaz Leskes 39ae16bc4c merge from master 2016-03-22 11:46:26 +01:00
Simon Willnauer 33521fc27c Detach IndexShard from node services
this is the last step to remove node level service from IndexShard.
This means that tests can now more easily create an IndexShard instance
without starting a node and removes the dependency between IndexShard and Client/ScriptService
2016-03-22 11:02:04 +01:00
javanna bf390a935e Merge branch 'master' into enhancement/remove_node_client_setting 2016-03-21 17:18:23 +01:00
Boaz Leskes 2d1152ebac Remove ClusterService interface, in favor of it's only production instance #17183
We current have a ClusterService interface, implemented by InternalClusterService and a couple of test classes. Since the decoupling of the transport service and the cluster service, one can construct a ClusterService fairly easily, so we don't need this extra indirection.

Closes #17183
2016-03-21 13:55:10 +01:00
Martijn van Groningen e3b7e5d75a percolator: Replace percolate api with the new percolator query
Also replaced the PercolatorQueryRegistry with the new PercolatorQueryCache.

The PercolatorFieldMapper stores the rewritten form of each percolator query's xcontext
in a binary doc values field. This make sure that the query rewrite happens only during
indexing (some queries for example fetch shapes, terms in remote indices) and
the speed up the loading of the queries in the percolator query cache.

Because the percolator now works inside the search infrastructure a number of features
(sorting fields, pagination, fetch features) are available out of the box.

The following feature requests are automatically implemented via this refactoring:

Closes #10741
Closes #7297
Closes #13176
Closes #13978
Closes #11264
Closes #10741
Closes #4317
2016-03-21 12:21:50 +01:00
Boaz Leskes 858610d0d1 merge from master 2016-03-19 13:57:40 +01:00
Ryan Ernst f71f0d6010 Revert "Build: Switch to maven-publish plugin"
This reverts commit a90a2b34fc.
2016-03-18 17:22:25 -07:00
Ryan Ernst 6af4c43c4f Merge pull request #17128 from rjernst/maven_publish
Build: Switch to maven-publish plugin
2016-03-17 11:53:50 -07:00
Simon Willnauer e91a141233 Prevent index level setting from being configured on a node level
Today we allow to set all kinds of index level settings on the node level which
is error prone and difficult to get right in a consistent manner.
For instance if some analyzers are setup in a yaml config file some nodes might
not have these analyzers and then index creation fails.

Nevertheless, this change allows some selected settings to be specified on a node level
for instance:
 * `index.codec` which is used in a hot/cold node architecture and it's value is really per node or per index
 * `index.store.fs.fs_lock` which is also dependent on the filesystem a node uses

All other index level setting must be specified on the index level. For existing clusters the index must be closed
and all settings must be updated via the API on each of the indices.

Closes #16799
2016-03-17 14:42:18 +01:00
Christoph Büscher 6ddf9ae92f Merge branch 'master' into feature-suggest-refactoring 2016-03-16 15:27:02 +01:00
Igor Motov b10db19595 Bring back tests for missing elements in the diff-serialized cluster state
We can add it back now that we improved our compression framework.

Closes #11257
2016-03-16 09:13:52 -04:00
Nik Everett 7197172047 [reindex] Properly register status
Without this commit fetching the status of a reindex from a node that isn't
coordinating the reindex will fail. This commit properly registers reindex's
status so this doesn't happen. To do so it moves all task status registration
into NetworkModule and creates a method to register other statuses which the
reindex plugin calls.
2016-03-16 07:40:49 -04:00
Christoph Büscher 39667b5793 Merge branch 'master' into feature-suggest-refactoring
Conflicts:
	docs/reference/migration/migrate_5_0/java.asciidoc
2016-03-16 12:06:42 +01:00
Ryan Ernst a90a2b34fc Build: Switch to maven-publish plugin
The build currently uses the old maven support in gradle. This commit
switches to use the newer maven-publish plugin. This will allow future
changes, for example, easily publishing to artifactory.

An additional part of this change makes publishing of build-tools part
of the normal publishing, instead of requiring a separate upload step
from within buildSrc. That also sets us up for a follow up to enable
precomit checks on the buildSrc code itself.
2016-03-15 19:16:37 -07:00
Jason Tedor 618441aea3 Merge pull request #17088 from jasontedor/simplify-bootstrap-settings
Bootstrap does not set system properties
2016-03-15 19:25:16 -04:00
Yannick Welsch f5e6db4090 Remove System.out.println and Throwable.printStackTrace from tests 2016-03-15 15:40:37 +01:00
Yannick Welsch e91fd09692 Enable jdk-system-out Forbidden API checks on test sources 2016-03-15 15:03:37 +01:00
Christoph Büscher 97638c95fc Merge branch 'master' into feature-suggest-refactoring
Conflicts:
	docs/reference/migration/migrate_5_0.asciidoc
2016-03-14 11:13:47 +01:00
Jason Tedor 8a05c2a2be Bootstrap does not set system properties
Today, certain bootstrap properties are set and read via system
properties. This action-at-distance way of managing these properties is
rather confusing, and completely unnecessary. But another problem exists
with setting these as system properties. Namely, these system properties
are interpreted as Elasticsearch settings, not all of which are
registered. This leads to Elasticsearch failing to startup if any of
these special properties are set. Instead, these properties should be
kept as local as possible, and passed around as method parameters where
needed. This eliminates the action-at-distance way of handling these
properties, and eliminates the need to register these non-setting
properties. This commit does exactly that.

Additionally, today we use the "-D" command line flag to set the
properties, but this is confusing because "-D" is a special flag to the
JVM for setting system properties. This creates confusion because some
"-D" properties should be passed via arguments to the JVM (so via
ES_JAVA_OPTS), and some should be passed as arguments to
Elasticsearch. This commit changes the "-D" flag for Elasticsearch
settings to "-E".
2016-03-13 20:09:15 -04:00
David Pilato 9acb0bb28c Merge branch 'master' into pr/16598-register-filter-settings
# Conflicts:
#	core/src/main/java/org/elasticsearch/cluster/service/InternalClusterService.java
#	core/src/main/java/org/elasticsearch/common/settings/IndexScopedSettings.java
#	core/src/main/java/org/elasticsearch/common/settings/Setting.java
2016-03-13 14:52:10 +01:00
Ryan Ernst 591fb8f028 Merge branch 'master' into cli-parsing 2016-03-11 10:45:05 -08:00
Ali Beyad 31dcb3e18b Merge pull request #16873 from abeyad/suggester-wiring-refactoring
Change internal representation of suggesters
2016-03-11 12:34:35 -05:00
Christoph Büscher daeffb149c Merge branch 'master' into feature-suggest-refactoring 2016-03-11 10:37:28 +01:00
Yannick Welsch 04e55ecf6b Make logging message String constant to allow static checks 2016-03-11 10:30:59 +01:00
Yannick Welsch b17f4b40ba Check that number of placeholders in log message matches number of parameters 2016-03-11 10:30:59 +01:00
Yannick Welsch f6ae9ec4f6 Remove ShardsAllocators and merge allocateUnassigned, moveShards and rebalance to improve performance 2016-03-11 08:42:25 +01:00
Nik Everett 64c4b5f3ca Merge pull request #17036 from dongjoon-hyun/fix_typo_in_test
Fix typos in comments/strings of `test` module.
2016-03-10 13:42:45 -05:00
Ryan Ernst ac73d97169 Merge branch 'master' into dots2 2016-03-10 09:22:31 -08:00
Ali Beyad 807d085218 Merge remote-tracking branch 'upstream/feature-suggest-refactoring' into suggester-wiring-refactoring 2016-03-10 09:56:08 -05:00
Jay Modi debd41b3ea Merge pull request #17049 from jaymode/single_node_test_settings
Allow additional settings for the node in ESSingleNodeTestCase
2016-03-10 07:42:47 -05:00
Martijn van Groningen 2fa33d5c47 Added ingest statistics to node stats API
The ingest stats include the following statistics:
* `ingest.total.count`- The total number of document ingested during the lifetime of this node
* `ingest.total.time_in_millis` - The total time spent on ingest preprocessing documents during the lifetime of this node
* `ingest.total.current` - The total number of documents currently being ingested.
* `ingest.total.failed` - The total number ingest preprocessing operations failed during the lifetime of this node

Also these stats are returned on a per pipeline basis.
2016-03-10 13:21:43 +01:00
jaymode 9f92325587 Allow additional settings for the node in ESSingleNodeTestCase
This change adds a method that extending classes can override to provide additional settings
for the node used in a single node test case.
2016-03-10 07:16:37 -05:00
Boaz Leskes cd12241e9f Decouple the TransportService and ClusterService #16872
Currently, the cluster service is tightly coupled to the transport service by both managing node connections and requiring the bound address in order to create the local disco node. This commit introduces a new NodeConnectionsService which is in charge of node connection management and makes it possible to remove all network related calls from the cluster service. The local DiscoNode is now created by DiscoveryNodeService and is set both the cluster service and the transport service during node start up.

Closes #16788
Closes #16872
2016-03-10 11:45:15 +01:00
Christoph Büscher 69c83b3459 Merge branch 'master' into feature-suggest-refactoring
Conflicts:
	core/src/main/java/org/elasticsearch/action/suggest/TransportSuggestAction.java
        core/src/main/java/org/elasticsearch/search/suggest/SuggestParseElement.java
	core/src/main/java/org/elasticsearch/search/suggest/SuggestionSearchContext.java
	core/src/main/java/org/elasticsearch/search/suggest/phrase/PhraseSuggester.java
2016-03-10 11:44:36 +01:00
Boaz Leskes 330f2919cb merge from master 2016-03-10 09:37:42 +01:00
Ali Beyad 06487b0ac5 Change internal representation of suggesters to instances of
SuggestBuilder instead of raw bytes.
2016-03-10 00:12:39 -05:00
Ryan Ernst c48e6b86f6 Limit update to fields and objects which were actually modified, and
simplify root update creation.
2016-03-09 13:24:46 -08:00
Ryan Ernst bbdbfe7373 Merge branch 'master' into dots2 2016-03-09 11:28:27 -08:00
Simon Willnauer 016bc3ea2b Merge pull request #17001 from s1monw/cut_over_to_uuid
Use index UUID to lookup indices on IndicesService

Today we use the index name to lookup index instances on the IndicesService
which applied to search requests but also to index deletion etc. This commit
moves the interface to expect an Index instance which is a tuple
and looks up the index by uuid rather than by name. This prevents accidental modification
of the wrong index if and index is recreated or searching from the wrong index in such a case.
Accessing an index that has the same name but different UUID will now result in an IndexNotFoundException.
2016-03-09 19:44:36 +01:00
Simon Willnauer e72dac91b3 Use index UUID to lookup indices on IndicesService
Today we use the index name to lookup index instances on the IndicesService
which applied to search reqeusts but also to index deletion etc. This commit
moves the interface to expcet and `Index` instance which is a <name, uuid> tuple
and looks up the index by uuid rather than by name. This prevents accidential modificaiton
of the wrong index if and index is recreated or searching from the _wrong_ index in such a case.
Accessing an index that has the same name but different UUID will now result in an IndexNotFoundException.

Closes #17001
2016-03-09 19:42:15 +01:00
Dongjoon Hyun db534347e5 Fix typos in comments/strings of `test` module. 2016-03-09 10:03:37 -08:00
Ryan Ernst fdce9d7c4d Merge branch 'master' into cli-parsing 2016-03-08 14:18:20 -08:00
Ryan Ernst 80198accc1 Removed old cli stuff, and add tests for new Command behavior 2016-03-08 14:13:55 -08:00
Ryan Ernst e5c852f767 Convert bootstrapcli parser to jopt-simple 2016-03-08 13:39:37 -08:00
Nik Everett 6d0efae713 Teach list tasks api to wait for tasks to finish
_wait_for_completion defaults to false. If set to true then the API will
wait for all the tasks that it finds to stop running before returning. You
can use the timeout parameter to prevent it from waiting forever. If you
don't set a timeout parameter it'll default to 30 seconds.

Also adds a log message to rest tests if any tasks overrun the test. This
is just a log (instead of failing the test) because lots of tasks are run
by the cluster on its own and they shouldn't cause the test to fail. Things
like fetching disk usage from the other nodes, for example.

Switches the request to getter/setter style methods as we're going that
way in the Elasticsearch code base. Reindex is all getter/setter style.

Closes #16906
2016-03-08 11:53:57 -05:00
Jason Tedor 2bd09c4625 Merge branch 'master' into enable_acked
* master: (350 commits)
  Note to configuration docs on number of threads
  Reduce maximum number of threads in boostrap check
  Limit generic thread pool
  Remove NodeService injection to Discovery
  Prevent closing index during snapshot restore
  [TEST] Fix newline issue in PluginCliTests on Windows
  ParseFieldMatcher should log when using deprecated settings. #16988
  fix checkstyle error
  Add test for the index_options on a keyword field. #16990
  Analysis : Allow string explain param in JSON
  Analysis : Allow string explain param in JSON
  fix typo
  Remove SNAPSHOT from versions in plugin descriptors
  Add support for alpha versions
  Enable unmap hack for java 9
  Simplify mock scripts
  Adding `time_zone` parameter to daterange-aggregation docs
  Adding tests for `time_zone` parameter for date range aggregation
  Added ingest info to node info API, which contains a list of available processors.
  Remove bw compat from size mapper
  ...
2016-03-08 10:19:56 -05:00
Christoph Büscher 1264f37a1b Merge branch 'master' into feature-suggest-refactoring 2016-03-08 11:10:08 +01:00
Yannick Welsch c9e1ccf610 [TEST] Fix newline issue in PluginCliTests on Windows 2016-03-08 10:28:20 +01:00
Ryan Ernst 45b5ab24fe Moved MockTerminal and created a base test case for cli commands. 2016-03-07 12:42:15 -08:00
Simon Willnauer f071f327db Remove old and unsupported version constants
All version <= 2.0 are not supported anymore. This commit removes all
uses of these versions.
2016-03-07 11:49:06 +01:00
Christoph Büscher 7ec5075a87 Merge branch 'master' into feature-suggest-refactoring 2016-03-07 10:45:43 +01:00
Robert Muir 54018a5d37 upgrade to lucene 6.0.0-snapshot-bea235f
Closes #16964

Squashed commit of the following:

commit a23f9d2d29220991aa498214530753d7a5a148c6
Merge: eec9c4e 0b0a251
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Mar 7 04:12:02 2016 -0500

    Merge branch 'master' into lucene6

commit eec9c4e5cd11e9c3e0b426f04894bb2a6dae4f21
Merge: bc67205 675d940
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 13:45:00 2016 -0500

    Merge branch 'master' into lucene6

commit bc67205bdfe1526eae277ab7856fc050ecbdb7b2
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 09:56:31 2016 -0500

    fix test bug

commit a60723b007ff12d97b1810cef473bd7b553a0327
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Mar 4 15:35:35 2016 +0100

    Fix SimpleValidateQueryIT to put braces around boosted terms

commit ae3a49d7ba7ced448d2a5262e5d8ec98671a9090
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Mar 4 15:27:25 2016 +0100

    fix multimatchquery

commit ae23fdb88a8f6d3fb7ba60fd1aaf3fd72d899aa5
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Mar 4 15:20:49 2016 +0100

    Rewrite DecayFunctionScoreIT to be independent of the similarity used

    This test relied a lot on the term scoring and compared scores
    that are dependent on the similarity. This commit changes the base query
    to be a predictable constant score query.

commit 366c2d518c35d31251033f1b6f6a93f6e2ae327d
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Mar 4 14:06:14 2016 +0100

    Fix scoring in tests due to changes to idf calculation.

    Lucene 6 uses a different default similarity as well as a different
    way to calculate IDF. In contrast to older version lucene 6 uses docCount per field
    to calculate the IDF not the # of docs in the index to overcome the sparse field
    cases.

commit dac99fd64ac2fa71b8d8d106fe68825e574c49f8
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 08:21:57 2016 -0500

    don't hardcoded expected termquery score

commit 6e9f340ba49ab10eed512df86d52a121aa775b0f
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 08:04:45 2016 -0500

    suppress deprecation warning until migrated to points

commit 3ac8908424b3fdad44a90a4f7bdb3eff7efd077d
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 07:21:43 2016 -0500

    Remove invalid test: all commits have IDs, and its illegal to do this.

commit c12976288124ad1a26467e7e848fb810548e7eab
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 07:06:14 2016 -0500

    don't test with unsupported back compat

commit 18bbfe76128570bc70883bf91ff4c44c82d27817
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 07:02:18 2016 -0500

    remove now invalid lucene 4 backcompat test

commit 7e730e572886f0ef2d3faba712e4256216ff01ec
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 06:58:52 2016 -0500

    remove now invalid lucene 4 backwards test

commit 244d2ab6868ba5ac9e0bcde3c2833743751a25ec
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 06:47:23 2016 -0500

    use 6.0 codec

commit 5f64d4a431a6fdaa1234adca23f154c2a1de8284
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 06:43:08 2016 -0500

    compile, javadocs, forbidden-apis, etc

commit 1f273cd62a7fe9ca8f8944acbbfc5cbdd3d81ccb
Merge: cd33921 29e3443
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Mar 4 10:45:29 2016 +0100

    Merge branch 'master' into lucene6

commit cd33921ac742ef9fb351012eff35f3c7dbda7264
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:58:37 2016 -0500

    fix hunspell dictionary loading

commit c7fdbd837b01f7defe9cb1c24e2ec65604b0dc96
Merge: 4d4190f d8948ba
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:41:53 2016 -0500

    Merge branch 'master' into lucene6

commit 4d4190fd82601aaafac6b8254ccb3edf218faa34
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:39:14 2016 -0500

    remove nocommit

commit 77ca69e288b1a41aa9595c921ed166c272a00ea8
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:38:24 2016 -0500

    clean up numericutils vs legacynumericutils

commit a466d696fbaad04b647ffbc0857a9439b583d0bf
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:32:43 2016 -0500

    upgrade spatial4j

commit 5412c747a8cfe638bacedbc8233163cb75cc3dc5
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:19:28 2016 -0500

    move to 6.0.0-snapshot-8eada27

commit b32bfe924626b87e540692375ece09e7c2edb189
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 11:30:09 2016 +0100

    Fix some test compile errors.

commit 6ccde35e9840b03c68d1a2cd47c7923a06edf64a
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 11:25:51 2016 +0100

    Current Lucene version is 6.0.0.

commit f62e1015d931b4cc04c778298a8fa1ba65e97ad9
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 11:20:48 2016 +0100

    Fix compile errors in NGramTokenFilterFactory.

commit 6837c6eabf96075f743649da9b9b52dd39611c58
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 10:50:59 2016 +0100

    Fix the edge ngram tokenizer/filter.

commit ccd7f070de5efcdfbeb34b9555c65c4990bf1ba6
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 10:42:44 2016 +0100

    The missing value is now accessible through a getter.

commit bd3b77f9b28e5b05daa3d49683a9922a6baf2963
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 10:41:51 2016 +0100

    Remove IndexCacheableQuery.

commit 05f3091c347aeae80eeb16349ac51d2b53cf86f7
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 10:39:43 2016 +0100

    Fix compilation of function_score queries.

commit 81cda79a2431ac78f56b0cc5a5765387f662d801
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 10:35:02 2016 +0100

    Fix compile errors in BlendedTermQuery.

commit 70994ce8dd1eca0b995870974a38e20f26f96a7b
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 23:33:03 2016 -0500

    add bug ID

commit 29d4f1a71f36f646b5a6060bed3db019564a279d
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 21:02:32 2016 -0500

    easy .store changes

commit 5e1a1e6fd665fa455e88d3a8987362fad5f44bb1
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 20:47:24 2016 -0500

    cleanups mostly around boosting

commit 333a669ec6c305ada5645d13ed1da0e19ec1d053
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 20:27:56 2016 -0500

    more simple fixes

commit bd5cd98a1e089c866b6b4a5e159400b110140ce6
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 19:49:38 2016 -0500

    more easy fixes and removal of ancient cruft

commit a68f419ee47da5f9c9ce5b372f01d707e902474c
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 19:35:02 2016 -0500

    cutover numerics

commit 4ca5dc1fa47dd5892db00899032133318fff3116
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 18:34:18 2016 -0500

    fix some constants

commit 88710a17817086e477c6c021ec346d0534b7fb88
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 18:14:25 2016 -0500

    Add spatial-extras jar as a core dependency

commit c8cd6726583e5ce3f546ed355d4eca037164a30d
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 18:03:33 2016 -0500

    update to lucene 6 jars
2016-03-07 04:12:23 -05:00
Ryan Ernst ed7934ee79 Cli: Simplify test terminals
This commit simplifies and consolidates the two different
implementations of terminals used in tests. There is now a single
MockTerminal which captures output, and allows accessing as one large
string (with unix style \n as newlines), as well as configuring
input.
2016-03-06 11:30:30 -08:00
javanna da19922489 [TEST] adapt test infra to node.client setting removal
No changes were really needed in our test infra as it didn't use `node.client`. Yet it didn't take into account ingest nodes, what we used to call client nodes in InternalTestCluster were actually ingest only nodes, which now become coordinating only nodes.
Also renamed some method to get rid of the node client terminology as much as possible in favour or coordinating only node.
2016-03-05 10:55:19 +01:00
javanna eb941d8005 Remove node.client setting
As discussed in #16565, the node.client setting is an unnecessary shortcut to node.data: false and node.master: false. We have places where we treat nodes with node.client set to true differently compared to master false and data false, which is not correct. Also, with the addition of node.ingest or potentially new roles, it becomes confusing to figure out if a node client should support ingestion or not.

This commit removes the node.client setting in favour being explicit using node.master, node.data and node.ingest instead.
2016-03-04 20:17:34 +01:00
David Pilato 2bb3846d1f Update after review:
* remove `ClusterScope`
* rename `ClusterSettings` to `NodeSettings`
* rename `SettingsProperty` to `Property`
2016-03-04 16:53:24 +01:00
David Pilato c11cf3bf1f Merge branch 'master' into pr/16598-register-filter-settings
# Conflicts:
#	core/src/main/java/org/elasticsearch/common/logging/ESLoggerFactory.java
#	core/src/main/java/org/elasticsearch/common/settings/Setting.java
#	core/src/test/java/org/elasticsearch/common/settings/SettingTests.java
2016-03-04 12:23:10 +01:00
Jason Tedor 166259db64 Use and test relative time in TransportBulkAction
This commit modifies TransportBulkAction to use relative time instead of
absolute time when measuring how long a bulk request took to be
processed, and adds tests for this functionality.

Closes #16916
2016-03-03 11:53:53 -05:00
David Pilato 5fbf1b95dc Merge branch 'master' into pr/16598-register-filter-settings
# Conflicts:
#	core/src/main/java/org/elasticsearch/common/logging/ESLoggerFactory.java
#	core/src/main/java/org/elasticsearch/discovery/DiscoveryService.java
#	core/src/main/java/org/elasticsearch/discovery/DiscoverySettings.java
#	core/src/main/java/org/elasticsearch/http/HttpTransportSettings.java
#	plugins/repository-azure/src/main/java/org/elasticsearch/cloud/azure/storage/AzureStorageService.java
2016-03-02 09:43:53 +01:00
Christoph Büscher aecf51cb42 Merge branch 'master' into feature-suggest-refactoring
Conflicts:
	core/src/main/java/org/elasticsearch/rest/action/search/RestSearchAction.java
2016-03-01 17:20:15 +01:00
Nik Everett c7c8bb357a Merge pull request #16861 from nik9000/reindex_is_ready
Reindex required some parsing changes for search requests to support
differing defaults from the regular search api.
2016-03-01 10:02:48 -05:00
Christoph Büscher a7053afdb9 Merge branch 'master' into feature-suggest-refactoring
Conflicts:
	docs/reference/migration/migrate_5_0.asciidoc
2016-03-01 12:37:05 +01:00
Nik Everett 95cc3e38fc Check test naming conventions on all modules
The big win here is catching tests that are incorrectly named and will
be skipped by gradle, providing a false sense of security.

The whole thing takes about 10 seconds on my Macbook Air, not counting
compiling the test classes, which seems worth it. Because this runs as
a gradle task with propery UP-TO-DATE handling it can be skipped if the
tests haven't been changed which should save some time.

I chose to keep this in test:framework rather than a new subproject of
buildSrc because ESIntegTestCase and doesn't inroduce any additional
dependencies.
2016-02-29 16:31:49 -05:00
Boaz Leskes 195b43d66e Remove DiscoveryService and reduce guice to just Discovery #16821
DiscoveryService was a bridge into the discovery universe. This is unneeded and we can just access discovery directly or do things in a different way.

One of those different ways, is not having a dedicated discovery implementation for each our dicovery plugins but rather reuse ZenDiscovery.

UnicastHostProviders are now classified by discovery type, removing unneeded checks on plugins.

Closes #16821
2016-02-29 20:23:38 +01:00
David Pilato d77daf3861 Use an SettingsProperty.Dynamic for dynamic properties 2016-02-28 11:06:45 +01:00
David Pilato 31b5e0888f Use an SettingsProperty enumSet
Instead of modifying methods each time we need to add a new behavior for settings, we can simply pass `SettingsProperty... properties` instead.

`SettingsProperty` could be defined then:

```
public enum SettingsProperty {
  Filtered,
  Dynamic,
  ClusterScope,
  NodeScope,
  IndexScope
 // HereGoesYours;
}
```

Then in setting code, it become much more flexible.

TODO: Note that we need to validate SettingsProperty which are added to a Setting as some of them might be mutually exclusive.
2016-02-28 00:48:04 +01:00
Ali Beyad a7f6488216 Merge remote-tracking branch 'upstream/master' into
feature-suggest-refactoring
2016-02-26 17:21:12 -05:00
Nik Everett c38119bae9 Merge branch 'master' into feature/reindex 2016-02-26 16:59:54 -05:00
Ryan Ernst 09df7379c5 Merge branch 'master' into dots2 2016-02-25 13:05:27 -08:00
Nik Everett 7c4420bf3d Merge pull request #16807 from nik9000/not_mvn
Remove mention of mvn from java source files
2016-02-25 11:05:46 -05:00
Nik Everett 284f41f7f5 Rename tests.maven to tests.gradle 2016-02-25 10:26:12 -05:00
jaymode c9c4cac0ac Testing: workaround HttpClient issue with IPv6 hostname verification
This commit works around an issue with hostname verification in HttpClient when using IPv6
addresses in URLs. When an IPv6 address is used in a URL it is typically wrapped with square
brackets. The hostname verifier for HttpClient does not recognize these as valid IPv6 addresses
and instead treats them as a DNS name. We wrap the strict hostname verifier for this version
of HttpClient and strip brackets if we need to.

The corresponding issue in HttpClient is https://issues.apache.org/jira/browse/HTTPCLIENT-1698
but the fix has not been released yet in a stable version.
2016-02-25 08:04:41 -05:00
Nik Everett 49f704fd6a Remove mention of mvn from ramaining java 2016-02-24 19:54:47 -08:00
Ryan Ernst 551e6bd66f Mapping: Moved dynamic field handling in doc parsing to end of parsing
Currently dynamic mappings propgate through call semantics, where deeper
dynamic mappings are merged into higher level mappings through
return values of recursive method calls. This makese it tricky
to handle multiple updates in the same method, for example when
trying to create parent object mappers dynamically for a field name
that contains dots.

This change makes the api for adding mappers a simple list
of new mappers, and moves construction of the root level mapping
update to the end of doc parsing.
2016-02-24 12:33:39 -08:00
Christoph Büscher d888b5c267 Merge branch 'master' into feature-suggest-refactoring 2016-02-22 11:22:28 -08:00
Simon Willnauer 387f0473dc Assert that we can write in all data-path on startup
Today we might start a node and some of the paths might not have the
required permissions. This commit goes through all data directories as
well as index, shard and state directories and ensures we have write access.
To make this work across all OS etc. we are trying to write a real file
and remove it again in each of those directories
2016-02-22 10:32:29 -08:00
Simon Willnauer 4f7dd1284f Remove SNAPSHOT notion from code
Today we have the notion of a snapshot inside Version.java which makes
releasing complicated since to do a release Version.java must be changed.
This commit removes all notions of snapshot from the code and allows to
switch between snapshot and release build by specifying a system property on
the build. For instance running:

```
gradle run -Dbuild.snapshot=false
```

will build and package a release build while the default always
builds snapshots. Calls to the main rest action will still get the snapshot
information rendered out with the response.
2016-02-20 16:06:26 -08:00
Ali Beyad 5320d538f1 Merge remote-tracking branch 'upstream/master' into feature-suggest-refactoring 2016-02-15 11:03:33 -05:00
Boaz Leskes 63ada9882e make index counter checking use assertBusy 2016-02-13 14:14:10 +01:00
Ali Beyad beea413820 Merge remote-tracking branch 'upstream/master' into feature-suggest-refactoring 2016-02-12 11:54:44 -05:00
Boaz Leskes 4bb5b4100d merge from master 2016-02-12 15:53:31 +01:00
Colin Goodheart-Smithe ed3f7903f4 Merge branch 'master' into feature/aggs-refactoring 2016-02-12 09:48:59 +00:00
Nik Everett 821a20f582 Merge branch 'master' into feature/reindex 2016-02-11 17:41:05 -05:00
Simon Willnauer d538dd64c2 Refactor IndicesRequestCache to make it testable.
This commit moves IndicesRequestCache into o.e.indics and makes all API in this
class package private. All references to SearchReqeust, SearchContext etc. have been factored
out and relevant glue code has been added to IndicesService. The IndicesRequestCache is not a
simple class without any hard dependencies on ThreadPool nor SearchService or IndexShard. This now
allows to add unittests.
This commit also removes two settings `indices.requests.cache.clean_interval` and `indices.fielddata.cache.clean_interval`
in favor of `indices.cache.clean_interval` which cleans both caches.
2016-02-11 16:51:44 +01:00
Colin Goodheart-Smithe 4a9c84d86d Merge branch 'master' into feature/aggs-refactoring
# Conflicts:
#	core/src/main/java/org/elasticsearch/search/SearchService.java
#	test/framework/src/main/java/org/elasticsearch/search/MockSearchService.java
2016-02-11 14:04:04 +00:00
Simon Willnauer 7835525f45 Move IndicesQueryCache and IndicesRequestCache into IndicesService
this is a minor cleanup that detaches `IndicesRequestCache` and `IndicesQueryCache`
from guice and moves it into `IndicesService`. It also decouples the `IndexShard` and `IndexService`
from these caches which are unnecessary dependencies.
2016-02-11 12:02:21 +01:00
Areek Zillur 2038429f63 initial refactoring of completion suggester 2016-02-10 16:21:55 -05:00
Nik Everett 7ca2651a7b Wrap another lone line 2016-02-10 15:41:40 -05:00
Nik Everett 0da30d5eae Merge branch 'master' into feature/reindex 2016-02-10 14:07:41 -05:00
Colin Goodheart-Smithe 1f760bd1bd Merge branch 'master' into feature/aggs-refactoring 2016-02-10 12:16:26 +00:00
Colin Goodheart-Smithe 54e2f31189 Parse Aggregations on Coordinating node 2016-02-10 10:00:16 +00:00
Dongjoon Hyun 1d26b4de22 Fix typos in the names of some variables
Closes #16566
2016-02-09 20:35:42 -05:00
Colin Goodheart-Smithe 80bbb4a385 Merge branch 'master' into feature/aggs-refactoring
# Conflicts:
#	core/src/main/java/org/elasticsearch/percolator/PercolateContext.java
#	core/src/main/java/org/elasticsearch/search/aggregations/bucket/filter/FilterParser.java
#	core/src/main/java/org/elasticsearch/search/aggregations/bucket/filters/FiltersParser.java
#	core/src/main/java/org/elasticsearch/search/aggregations/bucket/significant/SignificantTermsParametersParser.java
#	core/src/main/java/org/elasticsearch/search/internal/DefaultSearchContext.java
#	core/src/test/java/org/elasticsearch/search/aggregations/bucket/geogrid/GeoHashGridParserTests.java
#	test/framework/src/main/java/org/elasticsearch/test/TestSearchContext.java
2016-02-09 10:37:20 +00:00
Dongjoon Hyun 3bad16c1ca Rename variables.
This PR renames the following three variables to fix a typo `settting` into `setting`.
 * Rename a static class member:
   INDEX_TRANSLOG_FLUSH_THRESHOLD_SIZE_SETTTING -> INDEX_TRANSLOG_FLUSH_THRESHOLD_SIZE_SETTING
 * Rename a parameter: aSettting --> aSetting
 * Rename a local variable: indexSetttings -> indexSettings
2016-02-08 23:13:33 -08:00
Colin Goodheart-Smithe 02ecfd6279 Merge branch 'master' into feature/aggs-refactoring 2016-02-08 15:41:02 +00:00
Simon Willnauer 96bcb47fc9 Detach QueryShardContext from IndexShard and remove obsolete threadlocals
IndexShard currently holds an arbitraritly used `getQueryShardContext` that comes
out of a ThreadLocal. It's usage is undefined and arbitraty since there is also
such a method with different semantics on `IndexService` This commit removes the threadLocal on
IndexShard as well as on the context itself. It's types are now a member and the QueryShardContext
lifecycle is managed byt SearchContext which passes the types on from the SearchRequest.
2016-02-08 15:05:03 +01:00
Simon Willnauer b185209b63 Make IndicesWarmer a private class of IndexService
There is no need for IndicesWarmer to be a global accessible class. All it needs
access to is inside IndexService. It also doesn't need to be mutable once it's not a per node
instance. This commit move IndicesWarmer to IndexWarmer and makes the default impls like field data and
norms warming an impl detail. Also the IndexShard doesn't depend on this class anymore, instead it accepts
an Engine.Warmer as a ctor argument which delegates to the actual warmer from the index.
2016-02-05 15:24:17 +01:00
Colin Goodheart-Smithe f06f17f328 Merge branch 'master' into feature/aggs-refactoring
# Conflicts:
#	core/src/main/java/org/elasticsearch/common/io/stream/StreamInput.java
#	core/src/main/java/org/elasticsearch/common/io/stream/StreamOutput.java
2016-02-05 09:22:14 +00:00
Simon Willnauer 00be9e58f2 Simplify IndicesFieldDataCache and detach from guice
Indices level field data cacheing belongs into IndicesService and doesn't need to be
wired by guice. This commit also moves the async cache refresh out of the class into
IndicesService such that threadpool dependencies are removed and testing / creation becomes
simpler.
2016-02-05 09:43:33 +01:00
Yannick Welsch 4937531a17 Remove obsolete version in ShardRouting
Closes #16243
2016-02-04 15:50:25 +01:00
Nik Everett bb4d8b79fe Merge branch 'master' into feature/reindex 2016-02-04 09:23:49 -05:00
Ryan Ernst 5c0f09b03a Merge pull request #16443 from rjernst/cli_terminal
CliTool: Cleanup and document Terminal
2016-02-04 02:05:28 -08:00
Ryan Ernst e97345984d Remove format args from readText and readSecret 2016-02-04 02:03:17 -08:00
Simon Willnauer cc520b20e1 Merge branch 'master' into fix_settings_filter 2016-02-04 09:28:41 +01:00
Ryan Ernst b80200a363 CliTool: Cleanup and document Terminal
This change documents the Terminal abstraction that cli tools use, as
well as simplifies the api to be a minimal set of methods to interact
with a terminal.
2016-02-03 22:22:56 -08:00
Nik Everett a2f07679fd Add task status
Implements a simple task status for superclasses of ReplicationRequest to
show how you can do use the status.
2016-02-03 18:21:42 -05:00
Simon Willnauer 686627be5a Move ModuleTestCase into framework 2016-02-03 21:04:44 +01:00
Colin Goodheart-Smithe 5d9d91b761 Merge branch 'master' into feature/aggs-refactoring 2016-02-03 14:45:16 +00:00
Simon Willnauer 4a4e523357 Merge branch 'master' into make_settings_strict 2016-02-03 11:34:12 +01:00
Simon Willnauer ed492901ed Apply feedback from @bleskes and add fix some minor issues 2016-02-03 10:44:06 +01:00
Tanguy Leroux 865bbc2096 Remove string formatting from Terminal print methods
This can be trappy and wrong formating strings can throws format exceptions and hide the real message.
2016-02-02 19:57:16 +01:00
Yannick Welsch cf28d62fc4 [TEST] Fail test if dummy doc is not found
Reverts back 7d3da91 after fix in #15900

Closes #8706
2016-02-02 16:21:38 +01:00
Jason Tedor 69a3f7f590 Add random positive time value convenience method
This commit adds a convenience method for producing random positive time
values which can be useful for places where non-negative and non-zero
time values are expected.
2016-02-02 08:49:47 -05:00
Simon Willnauer 818a9eefb2 Make settings validation strict
This commit enableds strict settings validation on node startup. All settings
passed to elasticsearch either through system properties, yaml files or any other
way to pass settings must be registered and valid. Settings that are unknown ie. due to
typos or due to deprecation or removal will cause the node to NOT start up. Plugins
have to declare all their settings on the `SettingsModule#registerSetting` and settings for
plugins that are not installed must be removed.

This commit also removes the ability to specify the nodes name via `-Des.name` or just `name` in the
configuration files. The node name must be prefixed with the node prexif like `node.name: Boom`. Left over
usage of `name` will also cause startup to fail.
2016-02-02 11:32:44 +01:00
Yannick Welsch 10b5ffcda5 Add proper handoff between old and new copy of relocating primary shard
When primary relocation completes, a cluster state is propagated that deactivates the old primary and marks the new primary as active.
As cluster state changes are not applied synchronously on all nodes, there can be a time interval where the relocation target has processed
the cluster state and believes to be the active primary and the relocation source has not yet processed the cluster state update and
still believes itself to be the active primary. This commit ensures that, before completing the relocation, the reloction source deactivates
writing to its store and delegates requests to the relocation target.

Closes #15900
2016-02-02 09:46:55 +01:00
Nik Everett c1d4478067 Merge branch 'master' into feature/reindex 2016-02-01 18:33:09 -05:00
Ryan Ernst 01b80b100e Merge pull request #16333 from rjernst/single_node_static
Make single node utility methods non-static
2016-02-01 08:46:54 -08:00
Ryan Ernst 661e3c99f7 Merge pull request #16336 from rjernst/demangle
Reduce complexity of plugin cli
2016-02-01 07:20:04 -08:00
Colin Goodheart-Smithe a241983c9d Merge branch 'master' into feature/aggs-refactoring
# Conflicts:
#	core/src/test/java/org/elasticsearch/search/aggregations/AggregationsBinaryIT.java
#	core/src/test/java/org/elasticsearch/search/aggregations/bucket/FilterIT.java
#	core/src/test/java/org/elasticsearch/search/aggregations/bucket/FiltersIT.java
#	core/src/test/java/org/elasticsearch/search/aggregations/bucket/GeoDistanceIT.java
#	core/src/test/java/org/elasticsearch/search/aggregations/bucket/NestedIT.java
#	core/src/test/java/org/elasticsearch/search/aggregations/bucket/SamplerIT.java
#	core/src/test/java/org/elasticsearch/search/aggregations/pipeline/DerivativeIT.java
#	modules/lang-groovy/src/test/java/org/elasticsearch/messy/tests/BucketSelectorTests.java
#	modules/lang-groovy/src/test/java/org/elasticsearch/messy/tests/TDigestPercentileRanksTests.java
2016-02-01 13:38:33 +00:00
Boaz Leskes df80e8f215 Migrate the rest of NettyTransport settings to the new infra
Also does some consistency clean up, renaming trasnport.netty.* settings to transport.*

Closes #16307
2016-02-01 10:26:52 +01:00
Simon Willnauer af0e40ec7d Merge pull request #16316 from s1monw/isseus/13685_2nd_try
Ensure all resoruces are closed on Node#close()
2016-02-01 10:17:57 +01:00
Ryan Ernst a052dfeb38 Plugins: Reduce complexity of plugin cli
The plugin cli currently is extremely lenient, allowing most errors to
simply be logged. This can lead to either corrupt installations (eg
partially installed plugins), or confused users.

This change rewrites the plugin cli to have almost no leniency.
Unfortunately it was not possible to remove all leniency, due in
particular to how config files are handled.

The following functionality was simplified:
* The format of the name argument to install a plugin is now an official
  plugin name, maven coordinates, or a URL.
* Checksum files are required, and only checked, for official plugins
  and maven plugins. Checksums are also only SHA1.
* Downloading no longer uses a separate thread, and no longer has a timeout.
* Installation, and removal, attempts to be atomic. This only truly works
  when no config or bin files exist.
* config and bin directories are verified before copying is attempted.
* Permissions and user/group are no longer set on config and bin files.
  We rely on the users umask.
* config and bin directories must only contain files, no subdirectories.
* The code is reorganized so each command is a separate class. These
  classes already existed, but were embedded in the plugin cli class, as
  an extra layer between the cli code and the code running for each command.
2016-01-31 20:30:45 -08:00
Ryan Ernst a398ba839c Tests: Make single node utility methods non-static
With the recent change to allow ESSingleNodeTestCase to specify plugins
and Version for the node, node creation became lazy, where it now
happens before the first test to run. However, there were many static
methods on ESSingleNodeTestCase that still try to access the node. This
change makes those methods non-static.
2016-01-31 09:58:41 -08:00
Jason Tedor 105411060c Uppercase ells ('L') in long literals
This commit removes and forbids the use of lowercase ells ('l') in long
literals because they are often hard to distinguish from the digit
representing one ('1').

Closes #16329
2016-01-30 22:16:02 -05:00
Ryan Ernst 23f70296fb Merge pull request #16304 from rjernst/expect_throws
Tests: Add expectThrows utility to base test case
2016-01-29 09:01:19 -08:00
Ryan Ernst 90485d0635 Remove unnecessary unchecked cast 2016-01-29 09:00:27 -08:00
Simon Willnauer 15841081f6 Disambiguate TestCluster implementation since Client is now also Closeable and if we call IOUtils it might interpret it as a Iterable<Closeable> 2016-01-29 16:51:06 +01:00
Simon Willnauer 0bcee2df37 Use IOUtils#close() where needed 2016-01-29 16:30:50 +01:00
Simon Willnauer 87737d3e02 Ensure all resoruces are closed on Node#close()
We are leaking all kinds of resources if something during Node#close() barfs.
This commit cuts over to a list of closeables to release resources that
also closed remaining services if one or more services fail to close.

Closes #13685
2016-01-29 16:18:21 +01:00
Colin Goodheart-Smithe 859f9e69b7 Merge branch 'master' into feature/aggs-refactoring
# Conflicts:
#	core/src/main/java/org/elasticsearch/action/search/SearchRequestBuilder.java
#	core/src/main/java/org/elasticsearch/percolator/PercolateContext.java
#	core/src/main/java/org/elasticsearch/search/aggregations/bucket/significant/heuristics/ScriptHeuristic.java
#	core/src/main/java/org/elasticsearch/search/internal/DefaultSearchContext.java
#	core/src/test/java/org/elasticsearch/search/aggregations/bucket/significant/SignificanceHeuristicTests.java
2016-01-29 11:58:19 +00:00
Martijn van Groningen 6c34452409 inner hits: Query and top level inner hit definitions shouldn't overwrite each other.
Closes #16218
2016-01-29 10:52:04 +01:00
Ryan Ernst ccfe1bd2e0 Tests: Add expectThrows utility to base test case
In junit5, a neat assertion method is added which makes testing expected
failures a little more straightforward. The block of code that is
expected to throw is passed in with a lambda expression, and the caught
exception returned for inspection.

This change adds an implementation of expectThrows to ESTestCase. When
junit5 is eventually releassed and we switch to it, we can remove.
2016-01-29 01:51:29 -08:00
Igor Motov e035dabd4d Extend tracking of parent tasks to master node, replication and broadcast actions
Now MasterNodeOperations, ReplicationAllShards, ReplicationSingleShard,  BroadcastReplication and BroadcastByNode actions keep track of their parent tasks.
2016-01-28 12:13:36 -05:00
Boaz Leskes 68070074da Begin migration of NettyTransport setting to the new platform
Note for many of the settings we had a netty specific override to a generic transport setting. I have removed those as they are not needed imo (ex. "transport.connections_per_node.recovery" was overriden by "transport.netty.connections_per_node.recovery", which I removed). The netty prefix was kept for settings that are tied to the netty universe - ex. "transport.netty.max_cumulation_buffer_capacity"

Closes #16200
2016-01-28 14:56:55 +01:00
Boaz Leskes 2a137b5548 Make index uuid available in Index, ShardRouting & ShardId
In the early days Elasticsearch used to use the index name as the index identity. Around 1.0.0 we introduced a unique index uuid which is stored in the index setting. Since then we used that uuid in a few places but it is by far not the main identifier when working with indices, partially because it's not always readily available in all places.

This PR start to make a move in the direction of using uuids instead of name by making sure that the uuid is available on the Index class (currently just a wrapper around the name) and as such also available via ShardRouting and ShardId.

Note that this is by no means an attempt to do the right thing with the uuid in all places. In almost all places it falls back to the name based comparison that was done before. It is meant as a first step towards slowly improving the situation.

Closes #16217
2016-01-28 08:40:10 +01:00
Simon Willnauer cefa5da08c Merge branch 'master' into more_settings 2016-01-27 17:45:13 +01:00
Simon Willnauer 7ff99eb89d Merge branch 'master' into trash_context_and_headers
if (name == 'expamle-fixtures') return
2016-01-27 16:33:50 +01:00
Simon Willnauer bbba1e5d7f Convert `config.ignore_system_properties` to new settings infra 2016-01-27 16:10:06 +01:00
Simon Willnauer 1c1898cf42 Convert `cluster.name` to the new settings infra 2016-01-27 16:04:45 +01:00
Colin Goodheart-Smithe 187009c12c Merge branch 'master' into feature/aggs-refactoring 2016-01-27 14:54:12 +00:00
Simon Willnauer 1df7d4d5b5 Merge pull request #16257 from s1monw/no_fsync_on_every_operation
Remove the ability to fsync on every operation and only schedule fsync task if really needed
2016-01-27 15:48:30 +01:00
Simon Willnauer 84ce9f3618 Remove the ability to fsync on every operation and only schedule fsync task if really needed
This commit limits the `index.translog.sync_interval` to a value not less than `100ms` and
removes the support for fsync on every operation which used to be enabled if `index.translog.sync_interval` was set to `0s`
Now this pr also only schedules an async fsync if the durability is set to `async`. By default not async task is scheduled.

Closes #16152
2016-01-27 12:28:38 +01:00
Jason Tedor 284cc3a048 Script mode settings as booleans
This commit modifies the accept values for script mode settings from
"on", "off", and "sandbox" to "true", "false", and "sandbox".
2016-01-27 06:26:58 -05:00
Jason Tedor 9944573449 Rename methods on ScriptEngineService
This commit method renames the ScriptEngineService interface methods
types, extensions, and sandboxed to getTypes, getExtensions, and
isSandboxed, respectively.
2016-01-27 06:26:04 -05:00
Jason Tedor 087e55cc51 Script mode settings
This commit converts the script mode settings to the new settings
infrastructure. This is a major refactoring of the handling of script
mode settings. This refactoring is necessary because these settings are
determined at runtime based on the registered script engines and the
registered script contexts.
2016-01-27 06:26:04 -05:00
Jason Tedor eb6aaefae5 Script cache expiration setting
This commit converts the script cache expiration setting
"script.cache.expire" to the new settings infrastructure.
2016-01-27 06:26:04 -05:00
Simon Willnauer 71c3e57aee Merge branch 'master' into trash_context_and_headers 2016-01-27 11:42:42 +01:00
Simon Willnauer 0404db65e3 Merge branch 'master' into convert_some_settings 2016-01-27 11:31:00 +01:00
Simon Willnauer 9cb9bdb736 Merge pull request #16232 from s1monw/use_version_id_in_settings
Use Version#id to parse version from settings.
2016-01-27 11:10:44 +01:00
Jim Ferenczi aea7660e37 Add search_after parameter in the Search API.
The search_after parameter provides a way to efficiently paginate from one page to the next. This parameter accepts an array of sort values, those values are then used by the searcher to sort the top hits from the first document that is greater to the sort values.
This parameter must be used in conjunction with the sort parameter, it must contain exactly the same number of values than the number of fields to sort on.

NOTE: A field with one unique value per document should be used as the last element of the sort specification. Otherwise the sort order for documents that have the same sort values would be undefined. The recommended way is to use the field `_uuid` which is certain to contain one unique value for each document.

Fixes #8192
2016-01-27 09:42:58 +01:00
Simon Willnauer daadf8e04f Convert processors to new settings infrastructure 2016-01-26 17:06:35 +01:00
javanna 9cefeabd9e [TEST] make sure node.mode is used in TransportClientFactory 2016-01-26 15:38:16 +01:00
Simon Willnauer cf358e426c Use Version#id to parse version from settings.
When a Version is passed to `Settings#put(String, Version)` it's id is used as
an integer which should also be used to deserialize it on the consumer end.
Today AssertinLocalTransport expects Version#toString() to be used which can lead
to subtile bugs in tests.
2016-01-26 14:53:06 +01:00
javanna 61630c2b27 migrate node.local and node.mode to new Setting infra 2016-01-26 14:40:46 +01:00
javanna 80b74a4885 migrate node.client, node.data and node.master to new Setting infra 2016-01-26 14:40:13 +01:00
Colin Goodheart-Smithe 3b35754f59 Merge branch 'master' into feature/aggs-refactoring
# Conflicts:
#	core/src/test/java/org/elasticsearch/percolator/PercolateDocumentParserTests.java
2016-01-26 13:17:53 +00:00
javanna 36d98478bf Merge branch 'master' into feature/ingest 2016-01-25 18:01:09 +01:00
Colin Goodheart-Smithe d63b8a25f8 Merge branch 'master' into feature/aggs-refactoring 2016-01-25 10:42:59 +00:00
Colin Goodheart-Smithe cd8320b171 Merge branch 'master' into feature/aggs-refactoring
# Conflicts:
#	core/src/main/java/org/elasticsearch/search/aggregations/bucket/filter/FilterAggregator.java
#	core/src/main/java/org/elasticsearch/search/aggregations/bucket/filters/FiltersAggregator.java
#	core/src/main/java/org/elasticsearch/search/SearchModule.java
2016-01-25 10:42:20 +00:00
Ryan Ernst b349746bbe Fix repro line for tests to use language tag
See https://issues.apache.org/jira/browse/LUCENE-6978

Also fixed date histogram test to use its version constant as a string
setting.
2016-01-22 22:54:49 -08:00
Boaz Leskes ec31feca93 Settings: migrate network service to the new infra
This commit migrates all the settings under network service to the new settings infra.

It also adds some chaining utils to make fall back settings slightly less verbose.

Breaking (but I think acceptable)  - network.tcp.no_delay and network.tcp.keep_alive used to accept the value `default` which make us not set them at all on netty. Our default was true so we weren't using this feature. I removed it and now we only accept a true boolean.
2016-01-22 16:59:07 +01:00
markharwood d371ef35f4 Settings - change over to o.e.common.settings.Setting for http settings:
http.cors.allow-credentials
http.cors.enabled
http.detailed_errors.enabled
http.enabled
http.pipelining
http.cors.max-age
2016-01-22 15:13:11 +00:00
Simon Willnauer 9b3559f473 Remove `test.cluster.node.seed` and special case `tests.portsfile`
* `test.cluster.node.seed` was only used in one place where it wasn't adding value and was now replaced with a constant
 * `tests.portsfile` is now an official setting and has been renamed to `node.portsfile`

this commit also convert `search.default_keep_alive` and `search.keep_alive_interval` to the new settings infrastrucutre
2016-01-22 15:47:32 +01:00
Yannick Welsch 296b48b9d1 Move discovery.* settings to new setting infrastructure
Closes #16182
2016-01-22 15:35:00 +01:00
Daniel Mitterdorfer 9a1fdb6987 Merge remote-tracking branch 'danielmitterdorfer/convert-path-settings' 2016-01-22 15:18:18 +01:00
Daniel Mitterdorfer e9bb3d31a3 Convert "path.*" and "pidfile" to new settings infra 2016-01-22 15:14:13 +01:00
Jim Ferenczi 9c286b601e Convert "indices.*" settings to new infra. 2016-01-22 13:51:06 +01:00
Simon Willnauer 20c3cbf98d [TEST] only pass valid random settings 2016-01-22 11:42:18 +01:00
Simon Willnauer 1fb2f22f32 Convert to the new settings infra 2016-01-22 11:25:26 +01:00
Ryan Ernst df24019261 Merge pull request #16038 from rjernst/remove_site_plugin
Plugins: Remove site plugins
2016-01-21 12:32:22 -08:00
Martijn van Groningen 44465c94f8 Merge remote-tracking branch 'es/master' into feature/ingest 2016-01-21 11:46:27 +01:00
Simon Willnauer 5374bd8e34 Merge branch 'master' into trash_context_and_headers 2016-01-20 09:40:03 +01:00
Martijn van Groningen 602a0f183e Merge remote-tracking branch 'es/master' into feature/ingest 2016-01-19 22:01:38 +01:00
Simon Willnauer 40783b5d32 delete dead code 2016-01-19 21:06:23 +01:00
Simon Willnauer 3d0cedbabb Merge branch 'master' into trash_context_and_headers 2016-01-19 20:47:43 +01:00
Simon Willnauer eb498282dd Add infrastructure to wrap/filter all clients exposed by the test clusters.
This commit allows an integ test to wrap all clients that are exposed by the
InternalTestCluster which is useful for request interception or to add general headers
to request or to intercept client to server call even if the client calls are done from within
the test framework.
2016-01-19 20:12:47 +01:00
Nik Everett 0d6d77328d Merge branch 'master' into feature/reindex 2016-01-19 13:27:24 -05:00
Simon Willnauer fa0e1488aa Merge branch 'master' into new_index_settings 2016-01-19 10:50:11 +01:00
Jason Tedor 296ab3ec16 Fix transport exceptions for shard state actions
This commit fixes an issue in the handling of TransportExceptions in
ShardStateAction. There were two cases not being handled correctly.
 - when the local node is shutting down, handlers will be notified with
   a TransportException with a message starting "transport stopped"
 - when the remote node disconnects, handlers will be notified with a
   NodeDisconnectedException

In both of these cases, the cause of the exception will be null and this
was incorrectly being handled. The first case can passed to the listener
like any other critical non-channel failure, and the second case can be
handled by modifying the logic for detecting master channel exceptions.

There was a third case of NodeNotConnectedException that was not being
treated as a master channel exception but should be.

This commit adds an integration test that simulates the handling of a
shard failure request during a network partition. By isolating the
master from the cluster while a shard failed request is in flight, this
test simulates that we wait until a new master is elected and then retry
sending that shard failed request to the newly elected master.

This commit adds methods to CapturingTransport to separate local and
remote transport exceptions. The motivation for this change is that
local transport exceptions are delivered to listeners (usually, but not
always) wrapped in SendRequestTransportException while remote transport
exceptions are delivered to listeners wrapped in
RemoteTransportException. By making this distinction clear in the
CapturingTransport, this makes it less likely that tests will make
incorrect assumptions about the exceptions coming out of the transport
layer to listeners.

Closes #16057
2016-01-19 10:43:50 +01:00
Simon Willnauer fbfa9f4925 Merge branch 'master' into new_index_settings 2016-01-19 10:13:48 +01:00
Simon Willnauer b73c737608 apply feedback from @nik9000 2016-01-19 09:54:00 +01:00
Ryan Ernst ef4f0a8699 Test: Make rest test framework accept http directly for the test cluster
The rest test framework, because it used to be tightly integrated with
ESIntegTestCase, currently expects the addresses for the test cluster to
be passed using the transport protocol port. However, it only uses this
to then find the http address.

This change makes ESRestTestCase extend from ESTestCase instead of
ESIntegTestCase, and changes the sysprop used to tests.rest.cluster,
which now takes the http address.

closes #15459
2016-01-18 16:44:14 -08:00
Nik Everett 093600cd63 Cleanup ContextAndHeaderTransportTests
We have two similar tests with the same name, ContextAndHeaderTransportTests.
They shared lots of common code so I extracted much of it into
ActionRecordingPlugin, a plugin which records all action requests for later
inspection.

I also removed all the warnings from both tests. That made lang-mustache
compile cleanly without any custom -Xlint so I removed those. To remove
the warnings I had to add type parameters to ActionFilter which seemed
like a good idea anyway.
2016-01-18 16:14:58 -05:00
Simon Willnauer c2e8ba8970 Register index.version_created on unittest that set it 2016-01-18 15:04:35 +01:00
Simon Willnauer dce29d29c0 Normalize template index settings before validating 2016-01-18 14:39:21 +01:00
Martijn van Groningen 62089539db s/processorId/processorTag 2016-01-18 13:11:46 +01:00
Simon Willnauer 9562fb76bc expose default settings via rest API 2016-01-18 12:48:47 +01:00
Simon Willnauer 7925e2ef84 convert IndexModule settings 2016-01-18 09:23:35 +01:00
Simon Willnauer 79f4697f3e Register MockFSDirectoryService settings 2016-01-18 09:23:34 +01:00
Simon Willnauer a8eedd0457 fix all kinds of crazy test failures 2016-01-18 09:23:34 +01:00
Simon Willnauer 8a4fca2947 make all unittests pass 2016-01-18 09:23:33 +01:00
Simon Willnauer 04045a072f First steps of integrating converted settings into the cluster/index settings infrastructure
The old infa has been removed in this commit such that nothing uses `DynamicSettings` anymore
and all index-scoped settings require to be registered before the node has fully started up.
2016-01-18 09:23:33 +01:00
Simon Willnauer 211f80fcad first cut at integrating new settings infra 2016-01-18 09:23:33 +01:00
Simon Willnauer abfef7aa73 convert index allocation filtering 2016-01-18 09:23:31 +01:00
Britta Weber a9ff6bc405 convert compound_format 2016-01-18 09:23:31 +01:00
Britta Weber c4a019d93d convert translog.flush_threshold_size 2016-01-18 09:23:31 +01:00
Simon Willnauer 2e1bc1ea44 cut over index.routing.rebalance.enable and index.routing.allocation.enable 2016-01-18 09:23:30 +01:00
Simon Willnauer 0bef73269a cut over index.unassigned.node_left.delayed_timeout 2016-01-18 09:23:30 +01:00
Simon Willnauer b7b104a585 convert index.translog.durability 2016-01-18 09:23:29 +01:00
Simon Willnauer da0424c75f Convert index level setting to the new setting infrastrucutre
this is an initial commit of cutting over simple string key based settings
to a more contained scoped settings infrastructure.
2016-01-18 09:23:29 +01:00
Ryan Ernst 3b78267c71 Plugins: Remove site plugins
Site plugins used to be used for things like kibana and marvel, but
there is no longer a need since kibana (and marvel as a kibana plugin)
uses node.js. This change removes site plugins, as well as the flag for
jvm plugins. Now all plugins are jvm plugins.
2016-01-16 22:45:37 -08:00
Tal Levy 9f48df9736 Add on_failure support for verbose _simulate execution and introduce optional processor_tag to Processors 2016-01-15 14:56:20 -08:00
Nik Everett 645b053061 Merge branch 'master' into feature/reindex 2016-01-15 14:04:37 -05:00
javanna 9c06736dbd Merge branch 'master' into feature/ingest 2016-01-15 10:11:56 +01:00
Colin Goodheart-Smithe e6f9cbce8f Merge branch 'master' into feature/aggs-refactoring
# Conflicts:
#	core/src/main/java/org/elasticsearch/common/io/stream/StreamInput.java
#	core/src/main/java/org/elasticsearch/common/io/stream/StreamOutput.java
2016-01-14 15:00:26 +00:00
Simon Willnauer aaf7e55c5f Remove index.flush_on_close entirely
This undocumented setting was mainly used for testing and a safety net for
a new flushOnClose feature back in 1.x. We can now safely remove this setting.
The test usage now uses a mock plugin setting to achive the same.
2016-01-14 10:34:58 +01:00
Simon Willnauer 4526f117a2 Randomize which MockPlugins are used for IT 2016-01-13 23:00:59 +01:00
Simon Willnauer 437d7c179a Move all dynamic settings and their config classes to the index level
Today we maintain a lot of settings on the shard level which are all index level settings.
In order to cut over to the new settings API where we register update listener we have to move
all of them on to the index level otherwise we need a way to un-register listeners which is error-prone
and requires additional handling when shards are closed. It's simpler and also more accurate to handle all of
them on the index level where we can trash the entire registry for update listener once the index goes out of scope.
2016-01-13 14:25:37 +01:00
Simon Willnauer 574d1b35b3 Replace ContextAndHeaders with a ThreadPool based ThreadLocal implementation
ContextAndHeaders has a massive impact on the core infrastructure since it has to
be manually passed on to all relevant places across threads/network calls etc. For the same reason
it's also very error prone and easily forgotten on potentially relevant APIs.

The new ThreadContext is associated with a ThreadPool (node or transport client) and ensures that
headers and context registered on a current thread are inherited to new threads spawned, send across
the network to be deserialized on the receiver end as well as restored on the response handling thread
once the response is received.
2016-01-13 11:53:32 +01:00
javanna ea8065aa3d Merge branch 'master' into feature/ingest 2016-01-12 18:28:42 +01:00
Nik Everett 78dc1c32de Merge pull request #15899 from nik9000/xlint_serial
Ban Serializable
2016-01-12 10:04:50 -05:00