Commit Graph

352 Commits

Author SHA1 Message Date
Daniel Mitterdorfer 3688629e11 Adjust line-length of transport related classes to coding standard 2016-04-15 10:12:24 +02:00
Ali Beyad b87fd54ba9 Improvements to the IndicesService class
This commit contains the following improvements/fixes:
  1. Renaming method names and variables to better reflect the purpose
of the method and the semantics of the variable.
  2. For deleting indexes, replace the closed parameter passed to the
delete index/store methods with obtaining the index's state from the
IndexSettings that is already passed in.
  3. Added tests to the IndexWithShadowReplicaIT suite, some of which
show issues in the shadow replica delete process that are captured in
Github issue 17695.

Closes #17638
2016-04-14 11:14:02 -04:00
Nik Everett 64f5a4f848 Stop map collisions on FiltersTests
Adds randomUnique to generate unique things and uses it to make unique
keys.

The offending seed was 81AE616FEAD10F17.
2016-04-13 08:35:45 -04:00
Daniel Mitterdorfer 117bc68af3 Limit request size on HTTP level
With this commit we limit the size of all in-flight requests on
HTTP level. The size is guarded by the same circuit breaker that
is also used on transport level. Similarly, the size that is used
is HTTP content length.

Relates #16011
2016-04-13 09:58:08 +02:00
Daniel Mitterdorfer 52b2016447 Limit request size on transport level
With this commit we limit the size of all in-flight requests on
transport level. The size is guarded by a circuit breaker and is
based on the content size of each request.

By default we use 100% of available heap meaning that the parent
circuit breaker will limit the maximum available size. This value
can be changed by adjusting the setting

network.breaker.inflight_requests.limit

Relates #16011
2016-04-13 09:54:59 +02:00
Adrien Grand 226644ea2c Do not assume term queries use the inverted index. #17532
We have a couple places in the code base that assume that search is always done
on the inverted index. However with the new points API in Lucene 6, this is not
true anymore. This commit makes MappedFieldType.indexedValueForSearch protected
and fixes call sites to keep working for field types that use the inverted
index and either work differently ar throw an exception otherwise. For instance,
it will still be possible to run cross_fields multi match queries on numeric
fields, but the score contributions will not be blended as well as before, and
significant terms aggregations on long terms will not be possible anymore since
points do not record document frequencies.
2016-04-12 09:47:20 +02:00
Adrien Grand 0eb1a816c8 Allow the query cache to be disabled. #16268
This replaces the internal `index.queries.cache.type` setting with
a new `index.queries.cache.enabled` setting, which is documented.

Closes #15802
2016-04-11 18:06:16 +02:00
Alexander Reelsen da19ddf3e6 Ingest Attachment: Allow to prevent base64 conversions by using raw bytes (#16601)
CBOR is natively supported in Elasticsearch and allows for byte arrays.
This means, that by using CBOR the user can prevent base64 conversions
for the data being sent back and forth.

This PR adds support to extract data from a byte array in addition to
a string. This also required to add a ByteArrayValueSource class.
2016-04-11 14:14:56 +02:00
David Pilato 1e346d1ac1 Merge branch 'fix/17625-close-ingest-factory' 2016-04-11 10:00:19 +02:00
Nik Everett 525ce40d1c Give SearchContext a toString
and move the string capturing to capture time.
2016-04-10 20:55:31 -04:00
Nik Everett ac94e5f287 Provide more information about open contexts
Sometimes we get a test failure caused by search contexts left open.
The tests include a stack trace of the call that opened the context
but nothing else about the context. This adds more information about
the context that has been left open like what query it was running,
what shard it targeted, and whether or not it was a scroll.

Relates to #17582
2016-04-10 20:55:31 -04:00
David Pilato 24f48b86b5 Update after review and add a Test 2016-04-09 13:14:25 +02:00
Adrien Grand 42526ac28e Remove Settings.settingsBuilder.
We have both `Settings.settingsBuilder` and `Settings.builder` that do exactly
the same thing, so we should keep only one. I kept `Settings.builder` since it
has my preference but also it is the one that we use in examples of the Java API.
2016-04-08 18:10:02 +02:00
Chris Earle d97d5ebb8b Remove hostname from NetworkAddress.format
This removes the inconsistent output of IP addresses. The format was parsing-unfriendly and it makes it hard
to reason about API responses, such as to _nodes.

With this change in place, it will never print the hostname as part of the default format, which has the
added benefit that it can be used consistently for URIs, which was not the case when the hostname might
appear at the front with "hostname/ip:port".
2016-04-07 17:27:59 -04:00
Adrien Grand c33300c543 Make MappedFieldType responsible for providing a parser/formatter. #17546
Aggregations need to perform instanceof calls on MappedFieldType instances in
order to know how they should be parsed or formatted. Instead, we should let
the field types provide a formatter/parser that can can be used.
2016-04-07 16:57:50 +02:00
jaymode f9d1e8a5f3 Root rest api delegates to a transport action
This change makes the root (/) rest api delegate to a transport action to get the
data for the response. This aligns this rest api with all of the other apis, which
delegate to one or more actions.

In doing this, unit tests were added to provide coverage of the RestMainAction
and the associated classes.
2016-04-07 10:03:49 -04:00
Jason Tedor 0a69985153 Merge pull request #17038 from jasontedor/enable_acked
Prepare for enabling acked indexing
2016-04-06 18:13:28 -04:00
Jimmy Jones f157dae053 Disallow unquoted field names, fix testcases using unquoted JSON 2016-04-06 14:37:15 -06:00
Clinton Gormley cbbf80ca35 v2.3.0 has been released and no longer needs to be hardcoded as -SNAPSHOT 2016-04-04 19:03:43 +02:00
Jason Tedor c7c8b1d825 Merge branch 'master' into enable_acked
* master: (156 commits)
  Make JNA calls optional
  Added RPM metadata
  Remove PROTOTYPE from MLT.Item
  Remove PROTOTYPE from VersionType
  Fix mistake in TopHits change
  Remove PROTOTYPEs from highlighting
  Clean up some log messages
  Command line arguments with comma must be quoted on windows
  Cluster Health should run on applied states, even if waitFor=0 #17440
  ingest: make concrete processor impl final, like all other processor concrete impls.
  Improve some test method comments.
  Document task id's as string in the rest spec
  Replace FieldStatsProvider with a method on MappedFieldType. #17334
  cleanup test
  Remove MathUtils. #17454
  Addressing review comments
  fix javadocs
  Make TranslogConfig immutable and pass TranslogGeneration as a ctor arg to Translog
  [reindex] Don't get rejected
  Remove redundant commit - #openTranslog() already commits in that case
  ...
2016-04-02 13:56:00 -04:00
Christoph Büscher 9d68a515b8 Merge pull request #17453 from cbuescher/add-xcontent-randomization
Add randomization of XContentBuilder output to query tests
2016-04-01 15:02:01 +02:00
Christoph Büscher 7a1b06ce0b Improve some test method comments. 2016-04-01 11:04:56 +02:00
Christoph Büscher 1a697a1ae6 Addressing review comments 2016-03-31 21:46:17 +02:00
Simon Willnauer baa2d51e59 Merge pull request #17422 from s1monw/recovery_mem_buffer_access
Move translog recover outside of the engine

We changed the way we manage engine memory buffers to an
open model where each shard can essentially has infinite memory.
The indexing memory controller is responsible for moving memory to disk
when it's needed. Yet, this doesn't work today when we recover from store/translog
since the engine is not fully initialized such that IMC has no access to the engine,
neither to it's memory buffer nor can it move data to disk.

The biggest issue here is that translog recovery happends inside the Engine constructor
which is problematic by itself since it might take minutes and uses a not yet fully
initialzied engine to perform write operations on.

This change detaches the translog recovery and makes it the responsibility of the caller
to run it once the engine is fully constructed or skip it if not necessary.
2016-03-31 21:03:00 +02:00
Christoph Büscher bbb6d91147 Add randomization of XContentBuilder output to query tests
Currently our testing of parsing query builders is limited to the
default order of the parameters that each builders toXContent()
method produces. To better test real queries where the order of
parameters can be different, this change adds a helper
method to ESTestCase that takes a XContentBuilder and randomly
shuffles the order of the fields inside an object. This is
used in AbstractQueryTestCase, but it can be used in other similar
places in the future.
2016-03-31 18:17:39 +02:00
Simon Willnauer 1e06139584 Move translog recover outside of the engine
We changed the way we manage engine memory buffers to an
open model where each shard can essentially has infinite memory.
The indexing memory controller is responsible for moving memory to disk
when it's needed. Yet, this doesn't work today when we recover from store/translog
since the engine is not fully initialized such that IMC has no access to the engine,
neither to it's memory buffer nor can it move data to disk.

The biggest issue here is that translog recovery happends inside the Engine constructor
which is problematic by itself since it might take minutes and uses a not yet fully
initialzied engine to perform write operations on.

This change detaches the translog recovery and makes it the responsibility of the caller
to run it once the engine is fully constructed or skip it if not necessary.
2016-03-30 23:24:24 +02:00
javanna b9f9b2e3ee Merge branch 'master' into enhancement/discovery_node_one_getter 2016-03-30 17:22:40 +02:00
javanna 62ac7d219f Remove DiscoveryNodes#masterNode in favour of existing DiscoveryNodes#getMasterNode 2016-03-30 15:28:32 +02:00
javanna f8b5d1f5b0 Remove DiscoveryNodes#masterNodeId in favour of existing DiscoveryNodes#getMasterNodeId 2016-03-30 15:28:06 +02:00
javanna 2dbba45f2c Rename static DiscoveryNode#localNode(Settings) to DiscoveryNode#isLocalNode(Settings) 2016-03-30 15:27:26 +02:00
javanna 49e952e272 Rename static DiscoveryNode#dataNode(Settings) to isDataNode 2016-03-30 15:26:41 +02:00
javanna 2230fec9ea Rename static DiscoveryNode#masterNode(Settings) to isMasterNode 2016-03-30 15:26:10 +02:00
javanna a8bbdff3bc Remove DiscoveryNode#name in favour of existing DiscoveryNode#getName 2016-03-30 14:47:36 +02:00
javanna 9889f10e5e Remove DiscoveryNode#id in favour of existing DiscoveryNode#getId 2016-03-30 14:42:15 +02:00
Camilo Diaz Repka 7be11a36cd Refactor: replace all ocurrences of ESTestCase.getRandom() for random().
Remove getRandom().
2016-03-29 23:18:05 -04:00
javanna 061f09d9a4 Merge branch 'master' into enhancement/remove_node_client_setting 2016-03-29 20:19:33 +02:00
Jason Tedor c4324f9964 Merge branch 'master' into enable_acked
* master: (25 commits)
  Replication operation that try to perform the primary phase on a replica should be retried
  split long line in ConvertProcessorTests
  add  type conversion support to ConvertProcessor
  percolator: Make explain use the two phase iterator
  test: make sure we don't flush during indexing the percolator queries
  Added experimental annotation to the update-by-query and reindex docs
  Fixed bad YAML in reindex REST test: 50_routing.yaml
  Update-by-query rest tests: fixed bad yaml and deleted a client-dependent test
  Prevents exception being raised when ordering by an aggregation which wasn't collected
  The reindex body is now required, which changes the exception thrown by the REST test
  Docs: Included Nodes Task API and tidied reindex/update-by-query
  Rename update-by-query REST tests to update_by_query
  REST: The body is required in the reindex API
  The source parameter should not be defined in the delete-by-query REST spec
  Renamed update-by-query REST spec to update_by_query
  Fix test bug in TypeQueryBuilderTests.
  Add comment why it is safe to check the number of nested fields in MapperService.merge.
  Automatically add a sub keyword field to string dynamic mappings. #17188
  Type filters should not have a performance impact when there is a single type. #17350
  Add API to explain why a shard is or isn't assigned
  ...
2016-03-29 11:42:34 -04:00
Colin Goodheart-Smithe ff3fd99074 Prevents exception being raised when ordering by an aggregation which wasn't collected
If a terms aggregation was ordered by a metric nested in a single bucket aggregator which did not collect any documents (e.g. a filters aggregation which did not match in that term bucket) an ArrayOutOfBoundsException would be thrown when the ordering code tried to retrieve the value for the metric. This fix fixes all numeric metric aggregators so they return their default value when a bucket ordinal is requested which was not collected.

Closes #17225
2016-03-29 13:28:03 +01:00
javanna de5cbda8e7 Merge branch 'master' into enhancement/remove_node_client_setting 2016-03-29 10:48:47 +02:00
Lee Hinman 80ab366de4 Add API to explain why a shard is or isn't assigned
This adds a new `/_cluster/allocation/explain` API that explains why a
shard can or cannot be allocated to nodes in the cluster. Additionally,
it will show where the master *desires* to put the shard, according to
the `ShardsAllocator`.

It looks like this:

```
GET /_cluster/allocation/explain?pretty
{
  "index": "only-foo",
  "shard": 0,
  "primary": false
}
```

Though, you can optionally send an empty body, which means "explain the
allocation for the first unassigned shard you find".

The output when a shard is unassigned looks like this:

```
{
  "shard" : {
    "index" : "only-foo",
    "index_uuid" : "KnW0-zELRs6PK84l0r38ZA",
    "id" : 0,
    "primary" : false
  },
  "assigned" : false,
  "unassigned_info" : {
    "reason" : "INDEX_CREATED",
    "at" : "2016-03-22T20:04:23.620Z"
  },
  "nodes" : {
    "V-Spi0AyRZ6ZvKbaI3691w" : {
      "node_name" : "Susan Storm",
      "node_attributes" : {
        "bar" : "baz"
      },
      "final_decision" : "NO",
      "weight" : 0.06666675,
      "decisions" : [ {
        "decider" : "filter",
        "decision" : "NO",
        "explanation" : "node does not match index include filters [foo:\"bar\"]"
      } ]
    },
    "Qc6VL8c5RWaw1qXZ0Rg57g" : {
      "node_name" : "Slipstream",
      "node_attributes" : {
        "bar" : "baz",
        "foo" : "bar"
      },
      "final_decision" : "NO",
      "weight" : -1.3833332,
      "decisions" : [ {
        "decider" : "same_shard",
        "decision" : "NO",
        "explanation" : "the shard cannot be allocated on the same node id [Qc6VL8c5RWaw1qXZ0Rg57g] on which it already exists"
      } ]
    },
    "PzdyMZGXQdGhqTJHF_hGgA" : {
      "node_name" : "The Symbiote",
      "node_attributes" : { },
      "final_decision" : "NO",
      "weight" : 2.3166666,
      "decisions" : [ {
        "decider" : "filter",
        "decision" : "NO",
        "explanation" : "node does not match index include filters [foo:\"bar\"]"
      } ]
    }
  }
}
```

And when the shard *is* assigned, the output looks like:

```
{
  "shard" : {
    "index" : "only-foo",
    "index_uuid" : "KnW0-zELRs6PK84l0r38ZA",
    "id" : 0,
    "primary" : true
  },
  "assigned" : true,
  "assigned_node_id" : "Qc6VL8c5RWaw1qXZ0Rg57g",
  "nodes" : {
    "V-Spi0AyRZ6ZvKbaI3691w" : {
      "node_name" : "Susan Storm",
      "node_attributes" : {
        "bar" : "baz"
      },
      "final_decision" : "NO",
      "weight" : 1.4499999,
      "decisions" : [ {
        "decider" : "filter",
        "decision" : "NO",
        "explanation" : "node does not match index include filters [foo:\"bar\"]"
      } ]
    },
    "Qc6VL8c5RWaw1qXZ0Rg57g" : {
      "node_name" : "Slipstream",
      "node_attributes" : {
        "bar" : "baz",
        "foo" : "bar"
      },
      "final_decision" : "CURRENTLY_ASSIGNED",
      "weight" : 0.0,
      "decisions" : [ {
        "decider" : "same_shard",
        "decision" : "NO",
        "explanation" : "the shard cannot be allocated on the same node id [Qc6VL8c5RWaw1qXZ0Rg57g] on which it already exists"
      } ]
    },
    "PzdyMZGXQdGhqTJHF_hGgA" : {
      "node_name" : "The Symbiote",
      "node_attributes" : { },
      "final_decision" : "NO",
      "weight" : 3.6999998,
      "decisions" : [ {
        "decider" : "filter",
        "decision" : "NO",
        "explanation" : "node does not match index include filters [foo:\"bar\"]"
      } ]
    }
  }
}
```

Only "NO" decisions are returned by default, but all decisions can be
shown by specifying the `?include_yes_decisions=true` parameter in the
request.

Resolves #14593
2016-03-28 15:21:02 -06:00
Jason Tedor 4793630eb8 Merge branch 'master' into enable_acked
* master: (419 commits)
  Remove PROTOTYPE from ShapeBuilders
  Take filterNodeIds into consideration while sending tasks actions requests to nodes
  test: cleanup imports and method rename
  Remove PROTOTYPE from SortBuilders
  percolator: Add query extract support for the blended term query and the common terms query.
  Don't iterate over shard routing if it's null
  [TEST] Reduce size of random shapes
  Add some debug logging to testPrimaryRelocationWhileIndexing
  Order methods in IndicesClusterStateService according to execution
  Tidied up percolator doc annotations
  In cat.snapshots, repository is required
  Do not retrieve all indices stats when checking for cache resets
  Enforce `discovery.zen.minimum_master_nodes` is set when bound to a public ip #17288
  Port Primary Terms to master #17044
  Revert "Add debug logging for Vagrant upgrade test"
  Ownership for data, logs, and configs for packages
  add on_failure exception metadata to ingest document for verbose simulate
  Revert "Merge pull request #16843 from xuzha/s3-encryption"
  Update Format, add new settings into the setting test
  Update and rebase the init implementation.
  ...
2016-03-28 12:29:53 -04:00
javanna a9f4982c40 Merge branch 'master' into enhancement/remove_node_client_setting 2016-03-25 20:16:40 +01:00
javanna 93ce36a198 separated attributes from node roles in DiscoveryNode
Node roles are now serialized as well, they are not part of the node attributes anymore. DiscoveryNodeService takes care of dividing settings into attributes and roles. DiscoveryNode always requires to pass in attributes and roles separately.
2016-03-25 20:14:27 +01:00
Tanguy Leroux a22529cceb Do not retrieve all indices stats when checking for cache resets 2016-03-25 13:16:12 +01:00
Boaz Leskes fe43eef1b5 Port Primary Terms to master #17044
Primary terms is a way to make sure that operations replicated from stale primary are rejected by shards following a newly elected primary.

Original PRs adding this to the seq# feature branch #14062 , #14651 . Unlike those PR, here we take a different approach (based on newer code in master) where the primary terms are stored in the meta data only (and not in `ShardRouting` objects).

Relates to #17038

Closes #17044
2016-03-25 12:01:00 +01:00
javanna 27d4994aff Merge branch 'master' into enhancement/remove_node_client_setting 2016-03-24 18:10:11 +01:00
Alexander Reelsen b2573858b6 Version: Set version to 5.0.0-alpha1
Change version, required a minor fix in the RPM building.
In case of a alpha/beta version, the release will contain alpha/beta
as the RPM version cannot contains dashes/tildes.
2016-03-24 08:36:08 +01:00
Honza Král b139f4e0bf [TEST] Move yaml test requiring yaml, add skip:yaml
Clients don't ship with yaml (de)serializer by default so this test must
be optionally skipped
2016-03-23 14:50:23 +01:00
javanna 030453d320 Merge branch 'master' into enhancement/remove_node_client_setting 2016-03-23 11:25:34 +01:00
Adrien Grand e50eeeaffb Refactor fielddata mappings. #17148
The fielddata settings in mappings have been refatored so that:
 - text and string have a `fielddata` (boolean) setting that tells whether it
   is ok to load in-memory fielddata. It is true by default for now but the
   plan is to make it default to false for text fields.
 - text and string have a `fielddata_frequency_filter` which contains the same
   thing as `fielddata.filter.frequency` used to (but validated at parsing time
   instead of being unchecked settings)
 - regex fielddata filtering is not supported anymore and will be dropped from
   mappings automatically on upgrade.
 - text, string and _parent fields have an `eager_global_ordinals` (boolean)
   setting that tells whether to load global ordinals eagerly on refresh.
 - in-memory fielddata is not supported on keyword fields anymore at all.
 - the `fielddata` setting is not supported on other fields that text and string
   and will be dropped when upgrading if specified.
2016-03-23 09:48:13 +01:00
Simon Willnauer 1988b8b387 [TEST] Reuse EsTestCase#createAnalysisService in KuromojiAnalysisTests 2016-03-22 13:45:20 +01:00
Simon Willnauer 33521fc27c Detach IndexShard from node services
this is the last step to remove node level service from IndexShard.
This means that tests can now more easily create an IndexShard instance
without starting a node and removes the dependency between IndexShard and Client/ScriptService
2016-03-22 11:02:04 +01:00
javanna bf390a935e Merge branch 'master' into enhancement/remove_node_client_setting 2016-03-21 17:18:23 +01:00
Boaz Leskes 2d1152ebac Remove ClusterService interface, in favor of it's only production instance #17183
We current have a ClusterService interface, implemented by InternalClusterService and a couple of test classes. Since the decoupling of the transport service and the cluster service, one can construct a ClusterService fairly easily, so we don't need this extra indirection.

Closes #17183
2016-03-21 13:55:10 +01:00
Martijn van Groningen e3b7e5d75a percolator: Replace percolate api with the new percolator query
Also replaced the PercolatorQueryRegistry with the new PercolatorQueryCache.

The PercolatorFieldMapper stores the rewritten form of each percolator query's xcontext
in a binary doc values field. This make sure that the query rewrite happens only during
indexing (some queries for example fetch shapes, terms in remote indices) and
the speed up the loading of the queries in the percolator query cache.

Because the percolator now works inside the search infrastructure a number of features
(sorting fields, pagination, fetch features) are available out of the box.

The following feature requests are automatically implemented via this refactoring:

Closes #10741
Closes #7297
Closes #13176
Closes #13978
Closes #11264
Closes #10741
Closes #4317
2016-03-21 12:21:50 +01:00
Ryan Ernst f71f0d6010 Revert "Build: Switch to maven-publish plugin"
This reverts commit a90a2b34fc.
2016-03-18 17:22:25 -07:00
Ryan Ernst 6af4c43c4f Merge pull request #17128 from rjernst/maven_publish
Build: Switch to maven-publish plugin
2016-03-17 11:53:50 -07:00
Simon Willnauer e91a141233 Prevent index level setting from being configured on a node level
Today we allow to set all kinds of index level settings on the node level which
is error prone and difficult to get right in a consistent manner.
For instance if some analyzers are setup in a yaml config file some nodes might
not have these analyzers and then index creation fails.

Nevertheless, this change allows some selected settings to be specified on a node level
for instance:
 * `index.codec` which is used in a hot/cold node architecture and it's value is really per node or per index
 * `index.store.fs.fs_lock` which is also dependent on the filesystem a node uses

All other index level setting must be specified on the index level. For existing clusters the index must be closed
and all settings must be updated via the API on each of the indices.

Closes #16799
2016-03-17 14:42:18 +01:00
Christoph Büscher 6ddf9ae92f Merge branch 'master' into feature-suggest-refactoring 2016-03-16 15:27:02 +01:00
Igor Motov b10db19595 Bring back tests for missing elements in the diff-serialized cluster state
We can add it back now that we improved our compression framework.

Closes #11257
2016-03-16 09:13:52 -04:00
Nik Everett 7197172047 [reindex] Properly register status
Without this commit fetching the status of a reindex from a node that isn't
coordinating the reindex will fail. This commit properly registers reindex's
status so this doesn't happen. To do so it moves all task status registration
into NetworkModule and creates a method to register other statuses which the
reindex plugin calls.
2016-03-16 07:40:49 -04:00
Christoph Büscher 39667b5793 Merge branch 'master' into feature-suggest-refactoring
Conflicts:
	docs/reference/migration/migrate_5_0/java.asciidoc
2016-03-16 12:06:42 +01:00
Ryan Ernst a90a2b34fc Build: Switch to maven-publish plugin
The build currently uses the old maven support in gradle. This commit
switches to use the newer maven-publish plugin. This will allow future
changes, for example, easily publishing to artifactory.

An additional part of this change makes publishing of build-tools part
of the normal publishing, instead of requiring a separate upload step
from within buildSrc. That also sets us up for a follow up to enable
precomit checks on the buildSrc code itself.
2016-03-15 19:16:37 -07:00
Jason Tedor 618441aea3 Merge pull request #17088 from jasontedor/simplify-bootstrap-settings
Bootstrap does not set system properties
2016-03-15 19:25:16 -04:00
Yannick Welsch f5e6db4090 Remove System.out.println and Throwable.printStackTrace from tests 2016-03-15 15:40:37 +01:00
Yannick Welsch e91fd09692 Enable jdk-system-out Forbidden API checks on test sources 2016-03-15 15:03:37 +01:00
Christoph Büscher 97638c95fc Merge branch 'master' into feature-suggest-refactoring
Conflicts:
	docs/reference/migration/migrate_5_0.asciidoc
2016-03-14 11:13:47 +01:00
Jason Tedor 8a05c2a2be Bootstrap does not set system properties
Today, certain bootstrap properties are set and read via system
properties. This action-at-distance way of managing these properties is
rather confusing, and completely unnecessary. But another problem exists
with setting these as system properties. Namely, these system properties
are interpreted as Elasticsearch settings, not all of which are
registered. This leads to Elasticsearch failing to startup if any of
these special properties are set. Instead, these properties should be
kept as local as possible, and passed around as method parameters where
needed. This eliminates the action-at-distance way of handling these
properties, and eliminates the need to register these non-setting
properties. This commit does exactly that.

Additionally, today we use the "-D" command line flag to set the
properties, but this is confusing because "-D" is a special flag to the
JVM for setting system properties. This creates confusion because some
"-D" properties should be passed via arguments to the JVM (so via
ES_JAVA_OPTS), and some should be passed as arguments to
Elasticsearch. This commit changes the "-D" flag for Elasticsearch
settings to "-E".
2016-03-13 20:09:15 -04:00
David Pilato 9acb0bb28c Merge branch 'master' into pr/16598-register-filter-settings
# Conflicts:
#	core/src/main/java/org/elasticsearch/cluster/service/InternalClusterService.java
#	core/src/main/java/org/elasticsearch/common/settings/IndexScopedSettings.java
#	core/src/main/java/org/elasticsearch/common/settings/Setting.java
2016-03-13 14:52:10 +01:00
Ryan Ernst 591fb8f028 Merge branch 'master' into cli-parsing 2016-03-11 10:45:05 -08:00
Ali Beyad 31dcb3e18b Merge pull request #16873 from abeyad/suggester-wiring-refactoring
Change internal representation of suggesters
2016-03-11 12:34:35 -05:00
Christoph Büscher daeffb149c Merge branch 'master' into feature-suggest-refactoring 2016-03-11 10:37:28 +01:00
Yannick Welsch 04e55ecf6b Make logging message String constant to allow static checks 2016-03-11 10:30:59 +01:00
Yannick Welsch b17f4b40ba Check that number of placeholders in log message matches number of parameters 2016-03-11 10:30:59 +01:00
Yannick Welsch f6ae9ec4f6 Remove ShardsAllocators and merge allocateUnassigned, moveShards and rebalance to improve performance 2016-03-11 08:42:25 +01:00
Nik Everett 64c4b5f3ca Merge pull request #17036 from dongjoon-hyun/fix_typo_in_test
Fix typos in comments/strings of `test` module.
2016-03-10 13:42:45 -05:00
Ryan Ernst ac73d97169 Merge branch 'master' into dots2 2016-03-10 09:22:31 -08:00
Ali Beyad 807d085218 Merge remote-tracking branch 'upstream/feature-suggest-refactoring' into suggester-wiring-refactoring 2016-03-10 09:56:08 -05:00
Jay Modi debd41b3ea Merge pull request #17049 from jaymode/single_node_test_settings
Allow additional settings for the node in ESSingleNodeTestCase
2016-03-10 07:42:47 -05:00
Martijn van Groningen 2fa33d5c47 Added ingest statistics to node stats API
The ingest stats include the following statistics:
* `ingest.total.count`- The total number of document ingested during the lifetime of this node
* `ingest.total.time_in_millis` - The total time spent on ingest preprocessing documents during the lifetime of this node
* `ingest.total.current` - The total number of documents currently being ingested.
* `ingest.total.failed` - The total number ingest preprocessing operations failed during the lifetime of this node

Also these stats are returned on a per pipeline basis.
2016-03-10 13:21:43 +01:00
jaymode 9f92325587 Allow additional settings for the node in ESSingleNodeTestCase
This change adds a method that extending classes can override to provide additional settings
for the node used in a single node test case.
2016-03-10 07:16:37 -05:00
Boaz Leskes cd12241e9f Decouple the TransportService and ClusterService #16872
Currently, the cluster service is tightly coupled to the transport service by both managing node connections and requiring the bound address in order to create the local disco node. This commit introduces a new NodeConnectionsService which is in charge of node connection management and makes it possible to remove all network related calls from the cluster service. The local DiscoNode is now created by DiscoveryNodeService and is set both the cluster service and the transport service during node start up.

Closes #16788
Closes #16872
2016-03-10 11:45:15 +01:00
Christoph Büscher 69c83b3459 Merge branch 'master' into feature-suggest-refactoring
Conflicts:
	core/src/main/java/org/elasticsearch/action/suggest/TransportSuggestAction.java
        core/src/main/java/org/elasticsearch/search/suggest/SuggestParseElement.java
	core/src/main/java/org/elasticsearch/search/suggest/SuggestionSearchContext.java
	core/src/main/java/org/elasticsearch/search/suggest/phrase/PhraseSuggester.java
2016-03-10 11:44:36 +01:00
Ali Beyad 06487b0ac5 Change internal representation of suggesters to instances of
SuggestBuilder instead of raw bytes.
2016-03-10 00:12:39 -05:00
Ryan Ernst c48e6b86f6 Limit update to fields and objects which were actually modified, and
simplify root update creation.
2016-03-09 13:24:46 -08:00
Ryan Ernst bbdbfe7373 Merge branch 'master' into dots2 2016-03-09 11:28:27 -08:00
Simon Willnauer 016bc3ea2b Merge pull request #17001 from s1monw/cut_over_to_uuid
Use index UUID to lookup indices on IndicesService

Today we use the index name to lookup index instances on the IndicesService
which applied to search requests but also to index deletion etc. This commit
moves the interface to expect an Index instance which is a tuple
and looks up the index by uuid rather than by name. This prevents accidental modification
of the wrong index if and index is recreated or searching from the wrong index in such a case.
Accessing an index that has the same name but different UUID will now result in an IndexNotFoundException.
2016-03-09 19:44:36 +01:00
Simon Willnauer e72dac91b3 Use index UUID to lookup indices on IndicesService
Today we use the index name to lookup index instances on the IndicesService
which applied to search reqeusts but also to index deletion etc. This commit
moves the interface to expcet and `Index` instance which is a <name, uuid> tuple
and looks up the index by uuid rather than by name. This prevents accidential modificaiton
of the wrong index if and index is recreated or searching from the _wrong_ index in such a case.
Accessing an index that has the same name but different UUID will now result in an IndexNotFoundException.

Closes #17001
2016-03-09 19:42:15 +01:00
Dongjoon Hyun db534347e5 Fix typos in comments/strings of `test` module. 2016-03-09 10:03:37 -08:00
Ryan Ernst fdce9d7c4d Merge branch 'master' into cli-parsing 2016-03-08 14:18:20 -08:00
Ryan Ernst 80198accc1 Removed old cli stuff, and add tests for new Command behavior 2016-03-08 14:13:55 -08:00
Ryan Ernst e5c852f767 Convert bootstrapcli parser to jopt-simple 2016-03-08 13:39:37 -08:00
Nik Everett 6d0efae713 Teach list tasks api to wait for tasks to finish
_wait_for_completion defaults to false. If set to true then the API will
wait for all the tasks that it finds to stop running before returning. You
can use the timeout parameter to prevent it from waiting forever. If you
don't set a timeout parameter it'll default to 30 seconds.

Also adds a log message to rest tests if any tasks overrun the test. This
is just a log (instead of failing the test) because lots of tasks are run
by the cluster on its own and they shouldn't cause the test to fail. Things
like fetching disk usage from the other nodes, for example.

Switches the request to getter/setter style methods as we're going that
way in the Elasticsearch code base. Reindex is all getter/setter style.

Closes #16906
2016-03-08 11:53:57 -05:00
Jason Tedor 2bd09c4625 Merge branch 'master' into enable_acked
* master: (350 commits)
  Note to configuration docs on number of threads
  Reduce maximum number of threads in boostrap check
  Limit generic thread pool
  Remove NodeService injection to Discovery
  Prevent closing index during snapshot restore
  [TEST] Fix newline issue in PluginCliTests on Windows
  ParseFieldMatcher should log when using deprecated settings. #16988
  fix checkstyle error
  Add test for the index_options on a keyword field. #16990
  Analysis : Allow string explain param in JSON
  Analysis : Allow string explain param in JSON
  fix typo
  Remove SNAPSHOT from versions in plugin descriptors
  Add support for alpha versions
  Enable unmap hack for java 9
  Simplify mock scripts
  Adding `time_zone` parameter to daterange-aggregation docs
  Adding tests for `time_zone` parameter for date range aggregation
  Added ingest info to node info API, which contains a list of available processors.
  Remove bw compat from size mapper
  ...
2016-03-08 10:19:56 -05:00
Christoph Büscher 1264f37a1b Merge branch 'master' into feature-suggest-refactoring 2016-03-08 11:10:08 +01:00
Yannick Welsch c9e1ccf610 [TEST] Fix newline issue in PluginCliTests on Windows 2016-03-08 10:28:20 +01:00
Ryan Ernst 45b5ab24fe Moved MockTerminal and created a base test case for cli commands. 2016-03-07 12:42:15 -08:00
Simon Willnauer f071f327db Remove old and unsupported version constants
All version <= 2.0 are not supported anymore. This commit removes all
uses of these versions.
2016-03-07 11:49:06 +01:00
Christoph Büscher 7ec5075a87 Merge branch 'master' into feature-suggest-refactoring 2016-03-07 10:45:43 +01:00
Robert Muir 54018a5d37 upgrade to lucene 6.0.0-snapshot-bea235f
Closes #16964

Squashed commit of the following:

commit a23f9d2d29220991aa498214530753d7a5a148c6
Merge: eec9c4e 0b0a251
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Mar 7 04:12:02 2016 -0500

    Merge branch 'master' into lucene6

commit eec9c4e5cd11e9c3e0b426f04894bb2a6dae4f21
Merge: bc67205 675d940
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 13:45:00 2016 -0500

    Merge branch 'master' into lucene6

commit bc67205bdfe1526eae277ab7856fc050ecbdb7b2
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 09:56:31 2016 -0500

    fix test bug

commit a60723b007ff12d97b1810cef473bd7b553a0327
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Mar 4 15:35:35 2016 +0100

    Fix SimpleValidateQueryIT to put braces around boosted terms

commit ae3a49d7ba7ced448d2a5262e5d8ec98671a9090
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Mar 4 15:27:25 2016 +0100

    fix multimatchquery

commit ae23fdb88a8f6d3fb7ba60fd1aaf3fd72d899aa5
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Mar 4 15:20:49 2016 +0100

    Rewrite DecayFunctionScoreIT to be independent of the similarity used

    This test relied a lot on the term scoring and compared scores
    that are dependent on the similarity. This commit changes the base query
    to be a predictable constant score query.

commit 366c2d518c35d31251033f1b6f6a93f6e2ae327d
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Mar 4 14:06:14 2016 +0100

    Fix scoring in tests due to changes to idf calculation.

    Lucene 6 uses a different default similarity as well as a different
    way to calculate IDF. In contrast to older version lucene 6 uses docCount per field
    to calculate the IDF not the # of docs in the index to overcome the sparse field
    cases.

commit dac99fd64ac2fa71b8d8d106fe68825e574c49f8
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 08:21:57 2016 -0500

    don't hardcoded expected termquery score

commit 6e9f340ba49ab10eed512df86d52a121aa775b0f
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 08:04:45 2016 -0500

    suppress deprecation warning until migrated to points

commit 3ac8908424b3fdad44a90a4f7bdb3eff7efd077d
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 07:21:43 2016 -0500

    Remove invalid test: all commits have IDs, and its illegal to do this.

commit c12976288124ad1a26467e7e848fb810548e7eab
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 07:06:14 2016 -0500

    don't test with unsupported back compat

commit 18bbfe76128570bc70883bf91ff4c44c82d27817
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 07:02:18 2016 -0500

    remove now invalid lucene 4 backcompat test

commit 7e730e572886f0ef2d3faba712e4256216ff01ec
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 06:58:52 2016 -0500

    remove now invalid lucene 4 backwards test

commit 244d2ab6868ba5ac9e0bcde3c2833743751a25ec
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 06:47:23 2016 -0500

    use 6.0 codec

commit 5f64d4a431a6fdaa1234adca23f154c2a1de8284
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 06:43:08 2016 -0500

    compile, javadocs, forbidden-apis, etc

commit 1f273cd62a7fe9ca8f8944acbbfc5cbdd3d81ccb
Merge: cd33921 29e3443
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Mar 4 10:45:29 2016 +0100

    Merge branch 'master' into lucene6

commit cd33921ac742ef9fb351012eff35f3c7dbda7264
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:58:37 2016 -0500

    fix hunspell dictionary loading

commit c7fdbd837b01f7defe9cb1c24e2ec65604b0dc96
Merge: 4d4190f d8948ba
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:41:53 2016 -0500

    Merge branch 'master' into lucene6

commit 4d4190fd82601aaafac6b8254ccb3edf218faa34
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:39:14 2016 -0500

    remove nocommit

commit 77ca69e288b1a41aa9595c921ed166c272a00ea8
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:38:24 2016 -0500

    clean up numericutils vs legacynumericutils

commit a466d696fbaad04b647ffbc0857a9439b583d0bf
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:32:43 2016 -0500

    upgrade spatial4j

commit 5412c747a8cfe638bacedbc8233163cb75cc3dc5
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:19:28 2016 -0500

    move to 6.0.0-snapshot-8eada27

commit b32bfe924626b87e540692375ece09e7c2edb189
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 11:30:09 2016 +0100

    Fix some test compile errors.

commit 6ccde35e9840b03c68d1a2cd47c7923a06edf64a
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 11:25:51 2016 +0100

    Current Lucene version is 6.0.0.

commit f62e1015d931b4cc04c778298a8fa1ba65e97ad9
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 11:20:48 2016 +0100

    Fix compile errors in NGramTokenFilterFactory.

commit 6837c6eabf96075f743649da9b9b52dd39611c58
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 10:50:59 2016 +0100

    Fix the edge ngram tokenizer/filter.

commit ccd7f070de5efcdfbeb34b9555c65c4990bf1ba6
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 10:42:44 2016 +0100

    The missing value is now accessible through a getter.

commit bd3b77f9b28e5b05daa3d49683a9922a6baf2963
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 10:41:51 2016 +0100

    Remove IndexCacheableQuery.

commit 05f3091c347aeae80eeb16349ac51d2b53cf86f7
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 10:39:43 2016 +0100

    Fix compilation of function_score queries.

commit 81cda79a2431ac78f56b0cc5a5765387f662d801
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 10:35:02 2016 +0100

    Fix compile errors in BlendedTermQuery.

commit 70994ce8dd1eca0b995870974a38e20f26f96a7b
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 23:33:03 2016 -0500

    add bug ID

commit 29d4f1a71f36f646b5a6060bed3db019564a279d
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 21:02:32 2016 -0500

    easy .store changes

commit 5e1a1e6fd665fa455e88d3a8987362fad5f44bb1
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 20:47:24 2016 -0500

    cleanups mostly around boosting

commit 333a669ec6c305ada5645d13ed1da0e19ec1d053
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 20:27:56 2016 -0500

    more simple fixes

commit bd5cd98a1e089c866b6b4a5e159400b110140ce6
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 19:49:38 2016 -0500

    more easy fixes and removal of ancient cruft

commit a68f419ee47da5f9c9ce5b372f01d707e902474c
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 19:35:02 2016 -0500

    cutover numerics

commit 4ca5dc1fa47dd5892db00899032133318fff3116
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 18:34:18 2016 -0500

    fix some constants

commit 88710a17817086e477c6c021ec346d0534b7fb88
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 18:14:25 2016 -0500

    Add spatial-extras jar as a core dependency

commit c8cd6726583e5ce3f546ed355d4eca037164a30d
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 18:03:33 2016 -0500

    update to lucene 6 jars
2016-03-07 04:12:23 -05:00