Commit Graph

4822 Commits

Author SHA1 Message Date
Jason Tedor 17dd60dd31 Merge pull request #17208 from jasontedor/install-plugin-permissions
Install plugin permissions
2016-03-23 18:44:47 -04:00
Areek Zillur e16e113691 Remove suggest threadpool
In #17198, we removed suggest transport action, which
used the `suggest` threadpool to execute requests. Now
`suggest` threadpool is unused and suggest requests are
executed on the `search` threadpool.
2016-03-23 18:01:45 -04:00
Areek Zillur b1ceaaddf4 remove dead code 2016-03-23 16:37:57 -04:00
Areek Zillur de78621c95 simplify handling suggest-only search request 2016-03-23 16:37:56 -04:00
Areek Zillur 59513b308e rename to isSuggestOnly 2016-03-23 16:37:56 -04:00
Areek Zillur 91dd9b3301 Merge suggest stats into search stats 2016-03-23 16:37:56 -04:00
Areek Zillur ed49ec437f remove suggest transport action 2016-03-23 16:37:56 -04:00
Areek Zillur 0eb2032189 Disable DFS search types and request caching for suggest-only requests 2016-03-23 16:37:56 -04:00
Areek Zillur b0437c3f22 Optimize search execution for suggest-only requests
We skip context preprocessing and only execute
suggest phase for suggest-only requests
2016-03-23 16:37:56 -04:00
Areek Zillur 5ed2bb5f18 [REST] use search transport for suggest rest endpoint 2016-03-23 16:37:56 -04:00
Jason Tedor 0647638a99 Guard against null key for property placeholder 2016-03-23 14:57:01 -04:00
Ryan Ernst 6e74d1d0c9 Merge pull request #17289 from rjernst/cli-args
Cli: Reject positional argument for bin/elasticsearch
2016-03-23 11:53:32 -07:00
Ryan Ernst 0fe6c9f3b4 Add elasticsearch cli test for options plus illegal args 2016-03-23 11:52:25 -07:00
David Pilato 6e80b5f2dd Merge branch 'fix/17244-s3-chunk-buffer-sizes' 2016-03-23 18:51:54 +01:00
Jason Tedor 0654b84d2c Useful error message for null property placeholder
This commit adds the key to the error message when encountering a
missing property placeholder.
2016-03-23 13:25:39 -04:00
Jason Tedor 3f73ef9aa4 Protect max size virtual memory check from Windows 2016-03-23 12:35:01 -04:00
Ryan Ernst f91a046b6f Cli: Reject positional argument for bin/elasticsearch
This exits with a usage error when bin/elasticsearch recieves any
positional args.

closes #17287
2016-03-23 09:03:29 -07:00
Christoph Büscher a3fc4c0370 Merge pull request #17205 from cbuescher/sort-use-sortbuilders
Switch to using SortBuilder instead of BytesReference in serialization of SearchSource and elsewhere.
2016-03-23 16:25:02 +01:00
Christoph Büscher e319985d93 Adding test for parsing sort on single fields as list 2016-03-23 15:27:53 +01:00
Isabel Drost-Fromm 9e1059be2b Remove deprecated reverse option from sorting
Relates to #17047
2016-03-23 15:03:57 +01:00
Adrien Grand 940fc3f864 Move dynamic template matching logic to the MatchType enum. #17281 2016-03-23 14:53:26 +01:00
Adrien Grand c2ba4448b0 Fix test bug. 2016-03-23 14:42:36 +01:00
Christoph Büscher 41ddc6fa3f Merge branch 'master' into sort-use-sortbuilders
Conflicts:
	core/src/main/java/org/elasticsearch/common/io/stream/StreamInput.java
2016-03-23 14:30:53 +01:00
Jay Modi ae1a68c10a Merge pull request #17218 from jaymode/streaminput_availble
Implement available for all StreamInput classes
2016-03-23 09:15:24 -04:00
Christoph Büscher f8e7462725 SortBuilder#toXContent should render full object 2016-03-23 14:10:47 +01:00
Christoph Büscher b614e3e075 Switch to using refactored SortBuilder in SearchSourceBuilder and elsewhere
Switching from using list of BytesReference to real SortBuilder list in
SearchSourceBuilder, TopHitsAggregatorBuilder and TopHitsAggregatorFactory.
Removing SortParseElement and related sort parsers.
2016-03-23 13:49:52 +01:00
Simon Willnauer 4826782b37 wrap line at 140chars 2016-03-23 13:31:50 +01:00
Christoph Büscher ececff357b Merge pull request #17248 from cbuescher/sort-add-parse-list
Moving the current top level parsing for the "sort" element into a static "fromXContent" method in SortBuilder.
2016-03-23 13:04:37 +01:00
Christoph Büscher 2bb57b6f2c Remove SortBuilderParser interface, using abstract methods in SortBuilder 2016-03-23 12:47:10 +01:00
javanna 030453d320 Merge branch 'master' into enhancement/remove_node_client_setting 2016-03-23 11:25:34 +01:00
David Pilato e907b7c11e Check that S3 setting `buffer_size` is always lower than `chunk_size`
We can be better at checking `buffer_size` and `chunk_size` for S3 repositories.
For example, we know that:

* `buffer_size` should be more than `5mb`
* `chunk_size` should be no more than `5tb`
* `buffer_size` should be lower than `chunk_size`

Otherwise, setting `buffer_size` is useless.

For the record:

`chunk_size` is a Snapshot setting whatever the implementation is.
`buffer_size` is an S3 implementation setting.

Let say that you are snapshotting a 500mb file. If you set `chunk_size` to `200mb`, then Snapshot service will call S3 repository to snapshot 3 files with the following sizes:

* `200mb`
* `200mb`
* `100mb`

If you set `buffer_size` to `100mb` (AWS maximum size recommendation), the first file of `200mb` will be uploaded on S3 using the multipart feature in 2 chunks and the workflow is basically the following:

* create the multipart request and get back an `id` from AWS S3 platform
* upload part1: `100mb`
* upload part2: `100mb`
* "commit" the full upload using the `id`.

Closes #17244.
2016-03-23 10:39:54 +01:00
Adrien Grand a3bb409f03 Upgrade string fields to text/keyword also if `ignore_above` is set. #17273
Since this parameter is used in the logstash default template, it would be nice
to handle it.
2016-03-23 10:31:10 +01:00
Colin Goodheart-Smithe b8ac05149d Merge pull request #17264 from pjo256/master
Setting 'other' bucket on empty aggregation
2016-03-23 09:19:42 +00:00
Simon Willnauer 2f1af552a9 Bring back operation rollback on unexpected mapping change during recovery
We lost some accounting code in the translog recover code during refactoring
which triggers a very rare assertion. If we fail on a recovery target with an
illegal mapping update (which can happen if the clusterstate is behind), then
we miss to rollback the # of processed ops in that batch and once we resume
the batch we trip an assertion that the stats are off.

This commit brings back the code lost in 8bc2332d9a
and improves the comment that explains why we need this rollback logic.
2016-03-23 10:15:53 +01:00
Adrien Grand 252ae5f15a Upgrade dynamic templates that use a dynamic type. #17254
Now that string has been splitted into text and keyword, we use text as a
dynamic type when encountering string fields in a json document. However
this does not play well with existing templates that look like

```
{
  "mapping": {
    "index": "not_analyzed",
    "type": "{dynamic_type}"
  },
  "match": "*"
}
```

Since we want existing templates to keep working as much as possible in 5.0,
this commit adds a hack to dynamic templates so that elasticsearch will create
a keyword field if the `index` property is set and is either `no` or
`not_analyzed`, similarly to what was done in #16991.

While this will make upgrades easier, we still need to figure out a way to
allow users to create keyword fields when using dynamic types.
2016-03-23 09:54:19 +01:00
Adrien Grand e50eeeaffb Refactor fielddata mappings. #17148
The fielddata settings in mappings have been refatored so that:
 - text and string have a `fielddata` (boolean) setting that tells whether it
   is ok to load in-memory fielddata. It is true by default for now but the
   plan is to make it default to false for text fields.
 - text and string have a `fielddata_frequency_filter` which contains the same
   thing as `fielddata.filter.frequency` used to (but validated at parsing time
   instead of being unchecked settings)
 - regex fielddata filtering is not supported anymore and will be dropped from
   mappings automatically on upgrade.
 - text, string and _parent fields have an `eager_global_ordinals` (boolean)
   setting that tells whether to load global ordinals eagerly on refresh.
 - in-memory fielddata is not supported on keyword fields anymore at all.
 - the `fielddata` setting is not supported on other fields that text and string
   and will be dropped when upgrading if specified.
2016-03-23 09:48:13 +01:00
Adrien Grand 435558a5c0 Also map floating-point numbers as floats when numeric detection is on. #17104
I overlooked it in #15319 since numeric detection triggers a totally different
path in the code of dynamic mappings.
2016-03-23 08:20:22 +01:00
Philip Ottesen 1dff3a8210 Setting 'other' bucket on empty aggregation 2016-03-22 20:23:35 -04:00
Tal Levy 534caa8927 Handle regex parsing errors in Gsub and Grok Processors
Currently, both Gsub and Grok parse regex strings during
Pipeline creation. Thrown parsing exceptions were leaking out, this
commit wraps those exceptions in ElasticsearchParseExceptions.
2016-03-22 15:06:29 -07:00
Jason Tedor d5e408b273 Mock rlimit infinity in virtual memory size test
This commit mocks the value of rlimit infinity in the max size virtual
memory check test. This is to avoid attempting to load the native C
library during the test on Windows which would lead to a permissions
violation (the native C library needs to be loaded before the security
manager is setup).
2016-03-22 17:03:46 -04:00
Areek Zillur 866a350599 Merge pull request #17232 from areek/cleanup/handling_index_state
Cleanup writing upgraded index state
2016-03-22 14:57:49 -04:00
Adrien Grand d514977c75 Make dynamic template parsing less lenient. #17249
Today unknown parameters are ignored yet carried through serialization.
2016-03-22 18:52:25 +01:00
Boaz Leskes 20644666e5 RecoveryWhileUnderLoadIT: output specific missing doc ids and their shard routing on failure
Also increase logging levels to see when a doc was indexed
2016-03-22 18:29:09 +01:00
Christoph Büscher 64d362ab9d Add parsing of list of sort builders to SortBuilder
Moving the current parsing code for the whole "sort" element
in the seach source over to static "fromXContent" method in
SortBuilder.
2016-03-22 18:07:08 +01:00
Simon Willnauer 3ed4ff054f Merge pull request #17246 from s1monw/archive_persistent_settings
Archive cluster level settings if unknown or broken

We already archive index level settings if we find an unknown or invalid/broken
value for a setting on node startup. The same could potentially happen for persistent
cluster level settings if we remove a setting or if we add validation to a setting that
didn't exist in the past. To ensure that only valid settings are recovered into the cluster
state we archive them (prefix them with `archive.` and log a warning. Tools that check the
cluster settings can then warn users that they have broken settings in their clusterstate that
got archived.
2016-03-22 17:35:08 +01:00
Nik Everett da96b6e41d [reindex] Add thottling support
The throttle is applied when starting the next scroll request so that its
timeout can include the throttle time.
2016-03-22 12:34:14 -04:00
Simon Willnauer c0ef3189b7 add javadocs for isPrivate() 2016-03-22 17:33:51 +01:00
Colin Goodheart-Smithe ea93b803d2 Rewrite to unbounded range query if relation to query is WITHIN 2016-03-22 16:14:47 +00:00
Jason Tedor 8004c51c17 Add max size virtual memory check
This commit adds a bootstrap check on Linux and OS X for the max size of
virtual memory (address space) to the user running the Elasticsearch
process.

Closes #16935
2016-03-22 11:52:36 -04:00
Adrien Grand c52b1f3a7c An `exists` query on an object should query a single term.
Currently if you run an `exists` query on an object, it will resolve all sub
fields and create a disjunction for all those fields. However the `_field_names`
mapper indexes paths for objects so we could query object paths directly.

I also changed the query parser to reject `exists` queries if the `_field_names`
field is disabled since it would be a big performance trap.
2016-03-22 16:26:45 +01:00
Simon Willnauer 68d07fc01f Archive cluster level settings if unknown or broken
We already archive index level settings if we find an unknown or invalid/broken
value for a setting on node startup. The same could potentially happen for persistent
cluster level settings if we remove a setting or if we add validation to a setting that
didn't exist in the past. To ensure that only valid settings are recovered into the cluster
state we archive them (prefix them with `archive.` and log a warning. Tools that check the
cluster settings can then warn users that they have broken settings in their clusterstate that
got archived.
2016-03-22 16:17:06 +01:00
Luca Cavanna 3764b3ff80 Merge pull request #17145 from alexshadow007/fix-17101
Fix column aliases in _cat/indices, _cat/nodes and _cat/shards APIs
2016-03-22 15:37:21 +01:00
Jason Tedor 5dc48e71d0 Use mock filesystem during install plugins tests
This commit sets up the default filesystem used during install plugins
tests. A hack is neeeded to handle the temporary directory because the
system property "java.io.tmpdir" will have been initialized to a value
that is sensible for the default filesystem, but not necessarily to a
value that makes sense for the mock filesystem in use during the
tests. This property is restored after each test.
2016-03-22 10:25:27 -04:00
Boaz Leskes 533c967a2d Revert "Removed index level metadata election #17233"
This reverts commit 1264ee79b6.
2016-03-22 14:35:42 +01:00
Christoph Büscher 14f45c1784 Merge pull request #17146 from cbuescher/sort-add-build
For the refactoring of SortBuilders related to #10217, each SortBuilder needs to get a build()
method that produces a SortField according to the SortBuilder parameters on the shard.
2016-03-22 13:46:50 +01:00
Simon Willnauer 75d5b83367 Improve error message if resource files have illegal encoding
This commit fixes string formatting issues in the error handling and
provides a bettter error message if malformed input is detected.
This commit also adds tests for both situations.

Relates to #17212
2016-03-22 13:29:07 +01:00
Christoph Büscher 697174dcb0 Make sure to use nestedScope levels when building nested filters 2016-03-22 13:28:40 +01:00
Christoph Büscher 25da6b2f2e Merge branch 'master' into sort-add-build 2016-03-22 12:20:56 +01:00
Christoph Büscher ff021c60d9 Merge pull request #17238 from cbuescher/simplify-nestedInnerQueryParseSupport
Remove unused methods and fields in NestedInnerQueryParseSupport
2016-03-22 12:16:44 +01:00
Boaz Leskes b07a8185a7 Wait for metadata to stabilize before checking for it after opening indices in testMetaWrittenWhenIndexIsClosedAndMetaUpdated 2016-03-22 11:36:42 +01:00
Christoph Büscher 20417262e2 Remove unused methods and fields in NestedInnerQueryParseSupport 2016-03-22 11:32:24 +01:00
Simon Willnauer 33521fc27c Detach IndexShard from node services
this is the last step to remove node level service from IndexShard.
This means that tests can now more easily create an IndexShard instance
without starting a node and removes the dependency between IndexShard and Client/ScriptService
2016-03-22 11:02:04 +01:00
javanna eebd0cfccd Merge branch 'master' into enhancement/remove_node_client_setting 2016-03-22 10:34:40 +01:00
Martijn van Groningen 8f22a01bbd ingest: Give the `foreach` processor access to the rest of the document.
Closes #17147
2016-03-22 10:32:13 +01:00
Boaz Leskes 1264ee79b6 Removed index level metadata election #17233
When a master is elected, it reaches out to all master nodes for their cluster state, selecting the one with the highest version. At the moment, we do another round to select the index metadata with the highest version as well. This is not needed - the election of a cluster state is enough - we should just use whatever indices are in it.

Closes #17233
2016-03-22 10:28:01 +01:00
Areek Zillur ec5419048e cleanup writing upgraded index state
In #17187, we upgrade index state after upgrading
index folder structure. As we don't have to write
the upgraded state in the old index folder structure,
we can cleanup how we write upgraded index state.
2016-03-21 18:59:37 -04:00
Simon Willnauer a0c68c281c Improve error message if setting is not found
We can do better than just throwing an error when we don't find a
setting. It's actually trivial to leverage lucenes slow LD StringDistance
to find possible candiates for a setting to detect missspellings and suggest
a possible setting.
This commit adds error messages like:

 * `unknown setting [index.numbe_of_replica] did you mean [index.number_of_replicas]?`

rather than just reporting the setting as unknown
2016-03-21 23:13:24 +01:00
Simon Willnauer 8127a06b2e Recover broken IndexMetaData as closed
Today if something is wrong with the IndexMetaData we detect it very
late and most of the time if that happens we already allocated the index
and get endless loops and full log files on data-nodes. This change tries
to verify IndexService creattion during initial state recovery on the master
and if the recovery fails the index is imported as `closed` and won't be allocated
at all.

Closes #17187
2016-03-21 22:50:58 +01:00
Simon Willnauer 7f16a1d9a7 Improve upgrade experience of node level index settings
In 5.0 we don't allow index settings to be specified on the node level ie.
in yaml files or via commandline argument. This can cause problems during
upgrade if this was used extensively. For instance if analyzers where
specified on a node level this might cause the index to be closed when
imported (see #17187). In such a case all indices relying on this
must be updated via `PUT /${index}/_settings`. Yet, this API has slightly
different semantics since it overrides existing settings. To make this less
painful this change adds a `preserve_existing` parameter on that API to ensure
we have the same semantics as if the setting was applied on the node level.

This change also adds a better error message and a change to the migration guide
to ensure upgrades are smooth if index settings are specified on the node level.

If a index setting is detected this change fails the node startup and prints a message
like this:
```
*************************************************************************************
Found index level settings on node level configuration.

Since elasticsearch 5.x index level settings can NOT be set on the nodes
configuration like the elasticsearch.yaml, in system properties or command line
arguments.In order to upgrade all indices the settings must be updated via the
/${index}/_settings API. Unless all settings are dynamic all indices must be closed
in order to apply the upgradeIndices created in the future should use index templates
to set default values.

Please ensure all required values are updated on all indices by executing:

curl -XPUT 'http://localhost:9200/_all/_settings?preserve_existing=true' -d '{
  "index.number_of_shards" : "1",
  "index.query.default_field" : "main_field",
  "index.translog.durability" : "async",
  "index.ttl.disable_purge" : "true"
}'
*************************************************************************************
```
2016-03-21 20:12:18 +01:00
Areek Zillur bd44f37580 Merge pull request #17177 from areek/enhancement/change_shard_state_format
Always write shard state in SMILE format
2016-03-21 14:32:47 -04:00
Alexander Reelsen bf98a4455b PluginManager: Add xpack as official plugin
In order to be able to install `xpack` as a plugin and unless this works
for cross product packs, this adds xpack as an official plugin.
2016-03-21 18:51:17 +01:00
javanna 58775d885d adapt TribeServiceTests after merge 2016-03-21 18:05:44 +01:00
javanna 42ea2bb211 fix compile errors after merge 2016-03-21 18:00:31 +01:00
Jason Tedor e6eefcb142 Jimfs throws IAE when it should throw UOE
This commit adds a hack to detect when Jimfs throws an IAE where it
should be throwing an UOE. Namely, the method
FileSystemProvider#createDirectory should be throwing an UOE if an
attempt is made to set attributes that the filesystem does not support,
but instead Jimfs violates this and throws an IAE.
2016-03-21 12:44:29 -04:00
javanna bf390a935e Merge branch 'master' into enhancement/remove_node_client_setting 2016-03-21 17:18:23 +01:00
jaymode 98123e9775 Implement available for all StreamInput classes
There are some implementation of StreamInput that implement the available method
and there are others that do not implement this method. This change makes the
available method abstract in the StreamInput class and implements the method where
it was not previously implemented.
2016-03-21 09:30:20 -04:00
Jason Tedor 6db6c15d06 Add tests of POSIX handling for installing plugins
This commit refactors the unit tests for installing plugins to test
against mock filesystems (as well as the native filesystem) for better
test coverage. This commit also adds tests that cover the POSIX
attributes handling when installing plugins (e.g., ensuring that the
plugins directory has the right permissions, the bin directory has
execute permissions, and the config directory has the same owner and
group as its parent).
2016-03-21 09:02:25 -04:00
Jason Tedor 554eb8aa87 Refactor POSIX handling when installing plugins
This commit refactors the handling of POSIX filesystem attributes when
install plugins.
2016-03-21 09:02:12 -04:00
Boaz Leskes 2d1152ebac Remove ClusterService interface, in favor of it's only production instance #17183
We current have a ClusterService interface, implemented by InternalClusterService and a couple of test classes. Since the decoupling of the transport service and the cluster service, one can construct a ClusterService fairly easily, so we don't need this extra indirection.

Closes #17183
2016-03-21 13:55:10 +01:00
Colin Goodheart-Smithe 4335997017 Aggregations: Fixes the defaults for `keyed` in the percentiles aggregations
During the aggregation refactoring the default value for `keyed` in the `percentiles` and `percentile_ranks` aggregation was inadvertently changed from `true` to `false`. This change reverts the defaults to the old (correct) value
2016-03-21 11:46:55 +00:00
Martijn van Groningen e3b7e5d75a percolator: Replace percolate api with the new percolator query
Also replaced the PercolatorQueryRegistry with the new PercolatorQueryCache.

The PercolatorFieldMapper stores the rewritten form of each percolator query's xcontext
in a binary doc values field. This make sure that the query rewrite happens only during
indexing (some queries for example fetch shapes, terms in remote indices) and
the speed up the loading of the queries in the percolator query cache.

Because the percolator now works inside the search infrastructure a number of features
(sorting fields, pagination, fetch features) are available out of the box.

The following feature requests are automatically implemented via this refactoring:

Closes #10741
Closes #7297
Closes #13176
Closes #13978
Closes #11264
Closes #10741
Closes #4317
2016-03-21 12:21:50 +01:00
Jason Tedor 4fad18d3bc Fix line-length violations in InstallPluginCommand
This commit fixes the line-length checkstyle violations in
InstallPluginCommand.java and removes this from the list of files for
which the line-length check is suppressed.
2016-03-20 21:52:21 -04:00
Boaz Leskes 032678f0c3 Remove unused refreshScheduledFuture in IndexShard
It has been replaced by the logic in IndexService
2016-03-20 21:22:56 +01:00
Boaz Leskes ef4293a993 Disable more usage of PosixPermission on Windows in InstallPluginCommand
Releates to #17201
2016-03-19 12:02:42 +01:00
Boaz Leskes ee95c0a384 Don't use PosixPermission on Windows, when creating temp plugin folders 2016-03-19 11:35:05 +01:00
Ryan Ernst f71f0d6010 Revert "Build: Switch to maven-publish plugin"
This reverts commit a90a2b34fc.
2016-03-18 17:22:25 -07:00
Jason Tedor 96c6ac2d3f Merge pull request #17176 from jasontedor/its-not-easy-being-green
Vagrant tests should be green
2016-03-18 15:28:37 -04:00
Christoph Büscher 4c3d889de4 Adding tests for median sort mode to GeoDistanceSortBuilderIT
Also adding checks for median SortMode on non-numeric field types
to FieldSortBuilder, removing some unused code and switching
GeoDistanceSortBuilder to using ParseField.
2016-03-18 15:57:55 +01:00
Simon Willnauer 99321f068f Revert "Revert "Merge pull request #17182 from s1monw/issues/17090""
This reverts commit b693a520ee.
2016-03-18 11:24:55 +01:00
Simon Willnauer b693a520ee Revert "Merge pull request #17182 from s1monw/issues/17090"
This reverts commit 0fe47f8136, reversing
changes made to 3b17ddcd46.
2016-03-18 10:37:44 +01:00
Simon Willnauer 0fe47f8136 Merge pull request #17182 from s1monw/issues/17090
Provide better error message when an incompatible node connects to a node

We should give a better exception message when an incompatible node connects
and we receive a messeage. This commit adds a clear excpetion based on the
protocol version received instead of throwing cryptic messages about not fully reaed
buffer etc.

Relates to #17090
2016-03-18 10:17:05 +01:00
Martijn van Groningen 3b17ddcd46 Removed old 1.x parent/child logic that should have been removed.
`0` really means, don't match any child docs.
2016-03-18 10:07:27 +01:00
Martijn van Groningen 1dd2be81c3 nested / parent child: Removed `total` score mode in favour of `sum` score mode.
Closes #17083
2016-03-18 10:07:26 +01:00
Simon Willnauer 7898522514 Provide better error message when an incompatible node connects to a node
We should give a better exception message when an incompatible node connects
and we receive a messeage. This commit adds a clear excpetion based on the
protocol version received instead of throwing cryptic messages about not fully reaed
buffer etc.

Relates to #17090
2016-03-18 09:51:00 +01:00
Ryan Ernst 69ff402b62 Still create the archive name for core as elasticsearch 2016-03-17 13:35:16 -07:00
Areek Zillur 771a34d47d change shard state format to smile 2016-03-17 16:29:58 -04:00
Jason Tedor 9f73152940 Fix plugins permissions 2016-03-17 15:35:54 -04:00
Jason Tedor dacb96ba61 Fix plugin installation logging 2016-03-17 15:35:54 -04:00
Jason Tedor ebdacb1297 Fix list of modules and official plugins 2016-03-17 15:35:54 -04:00
Ryan Ernst 6af4c43c4f Merge pull request #17128 from rjernst/maven_publish
Build: Switch to maven-publish plugin
2016-03-17 11:53:50 -07:00
Christoph Büscher d9de129995 Adressing review comments 2016-03-17 18:57:45 +01:00
Zachary Tong 1010e1e543 [TEST] Make test less fragile by sorting only on _uid
The previous method sorted first by _score, then _uid.  In certain situations, this allowed
floating point errors to slightly alter the sort order, causing test failure.

We only sort on _uid now, which should be deterministic and allow comparison of ten
documents.  Not quite as useful, but less fragile and we still check to make sure num hits
and max score are identical.

Closes #17164
2016-03-17 13:51:52 -04:00
Boaz Leskes 6ae738a2eb Debug log on testMergesHappening 2016-03-17 17:22:47 +01:00
Martijn van Groningen 7589ed7fbd Merge pull request #17167 from martijnvg/parent_id_query_take_child_type_into_account
parent_id query should take the child type into account too
2016-03-17 17:01:36 +01:00
Martijn van Groningen 68282dd9e9 parent/child: `parent_id` query should take the child type into account too.
If this query doesn't take the child type into account then it can match other
child document types pointing to the same parent type and that have the same id too.
2016-03-17 14:58:57 +01:00
Simon Willnauer e91a141233 Prevent index level setting from being configured on a node level
Today we allow to set all kinds of index level settings on the node level which
is error prone and difficult to get right in a consistent manner.
For instance if some analyzers are setup in a yaml config file some nodes might
not have these analyzers and then index creation fails.

Nevertheless, this change allows some selected settings to be specified on a node level
for instance:
 * `index.codec` which is used in a hot/cold node architecture and it's value is really per node or per index
 * `index.store.fs.fs_lock` which is also dependent on the filesystem a node uses

All other index level setting must be specified on the index level. For existing clusters the index must be closed
and all settings must be updated via the API on each of the indices.

Closes #16799
2016-03-17 14:42:18 +01:00
Tanguy Leroux 6441852618 Fix OOM in AbstractXContentParser
This commit fixes an OOM error that happens when the XContentParser.readList() method is asked to parse a single value instead of an array. It fixes the UpdateRequest parsing as well as remove some leniency in the readList() method so that it expect to be in an array before parsing values.

closes #15338
2016-03-17 14:37:33 +01:00
Colin Goodheart-Smithe d17fd335e4 fix range query rewrite so it rewrites correctly when shard min value == shard max value 2016-03-17 10:24:11 +00:00
Colin Goodheart-Smithe 30cdc11d75 Enable the indices request cache by default
Now we have 16870 we can enable the request cache by default. The caching can still be disabled on a per request basis and can still be disabled in the settings, only the default value has changed. For now this is done regardless of whether the shard is active or inactive.

Closes #17134
2016-03-17 09:06:32 +00:00
Colin Goodheart-Smithe 0f6b8dd848 Adds a rewrite phase to queries on the shard level
This change adds a rewrite phase to the queries on the shard before they are assessed for caching or executed. This allows the opportunity to rewrite queries as faster running simpler queries based on attributes known to only the shard itself. The first query to implement this is the RangeQueryBuilder which will rewrite to a MatchAllQueryBuilder if the range of terms on the shard is a subset of the query and rewrites to a MatchNoneQueryBuilder if the range of terms on the shard is completely outside the query.
2016-03-17 08:26:38 +00:00
Areek Zillur da24bfe542 simplify handling top-level suggest results 2016-03-17 00:34:19 -04:00
Areek Zillur 22e12ab7c3 cleanup request parsing in RestSearchAction 2016-03-16 19:03:30 -04:00
Areek Zillur 44b3dc95a0 remove irrelvant tests 2016-03-16 18:48:02 -04:00
Areek Zillur d1eba4baf2 fix tests 2016-03-16 18:48:02 -04:00
Areek Zillur 7aeeb52cf6 Standardize state format type for global and index level metadata
Currently, global and index level state format type can be configured through gateway.format.
This commit removes the ability to configure format type for these states.
Now we always store these states in SMILE format and ensure we always write them
to disk in the most compact way.
2016-03-16 18:48:02 -04:00
Simon Willnauer 0a12e7bb5b Revert "Added version 2.4.0 to Version"
This reverts commit 8c4aa75b0c.
2016-03-16 22:17:28 +01:00
Clinton Gormley 8c4aa75b0c Added version 2.4.0 to Version 2016-03-16 17:46:16 +01:00
Nik Everett 80f638b56a Support scheduled commands in current context
Adds support for scheduling commands to run at a later time on another
thread pool in the current thread's context:

```java
Runnable someCommand = () -> {System.err.println("Demo");};
someCommand = threadPool.getThreadContext().preserveContext(someCommand);
threadPool.schedule(timeValueMinutes(1), Names.GENERAL, someCommand);
```

This happens automatically for calls to `threadPool.execute` but `schedule`
and `scheduleWithFixedDelay` don't do that, presumably because scheduled
tasks are usually context-less. Rather than preserve the current context
on all scheduled tasks this just makes it possible to preserve it using
the syntax above.

To make this all go it moves the Runnables that wrap the commands from
EsThreadPoolExecutor into ThreadContext.

This, or something like it, is required to support reindex throttling.
2016-03-16 12:29:06 -04:00
Christoph Büscher 1908d6dc73 Add build() method to SortBuilder implementations
For the current refactoring of SortBuilders related to #10217,
each SortBuilder should get a build() method that produces a
SortField according to the SortBuilder parameters on the shard.

This change also slightly refactors the current parse method in
SortParseElement to extract an internal parse method that returns
a list of sort fields only needs a QueryShardContext as input
instead of a full SearchContext. This allows using this internal
parse method for testing.
2016-03-16 17:21:06 +01:00
Alexander Kazakov 51ac97000b Fix column aliases in _cat/indices, _cat/nodes and _cat/shards APIs #17101 2016-03-16 19:18:43 +03:00
Christoph Büscher 6ddf9ae92f Merge branch 'master' into feature-suggest-refactoring 2016-03-16 15:27:02 +01:00
Nik Everett 7197172047 [reindex] Properly register status
Without this commit fetching the status of a reindex from a node that isn't
coordinating the reindex will fail. This commit properly registers reindex's
status so this doesn't happen. To do so it moves all task status registration
into NetworkModule and creates a method to register other statuses which the
reindex plugin calls.
2016-03-16 07:40:49 -04:00
Christoph Büscher 39667b5793 Merge branch 'master' into feature-suggest-refactoring
Conflicts:
	docs/reference/migration/migrate_5_0/java.asciidoc
2016-03-16 12:06:42 +01:00
Christoph Büscher 79356c8a3b Query DSL: `constant_score` should throw error on more than one filter
When specifying more than one `filter` in a `constant_score`
query, the last one will be the only one that will be
executed, overwriting previous filters. It should rather
raise a ParseException to notify the user that only one
filter query is accepted.

Closes #17126
2016-03-16 10:59:48 +01:00
Christoph Büscher 43f71d1ff5 Merge pull request #17111 from cbuescher/fix-potential-parseBug-GeoDistanceSort
Fix a potential parsing problem in GeoDistanceSortParser.
For an input like `{ [...], "coerce" = true, "ignore_malformed" = false }` the parser will 
fail to parse the ignore_malformed boolean flag and will fall through to the last 
else-branch where the boolean flag will be parsed as geo-hash and `ignore_malformed`
treated as field name.
2016-03-16 10:17:08 +01:00
Simon Willnauer 302087d686 [TEST] Wait for ongoing merges in testRenewSyncFlush
Now that we also renew on forceMerge we might get a concurrent
flush while we are flushing on the test level
2016-03-16 09:37:50 +01:00
Ryan Ernst a90a2b34fc Build: Switch to maven-publish plugin
The build currently uses the old maven support in gradle. This commit
switches to use the newer maven-publish plugin. This will allow future
changes, for example, easily publishing to artifactory.

An additional part of this change makes publishing of build-tools part
of the normal publishing, instead of requiring a separate upload step
from within buildSrc. That also sets us up for a follow up to enable
precomit checks on the buildSrc code itself.
2016-03-15 19:16:37 -07:00
Jason Tedor 0f00c14afc Remove dead code in FTL#simpleMatchToFullName
This commit removes some dead code that resulted from removing the
ability for a field to have different names (after enforcing that fields
have the same full and index name).

Closes #17127
2016-03-15 22:11:48 -04:00
Jason Tedor 618441aea3 Merge pull request #17088 from jasontedor/simplify-bootstrap-settings
Bootstrap does not set system properties
2016-03-15 19:25:16 -04:00
Jason Tedor 4ee90db13d Remove path.home command-line setting 2016-03-15 16:29:29 -04:00
Yannick Welsch 7fb07a9a53 Merge pull request #17112 from ywelsch/enhance/forbid-sysout-for-tests
Forbid test sources to use System.out.println and Throwable.printStackTrace
2016-03-15 16:36:34 +01:00
Jason Tedor d1b85f69ef Shorter name for test class
This commit renames the ElasticsearchCommandLineParsingTests to
ElasticsearchCliTests.
2016-03-15 11:22:53 -04:00
Christoph Büscher 52852bdf39 Fix a potential parsing problem in GeoDistanceSortParser
Test revealed a potential problem in the current GeoDistanceSortParser.
For an input like `{ [...], "coerce" = true, "ignore_malformed" = false }
the parser will fail to parse the `ignore_malformed` boolean flag and
will fall through to the last else-branch where the boolean flag will be
parsed as geo-hash and `ignore_malformed` treated as field name.

Adding fix and test that will fail with the old parser code.
2016-03-15 15:41:54 +01:00
Yannick Welsch f5e6db4090 Remove System.out.println and Throwable.printStackTrace from tests 2016-03-15 15:40:37 +01:00
Jason Tedor 088dea8863 Fix javadoc comment on Elasticsearch#init
This commit fixes a stale javadoc comment on Elasticsearch#init; the
method is now visible for testing.
2016-03-15 09:35:58 -04:00
Jason Tedor fac0f990fc Rename "daemonize" to "foreground"
This commit renames the Bootstrap#init parameter "daemonize" to
"foreground" for clarity.
2016-03-15 09:31:33 -04:00
Simon Willnauer b4300da816 Merge pull request #17108 from s1monw/issues/17019
Try to renew sync ID if `flush=true` on forceMerge

Today we do a force flush which wipes the sync ID if there is one which
can cause the lost of all benefits of the sync ID ie. fast recovery.
This commit adds a check to renew the sync ID if possible. The flush call
is now also not forced since the IW will show pending changes if the forceMerge added new segments.
if we keep using force we will wipe the sync ID even if no renew was actually needed.

Closes #17019
2016-03-15 14:16:25 +01:00
Simon Willnauer 52c283cdee [TEST] Add test that ensures we never bump the minor version of lucene in a bugfix release 2016-03-15 14:15:41 +01:00
Simon Willnauer 2390eb5ed4 Add 2.2.1 to the version lookup table 2016-03-15 14:07:20 +01:00
Christoph Büscher 7ccf7adbdf Merge pull request #17074 from cbuescher/sort-serialization-scriptsort
Adding methods and tests to ScriptSortBuilder that makes it implement NamedWritable and adds a fromXContent() method needed to read itseld from xContent. Also changing sortMode() setters in 
FieldSortBuilder, GeoDistanceSortBuilder and ScriptSortBuilder to accept an enum instead of a String
value.

Relates to #15178
2016-03-15 13:59:10 +01:00
Clinton Gormley 0793f00cb9 Added version 2.2.1 and bwc indices for 2.2.1 2016-03-15 13:48:38 +01:00
Simon Willnauer 8b201fda46 Merge pull request #17106 from s1monw/fix_can_delete
IndicesStore checks for `allocated elsewhere` for every shard not alocated on the local node

On each cluster-state update we check on the local node if we can delete some shards content.
For this we linearly walk all shards and check if they are allocated and started on another node
and if we can delete them locally. if we can delete them locally we go and ask other nodes if we can
delete them and then if the shared IS active elsewhere issue a state update task to delete it. Yet,
there is a bug in IndicesService#canDeleteShardContent which returns `true` even if that shards
datapath doesn't exist on the node which causes tons of unnecessary node to node communication and
as many state update task to be issued. This can have large impact on the cluster state processing
speed. 

**NOTE:** This only happens for shards that have at least one shard allocated on the node ie. if an `IndexService` exists.
2016-03-15 12:59:45 +01:00
Simon Willnauer 10333e2f05 IndicesStore checks for `allocated elsewhere` for every shard not allocated on the local node
On each clusterstate update we check on the local node if we can delete some shards content.
For this we linearly walk all shards and check if they are allocated and started on another node
and if we can delete them locally. if we can delete them locally we go and ask other nodes if we can
delete them and then if the shared IS active elsewhere issue a state update task to delete it. Yet,
there is a bug in IndicesService#canDeleteShardContent which returns `true` even if that shards
datapath doesn't exist on the node which causes tons of unnecessary node to node communciation and
as many state update task to be issued. This can have large impact on the cluster state processing
speed.

**NOTE:** This only happens for shards that have at least one shard allocated on the node
ie. if an `IndexService` exists.

Closes #17106
2016-03-15 12:58:36 +01:00
Simon Willnauer e8152d375e Try to renew sync ID if `flush=true` on forceMerge
Today we do a force flush which wipes the sync ID if there is one which
can cause the lost of all benefits of the sync ID ie. fast recovery.
This commit adds a check to renew the sync ID if possible. The flush call
is now also not forced since the IW will show pending changes if the forceMerge added new segments.
if we keep using force we will wipe the sync ID even if no renew was actually needed.

Closes #17019
2016-03-15 12:50:57 +01:00
Christoph Büscher bc84cdfed1 Using SortMode enum in all sort builders 2016-03-15 12:43:19 +01:00
Christoph Büscher b4b874f0d8 Merge branch 'master' into feature-suggest-refactoring 2016-03-15 12:11:39 +01:00
Isabel Drost-Fromm be09778dc0 Fix compile error
that's what you get for a "let me quickly try something out here"
2016-03-15 11:12:38 +01:00
Christoph Büscher 1f1f6861b7 Merge branch 'master' into sort-serialization-scriptsort
Conflicts:
	core/src/test/java/org/elasticsearch/search/sort/AbstractSortTestCase.java
2016-03-15 11:03:28 +01:00
Isabel Drost-Fromm 4b5783bdd9 Merge pull request #16127 from MaineC/enhancement/15178-sort-fieldsortbuilder
Refactor FieldSortBuilder to add serialization support
2016-03-15 10:48:20 +01:00
Simon Willnauer 29268f100b Reenable CreateIndexIT#testCreateAndDeleteIndexConcurrently
Since #16442 is merged we should be able to reenable this test as a followup
of #15853 - all issues blocking it have been resolved I guess.
2016-03-15 09:31:13 +01:00
Simon Willnauer 345e988bbc Merge pull request #17072 from s1monw/add_backwards_rest_tests
Add infrastructure to run REST tests on a multi-version cluster

This change adds the infrastructure to run the rest tests on a multi-node
cluster that users 2 different minor versions of elasticsearch. It doesn't implement
any dedicated BWC tests but rather leverages the existing REST tests.

Since we don't have a real version to test against, the tests uses the current version
until the first minor / RC is released to ensure the infrastructure works.

Given the amount of problems this change already found I think it's worth having this run with our test suite by default. The structure of this infra will likely change over time but for now it's a step into the right direction. We will likely want to split it up into integTests and integBwcTests etc. so each plugin can have it's own bwc tests but that's left for future refactoring.
2016-03-15 09:17:43 +01:00
Areek Zillur c3078f4d65 adapt tests to use index uuid as folder name 2016-03-14 23:24:24 -04:00
Areek Zillur 2b18a3ce1d use index uuid as folder name to decouple index folder name from index name 2016-03-14 23:24:19 -04:00
Areek Zillur 3daa83b2d2 remove redundant getters in MetaData 2016-03-14 23:24:12 -04:00
Areek Zillur 35f7cfb6c0 Add upgrader to upgrade old indices to new naming convention 2016-03-14 23:24:05 -04:00
Ali Beyad f527a034a3 Merge remote-tracking branch 'upstream/master' into feature-suggest-refactoring 2016-03-14 17:25:02 -04:00
Simon Willnauer bd96075f7f Merge branch 'master' into add_backwards_rest_tests 2016-03-14 21:18:42 +01:00
Christoph Büscher 17a420e6aa Adressing review comments, adding parsing tests 2016-03-14 15:24:49 +01:00
Christoph Büscher 40f3501d7f Merge branch 'master' into feature-suggest-refactoring 2016-03-14 14:57:57 +01:00
Simon Willnauer e4bed0c97e Improve validation error message on PutMappingRequest 2016-03-14 14:57:43 +01:00
Simon Willnauer 0ebcef0bb4 wrap line after 140 chars 2016-03-14 14:51:43 +01:00
Simon Willnauer 9f382da5d3 Add better validation error message and a dedicated test 2016-03-14 14:27:35 +01:00
Christoph Büscher 3fe07a9754 Adding enum for SortMode and use it in ScriptSortBuilder 2016-03-14 14:18:18 +01:00
Isabel Drost-Fromm 744de1f6cb Throw ParsingExceptions instead of IllegalArgument
... keeps track of the position of the error.
2016-03-14 14:02:22 +01:00
Isabel Drost-Fromm fb647e9bf4 Add sanity checks and support for reverse in FieldSortBuilder.parse(...)
After another round of input from @cbuescher this adds a few more sanity
checks to request parsing.

In addition adds (back) support for the reverse option.
2016-03-14 14:02:22 +01:00
Isabel Drost-Fromm 02e698bc43 Third round of comments 2016-03-14 14:02:22 +01:00
Isabel Drost-Fromm 720f47e87f Second round of comments 2016-03-14 14:02:22 +01:00
Isabel Drost-Fromm dd6e835e30 First comments. 2016-03-14 14:02:22 +01:00
Isabel Drost-Fromm 5bff6e4218 Refactor FieldSortBuilder
* adds json parsing,
* refactors json serialisation,
* adds writable parsing and serialisation,
* adds json and writable roundtrip test
2016-03-14 14:02:22 +01:00
Christoph Büscher f0074668db Merge branch 'master' into sort-serialization-scriptsort 2016-03-14 12:15:53 +01:00
Christoph Büscher 97638c95fc Merge branch 'master' into feature-suggest-refactoring
Conflicts:
	docs/reference/migration/migrate_5_0.asciidoc
2016-03-14 11:13:47 +01:00
Simon Willnauer 31740e279f Resolve index names to Index instances early
Today index names are often resolved lazily, only when they are really
needed. This can be problematic especially when it gets to mapping updates
etc. when a node sends a mapping update to the master but while the request
is in-flight the index changes for whatever reason we would still apply the update
since we use the name of the index to identify the index in the clusterstate.
The problem is that index names can be reused which happens in practice and sometimes
even in a automated way rendering this problem as realistic.
In this change we resolve the index including it's UUID as early as possible in places
where changes to the clusterstate are possible. For instance mapping updates on a node use a
concrete index rather than it's name and the master will fail the mapping update iff
the index can't be found by it's <name, uuid> tuple.

Closes #17048
2016-03-14 11:08:48 +01:00
Adrien Grand 071b396306 Fix test bug: norms are on by default on _all. 2016-03-14 09:06:11 +01:00
Adrien Grand c50c5a52d5 Rework norms parameters for 5.0. #16987
Changes:
 - no more option to configure eager/lazy loading of the norms (useless now
   that orms are disk-based)
 - only the `string`, `text` and `keyword` fields support the `norms` setting
 - the `norms` setting takes a boolean that decides whether norms should be
   stored in the index but old options are still supported to give users time
   to upgrade
 - setting a `boost` no longer implicitely enables norms (for new indices only,
   this is still needed for old indices)
2016-03-14 08:42:35 +01:00
Adrien Grand 5596e31068 Upgrade to lucene-6.0.0-f0aa4fc. #17075 2016-03-14 07:58:52 +01:00
Jason Tedor 8a05c2a2be Bootstrap does not set system properties
Today, certain bootstrap properties are set and read via system
properties. This action-at-distance way of managing these properties is
rather confusing, and completely unnecessary. But another problem exists
with setting these as system properties. Namely, these system properties
are interpreted as Elasticsearch settings, not all of which are
registered. This leads to Elasticsearch failing to startup if any of
these special properties are set. Instead, these properties should be
kept as local as possible, and passed around as method parameters where
needed. This eliminates the action-at-distance way of handling these
properties, and eliminates the need to register these non-setting
properties. This commit does exactly that.

Additionally, today we use the "-D" command line flag to set the
properties, but this is confusing because "-D" is a special flag to the
JVM for setting system properties. This creates confusion because some
"-D" properties should be passed via arguments to the JVM (so via
ES_JAVA_OPTS), and some should be passed as arguments to
Elasticsearch. This commit changes the "-D" flag for Elasticsearch
settings to "-E".
2016-03-13 20:09:15 -04:00
Boaz Leskes e472d7894b Log suppressed stack traces under DEBUG
To make API's output more easy to read we are suppressing stack traces (#12991) unless explicitly requested by setting `error_trace=true` on the request. To compensate we are logging the stacktrace into the logs so people can look it up even the error_trace wasn't enabled. Currently we do so using the `INFO` level which can be verbose if an api is called repeatedly by some automation.  For example, if someone tries to read from an index that doesn't exist we will respond with a 404 exception and log under info every time. We should reduce the level to `DEBUG` as we do with other API driven errors. Internal errors (rest codes >=500) are logged as WARN.

Closes #16627
2016-03-13 21:50:17 +01:00
David Pilato 25531b7299 Update after last review
We check for null. Test added as well.
2016-03-13 15:36:17 +01:00
David Pilato e9e1e25998 Fix after merge with master 2016-03-13 15:14:07 +01:00
David Pilato 9acb0bb28c Merge branch 'master' into pr/16598-register-filter-settings
# Conflicts:
#	core/src/main/java/org/elasticsearch/cluster/service/InternalClusterService.java
#	core/src/main/java/org/elasticsearch/common/settings/IndexScopedSettings.java
#	core/src/main/java/org/elasticsearch/common/settings/Setting.java
2016-03-13 14:52:10 +01:00
Yannick Welsch 625695a92a [TEST] MockRepository should also unblock repositories that are not blocked yet 2016-03-13 14:12:29 +01:00
Simon Willnauer 121e7c8ca4 Add infrastructure to run REST tests on a multi-version cluster
This change adds the infrastructure to run the rest tests on a multi-node
cluster that users 2 different minor versions of elasticsearch. It doesn't implement
any dedicated BWC tests but rather leverages the existing REST tests.

Since we don't have a real version to test against, the tests uses the current version
until the first minor / RC is released to ensure the infrastructure works.

Relates to #14406
Closes #17072
2016-03-13 10:52:39 +01:00
Ryan Ernst 8b26c260d1 Plugins: Enforce plugin zip does not contain zip entries outside of the unzip dir
When unzipping a plugin zip, the zip entries are resolved relative to
the directory being unzipped into. However, there are currently no
checks that the entry name was not absolute, or relatively points
outside of the plugin dir. This change adds a check for those two cases.
2016-03-11 14:53:14 -08:00
Areek Zillur 4b803d75cf nuke SuggestParseElement 2016-03-11 16:27:33 -05:00
Ryan Ernst 5f3d0067f8 Merge pull request #17024 from rjernst/cli-parsing
Cli: Switch to jopt-simple
2016-03-11 12:35:39 -08:00
Areek Zillur 97e2bab4cd clarify parsing logic in SuggestBuilder 2016-03-11 15:06:42 -05:00
Areek Zillur c9f30f2f3f scope internal methods in suggest builders 2016-03-11 15:06:36 -05:00
Ryan Ernst 5bd7da5659 Addressed PR feedback
* Fix tests still referring to -E
* add comment about missing classes
* rename writer constant
2016-03-11 11:46:23 -08:00
Yannick Welsch afb54bab44 [TEST] Wait on all data nodes to be blocked if blocks active
Fixes race condition in MockRepository where unblock happens before block
2016-03-11 20:20:07 +01:00
Ryan Ernst 591fb8f028 Merge branch 'master' into cli-parsing 2016-03-11 10:45:05 -08:00
Ali Beyad 31dcb3e18b Merge pull request #16873 from abeyad/suggester-wiring-refactoring
Change internal representation of suggesters
2016-03-11 12:34:35 -05:00
Christoph Büscher 5107388fe9 Added enum for script sort type 2016-03-11 17:32:39 +01:00
Christoph Büscher 8ab4d001e2 Make ScriptSortBuilder implement NamedWritable
This adds methods and tests to ScriptSortBuilder that
makes it implement NamedWritable and adds the fromXContent
method needed to read itseld from xContent.
2016-03-11 16:57:22 +01:00
Yannick Welsch 422df6089c [TEST] Unblock nodes if snapshot/restore test fails 2016-03-11 16:38:52 +01:00
Christoph Büscher 7aa29e3f7c Make Copy of collate parameter map
Test failures showed problems with passing down
the same collate parameter map reference from the
phrase suggestion builder to the context where.
This changes the collate parameter setters to
make a shallow copy of the map passed in.
2016-03-11 15:20:52 +01:00
Jason Tedor f465d98eb3 Add raw recovery progress to cat recovery API
This commit adds fields bytes_recovered and files_recovered to the cat
recovery API. These fields, respectively, indicate the total number of
bytes and files recovered. Additionally, for consistency, some totals
fields and translog recovery fields have been renamed.

Closes #17064
2016-03-11 08:27:09 -05:00
Christoph Büscher daeffb149c Merge branch 'master' into feature-suggest-refactoring 2016-03-11 10:37:28 +01:00
Yannick Welsch 04e55ecf6b Make logging message String constant to allow static checks 2016-03-11 10:30:59 +01:00
Yannick Welsch 718876a941 Fix wrong placeholder usage in logging statements 2016-03-11 10:30:59 +01:00
Yannick Welsch b17f4b40ba Check that number of placeholders in log message matches number of parameters 2016-03-11 10:30:59 +01:00
Adrien Grand a46d2f21c6 Fix dynamic mapper bug with deeply nested fields. 2016-03-11 09:59:38 +01:00
Ryan Ernst efd59f531d Merge pull request #17065 from rjernst/dynamic_mappers_are_hard
Fix dynamic mapper when its parent already has an update
2016-03-11 00:29:31 -08:00
Adrien Grand cb2ed50aeb Remove friction from the mapping changes in 5.0. #16991
This tries to remove friction to upgrade to 5.0 that would be caused by mapping
changes:
 - old ways to specify mapping settings (eg. store: yes instead of store:true)
   will still work but a deprecation warning will be logged
 - string mappings that only use the most common options will be upgraded
   automatically to text/keyword
2016-03-11 09:23:49 +01:00
Yannick Welsch 64e84dcc76 Reuse shard model across 3 phases in BalancedShardsAllocator 2016-03-11 08:42:25 +01:00
Yannick Welsch f6ae9ec4f6 Remove ShardsAllocators and merge allocateUnassigned, moveShards and rebalance to improve performance 2016-03-11 08:42:25 +01:00
Ryan Ernst 96ec48afcd Fix dynamic mapper when its parent already has an update
The change to move dynamic mapping handling to the end of document
parsing has an edge case which can cause dynamic mappings to fail
document parsing. If field a.b is added as an as part of the root update,
followed by a.c.d, then we need to expand the mappers on the stack,
since a is hidden inside the root update which exists on the stack.

This change adds a test for this case, as well as tries to better
document how the logic works for building up the stack before adding a
dynamic mapper.
2016-03-10 18:00:07 -08:00
Ryan Ernst 42a6869bb1 Merge pull request #17059 from elastic/fix/16864-attachment-doctypes
Fix attachments plugins with docx
2016-03-10 17:27:02 -08:00
Ryan Ernst 2f3efc3fe1 Add doc and docx rest test to mapper attachment along with
getClassLoader permission
2016-03-10 13:28:19 -08:00
Zachary Tong 9d340e6b08 Fix import formatting - do not use wildcard 2016-03-10 16:08:18 -05:00
Zachary Tong ae36262a95 Merge pull request #14745 from polyfractal/bugfix/deriv_npe
Fix NPE in Derivative Pipeline when current bucket value is null
2016-03-10 15:34:31 -05:00
Zachary Tong 39a067aa3f Fix NPE in Derivative Pipeline when current bucket is null
Sequence of events that lead to the NPE:

- avg metric returns NaN for buckets
- Movavg skips NaN or null buckets, and simply re-uses the existing bucket (e.g. doesn't add
a 'movavg' field)
- Derivative references Movavg, the bucket resolution returns null because Movavg wasn't added
to the bucket, NPE when trying to subtract null values
2016-03-10 15:17:26 -05:00
Ryan Ernst f3195cb514 Merge pull request #16798 from rjernst/dots2
Moved dynamic field handling in doc parsing to end of parsing
2016-03-10 10:03:52 -08:00
Ryan Ernst ac73d97169 Merge branch 'master' into dots2 2016-03-10 09:22:31 -08:00
Yannick Welsch 266394c3ab Fail closing or deleting indices during a full snapshot
Closes #16321
2016-03-10 18:11:47 +01:00
Ali Beyad 5a8d40e44a Fixes CompletionSuggesterBuilderTests to test for the correct expected
error message.
2016-03-10 10:39:46 -05:00
Ali Beyad 807d085218 Merge remote-tracking branch 'upstream/feature-suggest-refactoring' into suggester-wiring-refactoring 2016-03-10 09:56:08 -05:00
Martijn van Groningen 2fa33d5c47 Added ingest statistics to node stats API
The ingest stats include the following statistics:
* `ingest.total.count`- The total number of document ingested during the lifetime of this node
* `ingest.total.time_in_millis` - The total time spent on ingest preprocessing documents during the lifetime of this node
* `ingest.total.current` - The total number of documents currently being ingested.
* `ingest.total.failed` - The total number ingest preprocessing operations failed during the lifetime of this node

Also these stats are returned on a per pipeline basis.
2016-03-10 13:21:43 +01:00
Boaz Leskes cd12241e9f Decouple the TransportService and ClusterService #16872
Currently, the cluster service is tightly coupled to the transport service by both managing node connections and requiring the bound address in order to create the local disco node. This commit introduces a new NodeConnectionsService which is in charge of node connection management and makes it possible to remove all network related calls from the cluster service. The local DiscoNode is now created by DiscoveryNodeService and is set both the cluster service and the transport service during node start up.

Closes #16788
Closes #16872
2016-03-10 11:45:15 +01:00
Christoph Büscher 69c83b3459 Merge branch 'master' into feature-suggest-refactoring
Conflicts:
	core/src/main/java/org/elasticsearch/action/suggest/TransportSuggestAction.java
        core/src/main/java/org/elasticsearch/search/suggest/SuggestParseElement.java
	core/src/main/java/org/elasticsearch/search/suggest/SuggestionSearchContext.java
	core/src/main/java/org/elasticsearch/search/suggest/phrase/PhraseSuggester.java
2016-03-10 11:44:36 +01:00
Christoph Büscher b4db26eaf9 Sort: Move up `order` field to SortBuilder
Currently all SortBuilder implementations have their separate order
field. This PR moves this up to SortBuilder, together with setter
and getter and makes sure the default is set to SortOrder.ASC
except for `_score` sorting where the default is SortOrder.DESC.
2016-03-10 10:47:34 +01:00
Isabel Drost-Fromm 7621c8875f Merge pull request #16999 from MaineC/sort-parser-simplification
Moves SortParser:parse(...) to only require QueryShardContext
2016-03-10 10:21:02 +01:00
Ali Beyad a6662d78df Improves the organization of the suggestion builder tests 2016-03-10 00:14:27 -05:00
Ali Beyad 06487b0ac5 Change internal representation of suggesters to instances of
SuggestBuilder instead of raw bytes.
2016-03-10 00:12:39 -05:00
Nik Everett 7d35db6cc7 Merge fix for expressions starting with * 2016-03-09 18:27:40 -05:00
Ryan Ernst 1bc8d0249f Merge pull request #16286 from fforbeck/fix/15269
Skipping hidden files compilation for script service
2016-03-09 14:45:24 -08:00
Nicholas Knize 61f39e6c92 GeoPointV2 update docs and query builders
This commit updates the documentation for GeoPointField by removing all references to the coerce and doc_values parameters. DocValues are enabled in lucene GeoPointField by default (required for boundary filtering). The QueryBuilders are updated to automatically normalize points (ignoring the coerce parameter) for any index created onOrAfter version 2.2.
2016-03-09 16:09:44 -06:00
Ryan Ernst 0344af0750 Remove unnecessary comment 2016-03-09 13:26:33 -08:00
Ryan Ernst c48e6b86f6 Limit update to fields and objects which were actually modified, and
simplify root update creation.
2016-03-09 13:24:46 -08:00
Jason Tedor bd9e908b7c Waiting for primary terms 2016-03-09 15:17:23 -05:00
Jason Tedor 12a6f36a34 Log shard after translog snapshot during recovery 2016-03-09 15:15:58 -05:00
Ryan Ernst bbdbfe7373 Merge branch 'master' into dots2 2016-03-09 11:28:27 -08:00
Simon Willnauer 016bc3ea2b Merge pull request #17001 from s1monw/cut_over_to_uuid
Use index UUID to lookup indices on IndicesService

Today we use the index name to lookup index instances on the IndicesService
which applied to search requests but also to index deletion etc. This commit
moves the interface to expect an Index instance which is a tuple
and looks up the index by uuid rather than by name. This prevents accidental modification
of the wrong index if and index is recreated or searching from the wrong index in such a case.
Accessing an index that has the same name but different UUID will now result in an IndexNotFoundException.
2016-03-09 19:44:36 +01:00
Simon Willnauer e72dac91b3 Use index UUID to lookup indices on IndicesService
Today we use the index name to lookup index instances on the IndicesService
which applied to search reqeusts but also to index deletion etc. This commit
moves the interface to expcet and `Index` instance which is a <name, uuid> tuple
and looks up the index by uuid rather than by name. This prevents accidential modificaiton
of the wrong index if and index is recreated or searching from the _wrong_ index in such a case.
Accessing an index that has the same name but different UUID will now result in an IndexNotFoundException.

Closes #17001
2016-03-09 19:42:15 +01:00
Jason Tedor 563304df9e No old states polluting pending states queue
This commit adds a guard against an old cluster state that arrives out
of order from the last seen cluster state from the current master from
polluting the pending cluster states queue. Without this guard, such a
state can end up stuck in the pending states queue.
2016-03-09 13:12:19 -05:00
Areek Zillur e7cffa5e9f simplify mocking field type in SuggestionBuilderTests 2016-03-09 11:53:44 -05:00
Areek Zillur 55c58c56a8 incorporate feedback 2016-03-09 11:53:34 -05:00
Areek Zillur 987f2f5aa8 cleanup 2016-03-09 11:53:13 -05:00
Areek Zillur 5bb72dbcd2 construct suggestion context from query context 2016-03-09 11:52:59 -05:00
Alexander Kazakov 2ce90c001c Don't return all indices immediately if count of expressions >1 and first expression is * #17027 2016-03-09 19:43:56 +03:00
Nicholas Knize d09ee3f174 Remove .geohash suffix from GeoDistanceQuery and GeoDistanceRangeQuery
Occasionally the .geohash suffix in Geo{Distance|DistanceRange}Query would conflict with a mapping that defines a sub-field by the same name. This occurs often with nested and multi-fields a mapping defines a geo_point sub-field using the field name "geohash". Since the QueryParser already handles parsing geohash encoded geopoints without requiring the ".geohash" suffix, the suffix parsing can be removed altogether.

This commit removes the .geohash suffix parsing, adds explicit test coverage for the nested query use-case, and adds random distance queries to the nested query test suite.
2016-03-09 09:35:27 -06:00
Ali Beyad e411cbb060 Fixes the DiscoveryWithServiceDisruptionsIT#testIndicesDeleted test
In particular, this test ensures we don't restart the master node until
we know the index deletion has taken effect on master and the master
eligible nodes.

Closes #16917
Closes #16890
2016-03-09 10:28:12 -05:00
Christoph Büscher 11b18a9963 Sort: Make ScoreSortBuilder implement NamedWriteable and add fromXContent parsing
This change makes ScoreSortBuilder implement NamedWriteable, adds
equals() and hashCode() and also implements parsing ScoreSortBuilder
back from xContent. This is needed for the ongoing Search refactoring.
2016-03-09 14:42:06 +01:00
Simon Willnauer f8ab6a6669 [TEST] Make boost more prominent in test since with new default similarity it might score lower without the boost 2016-03-09 11:43:38 +01:00
Martijn van Groningen b9b5c15fe1 test: ensure the each node sees 2 nodes. 2016-03-09 11:39:35 +01:00
Simon Willnauer 5fca3b6126 Merge pull request #16985 from elastic/remove_leniency_in_commitpoint_checks
Remove leniency from segments info integrity checks

Closes #16973
2016-03-09 11:25:42 +01:00
Isabel Drost-Fromm 7b5b0d4511 Move missing() from SortBuilder interface to class
As mentioned by @cbuescher on #16151 this method is really implemented only in
the FieldSortBuilder. Moving the method down.

Relates to #15178
2016-03-09 10:18:31 +01:00
Simon Willnauer 716e7267f3 Remove unused test-only constructor from IndexingSlowLog 2016-03-09 10:10:38 +01:00
Simon Willnauer 997fccde09 Remove unused delete logger in IndexingSlowLog
The delete logger is a leftover and has no usage in this class.
2016-03-09 10:10:38 +01:00
Ryan Ernst dedc45ea62 Fix tribe integ test to not try to pass through path settings 2016-03-09 01:03:27 -08:00
Simon Willnauer 91308d74c6 Merge pull request #17026 from s1monw/issues/17025
Add missing index name to indexing slow log
2016-03-09 09:49:06 +01:00
Simon Willnauer 98249507cf Add missing index name to indexing slow log
This was lost in refactoring even on the 2.x branch. The slow-log
is not per index not per shard anymore such that we don't add the
shard ID as the logger prefix. This commit adds back the index
name as part of the logging message not as a prefix on the logger
for better testabilitly.

Closes #17025
2016-03-09 09:38:46 +01:00
Ryan Ernst 712043315d Use Setting objects for tribe node client passthrough, and add scripts
path to passthrough
2016-03-09 00:37:15 -08:00
Ryan Ernst b419a50381 Merge branch 'master' into more_tribe_node_settings 2016-03-09 00:19:43 -08:00
Ryan Ernst 80ae2b0002 Fix more licenses 2016-03-09 00:10:59 -08:00
Ryan Ernst d822c6558f Fix file rename to match class name 2016-03-08 23:17:35 -08:00
Ryan Ernst 1dafead2eb Fix precommit 2016-03-08 22:55:24 -08:00
Ryan Ernst 6cfdf9f440 Remove old commons-cli dep 2016-03-08 17:29:31 -08:00
Ryan Ernst 73ebe36ed0 More tests 2016-03-08 17:27:53 -08:00
Ryan Ernst fdce9d7c4d Merge branch 'master' into cli-parsing 2016-03-08 14:18:20 -08:00
Ryan Ernst 13424318db Remove old help files 2016-03-08 14:16:39 -08:00
Ryan Ernst 80198accc1 Removed old cli stuff, and add tests for new Command behavior 2016-03-08 14:13:55 -08:00
Ryan Ernst 3836f3a736 Remove reference to standalonerunner 2016-03-08 13:40:39 -08:00
Ryan Ernst e5c852f767 Convert bootstrapcli parser to jopt-simple 2016-03-08 13:39:37 -08:00
Nicholas Knize 496f50bfc3 Deprecate lat_lon and precision_step
With GeoPoinV2 lat_lon and precision_step parameters will be removed in 5.0. This PR adds deprecation logging for 2.x.
2016-03-08 14:45:54 -06:00
Nik Everett e32716b26f [test] Fix uncommon tests failure in TasksIT 2016-03-08 12:09:05 -05:00
Nik Everett 0745e19c29 Add uuid to Index's toString
This is useful because uuid is starting to matter more and more in index
operations.
2016-03-08 11:56:45 -05:00
Nik Everett 6d0efae713 Teach list tasks api to wait for tasks to finish
_wait_for_completion defaults to false. If set to true then the API will
wait for all the tasks that it finds to stop running before returning. You
can use the timeout parameter to prevent it from waiting forever. If you
don't set a timeout parameter it'll default to 30 seconds.

Also adds a log message to rest tests if any tasks overrun the test. This
is just a log (instead of failing the test) because lots of tasks are run
by the cluster on its own and they shouldn't cause the test to fail. Things
like fetching disk usage from the other nodes, for example.

Switches the request to getter/setter style methods as we're going that
way in the Elasticsearch code base. Reindex is all getter/setter style.

Closes #16906
2016-03-08 11:53:57 -05:00
Lee Hinman 678bc927ce Log when cancelling allocation of a replica because a new syncid was found
Currently the message stays in the `UnassignedInfo` for the shard,
however, it would be very useful to know the exact point (time-wise)
that the cancellation happened when diagnosing an issue.

Relates to debugging #16357
2016-03-08 09:12:04 -07:00
Jason Tedor 2bd09c4625 Merge branch 'master' into enable_acked
* master: (350 commits)
  Note to configuration docs on number of threads
  Reduce maximum number of threads in boostrap check
  Limit generic thread pool
  Remove NodeService injection to Discovery
  Prevent closing index during snapshot restore
  [TEST] Fix newline issue in PluginCliTests on Windows
  ParseFieldMatcher should log when using deprecated settings. #16988
  fix checkstyle error
  Add test for the index_options on a keyword field. #16990
  Analysis : Allow string explain param in JSON
  Analysis : Allow string explain param in JSON
  fix typo
  Remove SNAPSHOT from versions in plugin descriptors
  Add support for alpha versions
  Enable unmap hack for java 9
  Simplify mock scripts
  Adding `time_zone` parameter to daterange-aggregation docs
  Adding tests for `time_zone` parameter for date range aggregation
  Added ingest info to node info API, which contains a list of available processors.
  Remove bw compat from size mapper
  ...
2016-03-08 10:19:56 -05:00
Simon Willnauer 9a9eadd743 [TEST] Use actual target directory to fsync copied files in test
Apparently lucene6 is way more picky with respect to corrupting files
that are not fsynced that's why this test sometimes failed after the lucene6
upgrade.
2016-03-08 15:30:57 +01:00
Simon Willnauer 4603313c75 Merge remote-tracking branch 'origin/master' 2016-03-08 15:29:47 +01:00
Jason Tedor 930984eb4f Reduce maximum number of threads in boostrap check
This commit reduces the maximum number of threads required in the
bootstrap check. This limit can be reduced since the generic thread pool
is no longer unbounded.

Relates #17003
2016-03-08 09:14:49 -05:00
Jason Tedor 95b0a6a2cf Limit generic thread pool
The generic thread pool was previously configured to be able to create
an unlimited number of threads. The thinking is that tasks that are
submitted to its work queue must execute and should not block waiting
for a worker. However, in cases of heavy load, this can lead to an
explosion in the number of threads; this can even lead to a feedback
loop that exacerbates the problem. What is more, this can even bump into
OS limits on the number of threads that can be created.

This commit limits the number of threads in the generic thread pool to
four times the bounded number of processors.

Relates #17003
2016-03-08 09:14:34 -05:00
Simon Willnauer be176a1fed Add Json representation to raw group settings for better logging represetation. 2016-03-08 15:10:46 +01:00
Simon Willnauer 474fd26073 Prevent overriding built-in similarity and allow defining index global similarity
This merge commit merges #16682 which adds the ability to define a index-global
default similarity but at the same time prevents overriding built-in similarities
for new indices. Old indices where able to do this int the past which should not
be punished even though this is not possible anymore.

Closes #16594
Closes #16682
2016-03-08 14:52:59 +01:00
Yannick Welsch 20f5255670 Merge pull request #16933 from ywelsch/fix/snaprestore-fails-close
Prevent closing index during snapshot restore
2016-03-08 14:21:41 +01:00
Isabel Drost-Fromm 62867ad295 Merge pull request #16573 from MaineC/deprecation/field-sort-builder
Remove deprecated parameter from field sort builder.
2016-03-08 13:22:08 +01:00
Boaz Leskes fa6a3398a8 Remove NodeService injection to Discovery
This was only used by the multicast plugin which is now removed.

Closes #17000
2016-03-08 12:56:00 +01:00
Yannick Welsch 71ac12f4a9 Prevent closing index during snapshot restore
Closes #16933
2016-03-08 12:00:53 +01:00
Isabel Drost-Fromm a4b5fbedb8 Moves SortParser:parse(...) to only require QueryShardContext
This removes the need for accessing the SearchContext when parsing Sort elements
to queries. After applying the patch only a QueryShardContext is needed.

Relates to #15178
2016-03-08 11:11:48 +01:00
Christoph Büscher 1264f37a1b Merge branch 'master' into feature-suggest-refactoring 2016-03-08 11:10:08 +01:00
Adrien Grand 026519e81b ParseFieldMatcher should log when using deprecated settings. #16988
I always thought ParseFieldMatcher would log when using a deprecated setting,
but it does not.
2016-03-08 09:51:46 +01:00
Jun Ohtani 3d13c27fa0 fix checkstyle error 2016-03-08 17:32:14 +09:00
Adrien Grand ad7fbe7251 Add test for the index_options on a keyword field. #16990
This found a bug in the validation, which was checking the wrong IndexOptions.
2016-03-08 09:24:20 +01:00
Jun Ohtani 48cb81e30b Analysis : Allow string explain param in JSON
Fix typo
Remove unused import

Closes #16925
2016-03-08 16:19:02 +09:00
Jun Ohtani 071d578953 Analysis : Allow string explain param in JSON
Move some test methods from AnalylzeActionIT to RestAnalyzeActionTest
Allow string explain param if it can parse
Fix wrong param name in rest-api-spec

Closes #16925
2016-03-08 16:19:02 +09:00
Ryan Ernst 7e16afbbf2 Added transport network settings to whitelist for tribe node
Also fail on any path settings within tribe sections
2016-03-07 12:58:55 -08:00
Ryan Ernst 55cf49420c Merge branch 'master' into more_tribe_node_settings 2016-03-07 12:43:50 -08:00
Ryan Ernst 45b5ab24fe Moved MockTerminal and created a base test case for cli commands. 2016-03-07 12:42:15 -08:00
Robert Muir 2dd8ed90ab Merge pull request #16986 from rmuir/mmap_java9
Enable unmap hack for java 9
2016-03-07 15:22:55 -05:00
Simon Willnauer 53cc8bdc45 fix typo 2016-03-07 17:52:23 +01:00
Simon Willnauer 9cbc602487 Remove SNAPSHOT from versions in plugin descriptors
We removed leniencey from version parsing which caught problems with
-SNAPSHOT suffixes on plugin properies. This commit removes the -SNAPSHOT
from both es and the extension version and adds tests to ensure we can
parse older versions that allowed -SNAPSHOT in BWC way.
2016-03-07 17:41:38 +01:00
Simon Willnauer 887b69b58b Add support for alpha versions
Elasticsearch 5.0 will come with alpha versions which is not supported
in the current version scheme. This commit adds support for aplpha starting
with es 5.0.0 in a backwards compatible way.
2016-03-07 16:39:15 +01:00
Christoph Büscher 350e3a4850 Merge pull request #16955 from cbuescher/timezone-dateRangeAgg
Date range aggregations used to be unable to use a `time_zone` parameter to e.g. be applied 
int date math roundings like in `now/d` (see #10130 as an example). After the aggregation 
refactoring, the time_zone parameter has been pulled up to ValuesSourceAggregatorBuilder 
and can now be used in date range aggregations as well. 

This change adds randomized time zone settings to the existing IT tests to verify that the 
`time_zone` parameter is honored when calculating the bucket boundaries. Also moving 
the DateRangeTests from module-groovy/messy back to core as DateRangeIT, sharing 
common script mocks with DateHistogramIT and adding documentation for the 
`time_zone` parameter in the date range aggregation docs.

Closes #10130
2016-03-07 16:11:55 +01:00
Simon Willnauer a40587b377 Merge pull request #16982 from s1monw/remove_old_version
Remove old and unsupported version constants

All version <= 2.0 are not supported anymore. This commit removes all
uses of these versions.
2016-03-07 16:03:14 +01:00
Robert Muir a0191dff01 Enable unmap hack for java 9 2016-03-07 09:45:55 -05:00
Christoph Büscher ff46303f15 Simplify mock scripts 2016-03-07 15:39:35 +01:00
Christoph Büscher 5ff413074a Adding tests for `time_zone` parameter for date range aggregation 2016-03-07 15:38:24 +01:00
Martijn van Groningen 82d01e4315 Added ingest info to node info API, which contains a list of available processors.
Internally the put pipeline API uses this information in node info API to validate if all specified processors in a pipeline exist on all nodes in the cluster.
2016-03-07 14:44:50 +01:00
Simon Willnauer d9ddd3fa45 Remove leniency from segments info integrity checks
Closes #16973
2016-03-07 12:23:30 +01:00
Simon Willnauer 7e4c4cd8d9 bring back accidentially removed MockScriptEngine plugin 2016-03-07 12:17:32 +01:00
Simon Willnauer 46c295d029 use true/false rather than yes/no in tests 2016-03-07 12:03:05 +01:00
Simon Willnauer a29f734b75 Only assert that analyzers are the same if lucene version is the same 2016-03-07 11:55:56 +01:00
Simon Willnauer f071f327db Remove old and unsupported version constants
All version <= 2.0 are not supported anymore. This commit removes all
uses of these versions.
2016-03-07 11:49:06 +01:00
Martijn van Groningen 050bfc31b6 Don't call IR#leaves() after global field data has been constructed
This IR may already be closed and GlobalFieldData is cached and this can cause AlreadyClosedException while checking the assert.
2016-03-07 11:25:52 +01:00
Christoph Büscher 7ec5075a87 Merge branch 'master' into feature-suggest-refactoring 2016-03-07 10:45:43 +01:00
Adrien Grand c69fc008d4 Merge pull request #16972 from alexshadow007/fix-16812
Build empty extended stats aggregation if no docs collected for bucket
2016-03-07 10:44:20 +01:00
Robert Muir 54018a5d37 upgrade to lucene 6.0.0-snapshot-bea235f
Closes #16964

Squashed commit of the following:

commit a23f9d2d29220991aa498214530753d7a5a148c6
Merge: eec9c4e 0b0a251
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Mar 7 04:12:02 2016 -0500

    Merge branch 'master' into lucene6

commit eec9c4e5cd11e9c3e0b426f04894bb2a6dae4f21
Merge: bc67205 675d940
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 13:45:00 2016 -0500

    Merge branch 'master' into lucene6

commit bc67205bdfe1526eae277ab7856fc050ecbdb7b2
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 09:56:31 2016 -0500

    fix test bug

commit a60723b007ff12d97b1810cef473bd7b553a0327
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Mar 4 15:35:35 2016 +0100

    Fix SimpleValidateQueryIT to put braces around boosted terms

commit ae3a49d7ba7ced448d2a5262e5d8ec98671a9090
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Mar 4 15:27:25 2016 +0100

    fix multimatchquery

commit ae23fdb88a8f6d3fb7ba60fd1aaf3fd72d899aa5
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Mar 4 15:20:49 2016 +0100

    Rewrite DecayFunctionScoreIT to be independent of the similarity used

    This test relied a lot on the term scoring and compared scores
    that are dependent on the similarity. This commit changes the base query
    to be a predictable constant score query.

commit 366c2d518c35d31251033f1b6f6a93f6e2ae327d
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Mar 4 14:06:14 2016 +0100

    Fix scoring in tests due to changes to idf calculation.

    Lucene 6 uses a different default similarity as well as a different
    way to calculate IDF. In contrast to older version lucene 6 uses docCount per field
    to calculate the IDF not the # of docs in the index to overcome the sparse field
    cases.

commit dac99fd64ac2fa71b8d8d106fe68825e574c49f8
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 08:21:57 2016 -0500

    don't hardcoded expected termquery score

commit 6e9f340ba49ab10eed512df86d52a121aa775b0f
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 08:04:45 2016 -0500

    suppress deprecation warning until migrated to points

commit 3ac8908424b3fdad44a90a4f7bdb3eff7efd077d
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 07:21:43 2016 -0500

    Remove invalid test: all commits have IDs, and its illegal to do this.

commit c12976288124ad1a26467e7e848fb810548e7eab
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 07:06:14 2016 -0500

    don't test with unsupported back compat

commit 18bbfe76128570bc70883bf91ff4c44c82d27817
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 07:02:18 2016 -0500

    remove now invalid lucene 4 backcompat test

commit 7e730e572886f0ef2d3faba712e4256216ff01ec
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 06:58:52 2016 -0500

    remove now invalid lucene 4 backwards test

commit 244d2ab6868ba5ac9e0bcde3c2833743751a25ec
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 06:47:23 2016 -0500

    use 6.0 codec

commit 5f64d4a431a6fdaa1234adca23f154c2a1de8284
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 06:43:08 2016 -0500

    compile, javadocs, forbidden-apis, etc

commit 1f273cd62a7fe9ca8f8944acbbfc5cbdd3d81ccb
Merge: cd33921 29e3443
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Mar 4 10:45:29 2016 +0100

    Merge branch 'master' into lucene6

commit cd33921ac742ef9fb351012eff35f3c7dbda7264
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:58:37 2016 -0500

    fix hunspell dictionary loading

commit c7fdbd837b01f7defe9cb1c24e2ec65604b0dc96
Merge: 4d4190f d8948ba
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:41:53 2016 -0500

    Merge branch 'master' into lucene6

commit 4d4190fd82601aaafac6b8254ccb3edf218faa34
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:39:14 2016 -0500

    remove nocommit

commit 77ca69e288b1a41aa9595c921ed166c272a00ea8
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:38:24 2016 -0500

    clean up numericutils vs legacynumericutils

commit a466d696fbaad04b647ffbc0857a9439b583d0bf
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:32:43 2016 -0500

    upgrade spatial4j

commit 5412c747a8cfe638bacedbc8233163cb75cc3dc5
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:19:28 2016 -0500

    move to 6.0.0-snapshot-8eada27

commit b32bfe924626b87e540692375ece09e7c2edb189
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 11:30:09 2016 +0100

    Fix some test compile errors.

commit 6ccde35e9840b03c68d1a2cd47c7923a06edf64a
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 11:25:51 2016 +0100

    Current Lucene version is 6.0.0.

commit f62e1015d931b4cc04c778298a8fa1ba65e97ad9
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 11:20:48 2016 +0100

    Fix compile errors in NGramTokenFilterFactory.

commit 6837c6eabf96075f743649da9b9b52dd39611c58
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 10:50:59 2016 +0100

    Fix the edge ngram tokenizer/filter.

commit ccd7f070de5efcdfbeb34b9555c65c4990bf1ba6
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 10:42:44 2016 +0100

    The missing value is now accessible through a getter.

commit bd3b77f9b28e5b05daa3d49683a9922a6baf2963
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 10:41:51 2016 +0100

    Remove IndexCacheableQuery.

commit 05f3091c347aeae80eeb16349ac51d2b53cf86f7
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 10:39:43 2016 +0100

    Fix compilation of function_score queries.

commit 81cda79a2431ac78f56b0cc5a5765387f662d801
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 10:35:02 2016 +0100

    Fix compile errors in BlendedTermQuery.

commit 70994ce8dd1eca0b995870974a38e20f26f96a7b
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 23:33:03 2016 -0500

    add bug ID

commit 29d4f1a71f36f646b5a6060bed3db019564a279d
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 21:02:32 2016 -0500

    easy .store changes

commit 5e1a1e6fd665fa455e88d3a8987362fad5f44bb1
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 20:47:24 2016 -0500

    cleanups mostly around boosting

commit 333a669ec6c305ada5645d13ed1da0e19ec1d053
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 20:27:56 2016 -0500

    more simple fixes

commit bd5cd98a1e089c866b6b4a5e159400b110140ce6
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 19:49:38 2016 -0500

    more easy fixes and removal of ancient cruft

commit a68f419ee47da5f9c9ce5b372f01d707e902474c
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 19:35:02 2016 -0500

    cutover numerics

commit 4ca5dc1fa47dd5892db00899032133318fff3116
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 18:34:18 2016 -0500

    fix some constants

commit 88710a17817086e477c6c021ec346d0534b7fb88
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 18:14:25 2016 -0500

    Add spatial-extras jar as a core dependency

commit c8cd6726583e5ce3f546ed355d4eca037164a30d
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 18:03:33 2016 -0500

    update to lucene 6 jars
2016-03-07 04:12:23 -05:00
Ryan Ernst 7a49cd1287 Merge branch 'master' into cli-parsing 2016-03-06 13:29:56 -08:00
Ryan Ernst 8455624884 Merge branch 'master' into mock_terminal 2016-03-06 11:35:22 -08:00
Ryan Ernst 5a94f8b3e6 Remove outdated nocommit and tweak assert to output bad exception message on failure. 2016-03-06 11:34:10 -08:00
Ryan Ernst ed7934ee79 Cli: Simplify test terminals
This commit simplifies and consolidates the two different
implementations of terminals used in tests. There is now a single
MockTerminal which captures output, and allows accessing as one large
string (with unix style \n as newlines), as well as configuring
input.
2016-03-06 11:30:30 -08:00
Simon Willnauer 8ca284862d Notify GatewayRecoveryListener on failure
If the recovery throws an exception we miss to notify the recovery
listener and bubble up the uncaught exception. This commit uses
an AbstractRunnable that also catches rejected execution exceptions
etc. and notifies the listener accordingly.
2016-03-06 16:43:12 +01:00
Simon Willnauer 17e28d05b7 Delete ShardsAllocatorModuleIT, it's been replaced by ClusterModuleTests
ClusterModuleTests tests what ShardsAllocatorModuleIT tests without starting
a cluster. Unittests should be preferrred over IT tests anyway and the instantiation
of the balanced shards allocator is tested with every other integration test.
2016-03-06 13:42:56 +01:00
Simon Willnauer a283a2526b Merge pull request #16956 from s1monw/only_wait_unless_master
Only wait for initial state unless we already got a master
2016-03-06 09:16:07 +01:00
Alexander Kazakov 3674774b73 Build empty extended stats aggregation if no docs are collected for bucket #16812 2016-03-05 22:50:54 +03:00
Adrien Grand 0322d490a8 Temporarily re allow string fields to give Kibana more time to switch to text/keyword. 2016-03-05 19:09:45 +01:00
Simon Willnauer 9135c3c967 Only wait for initial state unless we already got a master
This seems to be an error introduced in refactoring around #16821
where we now wait 30seconds by default if the node already joined
a cluster and got a master. This can slow down tests dramatically
espeically on slow boxes and notebooks.

Closes #16956
2016-03-05 12:00:25 +01:00
David Pilato e35032950e Merge branch 'master' into pr/16598-register-filter-settings 2016-03-05 11:37:03 +01:00
javanna da19922489 [TEST] adapt test infra to node.client setting removal
No changes were really needed in our test infra as it didn't use `node.client`. Yet it didn't take into account ingest nodes, what we used to call client nodes in InternalTestCluster were actually ingest only nodes, which now become coordinating only nodes.
Also renamed some method to get rid of the node client terminology as much as possible in favour or coordinating only node.
2016-03-05 10:55:19 +01:00
javanna 9c4a5bbe7e adapt cluster stats api to node.client setting removal
The cluster stats api now returns counts for each node role. The `master_data`, `master_only`, `data_only` and `client` fields have been removed from the response in favour of `master`, `data`, `ingest` and `coordinating_only`. The same node can have multiple roles, hence contribute to multiple roles counts. Every node is implicitly a coordinating node, so whenever a node has no explicit roles, it will be counted as coordinating only.
2016-03-05 10:55:19 +01:00
javanna f786e9866c adapt _cat/nodes to node.client removal
_cat/nodes used to return `c` for client node or `d` for data node as part of the node.role column. This commit changes it to return `m` for master eligible, `d` for data and/or `i` for ingest. A node with no explicit roles will be a coordinating only node and marked with `-`. A node can obviously have multiple roles. The master column has been adapted to return only whether a node is the current master (`*`) or not (`-`).
2016-03-05 10:55:19 +01:00
javanna 4224371d8b introduced notion of Role in DiscoveryNode
A node can now have roles, Role is an enum made of master, data, ingest. A ndoe with no roles is simplicitly a coordinating only node. Roles are resolved once at construction time based on node attributes and never serialized. Moving DiscoveryNode to Writeable helps cleaning up the code, making fields final allow to easily see where roles need to be initialized and do it in one single place.
2016-03-05 10:55:18 +01:00
George P. Stathis 198a79edf5 Fix spacing, assert key contains period 2016-03-04 19:32:44 -05:00
David Pilato bde97e1d9c Move validation logic to `SettingsModule.registerSetting` 2016-03-04 21:44:01 +01:00
Ryan Ernst 687aadef00 Merge branch 'master' into cli-parsing 2016-03-04 12:13:17 -08:00
Ryan Ernst 8321d7c5c2 Catch option error during execution too, since OptionSet is passed there 2016-03-04 12:11:25 -08:00
javanna eb941d8005 Remove node.client setting
As discussed in #16565, the node.client setting is an unnecessary shortcut to node.data: false and node.master: false. We have places where we treat nodes with node.client set to true differently compared to master false and data false, which is not correct. Also, with the addition of node.ingest or potentially new roles, it becomes confusing to figure out if a node client should support ingestion or not.

This commit removes the node.client setting in favour being explicit using node.master, node.data and node.ingest instead.
2016-03-04 20:17:34 +01:00
Yannick Welsch 250db4999e Speed up shard balancer by reusing shard model while moving shards that can no longer be allocated to a node
Decommissioning a node or applying a filter inclusion / exclusion can potentially lead to many shards that need to be moved to other nodes. This commit reuses the model across all
shard movements in an allocation round: It calculates the shard model once and simulates the application of all shards that can be moved on this model.

Closes #16926
2016-03-04 19:10:56 +01:00
David Pilato 5c2ca3c9f5 Check that we must have one and only one scope for a Setting 2016-03-04 17:05:31 +01:00
David Pilato 2bb3846d1f Update after review:
* remove `ClusterScope`
* rename `ClusterSettings` to `NodeSettings`
* rename `SettingsProperty` to `Property`
2016-03-04 16:53:24 +01:00
Lee Hinman 809bb9e5a1 [TEST] Increase timeout for awaiting alias & document creation
Test waits 10 seconds for all 10 aliases and documents to be created,
but it can take longer that that on slower machines.
2016-03-04 08:25:40 -07:00
David Pilato 76719341dc Fix after merge 2016-03-04 13:24:39 +01:00
David Pilato c11cf3bf1f Merge branch 'master' into pr/16598-register-filter-settings
# Conflicts:
#	core/src/main/java/org/elasticsearch/common/logging/ESLoggerFactory.java
#	core/src/main/java/org/elasticsearch/common/settings/Setting.java
#	core/src/test/java/org/elasticsearch/common/settings/SettingTests.java
2016-03-04 12:23:10 +01:00
Simon Willnauer 8652cd8aa3 Handle RejectedExecution gracefully in TransportService
Today we might run into a rejected execution exception when
we shutdown the node while handling a transport exception. The
exception is run in a seperate thread but that thread might
not be able to execute due to the shutdown. Today we barf and fill
the logs with large exception. This commit catches this exception
and logs it as debug logging instead.
2016-03-04 12:18:07 +01:00
Jim Ferenczi 927303e7a9 Change the field mapping index time boost into a query time boost.
Index time boost will still be applied for indices created before 5.0.0.
2016-03-04 11:47:35 +01:00
Simon Willnauer 5008694ba1 Remove support for legacy checksums
Elasticsearch 5.0 doesn't support indices wiht legacy checksums anymore.
The last time we write legacy checksums was in 1.3.0 which was based
on lucene 4.9 already which means that all files have CRC32 checksums.
All indices that Elasticsearch can read today must be written with
lucene version >= 4.8 anyway so we can drop this layer of backwards
compatibility entirely.

Since we are close to upgrading to Lucene 6.0 we should get rid of this
in a more contiained change than the lucene upgrade.
2016-03-03 22:58:18 +01:00
Simon Willnauer 6a76fa3382 Use SleepingWrapper on shared filesytems
On shared FS / shadow replicas we rely on a lock retry if the lock has
not yet been relesed on a relocated primary. This commit adds this `hack`
for shared filesystems only.

Closes #16936
2016-03-03 21:43:12 +01:00
Christoph Büscher 30a788d87c Suggestions: Make field name mandatory ctor argument
The field name is a required argument for all suggesters, but
it was specified via a field() setter in SuggestionBuilder so far.
This changes field name to being a mandatory constructor argument
and lets suggestion builders throw an error if field name is missing
or the empty string.
2016-03-03 21:35:53 +01:00
Lee Hinman 6adbbff97c Fix organization rename in all files in project
Basically a query-replace of "https://github.com/elasticsearch/" with "https://github.com/elastic/"
2016-03-03 12:04:13 -07:00
Lee Hinman f9d1f95e84 [TEST] mark IndexWithShadowReplicasIT with awaitsfix 2016-03-03 11:14:06 -07:00
Jason Tedor 166259db64 Use and test relative time in TransportBulkAction
This commit modifies TransportBulkAction to use relative time instead of
absolute time when measuring how long a bulk request took to be
processed, and adds tests for this functionality.

Closes #16916
2016-03-03 11:53:53 -05:00
Jason Tedor e75a0da4d5 Add max number of processes check
This commit adds a bootstrap check on Linux for the max number of
processes available to the user running the Elasticsearch process.

Closes #16919
2016-03-03 11:40:10 -05:00
Simon Willnauer 244caa3618 Remove writeLockTimeout from InternalEngine
`writeLockTimeout` has been removed in Lucene 6 completely and since we have
the shard locking mechanism now for quite a while we don't need this anymore.
Shards should only be allocated once all resources are released such that there
can't be any other shard holding the lock to that index in any sane situation.
2016-03-03 16:40:24 +01:00
Martijn van Groningen 7a3da93551 Merge branch 'pr/16923' 2016-03-03 15:58:38 +01:00
Martijn van Groningen f0e80e1a7a replaced integration test with unit test.
Closes #16692
2016-03-03 15:31:57 +01:00
Adrien Grand 2c3e4840f2 Store _all payloads on 1 byte instead of 4. #16899
This changes the `_all` field to store per-field boosts using a single byte
similarly to norms.
2016-03-03 15:00:23 +01:00
Jim Ferenczi c996db731a Merge pull request #16922 from jimferenczi/fix_match_query_tests
Fix sporadic error on match query test when a fuzziness of 0s is used.
2016-03-03 14:29:47 +01:00
Jim Ferenczi c76949ddd5 After review 2016-03-03 12:19:07 +01:00
Christoph Büscher ef4db5c1e4 Merge branch 'master' into feature-suggest-refactoring 2016-03-03 12:18:03 +01:00
Jim Ferenczi 06bef0ca6b Fix sporadic error on match query test when a fuzziness of 0s is used on a date field. 2016-03-03 11:54:07 +01:00
Alexander Kazakov 98c164c3f0 Check that parent_type in HasParent query has child types #16692 2016-03-03 13:32:22 +03:00
Adrien Grand eef19be072 Deprecate string in favor of text/keyword. #16877
This commit removes the ability to use string fields on indices created on or
after 5.0. Dynamic mappings now generate text fields by default for strings
but there are plans to also add a sub keyword field (in a future PR).

Most of the changes in this commit are just about replacing string with
keyword or text. Some tests have been removed because they existed because of
corner cases of string mappings like setting ignore-above on a text field or
enabling term vectors on a keyword field which are now impossible.

The plan is to remove strings entirely in 6.0.
2016-03-03 10:20:56 +01:00
Daniel Mitterdorfer f70e5aca50 Merge remote-tracking branch 'danielmitterdorfer/simplify-azure-settings' 2016-03-03 10:02:35 +01:00
Daniel Mitterdorfer 52acf0e6e1 Use new settings infra to parse AzureStorageSettings
With this commit we simplify the parsing logic in AzureStorageSettings
by leveraging the new settings infrastructure.

Closes #16363
2016-03-03 10:01:14 +01:00
Jason Tedor f35bad808c Fix case in mlockall check error message 2016-03-02 22:43:20 -05:00
Jason Tedor 62638b8212 Fix case in mlockall check error message 2016-03-02 22:12:18 -05:00
javanna 45b7b5175c [TEST] add unit test for DiscoveryNodes node selectors 2016-03-02 21:59:20 +01:00
javanna 04947a5fef Remove DiscoveryNode#shouldConnectTo method
This method was originally introduced to prevent client nodes from connecting to other client nodes directly in the cluster. That said, it worked only if node.client was set to true and not when node.master and node.data were both set to false. It looks safe to remove, which allows to solve all kinds of problems around monitoring that happen wherever there are 2 or more clients nodes in the cluster, and a REST call hits one of them (node counts are off, clients nodes are missing).
2016-03-02 21:59:20 +01:00
Simon Willnauer 1e70e16c89 Merge pull request #16866 from s1monw/only_address_for_http
Use HttpInfo in cat/_nodes instead of node attribute
2016-03-02 21:48:35 +01:00
Simon Willnauer 8fbdd3334f Use HttpInfo in cat/_nodes instead of node attribute
The node attribute also renders the type of the inet address
as well as the host name which is unneeded in the cat action.
2016-03-02 21:33:36 +01:00
Michael McCandless 4df83dc9a5 Merge pull request #16661 from camilojd/feature/disk-segments-stats
Segments stats in indices stats API now optionally includes aggregated file sizes by file extension / index component
2016-03-02 13:58:20 -05:00
javanna 72e4c26d0e _cat/nodeattrs: remove unnecessary nodes stats call 2016-03-02 17:14:49 +01:00
javanna fd70d60457 Remove http_address from _cat/nodeattrs
This was recently added with #16770 but it turns out it is not a proper node attribute, so it is enough to print out as part of _cat/nodes only.
2016-03-02 17:14:49 +01:00
Jason Tedor 99aa4942d2 Add more missed final keywords in BootstrapChecks 2016-03-02 10:00:44 -05:00
Jason Tedor 0c914b51c7 Add missed final keywords in BootstrapChecks 2016-03-02 09:57:26 -05:00
Jason Tedor f2220c9786 Add mlockall bootstrap check
This commit adds a boostrap check for the situation bootstrap.mlockall
is enabled, but memory was not able to be locked.

Closes #16909
2016-03-02 09:51:31 -05:00
Jason Tedor 6caec9bc25 Display if node is snapshot build on startup
Recent simplifications to org.elasticsearch.Version led to "-SNAPSHOT"
to not be displayed in the node version upon startup. Yet, this is
useful information to have. This simple commit adds back the display of
"-SNAPSHOT" in the version on startup if the build is a snapshot build.

Closes #16908
2016-03-02 09:41:23 -05:00
Martijn van Groningen 27ea143c3f Revert "bulk api: use `System.nanoTime()` instead of `System.currentTimeMillis()` to compute the estimated took time"
This reverts commit 82567f1bdf.

`System.nanoTime()` was being called from multiple threads causing the timing to be incorrect.
2016-03-02 15:20:54 +01:00
David Pilato e4031932ed Use deprecation Logger 2016-03-02 15:10:32 +01:00
Jason Tedor 9ad5919002 Refactor bootstrap checks
This commit refactors the bootstrap checks into a dedicated class. The
refactoring provides a model for different limits per operating system,
and provides a model for unit tests for individual checks.

Closes #16844
2016-03-02 08:47:16 -05:00
Christoph Büscher c0e5e40e02 Merge branch 'master' into feature-suggest-refactoring 2016-03-02 14:22:03 +01:00
Christoph Büscher 299b9d9b63 Fix NPE in SearchSourceBuilder
Prevent a NPE when parsing `fields` parameter and internal field name
list has not been initialized.

Closes #16902
2016-03-02 12:25:29 +01:00
David Pilato 3f71c1d6a5 Replace `s -> s` by `Function.identity()` 2016-03-02 10:12:40 +01:00
David Pilato c103e40e72 Add support for deprecated settings
This is a backport of #16845 in this branch.

We now also support marking settings with `SettingsProperty.Deprecated`.

If the setting is still used, it will print a `warn` to the user.
2016-03-02 10:06:53 +01:00
David Pilato e4d9e46508 Fix merge with master 2016-03-02 09:55:09 +01:00
David Pilato 5fbf1b95dc Merge branch 'master' into pr/16598-register-filter-settings
# Conflicts:
#	core/src/main/java/org/elasticsearch/common/logging/ESLoggerFactory.java
#	core/src/main/java/org/elasticsearch/discovery/DiscoveryService.java
#	core/src/main/java/org/elasticsearch/discovery/DiscoverySettings.java
#	core/src/main/java/org/elasticsearch/http/HttpTransportSettings.java
#	plugins/repository-azure/src/main/java/org/elasticsearch/cloud/azure/storage/AzureStorageService.java
2016-03-02 09:43:53 +01:00
Martijn van Groningen 82567f1bdf bulk api: use `System.nanoTime()` instead of `System.currentTimeMillis()` to compute the estimated took time
In this case we compute elapsed time and `System.nanoTime()` is designed to do just do that.
The absolute timing `System.currentTimeMillis()` provides in not needed and the relative timing `System.nanoTime()` provides is likely to be more accurate.
2016-03-02 09:20:24 +01:00
Camilo Díaz Repka 356364810c Implementation of Segment disk stats aggregating sizes by index file extension.
Use 'includeSegmentFileSizes' as the flag name to report disk usage.
Added test that verifies reported segment disk usage is growing accordingly after adding a document.
Documentation: Reference the new parameter as part of indices stats.
2016-03-01 23:16:49 -03:00
Ryan Ernst 3bd33f839f Add test for ignoring path settings in tribe client 2016-03-01 15:07:59 -08:00
Ryan Ernst 6566327159 Add test for per tribe client overrides 2016-03-01 14:49:27 -08:00
Ryan Ernst 6ca7666646 Tribe: Passthrough environment and network settings to tribe client nodes
In 2.2, the client nodes created internally by a tribe node were changed
to be explicit about which settings the client nodes use, no longer
loading all settings from elasticsearch.yml. However, some settings were
missed, notably network bind settings. This change adds those settings
to be passed through, as well as adds unit tests for building the tribe
client node settings.
2016-03-01 14:42:28 -08:00
Chris Earle ea2792e278 Adjusting based on comments. 2016-03-01 17:18:09 -05:00
Chris Earle 8fa0d8e905 Add LifecycleRunnable
I've noticed throughout the code that we have a need to remove the boilerplate lifecycle check when starting/rescheduling certain runnables. This provides a simpler implementation to get this functionality without duplicating it.
2016-03-01 17:18:09 -05:00
Jason Tedor aa8ee74c6c Bump Elasticsearch version to 5.0.0-SNAPSHOT
This commit bumps the Elasticsearch version to 5.0.0-SNAPSHOT in line
with the alignment of versions across the stack.

Closes #16862
2016-03-01 17:03:47 -05:00
Ryan Ernst 354ede717b Removed old help files and improved plugin cli tests 2016-03-01 11:48:52 -08:00
Ali Beyad 72ed01c304 Add AwaitsFix annotation to the sporadically failing test
DiscoveryWithServiceDisruptionIT.testIndicesDeleted pending a fix for
that test (see issue #16890).
2016-03-01 14:04:04 -05:00
Martijn van Groningen 75387001df Added `ingest_took` to bulk response to indicate how much time was spent on ingest preprocessing.
The `ingest_took` is separate from `took`, which keeps track how much time is spent on indexing/deleting/updating.
The `ingest_took` is only visible in the rest response if at least for one bulk item has ingest enabled.
2016-03-01 18:24:26 +01:00
Christoph Büscher be8ed737bc Suggest: Move name of suggestion to SuggestBuilder
Currently each suggestion keeps track of its own name. This has
the disadvantage of having to pass down the parsed name property
in the suggestions fromXContent() and the serialization methods
as an argument, since we need it in the ctor.

This change moves the naming of the suggestions to the surrounding
SuggestBuilder and by this eliminates the need for passind down
the names in the parsing and serialization methods. By making
`name` a required argument in SuggestBuilder#addSuggestion() we
also make sure it is always set and prevent using the same name twice,
which wasn't possible before.
2016-03-01 17:34:47 +01:00
Christoph Büscher aecf51cb42 Merge branch 'master' into feature-suggest-refactoring
Conflicts:
	core/src/main/java/org/elasticsearch/rest/action/search/RestSearchAction.java
2016-03-01 17:20:15 +01:00
Yannick Welsch d5304c7d41 Add additional fallback to http.publish_port and transport.publish_port
In case where the publish address is different to all bound addresses and an explicit publish_port
setting is not provided, the publish_port is now selected as the unique port of bound addresses.
An exception is thrown in case of ambiguities.

Closes #16626
2016-03-01 16:28:13 +01:00
Nik Everett c7c8bb357a Merge pull request #16861 from nik9000/reindex_is_ready
Reindex required some parsing changes for search requests to support
differing defaults from the regular search api.
2016-03-01 10:02:48 -05:00
Ali Beyad 83d1e0985d Index deletes not applied when cluster UUID has changed
If a node was isolated from the cluster while a delete was happening,
the node will ignore the deleted operation when rejoining as we couldn't
detect whether the new master genuinely deleted the indices or it is a
new fresh "reset" master that was started without the old data folder.
We can now be smarter and detect these reset masters and actually delete
the indices on the node if its not the case of a reset master.

Note that this new protection doesn't hold if the node was shut down. In
that case it's indices will still be imported as dangling indices.

Closes #16825
Closes #11665
2016-03-01 09:49:42 -05:00
javanna 5fb4d73e89 Cat nodes api: fix typo in help output 2016-03-01 15:36:54 +01:00
javanna a70be3a10f Cat nodes api: prevent NPEs in case there are multiple client nodes
If you connect to a client node and call _cat/nodes, and there is at least another client node in the cluster, the http address cannot be retrieved thus we get an NPE. This commit prevents an NPE from being thrown. This bug was introduced with #16770
2016-03-01 15:36:42 +01:00
Martijn van Groningen 8182a282d4 removed unused imports and an unused field 2016-03-01 14:12:26 +01:00
Martijn van Groningen a5c0f77f1d ingest: use diffs for ingest metadata 2016-03-01 13:57:31 +01:00
Luca Cavanna d81bfe6f30 Merge pull request #16880 from javanna/enhancement/rename_search_service_transport_action
Rename SearchServiceTransportAction to SearchTransportService
2016-03-01 12:58:28 +01:00
Boaz Leskes b224cc8898 Node.stop() should stop Discovery
It used to do so through the DiscoveryService but that was removed in #16821 , but I forgot to add an explicit discovery stop
2016-03-01 12:51:38 +01:00
Felipe Forbeck 95a76a5921 testing script compiled once dot files detected 2016-03-01 08:38:28 -03:00
Felipe Forbeck f65f84e0ef Ignoring hidden script files and files with invalid names 2016-03-01 08:38:28 -03:00
Felipe Forbeck 5478c97a36 Minor fixes after review 2016-03-01 08:38:27 -03:00
Felipe Forbeck cc743049cf Skipping hidden files compilation for script service 2016-03-01 08:38:27 -03:00
Christoph Büscher a7053afdb9 Merge branch 'master' into feature-suggest-refactoring
Conflicts:
	docs/reference/migration/migrate_5_0.asciidoc
2016-03-01 12:37:05 +01:00
javanna b7597d7aea Rename SearchServiceTransportAction to SearchTransportService
The suffix TransportAction is misleading as it may make think that it extends TransportAction, but it does not. This class makes accessible the different search operations exposed by SearchService through the transport layer. Also resolved few compiler warnings in the class itself.
2016-03-01 11:35:04 +01:00
Ryan Ernst 8cd919c687 Added jopt simple option parser and switched plugin cli to use it 2016-02-29 19:52:42 -08:00
Nik Everett 95cc3e38fc Check test naming conventions on all modules
The big win here is catching tests that are incorrectly named and will
be skipped by gradle, providing a false sense of security.

The whole thing takes about 10 seconds on my Macbook Air, not counting
compiling the test classes, which seems worth it. Because this runs as
a gradle task with propery UP-TO-DATE handling it can be skipped if the
tests haven't been changed which should save some time.

I chose to keep this in test:framework rather than a new subproject of
buildSrc because ESIntegTestCase and doesn't inroduce any additional
dependencies.
2016-02-29 16:31:49 -05:00
Boaz Leskes 195b43d66e Remove DiscoveryService and reduce guice to just Discovery #16821
DiscoveryService was a bridge into the discovery universe. This is unneeded and we can just access discovery directly or do things in a different way.

One of those different ways, is not having a dedicated discovery implementation for each our dicovery plugins but rather reuse ZenDiscovery.

UnicastHostProviders are now classified by discovery type, removing unneeded checks on plugins.

Closes #16821
2016-02-29 20:23:38 +01:00
Nik Everett e17d8bda93 Make excuses for SearchSourceBuilder.parseXContent
Its useful when you need to have defaults that don't match
SearchSourceBuilder's defaults. You build your SearchSourceBuilder, set the
defaults, and then layer the XContent on top of that.
2016-02-29 13:50:46 -05:00
Nik Everett e9ce61910f Remove reindex from plugins list
It isn't a plugin any more - it is a module.
2016-02-29 10:18:26 -05:00
Nik Everett 356b7cd12b Merge reindex to master because it is ready! 2016-02-29 10:08:29 -05:00
Nik Everett d587a74533 Fix reindex to work with master branch
Stuff changed, reindex's got to change.
2016-02-29 10:04:16 -05:00
Luca Cavanna 6c76ac3e37 Merge pull request #16758 from javanna/enhancement/remove_action_search_type_package
Cleanup search sub transport actions and collapse o.e.action.search.type package into o.e.action.search
2016-02-29 14:52:08 +01:00
javanna 9738ef3a90 Cleanup search sub transport actions and collapse o.e.action.search.type package into o.e.action.search
TransportSearchTypeAction and subclasses are not actually transport actions, but just support classes useful for their inner async actions that can easily be extracted out so that we get rid of one too many level of abstraction.

Same pattern can be applied to TransportSearchScrollQueryAndFetchAction & TransportSearchScrollQueryThenFetchAction which we could remove in favour of keeping only their inner classes named SearchScrollQueryAndFetchAsyncAction and SearchScrollQueryThenFetchAsyncAction.

Remove org.elasticsearch.action.search.type package, collapsed remaining classes into existing org.elasticsearch.action.search package

Make also ParsedScrollId ScrollIdForNode and TransportSearchHelper classes and their methods package private.

Closes #11710
2016-02-29 14:11:18 +01:00
Christoph Büscher 379bd94c2a Merge branch 'master' into feature-suggest-refactoring 2016-02-29 14:06:30 +01:00
Yannick Welsch d76161d51a Write shard state metadata as soon as shard is created / initializing
As we rely on active allocation ids persisted in the cluster state to select
the primary shard copy, we can write shard state metadata on the allocated node
as soon as the node knows about receiving this shard. This also ensures that
in case of primary relocation, when the relocation target is marked as started
by the master node, the shard state metadata with the correct allocation id has
already been written on the relocation target. Before this change, shard state
metadata was only written once the node knows it is marked as started. In case
of failures between master marking the node as started and the node
receiving and processing this event, the relation between the shard copy on disk
and the cluster state could get lost. This means that manual allocation of
the shard using the reroute command allocate_stale_primary was necessary.

Closes #16625
2016-02-29 13:43:58 +01:00
Boaz Leskes c5a290550f Fix exception assertions in MetaDataStateFormatTests.testLoadState (again) 2016-02-29 12:11:10 +01:00
Boaz Leskes e5e21480d4 Fix exception assertions in MetaDataStateFormatTests.testLoadState
Got broken with the change in #16850
2016-02-29 12:03:48 +01:00
Boaz Leskes 4602d8c0dc Added file name to exceptions when failing to read index state #16850
Closes #16713
Closes #16850
2016-02-29 11:42:53 +01:00
David Pilato cadd8664bb Fix regression in test 2016-02-29 10:00:05 +01:00
David Pilato 7a7f112e89 Check mutually exclusive scopes
We want to make sure that a developer does not put more than one scope on a given setting.
2016-02-28 11:33:24 +01:00
David Pilato d77daf3861 Use an SettingsProperty.Dynamic for dynamic properties 2016-02-28 11:06:45 +01:00
George P. Stathis f8d2400ee6 First pass at validating similarities insite the Settings infrastructure. 2016-02-27 18:57:29 -05:00
George P. Stathis a70df69af4 Allow pre v3 indices to overwrite built-in similarities. 2016-02-27 18:56:53 -05:00
David Pilato 31b5e0888f Use an SettingsProperty enumSet
Instead of modifying methods each time we need to add a new behavior for settings, we can simply pass `SettingsProperty... properties` instead.

`SettingsProperty` could be defined then:

```
public enum SettingsProperty {
  Filtered,
  Dynamic,
  ClusterScope,
  NodeScope,
  IndexScope
 // HereGoesYours;
}
```

Then in setting code, it become much more flexible.

TODO: Note that we need to validate SettingsProperty which are added to a Setting as some of them might be mutually exclusive.
2016-02-28 00:48:04 +01:00
Igor Motov 863fab4007 Add start time and duration to tasks
Tasks now contain timestamps indicating when the tasks were created and current run time
2016-02-27 18:41:51 -05:00
Luca Cavanna 0aba19f4fa Merge pull request #16675 from javanna/enhancement/bulk_delete_routing_required
Bulk api: fail deletes when routing is required but not specified
2016-02-27 19:03:33 +01:00
Boaz Leskes 4ecbce1568 Disable testEnforceMaxFileDescriptorLimits for now
See #16835
2016-02-27 18:56:21 +01:00
javanna ba40c5bc9a Excpetion => Exception 2016-02-27 18:35:32 +01:00
javanna 5ff702b212 [TEST] remove needless index operation in SimpleRoutingIT 2016-02-27 18:35:32 +01:00
javanna 65f5cbe568 remove boolean return type from resolveRequest and needless listener arg 2016-02-27 18:35:32 +01:00
javanna 0fe73b51f9 Bulk api: fail deletes when routing is required but not specified
As part of #10136 we removed the transport action for broadcast deletes in case routing is required but not specified. Bulk api worked differently though and kept on doing the broadcast delete internally in that case. This commit makes sure that delete items are marked as failed in such cases. Also the check has been moved up in the code together with the existing check for the update api, and we now make sure that the exception is the same as the one thrown for single document apis (delete/update).

Note that the failure for the update api contained the wrong optype (the type of the document rather than "update"), that's been fixed too and tested.

Closes #16645
2016-02-27 18:35:31 +01:00
Yannick Welsch af918ebf2b Upgrade groovy dependency in lang-groovy module to version 2.4.6
Closes #16830
2016-02-27 17:41:15 +01:00
Jason Tedor f871b9bba2 Disable production limits on snapshot builds
This commit disables the production limits checks on snapshot
builds. This is at a minimum short-term relief for developers that do in
fact bind to external network interfaces, and is possibly a long-term
fix as well. The situation with using the JVM flag MaxFDLimit is far too
complicated.

Closes #16835
2016-02-26 19:55:10 -05:00
Ryan Ernst 03754ade83 Merge branch 'master' into dots2 2016-02-26 15:33:03 -08:00
Ryan Ernst 9c259aca41 Core: Remove log4j exception hiding
At some time in the distant past, Bootstrap could be used by those
embedding elasticsearch. However, it is no longer allowed (now package
private), and so any errors (for example, log4j missing, or any other
exception while initializing) should be exposed directly and cause
elasticsearch to fail to start. This change removes hiding of logging
initialization exceptions.
2016-02-26 14:52:53 -08:00
Ali Beyad a7f6488216 Merge remote-tracking branch 'upstream/master' into
feature-suggest-refactoring
2016-02-26 17:21:12 -05:00
Nik Everett c38119bae9 Merge branch 'master' into feature/reindex 2016-02-26 16:59:54 -05:00
Nik Everett ba5be0332d Remove optional logger wrappers
Removes all our logger wrappers except the wrapper for log4j1.2. If you
depend on Elasticsearch's jar in your application you'll need to declare
log4j 1.2 and/or some bridge to your favorite logger.

We did this to simplify our builds and code. No more commons-logging like
log implementation sniffing. No more optional dependency hacks in gradle.

We might one day want to use j.u.l instead of log4j. If we do want that
we can recover its wrapper by studying this commit. We didn't go directly
to j.u.l in this commit because that is a bigger change. Our logging
configuration is based on log4j1.2 and people are used to it. So it'd
be a much more fraught breaking change to do that conversion.
2016-02-26 16:41:07 -05:00
Nik Everett e3aba38bac Javadoc for common.io.stream
Adds some javadoc to Streamable, Writeable, and friends.
2016-02-26 16:28:44 -05:00
Lee Hinman 5d359f737f Merge branch 'pr/16801' 2016-02-26 13:22:14 -07:00
Adrien Grand 28b0965cc6 Remove `optimize_single_shard` settings. #15978
This optimization has existed for years with no complaints, I suggest that we
remove the settings that allow to disable it.
2016-02-26 14:30:28 +01:00
jaymode 5963f276f3 Settings with complex matchers should not overlap
It is possible to register multiple settings with complex matchers that could both match
a given key. The behavior when this occurs can lead to issues and depends on the
number of settings that have been registered. In order to identify the setting for a given
key, we iterate over the values in a map to find the first setting that matches the given key
and iteration order of a map should not be relied upon.

This commit checks complex settings when adding them and if the keys for these overlap,
an IllegalArgumentException is now thrown.
2016-02-26 06:31:37 -05:00
Daniel Mitterdorfer 9e473c98c6 Merge branch 'pr/16649'
This PR fixes a bunch of typos across the code base.
2016-02-26 11:31:45 +01:00
Ryan Ernst 09df7379c5 Merge branch 'master' into dots2 2016-02-25 13:05:27 -08:00
Jason Tedor d94e391e71 Use System#lineSeparator and not system property
This commit replaces a use of the system property "line.separator" and
replaces it with a dedicated method that provides the same value.

Closes #16776
2016-02-25 12:22:57 -05:00
Nik Everett 7c4420bf3d Merge pull request #16807 from nik9000/not_mvn
Remove mention of mvn from java source files
2016-02-25 11:05:46 -05:00
George P. Stathis f953d34cba Adds unit tests for #16594 and removes prior integration tests. Throws exception on redefining built-in similarities only for indices created on or after v3. 2016-02-25 01:35:16 -05:00
Nik Everett 74754952db Remove mention of mvn from RestIT and Security 2016-02-24 19:43:56 -08:00
James Bertouch cb35455745 Upgrade to Jackson 2.7.0
Closes #16294
2016-02-24 18:06:38 -05:00