Commit Graph

29781 Commits

Author SHA1 Message Date
Tim Brooks da5f52a2fc
Add test for writer operation buffer accounting (#27707)
This is a follow up to #27695. This commit adds a test checking that
across multiple writes using multiple buffers, a write operation
properly keeps track of which buffers still need to be written.
2017-12-07 12:48:49 -07:00
Lee Hinman cca54b811d [TEST] Wait for merging to complete before testing breaker
It's possible that a merge may be ongoing when we check the breaker and segment
stats' memory usage, this causes the test to fail. Instead, we should wait for
merging to complete.

Resolves #27651
2017-12-07 11:57:22 -07:00
olcbean bcc33f391f Add Open Index API to the high level REST client (#27574)
Add _open to the high level REST client

Relates to #27205
2017-12-07 18:16:03 +01:00
Christoph Büscher b83e14858a Correcting some minor typos in comments 2017-12-07 16:39:23 +01:00
Yannick Welsch 5a53798f83 Add unreleased v5.6.6 version 2017-12-07 14:59:57 +01:00
Christoph Büscher 52cb6c8ef2 Merge branch 'master' into rankeval 2017-12-07 14:22:46 +01:00
Robin Neatherway 057efea893 Correct two equality checks on incomparable types (#27688) 2017-12-07 14:18:11 +01:00
markwalkom ee21045697 [Docs] Add date math examples to api-conventions.asciidoc (#25217) 2017-12-07 14:10:51 +01:00
Yannick Welsch 69dd667f5e Add unreleased v6.0.2 version 2017-12-07 11:54:22 +01:00
Catalin Ursachi f823cea79c Added Create Index support to high-level REST client (#27351)
Relates to #27205
2017-12-07 11:39:59 +01:00
Yannick Welsch 0b102f6372 [TEST] Fix testOpenWaitingForActiveShardsFailed
This test periodically fails if the nodes that apply the cluster state fail to ack the change within 100ms. This commit changes the checks on the test so that
it still checks that the open command has taken effect, but that the wait for active shards has actually failed.
2017-12-07 10:23:36 +01:00
Tim Brooks 5b3230cbae
Fix issue where the incorrect buffers are written (#27695)
This is a followup to #27551. That commit introduced a bug where the
incorrect byte buffers would be returned when we attempted a write. This
commit fixes the logic.
2017-12-06 20:57:46 -07:00
Jason Tedor 6c7374804f
Extend JVM options to support multiple versions
JDK 9 has removed JVM options that were valid in JDK 8 (e.g., GC logging
flags) and replaced them with new flags that are not available in JDK
8. This means that a single JVM options file can no longer apply to JDK
8 and JDK 9, complicating development, complicating our packaging story,
and complicating operations. This commit extends the JVM options syntax
to specify the range of versions the option applies to. If the running
JVM matches the range of versions, the flag will be used to start the
JVM otherwise the flag will be ignored.

We implement this parser in Java for simplicity, and with this we start
our first step towards a Java launcher.

Relates #27675
2017-12-06 18:03:13 -05:00
Tim Brooks 2aa62daed4
Introduce resizable inbound byte buffer (#27551)
This is related to #27563. In order to interface with java nio, we must
have buffers that are compatible with ByteBuffer. This commit introduces
a basic ByteBufferReference to easily allow transferring bytes off the
wire to usage in the application.

Additionally it introduces an InboundChannelBuffer. This is a buffer
that can internally expand as more space is needed. It is designed to
be integrated with a page recycler so that it can internally reuse pages.
The final piece is moving all of the index work for writing bytes to a
channel into the WriteOperation.
2017-12-06 11:02:25 -07:00
Boaz Leskes e0e698bc26 testCorruptTranslogTruncation: add logging 2017-12-06 14:46:39 +01:00
Jim Ferenczi caea6b70fa
Add a new cluster setting to limit the total number of buckets returned by a request (#27581)
This commit adds a new dynamic cluster setting named `search.max_buckets` that can be used to limit the number of buckets created per shard or by the reduce phase. Each multi bucket aggregator can consume buckets during the final build of the aggregation at the shard level or during the reduce phase (final or not) in the coordinating node. When an aggregator consumes a bucket, a global count for the request is incremented and if this number is greater than the limit an exception is thrown (TooManyBuckets exception).
This change adds the ability for multi bucket aggregator to "consume" buckets in the global limit, the default is 10,000. It's an opt-in consumer so each multi-bucket aggregator must explicitly call the consumer when a bucket is added in the response.

Closes #27452 #26012
2017-12-06 09:15:28 +01:00
Simon Willnauer 70f8ea367b
Allow index settings to be reset by wildcards (#27671)
Index settings didn't support reset by wildcard which also causes
issues like #27537 where archived settings can't be reset. This change
adds support for wildcards like `archived.*` to be used to reset setting to their
defaults or remove them from an index.

Closes #27537
2017-12-06 07:35:37 +01:00
javanna 234e09a105 Fix UpdateMappingIntegrationIT test failures
The mappings can be submitted wrapped in a type object or not. They need to be returned in the same way as they were submitted. When applying field filters, we need to make sure that the format is preserved. MappingMetaData#getSourceAsMap removes the root level if it's the type object, which would make us overwrite the original mappings with filtered mappings but without the original root object.

Closes #27678
2017-12-06 01:43:17 +01:00
Ryan Ernst c51e48bec0
Correct docs for binary fields and their default for doc values (#27680)
closes #27240
2017-12-05 15:10:18 -08:00
Catalin Ursachi e584c52017 Clarify IntelliJ IDEA Jar Hell fix (#27635) 2017-12-05 14:39:01 -08:00
Ryan Ernst 8139e3a1c7
Add validation of keystore setting names (#27626)
This commit restricts settings added to the keystore to have a lowercase
ascii name. The java Keystore javadocs state that case sensitivity of
key alias names are implementation dependent. This ensures regardless of
case sensitivity in a jvm implementation, the keys will be stored as we
expect.
2017-12-05 14:30:36 -08:00
Nhat Nguyen ed2caf2bad
Prevent constructing index template without patterns (#27662)
Today, we prevent the system from storing a broken index template in the
transport layer, however we don't prevent this in XContent. A broken
index template can break the whole cluster state.

This commit attempts to prevent the system from constructing an index
template without a proper index patterns.
2017-12-05 15:38:33 -05:00
Deb Adair 2f9a882061 [DOCS] Fixed typos and broken attribute. 2017-12-05 11:46:40 -08:00
Luca Cavanna f4fb4d3bf5
Add support for filtering mappings fields (#27603)
Add support for filtering fields returned as part of mappings in get index, get mappings, get field mappings and field capabilities API.

Plugins can plug in their own function, which receives the index as argument, and return a predicate which controls whether each field is included or not in the returned output.
2017-12-05 20:31:29 +01:00
debadair 25ec068aef
[DOCS] Added link to upgrade guide and bumped the upgrade topic up to the top level (#27621)
* [DOCS] Added link to the upgrade guide & tweaked the intro.

* [DOCS] Bumped upgrade topic up to the top level of the TOC
2017-12-05 10:58:52 -08:00
Nicholas Knize 8bcf5393f2 [Geo] Add Well Known Text (WKT) Parsing Support to ShapeBuilders
This commit adds WKT support to Geo ShapeBuilders.

This supports the following format:

POINT (30 10)
LINESTRING (30 10, 10 30, 40 40)
BBOX (-10, 10, 10, -10)
POLYGON ((30 10, 40 40, 20 40, 10 20, 30 10))
POLYGON ((35 10, 45 45, 15 40, 10 20, 35 10), (20 30, 35 35, 30 20, 20 30))
MULTIPOINT ((10 40), (40 30), (20 20), (30 10))
MULTIPOINT (10 40, 40 30, 20 20, 30 10)
MULTILINESTRING ((10 10, 20 20, 10 40),(40 40, 30 30, 40 20, 30 10))
MULTIPOLYGON (((30 20, 45 40, 10 40, 30 20)), ((15 5, 40 10, 10 20, 5 10, 15 5)))
MULTIPOLYGON (((40 40, 20 45, 45 30, 40 40)), ((20 35, 10 30, 10 10, 30 5, 45 20, 20 35), (30 20, 20 15, 20 25, 30 20)))
GEOMETRYCOLLECTION (POINT (30 10), MULTIPOINT ((10 40), (40 30), (20 20), (30 10)))

closes #9120
2017-12-05 10:56:41 -06:00
David Turner 5b03e3b53d
Fix up tests now that GeoDistance.*.calculate works (#27541)
This resolves a longstanding @AwaitsFix
2017-12-05 16:19:33 +00:00
Glen Smith 14e972fa42 [Docs] Fix parameter name (#27656) 2017-12-05 14:30:22 +01:00
Jason Tedor 42a4ad35da
Add node name to thread pool executor name
This commit adds the node name to the names of thread pool executors so
that the node name is visible in rejected execution exception messages.

Relates #27663
2017-12-05 07:45:40 -05:00
Jason Tedor 144e1698cf Do not pass executor shutdown to super constructor
The main constructor for rejected execution exception its executor
shutdown constructor parameter to the super constructor where it would
be used as a formatting parameter. This is a mistake so this commit
fixes this issue.
2017-12-05 07:11:14 -05:00
Boaz Leskes 83c8daa5e8 FullClusterRestartIT.testRecovery - add more info on failure 2017-12-05 13:00:42 +01:00
Jason Tedor eb574425b7
Simplify rejected execution exception
This exception type has several unnecessary constructor overrides so
this commit removes them.

Relates #27664
2017-12-05 06:58:34 -05:00
Andrew Banchich 1be286c592 [Docs] Grammatical fix in bootstrap-checks.asciidoc (#27655) 2017-12-05 11:46:19 +01:00
Martijn van Groningen b447967104
removed redundant import 2017-12-05 11:01:25 +01:00
Clinton Gormley 0bba2a8438 Update removal_of_types.asciidoc
Corrected  `include_in_type` to `include_type_name`
2017-12-05 10:44:48 +01:00
Martijn van Groningen 4d78e1a9ad
Added msearch api to high level client 2017-12-05 10:17:47 +01:00
Boaz Leskes c203cff692 fix java9 compilation 2017-12-05 09:28:56 +01:00
Jason Tedor 99db391344
Add explicit coreutils dependency
The RPM and Debian packages depend on coreutils (for mktemp among
others). This commit adds an explicit package dependency on coreutils.

Relates #27660
2017-12-04 21:21:57 -05:00
Jason Tedor 2208a1a7b5
Detect mktemp from coreutils
GNU mktemp and BSD mktemp have different command line flags. On some
macOS systems users have mktemp from coreutils in their PATH overriding
the system mktemp from BSD. This commit adds detection for the coreutils
mktemp versus the BSD mktemp and uses the appropriate syntax based on
the detection.

Relates #27659
2017-12-04 19:53:14 -05:00
Jim Ferenczi 8635f68ece Fix term vectors generator with keyword and normalizer (bis)
Fallback on the index analyzer if the normalizer is null.

Closes #27320
2017-12-05 01:33:14 +01:00
Jason Tedor 963ed25cf5
Only fsync global checkpoint if needed
In the global checkpoint sync action, we fsync the translog. However,
the last synced global checkpoint might already be equal to the current
global checkpoint in which case the fsyncing the translog is unnecessary
as either the sync needed guard in the translog will skip the translog,
or the translog needs an fsync for another reason that will be picked up
elsewhere (e.g., at the end of a bulk request).

Relates #27652
2017-12-04 18:10:04 -05:00
Lee Hinman d3721c48c3 [TEST] Add AwaitsFix for FullClusterRestartIT.testRecovery
See: #27649
2017-12-04 15:42:43 -07:00
Jason Tedor 72800bb90b
Fix Lucene version for 6.2.0 constant in master
This commit fixes the Lucene version constant in master for the 6.2.0
version.

Relates #27658
2017-12-04 16:38:45 -05:00
Nhat Nguyen 4b558636f0
TEST: Rewrite GeoPointParsingTests#testEqualsHashCodeContract (#27634)
The hashCode contract states that equal objects must have equal hash
codes, however the unequal objects are not required to have unequal
hashCodes.

This commit rewrites GeoPointParsingTests#testEqualsHashCodeContract
using#checkEqualsAndHashCode helper.

Closes #27633
2017-12-04 16:34:34 -05:00
Lee Hinman 1ff5ef9055 [TEST] Check accounting breaker is equal to segment stats rather than 0
If there are existing indices, it may not be 0
2017-12-04 14:15:23 -07:00
Jim Ferenczi e0b1a6544d
Fix term vectors generator with keyword and normalizer (#27608)
This change applies the normalizer defined on the field when building term vectors dynamically on a keyword field.

Fixes #27320
2017-12-04 21:32:16 +01:00
Jim Ferenczi da50fa4540
Fix highlighting on a keyword field that defines a normalizer (#27604)
* Fix highlighting on a keyword field that defines a normalizer

The `plain` and sometimes the `unified` highlighters need to re-analyze the content to highlight a field
This change makes sure that we don't ignore the normalizer defined on the keyword field for this analysis.
2017-12-04 21:04:14 +01:00
Jason Tedor 17a2d574de
Obey translog durability in global checkpoint sync
After write operations in some situations we fire a post-operation
global checkpoint sync. The global checkpoint sync unconditionally
fsyncs the translog and this can then look like an fsync
per-request. This violates the translog durability settings on the index
if this durability is set to async. This commit changes the global
checkpoint sync to observe the translog durability.

Relates #27641
2017-12-04 12:14:25 -05:00
Boaz Leskes 0b50b313d2 RecoveryIT.testRecoveryWithConcurrentIndexing should check for 110 docs in an upgraded cluster
Closes #27650
2017-12-04 18:06:02 +01:00
Simon Willnauer 84ec472428
Include internal refreshes in refresh stats (#27615)
Today we exclude internal refreshes in the refresh stats. Yet, it's very much
confusing to not take these into account. This change includes internal refreshes
into the stats until we have a dedicated stats for this.
2017-12-04 16:33:47 +01:00