Commit Graph

47002 Commits

Author SHA1 Message Date
David Kyle d1280339a8
specifies which index to search in docs for various queries (#43307) (#43428)
the geo-bounding-box and phrase-suggest docs were susceptible to
failing due to other indices in the cluster. This change restricts
the queries to the index that is set up for the test.

relates to #43271.
2019-06-21 10:15:51 +01:00
Simon Willnauer 424ef4f158 SecurityIndexSearcherWrapper doesn't always carry over caches and similarity (#43436)
If DocumentLevelSecurity is enabled SecurityIndexSearcherWrapper doesn't carry
over the cache, cache policy and similarity from the incoming searcher.
2019-06-21 10:19:10 +02:00
Tim Vernum 059eb55108
Use SecureString for password length validation (#43465)
This replaces the use of char[] in the password length validation
code, with the use of SecureString

Although the use of char[] is not in itself problematic, using a
SecureString encourages callers to think about the lifetime of the
password object and to clear it after use.

Backport of: #42884
2019-06-21 17:11:07 +10:00
Yu c88f2f23a5 Make Recovery API support `detailed` params (#29076)
Properly forwards the `detailed` parameter to show the recovery stats details.

Closes #28910
2019-06-21 09:05:33 +02:00
Andrei Stefan 90e151edeb
Mute MultiSearchRequestTests.java tests (#43467) 2019-06-21 08:38:21 +03:00
Jim Ferenczi cc6c114cb8 Fix round up of date range without rounding (#43303)
Today when searching for an exclusive range the java date math parser rounds up the value
with the granularity of the operation. So when searching for values that are greater than
"now-2M" the parser rounds up the operation to "now-1M". This behavior was introduced when
we migrated to java date but it looks like a bug since the joda math parser rounds up values
but only when a rounding is used. So "now/M" is rounded to "now-1ms" (minus 1ms to get the largest inclusive value)
in the joda parser if the result should be exclusive but no rounding is applied if the input
is a simple operation like "now-1M". This change restores the joda behavior in order to have
a consistent parsing in all versions.

Closes #43277
2019-06-20 23:59:08 +02:00
Tim Brooks 827f8fcbd5
Move reindex request parsing into request (#43450)
Currently the fromXContent logic for reindex requests is implemented in
the rest action. This is inconsistent with other requests where the
logic is implemented in the request. Additionally, it requires access to
the rest action in order to parse the request. This commit moves the
logic and tests into the ReindexRequest.
2019-06-20 17:49:11 -04:00
Ryan Ernst 7b0a259b2c Clarify unsupported secure settings behavior (#43454)
This commit tweaks the docs for secure settings to ensure the user is
aware adding non secure settings to the keystore will result in
elasticsearch not starting.

fixes #43328

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2019-06-20 14:27:27 -07:00
Armin Braun 21515b9ff1
Fix IpFilteringIntegrationTests (#43019) (#43434)
* Increase timeout to 5s since we saw 500ms+ GC pauses on CI
* closes #40689
2019-06-20 22:31:59 +02:00
Armin Braun 1053a89b79
Log Blocked IO Thread State (#43424) (#43447)
* Let's log the state of the thread to find out if it's dead-locked or just stuck after being suspended
* Relates #43392
2019-06-20 22:31:36 +02:00
Deb Adair 6b1e45b5b3 [DOCS] Updated the URL for starting in the cloud. 2019-06-20 13:09:21 -07:00
sandmannn cf610b5e81 Added parsing of erroneous field value (#42321) 2019-06-20 15:24:04 -04:00
Armin Braun 6e7a0e1b2a
Remove Azure Container Exists Check (#43288) (#43433)
* This check is redundant, if the container doesn't exist subsequent operations will fail anyway. Since we are not running this exists check during verification I don't think there's much point to having it in snapshot initialization.
* This PR is mainly motivated by the fact that this forces more permissions to be available in shared environments
2019-06-20 20:23:51 +02:00
debadair 2319fe74c3 [DOCS] Fixed path to install directory. (#43443) 2019-06-20 10:36:28 -07:00
Jake Landis 2f2d0a198f
add version 6.8.2 2019-06-20 12:07:55 -05:00
Yannick Welsch 29d76baf7d Increase timeout for assertSeqNos
Helps with tests that do async translog syncing
2019-06-20 19:06:49 +02:00
Jack Conradson 7ef62a6cbc Remove old Painless doc generator (#43404)
This removes the previous Painless API Doc Generator prior to contexts 
existing. It has been replaced with the new doc generator that uses the 
documentation rest API.
2019-06-20 08:48:52 -07:00
Zachary Tong a8a81200d0 Better support for unmapped fields in AggregatorTestCase (#43405)
AggregatorTestCase will NPE if only a single, null MappedFieldType
is provided (which is required to simulate an unmapped field).  While
it's possible to test unmapped fields by supplying other, non-related
field types... that's clunky and unnecessary.  AggregatorTestCase
just needs to filter out null field types when setting up.
2019-06-20 11:31:49 -04:00
Yannick Welsch 8c856d6d91 Adapt local checkpoint assertion
With async durability, it does not hold true anymore after #43205. This is fine.
2019-06-20 17:29:53 +02:00
Lisa Cawley 5f8db95d60 [DOCS] Describe setup for monitoring logs (#42655) 2019-06-20 08:17:27 -07:00
debadair 7b740b4ea3 [DOCS] Add brew install instructions. Closes #42914 (#42915) 2019-06-20 07:56:49 -07:00
David Kyle 12bc38d9e6 Mute put-transform docs test
Relates to #43271
2019-06-20 15:54:24 +01:00
Armin Braun b7322812e0
Upgrade AWS SDK to Latest Version (#42708) (#43422)
* Just staying up to data on the SDK version
* Use `AbstractAmazonEC2` to shorten code
2019-06-20 16:43:42 +02:00
Armin Braun 99a44a04f7
Fix Infinite Loops in ExceptionsHelper#unwrap (#42716) (#43421)
* Fix Infinite Loops in ExceptionsHelper#unwrap

* Keep track of all seen exceptions and break out on loops
* Closes #42340
2019-06-20 16:38:28 +02:00
Armin Braun 39fef8379b
Fix FsRepositoryTests.testSnapshotAndRestore (#42925) (#43420)
* The commit generation can be 3 or 2 here -> fixed by checking the actual generation on the second commit instead of hard coding 2
* Closes #42905
2019-06-20 16:36:40 +02:00
Armin Braun 7d1983a7e3
Fix Operation Timestamps in Tests (#43155) (#43419)
* For the issue in #43086 we were running into inactive shards because the random timestamps previously used would randomly make `org.elasticsearch.index.shard.IndexShard#checkIdle` see an incorrect+huge  inactive time
* Also fixed one other spot in tests that passed `ms` instead of `ns` for the same timestamp on an index op to correctly use relative `ns`
* Closes #43086
2019-06-20 16:36:17 +02:00
Armin Braun 21e74dd7d2
Upgrade GCS Repository Dependencies (#43142) (#43418)
* Upgrade to latest GCS SDK and transitive dependencies (I chose the later version here on conflict)
* Remove now unnecessary hack for custom endpoints (the linked bugs were both resolved in the SDK)
2019-06-20 16:35:54 +02:00
synical b4c4018d00 Remove Confusing Comment (#43400) 2019-06-20 15:02:37 +01:00
David Turner c8eb09f158 Fail connection attempts earlier in tests (#43320)
Today the `DisruptibleMockTransport` always allows a connection to a node to be
established, and then fails requests sent to that node such as the subsequent
handshake. Since #42342, we log handshake failures on an open connection as a
warning, and this makes the test logs rather noisy. This change fails the
connection attempt first, avoiding these unrealistic warnings.
2019-06-20 14:45:24 +01:00
Armin Braun 5af9387fad Fix Stuck IO Thread Logging Time Precision (#42882)
* The precision of the timestamps we get from the cached time thread is only 200ms by default resulting in a number of needless ~200ms slow network thread execution logs
  * Fixed by making the warn threshold a function of the precision of the cached time thread found in the settings
2019-06-20 14:26:20 +02:00
Andrei Stefan 84d3de1fee
Mute DieWithDignityIT test. (#43415) 2019-06-20 15:10:21 +03:00
Yannick Welsch e04a2258fc Fix testGlobalCheckpointSync
The test needed adaption after #43205, as the ReplicationTracker now distinguishes between the
knowledge of the persisted global checkpoint and the computed global checkpoint on the primary

Follow-up to #43205
2019-06-20 14:00:00 +02:00
Yannick Welsch a76c034866 Reduce shard started failure logging (#43330)
If the master is stepping or shutting down, the error-level logging can cause quite a bit of noise.
2019-06-20 13:23:05 +02:00
Yannick Welsch 7f8e1454ab Advance checkpoints only after persisting ops (#43205)
Local and global checkpoints currently do not correctly reflect what's persisted to disk. The issue is
that the local checkpoint is adapted as soon as an operation is processed (but not fsynced yet). This
leaves room for the history below the global checkpoint to still change in case of a crash. As we rely
on global checkpoints for CCR as well as operation-based recoveries, this has the risk of shard
copies / follower clusters going out of sync.

This commit required changing some core classes in the system:

- The LocalCheckpointTracker keeps track now not only of the information whether an operation has
been processed, but also whether that operation has been persisted to disk.
- TranslogWriter now keeps track of the sequence numbers that have not been fsynced yet. Once
they are fsynced, TranslogWriter notifies LocalCheckpointTracker of this.
- ReplicationTracker now keeps track of the persisted local and persisted global checkpoints of all
shard copies when in primary mode. The computed global checkpoint (which represents the
minimum of all persisted local checkpoints of all in-sync shard copies), which was previously stored
in the checkpoint entry for the local shard copy, has been moved to an extra field.
- The periodic global checkpoint sync now also takes async durability into account, where the local
checkpoints on shards only advance when the translog is asynchronously fsynced. This means that
the previous condition to detect inactivity (max sequence number is equal to global checkpoint) is
not sufficient anymore.
- The new index closing API does not work when combined with async durability. The shard
verification step is now requires an additional pre-flight step to fsync the translog, so that the main
verify shard step has the most up-to-date global checkpoint at disposition.
2019-06-20 11:12:38 +02:00
Christoph Büscher adab7eae71 [Docs] Remove boost parameter from intervals-query example (#43331)
The boost factor doesn't seem to be needed and can be removed.
2019-06-20 10:34:14 +02:00
Tanguy Leroux 24cfca53fa Reconnect remote cluster when seeds are changed (#43379)
The RemoteClusterService should close the current 
RemoteClusterConnection and should build it again if 
the seeds are changed, similarly to what is done when 
the ping interval or the compression settings are changed.

Closes #37799
2019-06-20 10:30:02 +02:00
Andrei Stefan d684119618 Remove mentions of "fields with the same name in the same index" (#43077)
Together with types removal, any mention of "fields with the same name in the same index" doesn't make sense anymore.

(cherry picked from commit c5190106cbd4c007945156249cce462956933326)
2019-06-20 11:26:12 +03:00
Andrei Stefan fe0f9055d8 Fix NPE in case of subsequent scrolled requests for a CSV/TSV formatted response (#43365)
(cherry picked from commit 0ef7bb0f8b07cd0392d37f96ca9360821b19315a)
2019-06-20 11:26:11 +03:00
Luca Cavanna 94a4bc9933 SearchPhaseContext to not extend ActionListener (#43269)
The fact that SearchPhaseContext extends ActionListener makes it hard
to reason about when the original listener is notified and to trace
those calls. Also, the corresponding onFailure and onResponse were
only needed in two places, one each, where they can be replaced by a
more intuitive call, like sendSearchResponse for onResponse.
2019-06-20 10:21:24 +02:00
Jim Ferenczi c33d62adbc Reduce the number of docvalues iterator created in the global ordinals fielddata (#43091)
Today the fielddata for global ordinals re-creates docvalues readers of each segment
when building the iterator of a single segment. This is required because the lookup of
global ordinals needs to access the docvalues's TermsEnum of each segment to retrieve
the original terms. This also means that we need to create NxN (where N is the number of segment in the index) docvalues iterators
each time we want to collect global ordinal values. This wasn't an issue in previous versions since docvalues readers are stateless
before 6.0 so they are reused on each segment but now that docvalues are iterators we need to create a new instance each time
we want to access the values. In order to avoid creating too many iterators this change splits
the global ordinals fielddata in two classes, one that is used to cache a single instance per directory reader and one
that is created from the cached instance that can be used by a single consumer. The latter creates the TermsEnum of each segment
once and reuse them to create the segment's iterator. This prevents the creation of all TermsEnums each time we want to access
the value of a single segment, hence reducing the number of docvalues iterator to create to Nx2 (one iterator and one lookup per segment).
2019-06-20 08:44:07 +02:00
Jason Tedor 1f1a035def
Remove stale test logging annotations (#43403)
This commit removes some very old test logging annotations that appeared
to be added to investigate test failures that are long since closed. If
these are needed, they can be added back on a case-by-case basis with a
comment associating them to a test failure.
2019-06-19 22:58:22 -04:00
Lee Hinman c2bf628a6d
[7.x] Narrow period of Shrink action in which ILM prevents stopping (#43254) (#43393)
* Narrow period of Shrink action in which ILM prevents stopping

Prior to this change, we would prevent stopping of ILM if the index was
anywhere in the shrink action. This commit changes
`IndexLifecycleService` to allow stopping when in any of the innocuous
steps during shrink. This changes ILM only to prevent stopping if
absolutely necessary.

Resolves #43253

* Rename variable for ignore actions -> ignore steps

* Fix comment

* Factor test out to test *all* stoppable steps
2019-06-19 16:37:41 -06:00
Lee Hinman 6b084e55c5
[7.x] Prevent NullPointerException in TransportRolloverAction (#43353) (#43397)
It's possible for the passed in `IndexMetaData` to be null (for
instance, cluster state passed in does not have the index in its
metadata) which in turn can cause a `NullPointerException` when
evaluating the conditions for an index. This commit adds null protection
and unit tests for this case.

Resolves #43296
2019-06-19 16:07:28 -06:00
Benjamin Trent 77ce3260dd
[ML][Data Frame] make response.count be total count of hits (#43241) (#43389)
* [ML][Data Frame] make response.count be total count of hits

* addressing line length check

* changing response count for filters

* adjusting serialization, variable name, and total count logic

* making count mandatory for creation
2019-06-19 16:19:06 -05:00
Benjamin Trent b333ced5a7
[7.x] [ML][Data Frame] adds new pipeline field to dest config (#43124) (#43388)
* [ML][Data Frame] adds new pipeline field to dest config (#43124)

* [ML][Data Frame] adds new pipeline field to dest config

* Adding pipeline support to _preview

* removing unused import

* moving towards extracting _source from pipeline simulation

* fixing permission requirement, adding _index entry to doc

* adjusting for java 8 compatibility

* adjusting bwc serialization version to 7.3.0
2019-06-19 16:18:27 -05:00
Jim Ferenczi b957aa46ce Allocate memory lazily in BestBucketsDeferringCollector (#43339)
While investigating memory consumption of deeply nested aggregations for #43091
the memory used to keep track of the doc ids and buckets in the BestBucketsDeferringCollector
showed up as one of the main contributor. In my tests half of the memory held in the
 BestBucketsDeferringCollector is associated to segments that don't have matching docs
 in the selected buckets. This is expected on fields that have a big cardinality since each
 bucket can appear in very few segments. By allocating the builders lazily this change
 reduces the memory consumption by a factor 2 (from 1GB to 512MB), hence reducing the
impact on gcs for these volatile allocations. This commit also switches the PackedLongValues.Builder
with a RoaringDocIdSet in order to handle very sparse buckets more efficiently.

I ran all my tests on the `geoname` rally track with the following query:

````
{
    "size": 0,
    "aggs": {
        "country_population": {
            "terms": {
                "size": 100,
                "field": "country_code.raw"
            },
            "aggs": {
                "admin1_code": {
                    "terms": {
                        "size": 100,
                        "field": "admin1_code.raw"
                    },
                    "aggs": {
                        "admin2_code": {
                            "terms": {
                                "size": 100,
                                "field": "admin2_code.raw"
                            },
                            "aggs": {
                                "sum_population": {
                                    "sum": {
                                        "field": "population"
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}
````
2019-06-19 22:10:59 +02:00
Christos Soulios d1637ca476
Backport: Refactor aggregation base classes to remove doEquals() and doHashCode() (#43363)
This PR is a backport a of #43214 from v8.0.0

A number of the aggregation base classes have an abstract doEquals() and doHashCode() (e.g. InternalAggregation.java, AbstractPipelineAggregationBuilder.java).

Theoretically this is so the sub-classes can add to the equals/hashCode and don't need to worry about calling super.equals(). In practice, it's mostly just confusing/inconsistent. And if there are more than two levels, we end up with situations like InternalMappedSignificantTerms which has to call super.doEquals() which defeats the point of having these overridable methods.

This PR removes the do versions and just use equals/hashCode ensuring the super when necessary.
2019-06-19 22:31:06 +03:00
Armin Braun be42b2c70c
Fix NetworkUtilsTests (#43295) (#43378)
* Follow up to #42109:
   * Adjust test to only check that interface lookup by name works not actually lookup IPs which is brittle since virtual interfaces can be destroyed/created by Docker while the tests are running

Co-authored-by:  Jason Tedor <jason@tedor.me>
2019-06-19 21:23:09 +02:00
James Baiera 1dde6ba1db Muting DataFrameTransformIT.testGetStats
See #43324
2019-06-19 13:58:13 -04:00
Jason Tedor bf74d38782
Fix GeoIP custom database directory in docs (#43383)
These docs were misleading for package installations of
Elasticsearch. Instead, we should refer to $ES_CONFIG/ingest-geoip as
the path to place the custom database files. For non-package
installations, this is the same as $ES_HOME/config, but for package
installations this is not the case as the config directory for package
installations is /etc/elasticsearch, and is not relative to
$ES_HOME. This commit corrects the docs.
2019-06-19 13:26:07 -04:00