Commit Graph

28805 Commits

Author SHA1 Message Date
Jason Tedor 530d80fdc1 Fix global checkpoint sync log message
The log message here is incorrect, a failure here is occuring on the
post-operation global checkpoint sync, not the background sync. This
commit fixes the log message.
2017-09-25 16:38:58 -04:00
Nik Everett eb754a71be Fix update_by_query's default size parameter (#26784)
We were accidentally defaulting it to the scroll size.
Untwists some of the tricks that we play with parsing
so that the size is no longer scrambled.

Closes #26761
2017-09-25 16:25:27 -04:00
Jay Modi b8cd82e5c2 Increase time to wait for green in rolling upgrade tests (#26781)
This commit increases the amount of time to wait for green to accound for unassigned shards that
have been delayed. The default delay is 60s, so we need to wait longer than that. Previously, the
wait would timeout at 30s due to the rest client and the default for the cluster health api.

Closes #26742
2017-09-25 12:39:33 -06:00
David Pilato 3f71772cd2 Azure snapshots can not be restored anymore (#26778)
While working on #26751 and doing some manual integration testing I found that this #22858 removed an important line of our code:

`AzureRepository` overrides default `initializeSnapshot` method which creates metadata files and do other stuff.

But with PR #22858, I wrote:

```java
    @Override
    public void initializeSnapshot(SnapshotId snapshotId, List<IndexId> indices, MetaData clusterMetadata) {
        if (blobStore.doesContainerExist(blobStore.container()) == false) {
            throw new IllegalArgumentException("The bucket [" + blobStore.container() + "] does not exist. Please create it before " +
                " creating an azure snapshot repository backed by it.");
        }
    }
```

instead of

```java
    @Override
    public void initializeSnapshot(SnapshotId snapshotId, List<IndexId> indices, MetaData clusterMetadata) {
        if (blobStore.doesContainerExist(blobStore.container()) == false) {
            throw new IllegalArgumentException("The bucket [" + blobStore.container() + "] does not exist. Please create it before " +
                " creating an azure snapshot repository backed by it.");
        }
        super.initializeSnapshot(snapshotId, indices, clusterMetadata);
    }
```

As we never call `super.initializeSnapshot(...)` files are not created and we can't restore what we saved.

Closes #26777.
2017-09-25 14:35:27 -04:00
Jason Tedor 57aee93693 Add shard ID to failed global checkpoint messages
When a global checkpoint sync fails we should log the shard ID for the
shard the sync failed for. This commit causes this to be the case.
2017-09-25 14:32:55 -04:00
Boaz Leskes ec0c621072 `IndexShard.routingEntry` should only be updated once all internal state is ready (#26776)
The routing entry is used by external components to check whether the shard is ready to perform as primary. Most notably, the peer recovery source handler delays recoveries until the shard routing entry says the shard is ready. 

When a shard is promoted to primary, we currently update the shard's routing entry before we finish all the work relating to the promotion. This can cause recoveries to fail later on because the `GlobalCheckpointTracker` isn't set (yet) to primary mode. 

This commit fixes this issue by updating the routing entry last.
2017-09-25 19:57:16 +02:00
Adrien Grand 579cca9adb Allow copying from a field to another field that belongs to the same nested object. (#26774)
The previous test was too strict and enforced that the target object was a
parent. It has been relaxed so that fields that belong to the same nested
object can copy to each other.

The commit also improves error handling in case of multi-fields. The current
validation works but may throw confusing error messages since it assumes that
only object fields may introduce dots in fields names while multi fields may
too.

Closes #26763
2017-09-25 18:33:26 +02:00
Luca Cavanna 25f154b8c6 [DOCS] improve encrypted communication example in java REST low-level client docs (#26705)
Include SSLContext creation and add the missing "https" scheme to the host creation.

Relates to #26534
Closes #26670
2017-09-25 18:03:57 +02:00
Christoph Büscher 3827918417 Add configurable `maxTokenLength` parameter to whitespace tokenizer (#26749)
Other tokenizers like the standard tokenizer allow overriding the default
maximum token length of 255 using the `"max_token_length` parameter. This change
enables using this parameter also with the whitespace tokenizer. The range that
is currently allowed is from 0 to StandardTokenizer.MAX_TOKEN_LENGTH_LIMIT,
which is 1024 * 1024 = 1048576 characters.

Closes #26643
2017-09-25 17:21:19 +02:00
Adrien Grand 324ee8cc96 Use the 6.6.1 Lucene version constant. (#26768)
It is only possible since we moved to Lucene 7.0.0 GA. Previous snapshots did
not know about it.
2017-09-25 17:06:16 +02:00
kel 8f5f63452a Fix typo in date format (#26503) 2017-09-25 17:01:39 +02:00
Jiri Tyr 76f8701eec Fixing Grok pattern for Apache 2.4 (#26635) 2017-09-25 07:59:37 -07:00
Simon Willnauer aab4655e63 Unify Settings xcontent reading and writing (#26739)
This change adds a fromXContent method to Settings that allows to read
the xcontent that is produced by toXContent. It also replaces the entire settings
loader infrastructure and removes the structured map representation. Future PRs will
also tackle the `getAsMap` that exposes the internal represenation of settings for
better encapsulation.
2017-09-25 13:23:01 +02:00
javanna dee2ae1023 [DOCS] Replace mention of string field type with text and keyword
Closes #25713
2017-09-25 11:12:06 +02:00
Boaz Leskes cd2a4372b4 RecoveryIT should wait for green when in mixed cluster to avoid unassigned shards
The test starts with two old nodes and creates indices (without waiting for green, which is fixed here too). Then it restarts one of the nodes and waits for it to join the cluster. This wait condition only uses wait for yellow as our generic infra doesn't how many nodes are there in total. Once the restarted node is part of the cluster (mixed mode) the second old node is restarted. If indices are not fully allocated when that happens, the shards will go into delayed unassigned mode. If the recovery of the replica never completed we may end up with corrupted / no secondary copy on the node. This will cause the shards to be delayed for 1m before being reassigned and the test will time out.
2017-09-24 22:38:20 +02:00
Boaz Leskes 2b6f75730e RecoveryIT up client time out to 40s to see response in a 30s time 2017-09-24 21:33:20 +02:00
Jason Tedor d87ab67e44 Fix global checkpoint sync test
This commit fixes issues with the global checkpoint sync test. The test
was off in initializing the maximum sequence number on the primary
shard, and off setting the local knowledge on the primary of the global
checkpoint on the replica.
2017-09-22 17:12:32 -04:00
Jason Tedor d8bb413b1b Configure heap dump path out of the box
The JVM defaults to dumping the heap to the working directory of
Elasticsearch. For the RPM and Debian packages, this location is
/usr/share/elasticsearch. This directory is not writable by the
elasticsearch user, so by default heap dumps in this situation are
lost. This commit modifies the packaging for the RPM and Debian packages
to set the heap dump path to /var/lib/elasticsearch as the default
location for dumping the heap. This location is writable by the
elasticsearch user by default. We add documentation of this important
setting if /var/lib/elasticsearch is not suitable for receiving heap
dumps.

Relates #26755
2017-09-22 14:22:03 -04:00
Jason Tedor 2e63a13c0a Upgrade to Log4j 2.9.1
This commit upgrades the Log4j dependency, picking up a fix for an issue
with handling stack traces on JDK 9.

Relates #26750
2017-09-22 11:57:06 -04:00
Yannick Welsch 31f43f5c34 Reenable BWC tests
Relates to #26682
2017-09-22 16:54:00 +02:00
Yannick Welsch df5c450e89 Add v6.1 BWC layer for adding wait_for_active_shards to index open command
This commit disables BWC tests while adding a v6.1 BWC layer for the PR #26682
2017-09-22 16:30:07 +02:00
Martijn van Groningen a056c5d469
aggs: Changed how top_hits initialises leaf collectors
Both TopDocsCollector and LeafCollector were being kept around at the aggregator level. In case the nested aggregator would do a post collection then this could cause pushing down docids to top hits child aggregators that already moved the next LeafCollector (causing assertions to trip and incorrect results).

By keeping track of the LeafCollector in a seperate map at the leaf bucket level this problem can simply not happen any more as the place holding LeafCollector is no longer shared.

Also LeafCollector instances for TopDocsCollectors are no longer pre-created as the beginning a new segment is evaluated. There is no guarantee that TopHitsAggregator encounters a document for a particular bucket and there has to be logic to create LeafCollector instances which have not been seen before.

Closes #26738
2017-09-22 15:59:43 +02:00
Ryan Ernst 5b711c283d Plugins: Add backcompat for sha1 checksums (#26748)
With 6.0 rc1 we now publish sha512 checksums for official plugins.
However, in order to ease the pain for plugin authors, this commit adds
backcompat to still allow sha1 checksums.  Also added tests for
checksums.

Closes #26746
2017-09-22 11:26:32 +02:00
Alexander Kazakov ff737a880c Add wait_for_active_shards parameter to index open command (#26682)
Adds the wait_for_active_shards parameter to the index open command. Similar to the index creation command, the index open command will now, by default, wait until the primaries have been allocated.

Closes #20937
2017-09-22 11:15:03 +02:00
Martijn van Groningen 109c6c2717
aggs: Do not delegate a null scorer to LeafBucketCollectors
Closes #26611
2017-09-22 09:20:57 +02:00
Yannick Welsch 76e1b7437c [TEST] Remove assertSeqNos from testAckedIndexing 2017-09-22 08:31:36 +02:00
Jason Tedor e0db89bc35 Upgrade to Lucene 7.0.0
This commit upgrades to the GA release of Luence 7!

Relates #26744
2017-09-21 19:19:33 -04:00
Jason Tedor 954fb1c80d Reenable BWC tests after global checkpoint sync
This commit reenables the BWC tests after the introduction of the
post-operation and background global checkpoint sync.

Relates #26591
2017-09-21 15:44:36 -04:00
Jason Tedor f35d1de502 Introduce global checkpoint background sync
It is the exciting return of the global checkpoint background
sync. Long, long ago, in snapshot version far, far away we had and only
had a global checkpoint background sync. This sync would fire
periodically and send the global checkpoint from the primary shard to
the replicas so that they could update their local knowledge of the
global checkpoint. Later in time, as we sped ahead towards finalizing
the initial version of sequence IDs, we realized that we need the global
checkpoint updates to be inline. This means that on a replication
operation, the primary shard would piggy back the global checkpoint with
the replication operation to the replicas. The replicas would update
their local knowledge of the global checkpoint and reply with their
local checkpoint. However, this could allow the global checkpoint on the
primary to advance again and the replicas would fall behind in their
local knowledge of the global checkpoint. If another replication
operation never fired, then the replicas would be permanently behind. To
account for this, we added one more sync that would fire when the
primary shard fell idle. However, this has problems:
 - the shard idle timer defaults to five minutes, a long time to wait
   for the replicas to learn of the new global checkpoint
 - if a replica missed the sync, there was no follow-up sync to catch
   them up
 - there is an inherent race condition where the primary shard could
   fall idle mid-operation (after having sent the replication request to
   the replicas); in this case, there would never be a background sync
   after the operation completes
 - tying the global checkpoint sync to the idle timer was never natural

To fix this, we add two additional changes for the global checkpoint to
be synced to the replicas. The first is that we add a post-operation
sync that only fires if there are no operations in flight and there is a
lagging replica. This gives us a chance to sync the global checkpoint to
the replicas immediately after an operation so that they are always kept
up to date. The second is that we add back a global checkpoint
background sync that fires on a timer. This timer fires every thirty
seconds, and is not configurable (for simplicity). This background sync
is smarter than what we had previously in the sense that it only sends a
sync if the global checkpoint on at least one replica is lagging that of
the primary. When the timer fires, we can compare the global checkpoint
on the primary to its knowledge of the global checkpoint on the replicas
and only send a sync if there is a shard behind.

Relates #26591
2017-09-21 15:34:13 -04:00
James Baiera c760eec054 Add permission checks before reading from HDFS stream (#26716)
Add checks for special permissions before reading hdfs stream data. Also adds test from 
readonly repository fix. MiniHDFS will now start with an existing repository with a single snapshot 
contained within. Readonly Repository is created in tests and attempts to list the snapshots 
within this repo.
2017-09-21 11:55:07 -04:00
Martijn van Groningen fda8f8b827
muted test 2017-09-21 17:24:18 +02:00
wasserman 67845134de [Docs] Fixed typo of *configuration* (#25058) 2017-09-21 16:49:00 +02:00
kel 601be4f83e Add azure storage endpoint suffix #26432 (#26568)
Allow specifying azure storage endpoint suffix for an azure client.
2017-09-20 22:26:19 -07:00
lcawley 06551a8549 [DOCS] Added index-shared4 and index-shared5.asciidoc 2017-09-20 10:54:26 -07:00
Jay Modi c47f24d406 BulkProcessor flush runnable preserves the thread context from creation time (#26718)
When using a bulk processor, the thread context was not preserved for the flush runnable which is
executed in another thread in the thread pool. This change wraps the flush runnable in a context
preserving runnable so that the headers and transients from the creation time of the bulk processor
are available during the execution of the flush.

Closes #26596
2017-09-20 10:19:42 -06:00
Simon Willnauer b9c0d4447c Catch exceptions and inform handler in RemoteClusterConnection#collectNodes (#26725)
This adds a missing catch block to invoke the action listener instead of bubbeling
up the exception.

Closes #26700
2017-09-20 17:53:12 +02:00
Tahmim Ahmed Shibli 34662c9e6d [Docs] Fix name of character filter in example. (#26724) 2017-09-20 17:08:43 +02:00
Christoph Büscher 86b00b84bc Remove parse field deprecations in query builders (#26711)
The `fielddata` field and the use of the `_name` field in the short syntax of the range 
query have been deprecated in 5.0 and can be removed.

The same goes for the deprecated `score_mode` field in HasParentQueryBuilder,
the deprecated `like_text`, `ids` and `docs` parameter in the `more_like_this` query,
the deprecated query name in the short version of the `regexp` query, and several
deprecated alternative field names in other query builders.
2017-09-20 16:22:21 +02:00
Christoph Büscher 3d67915ed5 #26720: Set the correct bwc version after backport to 6.0 2017-09-20 16:14:11 +02:00
Christoph Büscher 22e200e79a Remove deprecated type and slop field in MatchQueryBuilder (#26720)
The `type` field has been deprecated in 5.0 and can be removed. It has been
replaced by using the MatchPhraseQueryBuilder or the
MatchPhrasePrefixQueryBuilder. The `slop` field has also been deprecated and can
be removed, the phrase and phrase prefix query builders still provide this
parameter.
2017-09-20 14:24:30 +02:00
Yannick Welsch 5f407062ad Refactoring of Gateway*** classes (#26706)
- Removes mutual dependency between GatewayMetaState and TransportNodesListGatewayMetaState
- Deguices MetaDataIndexUpgradeService
- Deguices GatewayMetaState
- Makes Gateway the master-level component that is only responsible for coordinating the state recovery
2017-09-20 12:51:58 +02:00
Tanguy Leroux b3819e7f30 Make RestHighLevelClient's Request class public (#26627)
Request class is currently package protected, making it difficult for
the users to extend the RestHighLevelClient and to use its protected
methods to execute requests. This commit makes the Request class public
and changes few methods of RestHighLevelClient to be protected.
2017-09-20 11:36:10 +02:00
Yannick Welsch ff1e26276d Deguice ActionFilter (#26691)
Allows to instantiate TransportAction instances without Guice.
2017-09-20 10:30:21 +02:00
Martijn van Groningen 61849a1150
aggs: Allow aggregation sorting via nested aggregation.
The nested aggregator now buffers all bucket ords per parent document and
emits all bucket ords for a parent document's nested document once. This way
the nested documents document DocIdSetIterator gets used once per bucket
instead of wrapping the nested aggregator inside a multi bucket aggregator,
which was the current solution upto now. This allows sorting by buckets
under a nested bucket.

Closes #16838
2017-09-20 07:44:53 +02:00
Ryan Ernst a1c766c75c Build: Set bwc builds to always set snapshot (#26704)
This commit enforces bwc builds always generate snapshot versions, even
when testing release versions in CI.

closes #26702
2017-09-19 17:41:51 -07:00
Ryan Ernst bebff47b5b File Discovery: Remove fallback with zen discovery (#26667)
When adding file based discovery, we added a fallback when the discovery
type was set to zen (the default, so everyone got this warning). This
commit removes the fallback for 6.0. Setting file discovery should now
happen explicitly through the hosts_provider setting.

closes #26661
2017-09-19 16:32:34 -07:00
Jason Tedor 581a873124 Remove assertion from checkpoint tracker invariants
This assertion is wrong because the global checkpoint on a promoted
primary can be lagging the replicas until it catches up after through
resyncs, ongoing indexing operations and removing the old primary from
the in-sync set.
2017-09-19 17:52:41 -04:00
Igor Motov 5090260119 Upgrade API: fix excessive logging and unnecessary template updates (#26698)
TemplateUpgradeService might get stuck in repeatedly upgrading templates after upgrade to 5.6.0. This is caused by shuffling mappings definition in the template during template serialization. This commit makes the template serialization consistent.

Closes #26673
2017-09-19 16:32:17 -04:00
Boaz Leskes 04385a9ce9 Restoring from snapshot should force generation of a new history uuid (#26694)
Restoring a shard from snapshot throws the primary back in time violating assumptions and bringing the validity of global checkpoints in question. To avoid problems, we should make sure that a shard that was restored will never be the source of an ops based recovery to a shard that existed before the restore. To this end we have introduced the notion of `histroy_uuid` in #26577 and required that both source and target will have the same history to allow ops based recoveries. This PR make sure that a shard gets a new uuid after restore.

As suggested by @ywelsch , I derived the creation of a `history_uuid` from the `RecoverySource` of the shard. Store recovery will only generate a uuid if it doesn't already exist (we can make this stricter when we don't need to deal with 5.x indices). Peer recovery follows the same logic (note that this is different than the approach in #26557, I went this way as it means that shards always have a history uuid after being recovered on a 6.x node and will also mean that a rolling restart is enough for old indices to step over to the new seq no model). Local shards and snapshot force the generation of a new translog uuid.

Relates #10708
Closes #26544
2017-09-19 15:58:36 +02:00
Martijn van Groningen 332b4d12fa
test: Use a single primary shard so that the exception can caught in the same way 2017-09-19 15:14:24 +02:00