test no longer depends on changing static non-final non-volatile variables used by multiple threads
test also no longer depends on arbitrary sleep calls, instead threads await/poll on concurrent signaling objects/queues
Prior to this commit, ReplicationHandler had a few inconsistencies
in how it reported errors:
- Sometimes the 'status' field was used as an enum (e.g. 'success'
v. 'failure'. Elsewhere it is used to hold error messages.
- Sometimes an explicit 'message' field was used, but often not.
- Sometimes a stack trace was provided in place of an error message.
This commit tweaks the various error cases in ReplicationHandler to
report errors consistently. 'status' is always an enum-type value. A
'message' field is provided for all errors, with an optional 'exception'
field.
This commit replaces the `try { doX(); fail(); }` pattern with the
`expectThrows` test helper, which was created for this purpose.
This commit makes these changes in the core package:
`o.a.solr.core`.
SOLR-12804: Remove static modifier from Overseer queue access.
SOLR-12896: Introduce more checks for shutdown and closed to improve clean close and shutdown. (Partial)
SOLR-12897: Introduce AlreadyClosedException to clean up silly close / shutdown logging. (Partial)
SOLR-12898: Replace cluster state polling with ZkStateReader#waitFor. (Partial)
SOLR-12923: The new AutoScaling tests are way too flaky and need special attention. (Partial)
SOLR-12932: ant test (without badapples=false) should pass easily for developers. (Partial)
SOLR-12933: Fix SolrCloud distributed commit.
MemoryIndex: compute/cache up-front
Solr Collapse/Expand with top_fc: compute/cache up-front
Json Facets numerics / hash DV: use the cached fieldInfos on SolrIndexSearcher
SolrIndexSearcher: move the cached FieldInfos to SlowCompositeReaderWrapper
This commit replaces the `try { doX(); fail(); }` pattern with the
`expectThrows` test helper, which was created for this purpose.
This commit makes these changes in the core package: `o.a.solr.search`.
Closes#464
This should prevent some non-reproducible failures where (i think) the problem was inconsistnet background merges between collections being compared, resulting in facet counts (of 0) for terms that weren't in equivilent (merged) indexes.
since the purpose of the test has nothing to do with mincount, force facet.mincount=0 and be more exact about how many terms we expect (in case the root problem lies somewhere else, be more assertive about finding it)
Highlighter doesn't support ToParent and ToChildBlockJoinQuery out of the
box anymore. In oder to highlight on Block-Join Queries a custom WeightedSpanTermExtractor
should be used.
This fixes the CollectionsAPIDistributedZkTest.testCoresAreDistributedAcrossNodes test failures. Due to the various tests for exceptional conditions, there were times where the session was not released causing stale data to remain in the policy session cache.
This test too makes assumptions about how replicas are placed. In the legacy assignment strategy, the replica of a given collection are spread equally across all nodes but with the new policy based strategy, all cores across collections are spread out. Therefore the assumptions in this test were wrong. I've changed this test to use the legacy assignment policy because testing the autoAddReplicas feature doesn't have to depend on new replica assignment strategies. This change also fixes a bug in Assign which used "collection" key instead of "cluster" to figure out which strategy to use.
Previously, the maxShardsPerNode parameter was not allowed on collections when autoscaling policy was configured. Also if an autoscaling policy was configured then the default was to set an unlimited maxShardsPerNode automatically. Now the maxShardsPerNode parameter is always allowed during collection creation and maxShardsPerNode should be set correctly (if required) regardless of whether autoscaling policies are in effect or not. The default value of maxShardsPerNode continues to be 1 as before. It can be set to -1 during collection creation to fall back to the old behavior of unlimited maxShardsPerNode when using autoscaling policy. This patch also fixes PolicyHelper to find the free disk space requirements of a new replica from the leader only if said leader node is alive.
Wrapping is now a bit more lightweight: Does not create FieldInfo for fields that
can't be uninverted (saves mem) and can avoid wrapping the reader altogether if there's nothing to uninvert.
IndexSchema.getUninversionMap refactored to getUninversionMapper and no longer merges FieldInfos.
The cluster wide defaults structure has changed from {collectionDefaults: {nrtReplicas : 2}} to {defaults : {collection : {nrtReplicas : 2}}}. The old format continues to be supported and can be read from ZK as well as written using the V2 set-obj-property syntax but it is deprecated and will be removed in Solr 9. We recommend that users change their API calls to use the new format going forward.
This commit deprecates the min_rf parameter. Solr now always includes the achieved replication
factor in the update requests (as if min_rf was always specified). Also, reverts the changes
introduced in SOLR-8034, replicas that don't ack an update will have to recover to prevent
inconsistent shards.
* added "ignored_*" to the default configSet
* Updated Ref Guide info on Solr Cell to demonstrate usage without using the techproducts configSet
Closes#438
Now, assignment is done with the help of a builder class instead of calling a method with large number of arguments. The number of special cases that had to be handled have been cut down as well.
The API now supports 'nrtReplicas', 'tlogReplicas', 'pullReplicas' parameters as well 'createNodeSet' parameter. As part of this change, the CREATESHARD API now delegates placing replicas entirely to the ADDREPLICA command and uses the new parameters to add all the replicas in one API call.
When retrying internal update requests, backoff only once for the full batch of retries
instead of for every request.
Make backoff linear with the number of retries
Simplified test utility TrackingUpdateProcessorFactory.
Reverted some attempts the TRA used to make in avoiding overseer communication (too complicated).
Closes#433
Cluster properties restriction of known keys only is relaxed, and now unknown properties starting with "ext."
will be allowed. This allows custom to plugins set their own cluster properties.
A few tests have failed on jenkins where the very first call to Solr to set the policy fails because no live nodes were found. This commit adds a 30 second (max) wait for nodes to be registered live in ZK before attempting to run any test.
This commit adds support for preferredOperation configuration for NodeLostTrigger. The ComputePlanAction now creates DeleteNodeSuggester for each lost node serially when preferredOperation=deletenode. A new section for node lost trigger with exampls is added to the ref guide.
This commit adds support for preferredOperation configuration parameter which defaults to movereplica. Changes ComputePlanAction to add all (collection,shard) pair as hints to AddReplicaSuggester when addreplica is selected as the preferred operation.
This is due to a change in behavior made in this issue where collection metadata creation was moved before attempting to use policy to place replicas. This change reverts the behavior by explicitly deleting the metadata when an AssignmentException occurs. Since the policy depends on collection metadata such as WITH_COLLECTION and COLOCATED_WITH properties, moving metadata creation after policy use would have resulted in passing these properties through the entire call stack quite invasively. Therefore the current approach is adopted.
Fixed SolrDocument's confusion of field-attached child documents in addField()
Fixed AtomicUpdateDocumentMerger's confusion of field-attached child documents in isAtomicUpdate()
Extracted a new method to ensure a given number of jettys are always running in the test setup. Also, delete all collections created by other test methods.
While the startJettySolrRunner adds the given jetty instance to the internal list of jetty instances, the stopJettySolrRunner method does not remove the given instance from the list. This leads to inconsistencies such as stopped jettys retained in the internal list and duplicate (stopped) jettys. This commit also fixes TestCollectionsAPIViaSolrCloudCluster to deal with this change.
Sometimes deletion of the withCollection succeeds even though the linked collection exists, if the delete request is sent immediately after the creation of the linked collection. This is because of a race between the delete request and the zookeeper update made by create collection such that the COLOCATED_WITH property is not visible to the overseer collection processor during the delete request. Therefore, we now wait until the property is visible before returning a success to the user. The test also adds more asserts so that a similar race will fail the test immediately with a proper assert message.
The testSplitAfterSplit and testSplitAfterSplit2 tests only differ in the kind of fault injection being used. The rest of the test code is exactly the same so the common code is extracted into a new method called splitAfterSplit.
This commit replaces the `try { doX(); fail(); }` pattern with the
`expectThrows` test helper, which was created for this purpose. This
commit makes these changes in the core package: `o.a.solr.cloud`.
Closes#425
When waiting for follower replicas to be in sync with the leader, TestInjection.waitForInSyncWithLeader
now keeps waiting even if it gets an exception while trying to get leader's replication details. The
method will only exit when it exhausts the number of retries or succeeds.
A collection may be co-located with another collection during collection creation time by specifying a
'withCollection' parameter. It can also be co-located afterwards by using the modify collection API.
The co-location guarantee is enforced regardless of future cluster operations whether they are invoked
manually via the Collection API or automatically by the Autoscaling framework.
Squashed commit of the following:
commit 3827703b38c598f1247c90ab57d3d640ab3a9e21
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date: Sat Jul 28 11:54:10 2018 +0530
SOLR-11990: Added change log entry
commit 7977222e07ba47274062cb8d8a69e7956d644000
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date: Sat Jul 28 11:52:17 2018 +0530
SOLR-11990: Added change log entry
commit 1857075fdb9d535b6149ad4369fed8b64b0c01f6
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date: Sat Jul 28 11:49:51 2018 +0530
SOLR-11990: Added note about co-location guarantees being one way only
commit 8557cbc8a511f21d1fcad99e11ea9d2104d0bef4
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date: Sat Jul 28 10:43:37 2018 +0530
SOLR-11990: Remove unused import
commit 864b013fd744edca9b6b84a8a7573fab3c5310d5
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date: Sat Jul 28 10:21:59 2018 +0530
SOLR-11990: Fixing compilation issues after merging master
commit dd840a2f7e765ee96c899d4d9ea89b6b67c5ae62
Merge: bb4ffb3 828d281
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date: Sat Jul 28 10:03:50 2018 +0530
Merge branch 'master' into jira/solr-11990
# Conflicts:
# solr/solr-ref-guide/src/collections-api.adoc
# solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Clause.java
# solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Suggestion.java
commit bb4ffb32c4960a2809ac8927e214e1e012204a73
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date: Fri Jul 27 14:09:44 2018 +0530
SOLR-11990: Ensure that the suggestion are validated by the policy engine otherwise move to the next candidate replica or the next candidate node
commit a97d45b22f9c232e939f979502c761001be9ae24
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date: Fri Jul 27 13:22:10 2018 +0530
SOLR-11990: Autoscaling suggestions for withCollection violations should prefer moving replicas before adding replicas
commit 7b5a84338dfe7335599a5e96aff2d26cb4eeaac6
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date: Fri Jul 27 12:22:45 2018 +0530
SOLR-11990: Fix statement about the behavior of the modify collection API when modifying the withCollection parameter
commit 63aec4fe0de7025c16b6ebc47dad1004531ecee1
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date: Thu Jul 26 07:29:07 2018 +0530
SOLR-11990: Added new page to the reference guide describing how to colocate collections together including guarantees and limitations
commit 6bfcd0786bb30353de9c26a01ec97ce3191b58f8
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date: Wed Jul 25 21:42:25 2018 +0530
SOLR-11990: Added another test which creates two collections which are colocated with two different collections and ensures that create collection and add replica operations work correctly
commit 4cead778f0044b6fb4012b085abf7b60350f495b
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date: Wed Jul 25 21:07:47 2018 +0530
SOLR-11990: Stop or start jettys in test setup to ensure that we always have exactly 2 replicas running before a test starts
commit 70dbfd042c2164fcd76d406eeab1518e4d3147fb
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date: Wed Jul 25 19:19:07 2018 +0530
SOLR-11990: Added description of the new withCollection parameter in the reference guide
commit 9d8260852b9d667d4d8e026432fd7727b7789393
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date: Wed Jul 25 19:16:46 2018 +0530
SOLR-11990: Reset count down latch during test setup
commit ae508165571b1afde54337859b8d5fdbb1d67312
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date: Wed Jul 25 15:43:54 2018 +0530
SOLR-11990: Add support for withCollection in simulated create collection API
commit 84f026b8c4cc25edb548430b8f5ad09d2486b3b5
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date: Tue Jul 24 17:21:33 2018 +0530
SOLR-11990: Ported the refactoring made in CreateCollectionCmd to the simulated version so that simulation tests are able to create collections correctly
commit defe111c9d31c8e4f0f00b4f2f3c875f5b2fa602
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date: Tue Jul 24 16:17:52 2018 +0530
SOLR-11990: Add missing javadoc for return statement
commit 8e47d5bc4545548c5441909c3fcc1a7901b38185
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date: Tue Jul 24 16:11:45 2018 +0530
SOLR-11990: Replace usage of forbidden Charsets with StandardCharsets class
commit 2d1b9eb25ea96a3a42c000ae654400ed44c17554
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date: Tue Jul 24 16:07:36 2018 +0530
SOLR-11990: Extract ConditionType to an interface VarType along with a WithCollectionVarType implementation
commit 1de2a4f52a59afca28de75bfa5156a3d6567a4f5
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date: Tue Jul 24 12:53:26 2018 +0530
SOLR-11990: Pass strict-ness parameter to the ConditionType so that WITH_COLLECTION can choose not to project add replica in strict mode.
This ensures that add replica or move replica suggesters always choose nodes that already have withCollection replicas first unless there are violations in doing so. Only if the first pass fails to find a suitable replica, do we go to the other nodes in the cluster. This also removes the need for the majority of changes in AddReplicaSuggester and so they've been reverted.
commit 0d616ed9e9bad791548c87086cba7760d724350d
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date: Tue Jul 24 11:36:34 2018 +0530
SOLR-11990: Minor changes to formatting and code comments
commit 1228538f934f35f15797d89c2c66f2deb9cddd8c
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date: Mon Jul 23 14:26:19 2018 +0530
SOLR-11990: Added a test which simulates a lost node and asserts that move replica suggester moves the replica on the lost node to a node already having the withCollection present
commit 582f1fd98de93ab73c74a1f623749dd031beb381
Author: Noble Paul <noble@apache.org>
Date: Mon Jul 23 18:35:22 2018 +1000
SOLR-11990: NPE removing unnecessary System.out.println
commit 501bc6c1d066321b344bbb8b1de3c2ead52f8c49
Author: Noble Paul <noble@apache.org>
Date: Mon Jul 23 18:31:07 2018 +1000
SOLR-11990: NPE during class init
commit acbf4a69321e16cff11cc7cf0a1f076fd9ac0037
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date: Mon Jul 23 13:55:30 2018 +0530
SOLR-11990: Added asserts on the nodes that should be selected by the add replica suggester
commit 4824933fd6eb7d1773acbff1a1a0c5e670226e0b
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date: Fri Jul 20 14:30:52 2018 +0530
SOLR-11990: Added WITH_COLLECTION to global tags. Fixed implementation of addViolatingReplicas and getSuggestions in the clause impl. Added more asserts in testWithCollectionSuggestions.
commit dbadb33211c190026e08d8e3ea587b6f8df8720b
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date: Fri Jul 20 13:44:36 2018 +0530
SOLR-11990: Added support for comparing violations, generating suggestions and adding violating replicas
commit ada1f17d5c93a4186260473e4822d2bee1da0e16
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date: Wed Jul 18 19:14:56 2018 +0530
SOLR-11990: Fix mock node state provider in TestPolicy to use the right cluster state. Added nocommits to ensure that we return the right suggestions for this feature.
commit ef2d61812e0d96eb2275b3411906d9de57ab835e
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date: Wed Jul 18 18:39:51 2018 +0530
SOLR-11990: Add missing node in nodeValues configuration
commit 34841fc01fea4a9f1e6a9f64050e576f2247a72b
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date: Wed Jul 18 16:32:57 2018 +0530
SOLR-11990: Make it possible to co-locate replicas of multiple collections together in a node
Chris Hostetter
markrmiller
Cao Manh Dat
Noble Paul
Jason Gerlowski
Erick Erickson
Gus Heck
markrmiller
javanna
David Smiley
Moshe
Mikhail Khludnev
Alan Woodward
Christine Poerschke
Jim Ferenczi
Christophe Bismuth
Adrien Grand
Tim Underwood
Alexandre Rafalovitch
Steve Rowe
Andrzej Bialecki
Tommaso Teofili
Varun Thacker
Anshum Gupta
Simon Willnauer
Shalin Shekhar Mangar
Nicholas Knize
Tomas Fernandez Lobbe
Charles Sanders
Peter Somogyi
Joel Bernstein
Erick
Dawid Weiss
Anshum Gupta
Jan Høydahl
Bar Rotstein
Uwe Schindler
Erick
Jason Gerlowski