Reader attributes allows a per IndexReader configuration of codec internals.
For instance this allows a per reader configuration if FSTs are loaded into memory or are left
on disk.
should help from local tests/profile, but maybe it helps in more memory
constrained build servers, this and prior version both beasted success
10 rounds of 5 on a machine with lots of memory.
Before we can expose options to configure this postings format
on a per-reader basis we need to expose the option to load the terms
index FST off or on heap on the postings format. This already allows to
change the default in a per-field posting format if an expert user
wants to change the defaults. This essentially provides the ability to change
defaults globally while still involving some glue code.
The racent change introducing ByteArrayUtf8CharSequence altered the
NamedLists produced by atomic-update requests so that they include
instances of this class for requests coming in as javabin. This is a
problem for 'remove' atomic-updates, which need to be able to compare
these ByteArrayUtf8CharSequence instances with existing field values
represented as Strings. equals() would always return false, and
'remove' operations would have no effect.
This commit converts items as necessary to allow atomic-update
operations to work as expected.
The previous version of this test had a chicken/egg problem (needed to init the servers w/the whitelist but didn't know the port nums until after init)
that caused it to require 'restarting' the servers -- leading to the possibility of 'Address already in use' errors from jenkins machines if the OS reclaimed the port between the stop/start of the jetty instance
Prior to this commit, RuleBasedAuthorizationPlugin would check for the
predefined 'ALL' permission only when the endpoint being hit wasn't
associated with another predefined-permission.
This resulted in some very unintuitive behavior. For example, the
permission {name:all, role:admin} would correctly prevent a
role:foo user from accessing /admin/info/properties, but would allow
write access to /admin/authorization because of the SECURITY_EDIT
predefined perm associated with that endpoint.
This commit fixes this bug so that the 'all' permission is always
consulted whether or not the endpoint is associated with other predefined
permissions.
FilterDirectory.getPendingDeletions() did not delegate the call, which
resulted in a new IndexWriter on same directory not considering pending
delete files. This could in turn result in a FileAlreadyExistsException
when running windows.
Related JIRAs:
* SOLR-11010
* SOLR-11381
* SOLR-12040
* SOLR-13297
Changes:
* Consolidate hdfs configuration into HdfsTestUtil
* Ensure socketTimeout long enough for HDFS tests
* Ensure HdfsTestUtil.getClientConfiguration used in tests
* Replace deprecated HDFS calls
* Use try-with-resources to ensure closing of HDFS resources
Signed-off-by: Kevin Risden <krisden@apache.org>
also decrease the number of iters while increase the cluster shape wait time to reduce the risk of spurious failures on machines under heavy contention w/o making the the test any slower on average
There are 3 tightly related bug fixes in these changes:
1) ConcurrentModificationExceptions were being thrown by some SimClusterStateProvider methods when
creating collections/replicas due to the use of ArrayLists nodeReplicaMap. These ArrayLists were changed
to use synchronizedList wrappers.
2) The Exceptions from #1 were being swallowed/hidden by code using SimCloudManager.submit() w/o checking
the result of the resulting Future object. (As a result, tests waiting for a particular ClusterShape
would timeout regardless of how long they waited.) To protect against "silent" failures like this,
this SimCloudManager.submit() has been updated to wrap all input Callables such that any uncaught errors
will be logged and "counted." SimSolrCloudTestCase will ensure a suite level failure if any such failures
are counted.
3) The changes in #2 exposed additional concurrency problems with the Callables involved in leader election:
These would frequently throw IllegalStateExceptions due to assumptions about the state/existence of
replicas when the Callables were created vs when they were later run -- notably a Callable may have been
created that held a reference to a Slice, but by the time that Callable was run the collection (or a
node, etc...) refered to by that Slice may have been deleted. While fixing this, the leader election
logic was also cleaned up such that adding a replica only triggers leader election for that shard, not
every shard in the collection.
While auditing this code, cleanup was also done to ensure all usage of SimClusterStateProvider.lock was
also cleaned up to remove all risky points where an exception may have been possible after aquiring the
lock but before the try/finally that ensured it would be unlocked.
This commit adds an introspection API to Query, allowing users to traverse
the nested structure of a query and examine its leaves. It replaces the existing
`extractTerms` method on Weight, and alters some highlighting code to use
the new API
default inside of the 'expr' parameter, add InjectionDefense class
for safer handling of untrusted data in streaming expressions and add
-DStreamingExpressionMacros system property to revert to legacy behavior
This is a fairly old test that isn't particularlay valuable, all it really does is stress test indexing, but w/o actually validating the results.
In it's current state, it can suffer from exceeding file handle limits depending on merge settings, and isn't reliably reproducible due to it's
approach of "do (a variable) amount of work until a fixed amount of wall clock time has elapsed"
- added logging
- ensure start/finish trigger action counters are incremented before latches are released
- replace arbitrary sleep calls with a trigger listener countdown latch
- increase all await() times: This means that 'real' failures (which should be rare and hopefully
reproducible) will be 'slow', but the trade off will be less hard to reproduce 'false failures'
due to thread contention on slow or heavily loaded (ie: jenkins) machines
This means that 'real' failures (which should be rare and hopefully reproducile) will be 'slow', but the trade off will be less hard to reproduce 'false failures' due to thread contention on slow or heavily loaded (ie: jenkins) machines
This means that 'real' failures (which should be rare and hopefully reproducile) will be 'slow', but the trade off will be less hard to reproduce 'false failures' due to thread contention on slow or heavily loaded (ie: jenkins) machines
When performing an async CREATE requests, responses to internal requests are also included inside the "success" or "failed" elements of the general response. This will make the operation cleanup in case of failure, the same way as we do when the request is synchronous.
Solr 7.5 enabled autoscaling based replica placement by default but in the absence of default cluster policies, autoscaling can place more than 1 replica of the same shard on the same node. Also, the maxShardsPerNode and createNodeSet was not respected. Due to these reasons, this issue reverts the default replica placement policy to the 'legacy' assignment policy that was the default until Solr 7.4.
The use of closeQuietly should hopefully prevent failures closing one HttpClient from resulting in other client objects being leaked
The setup changes are unlikey to improve test reliability, but will hopefully help diagnose where/how NPEs are coming from that currently cause some suite failures.
* new NestPathField encapsulating details for how _nest_path_ is indexed
** tweaked the analysis to index 1 token instead of variable
* TokenizerChain has new CustomAnalyzer copy-constructor
A new config option, "tombstoneConfig" allows the DocBasedVersionConstraintsProcessor
to add extra fields to the tombstone generated when a document is deleted. This can
be useful when the schema has required fields.
This follows the spirit of the change Mark intended in his previous commit to this test, but his solution wasn't covering all cases on backcompat to branch_7x
(see also: SOLR-12313)