This option allows to force the xcontent type to use to store the `_source`
document. The default is to use the same format as the input format.
This commit makes this option ignored for 2.x indices and rejected for 3.0
indices.
This commit removes and now forbids all uses of
Collections#shuffle(List) and Random#<init>() across the codebase. The
rationale for removing and forbidding these methods is to increase test
reproducibility. As these methods use non-reproducible seeds, production
code and tests that rely on these methods contribute to
non-reproducbility of tests.
Instead of Collections#shuffle(List) the method
Collections#shuffle(List, Random) can be used. All that is required then
is a reproducible source of randomness. Consequently, the utility class
Randomness has been added to assist in creating reproducible sources of
randomness.
Instead of Random#<init>(), Random#<init>(long) with a reproducible seed
or the aforementioned Randomess class can be used.
Closes#15287
In commit fafeb3a, we've refactored REST response handling logic
and returned HTTP status names instead of HTTP status codes for
bulk item responses. With this commit we restore the original
behavior.
Checked with @bleskes.
This method currently allows to write arbitrary bytes in an xcontent stream.
I changed it so that it can only write data to the same stream as the xcontent
(the bos parameter is removed) and that it yells at you if you try to write
raw bytes that can't be recognized as xcontent. Also the logic to copy the
structure instead of appending the bytes directly if the source and target
are of a different xcontent type have been moved to the low-level
XContentGenerator.
Tons of ancient "benchmarks" exist in elasticsearch. These are main
methods that do some kind of construction of ES classes and time various
things. The problem with these is they are not maintained, and not run.
Refactorings that touch anything that is common in these classes is very
painful. Going through these, almost all would simply not work in 2.x
without modifications (because they do not set path.home).
This change removes the entire benchmark package. If someone needs to
run a benchmark like this, they can look at history for examples if
necessary (although these examples are often not realistic and should
just start real elasticsearch processes in a shell script). Longer term,
we should make this easier to do by having the build support adding real
benchmarks which can be run in jenkins (so we know they actually run,
instead of doing refactorings with pure guesswork as to whether the
benchmark would run correctly).
we are not ready for this yet:
```
if (shardRouting.primary() && shardRouting.isRelocationTarget() == false) {
throw new IllegalIndexShardStateException(shardId, state, "shard is not a replica");
}
```
IndexResponse, DeleteResponse and UpdateResponse share some logic. This can be unified to a single DocWriteResponse base class. On top, some replication actions are now not about write operations anymore. This commit renames ActionWriteResponse to ReplicationResponse
Last some toXContent is moved from the Rest layer to the actual response classes, for more code re-sharing.
Closes#15334
The test configuration with seed A23029712A7EFB34 overwhelmed the pool which is invoked
in TransportService#sendLocalRequest().
With this commit we reduce the maximum number of concurrent requests from 10 to 7 and
add the failure message to the test output on the failing assertion for easier analysis.
As a replacement use ExistsQueryBuilder inside a mustNot() clause.
So instead of using `new ExistsQueryBuilder(name)` now use:
`new BoolQueryBuilder().mustNot(new ExistsQueryBuilder(name))`.
Closes#14112
This makes some minor improvements (does not fix all problems!)
It reorders unicast disco in elasticsearch.yml to be right after the network host,
for better locality.
It removes the warning (unreleased) about publish addresses, lets try to really discourage setting
that unless you need to (behind a proxy server). Most people should be fine with `network.host`
Finally it reorganizes the network docs page a bit:
We add a table of 4 "basic" settings at the very beginning:
* network.host
* discovery.zen.ping.unicast.hosts
* http.port
* transport.tcp.port
The first two being the most important, which addresses to bind and talk to, and the other two
being the port numbers.
The rest of the stuff I tried to simplify and reorder under "advanced" headers.
This is just a quick stab, I still think we need more effort into this thing, but we gotta start somewhere.
The NodeBuilder is currently used to construct a Node. However, this is
really just yet-another-builder that wraps around a Settings.Builder
witha couple convenience methods. But there are very few uses of these
convenience methods. This change removes NodeBuilder, in favor of just
using the Node constructor.
Follow up for #15340
We test that bind with wilcard IP + fixed IP it raises an exception
We test binding multiple IPs
(cherry picked from commit 2cc5bb7)
The goal of this method is to know whether the xcontent impl knows how to
differenciate floats from doubles or longs from ints or if it's just guessing.
However, all implementations return true (which is correct for yaml and json,
but cbor and smile should be able to differenciate). I first tried to implement
this method correctly but it raised many issues because eg. most impls write a
long as an integer when it is small enough. So I suggest that we remove this
method and just treat cbor and smile like yaml and json, which is already what
is happening today anyway.
The HighlightBuilder should be able to procude a SeachContextHighlight
object which contains the merged global and field options, also
contains objects that can only be created on the index shard (like
the actual lucene Query object used during highlighting).
This is done by the build() method of the HighlighBuilder. Also
adding tests that make sure the produced SearchContextHighlighter is
similar to the one we would get when parsing the xContent directly
with the current HighlightParseElement.
RecoverySource uses the RateLimiter under a cancelable thread. The SimpleRateLimiter used in throws ThreadInterruptedException on interruption. We should treat it as InterruptedException
Currently almost all our fields accept the `analyzer` and `term_vector` settings
although they only make sense on text fields. This commit forbids those settings
on all fields but `string` and `_all` for indices created on or after version
2.2.0.
throw exception if a copy_to is within a multi field
Copy to within multi field is ignored from 2.0 on, see #10802.
Instead of just ignoring it, we should throw an exception if this
is found in the mapping when a mapping is added. For already
existing indices we should at least log a warning.
We remove the copy_to in any case.
related to #14946
This commit adds the infrastructure to make settings that are updateable
resetable and changes the application of updates to be transactional. This means
setting updates are either applied or not. If the application failes all values are rejected.
This initial commit converts all dynamic cluster settings to make use of the new infrastructure.
All cluster level dynamic settings are not resettable to their defaults or to the node level settings.
The infrastructure also allows to list default values and descriptions which is not fully implemented yet.
Values can be reset using a list of key or simple regular expressions. This has only been implemented on the java
layer yet. For instance to reset all recovery settings to their defaults a user can just specify `indices.recovery.*`.
This commit also adds strict settings validation, if a setting is unknown or if a setting can not be applied the entire
settings update request will fail.
this ensures the codebase URL matches the permission grant (see matching toRealPath in Security.java)
in the case of symlinks or other shenanigans.
this is best effort, if we really want to support symlinks in any way, we need
e.g. qa or vagrant tests that configure a bunch of symlinks for things and ensure that in jenkins.
this should be easier to do with gradle, as we can just create a symlink'd home if we want
Today we only handle correctly if the `ExecutionCancelledException` comes from the
local execution. Yet, this can also come from remove and should be handled identically.
This commit restores the chunk size of 512kb lost in a previous but unreleased
refactoring. At the same time it removes the configurability of:
* `indices.recovery.file_chunk_size` - now fixed to 512kb
* `indices.recovery.translog_ops` - removed without replacement
* `indices.recovery.translog_size` - now fixed to 512kb
* `indices.recovery.compress` - file chunks are not compressed due to lucene's compression but translog operations are.
The compress option is gone entirely and compression is used where it makes sense. On sending files of the index
we don't compress as we rely on the lucene compression for stored fields etc.
Relates to #15161
This commit cherry picks some infrastructure changes from the `feature/seq_no` branch to make merging from master easier.
More explicitly, IndexShard current have prepareIndex and prepareDelete methods that are called both on the primary as the replica, giving it a different origin parameter. Instead, this commits creates two explicit prepare*OnPrimary and prepare*OnReplica methods. This has the extra added value of not expecting the caller to use an Engine enum.
Also, the commit adds some code reuse between TransportIndexAction and TransportDeleteAction and their TransportShardBulkAction counter parts.
Closes#15282
The tribe node creates one local client node for each cluster it
connects to. Refactorings in #13383 broke this so that each local client
node now tries to load the full elasticsearch.yml that the real tribe
node uses.
This change fixes the problem by adding a TribeClientNode which is a
subclass of Node. The Environment the node uses is now passed in (in
place of Settings), and the TribeClientNode simply does not use
InternalSettingsPreparer.prepareEnvironment.
The tests around tribe nodes are not great. The existing tests pass, but
I also manually tested by creating 2 local clusters, and configuring and
starting a tribe node. With this I was able to see in the logs the tribe
node connecting to each cluster.
closes#13383
I don't recall of this property of any of our field mappers and it's not in our
docs so I suspect it's very old. The removal of this property will not fail
version upgrades since none of the field mappers use it in toXContent.
This commit removes some unneeded null checks from
IndexingMemoryController that were left over from the work in #15251,
and simplifies the try-catch block in
IndexingMemoryController#updateShardBuffers.
For the search refactoring the HighlightBuilder needs a way to
create new instances by parsing xContent. For bwc this PR start
by moving over and slightly modifying the parsing from
HighlighterParseElement and keeps parsing for top level highlighter
and field options separate. Also adding tests for roundtrip
of random builder (rendering it to xContent and parsing it and
making sure the original builder properties are preserved)
Since 2.2 we run all scripts with minimal privileges, similar to applets in your browser.
The problem is, they have unrestricted access to other things they can muck with (ES, JDK, whatever).
So they can still easily do tons of bad things
This PR restricts what classes scripts can load via the classloader mechanism, to make life more difficult.
The "standard" list was populated from the old list used for the groovy sandbox: though
a few more were needed for tests to pass (java.lang.String, java.util.Iterator, nothing scary there).
Additionally, each scripting engine typically needs permissions to some runtime stuff.
That is the downside of this "good old classloader" approach, but I like the transparency and simplicity,
and I don't want to waste my time with any feature provided by the engine itself for this, I don't trust them.
This is not perfect and the engines are not perfect but you gotta start somewhere. For expert users that
need to tweak the permissions, we already support that via the standard java security configuration files, the
specification is simple, supports wildcards, etc (though we do not use them ourselves).
This commit simplifies shard inactive debug logging to only log when the
physical shard is marked as inactive. This eliminates duplicate logging
that existed in IndexShard#checkIdle and
IndexingMemoryController#checkIdle, and eliminates excessive logging
that was occurring when the shard was already inactive as a result of
the work in #15252.
Currently, when a user tries to install an old plugin (pre 2.x) on a 2.x
node, the error message is cryptic (just printing the file path that was
missing, when looking for the descriptor). This improves the message to
be more explicit that the descriptor is missing, and suggests the
problem might be the plugin was built before 2.0.
closes#15197
This commit addresses some issues that arose during the review of #14899
but were lost during squash while integrating into master.
- the number of test threads is dropped to at most eight
- a local variable is renamed for clarity
- task priorities are randomized
This commit fixes a test bug in
ClusterService#testClusterStateBatchedUpdates. In particular, in the
case that an executor did not receive a task assignment from the random
assignments, it would not have an entry in the map of executors to
counts of assigned tasks. The fix is to just check if each executor has
an entry in the counts map.
This commit modifies IndexingMemoryController to be stateless. Rather
than statefully tracking the indexing status of shards,
IndexingMemoryController can grab all available shards, check their idle
state, and then resize the buffers based on the number of and which
shards are not idle.
The driver for this change is a performance regression that can arise in
some scenarios after #13918. One scenario under which this performance
regression can arise is if an index is deleted and then created
again. Because IndexingMemoryController was previously statefully
tracking the state of shards via a map of ShardIds, the new shards with
the same ShardIds as previously existing shards would not be detected
and therefore their version maps would never be resized from the
defaults. This led to an explosion in the number of merges causing a
degradation in performance.
Closes#15225
Today we only check mapping compatibility when adding mappers to the
lookup structure. However, at this stage, the mapping has already been merged
partially, so we can leave mappings in a bad state. This commit removes the
compatibility check from Mapper.merge entirely and performs it _before_ we
call Mapper.merge.
One minor regression is that the exception messages don't group together errors
that come from MappedFieldType.checkCompatibility and Mapper.merge. Since we
run the former before the latter, Mapper.merge won't even have a chance to let
the user know about conflicts if conflicts were discovered by
MappedFieldType.checkCompatibility.
Close#15049
The `translated` flag makes LineStringBuilder stateful and gets set
to true under certain conditions when building a Shape or Geometry
from the ShapeBuilder. This makes building operations not be idempotent,
so calling build() more than once on a LineStringBuilder might change the
builder itself. This PR fixes this by replacing the instance variable by
a local `translated` flag that is only updated internally during the
building process and created again on any subsequent calls to build()
or buildGeometry().
Failures to merge a mapping can either come as a MergeMappingException if they
come from Mapper.merge or as an IllegalArgumentException if they come from
FieldTypeLookup.checkCompatibility. I think we should settle on one: this pull
request replaces all usage of MergeMappingException with
IllegalArgumentException.
The ttl could be specified as a time value only via the REST layer. That is now possible via java api too, either as a string or as a proper TimeValue. The internal format in IndexRequest becomes now TimeValue, which will then still converted to a long before storing the document.
Closes#15047
- Supports ImmutableOpenIntMap besides java.util.Map and ImmutableOpenMap
- Map keys can be any value (not only String)
- Map values do not have to implement Diffable interface. In that case custom value serializer needs to be provided.
Several settings have been deprecated or are replaced with new settings after refactorings
in version 1.x. This commit removes the support for these settings.
The settings are:
* `index.shard.recovery.translog_size`
* `index.shard.recovery.translog_ops`
* `index.shard.recovery.file_chunk_size`
* `index.shard.recovery.concurrent_streams`
* `index.shard.recovery.concurrent_small_file_streams`
* `indices.recovery.max_size_per_sec`
When not in debug mode, we currently only print the message of an
exception. However, this is not usually useful without knowing what the
exception type was. This change makes cli tools use toString() on the
exception so we get the type + message.
This commit wraps the trace logging statements in
TransportBroadcastByNodeAction in trace enabled checks to avoid
unnecessarily allocating objects.
The most egregious offenders were the two trace logging statements in
BroadcastByNodeTransportRequestHandler#onShardOperation. Aside from the
usual object allocations that occur when invoking ESLogger#trace (the
allocated object array for the varargs Object... parameter), these two
logging statements were invoking ShardRouting#shortSummary generating a
bunch of char arrays and Strings (from the StringBuilder, and so a bunch
of array copies as well). In a scenario where there are a lot of shards
and this method is being invoked frequently (e.g., constantly hitting
the _stats endpoint), these two unprotected trace logging statements
were generating a lot of unnecessary allocations.
This commit modifies the handling of shard started cluster state updates
to use the general cluster state batching mechanism. An advantage of
this approach is we now get correct per-listener notification on
failures.
This commit removes a simple early-out check in
MetaDataMappingService#executeRefresh. The early-out is unnecessary
because the cluster state task execution framework will not invoke
ClusterStateTaskExecutor#execute if the list of tasks is empty.
This commit updates a stale Javadoc on
MetaDataMappingService#executeRefresh. Previously this method handled
refresh and update tasks. Update tasks have been removed and the method
was renamed, but the Javadoc was not updated to reflect this.
This is due to the fact that the query cache will still call the
onDocIdSetEviction callback in this case but with a number of entries equal to
zero.
Close#15043
This commit removes and now forbids all uses of the type-unsafe empty
Collections fields Collections#EMPTY_LIST, Collections#EMPTY_MAP, and
Collections#EMPTY_SET. The type-safe methods Collections#emptyList,
Collections#emptyMap, and Collections#emptySet should be used instead.
Today we try to have type-level granularity when dealing with mappings. This
does not play well with the cross-type validations that we are adding. For
instance we prevent the `_parent` field to point to an existing type. This
validation would be skipped today in the case of dedicated master nodes, since
those master nodes would only create the type that is being updated when
updating a mapping.
In 1.x it is possible via index templates to create an index with an alias with the same name as the index. The index name must match the index template and have an alias with the same name as the index being created.
This change attempts to simplify the gradle tasks for precommit. One
major part of that is using a "less groovy style", as well as being more
consistent about how tasks are created and where they are configured. It
also allows the things creating the tasks to set up inter task
dependencies, instead of assuming them (ie decoupling from tasks
eleswhere in the build).
This adds safety that you can't index into the `_default_` type (it was possible
before), and can't add default mappers to the field type lookups (was not
happening in tests but I think this is still a good check).
Also MapperService.types() now excludes `_default` so that eg. the `ids` query
does not try to search on this type anymore.
This commit modifies the handling of shard failure cluster state updates
to use the general cluster state batching mechanism. An advantage of
this approach is we now get correct per-listener notification on
failures.
This change pulls out the common fields that HighlighterBuilder shares with
its nested Field class into a new abstract CommonHighlighterOptions superclass
which also gets equals() and hashCode() method and methods to serialize the
common fields to a StreamOutput and read them from a stream.
Relates to #15044
Validation is not done as part of the distance setter method and tested in GeoDistanceQueryBuilderTests. Fixed GeoDistanceTests to adapt to the new validation.
Closes#15135
When creating an index on master for the purpose of updating mappings, a
mapping being updated could needlessly be merged multiple times. This
commit ensures that each mapping is merged at most once while preparing
to update mappings.
When creating an index on master for the purpose of updating mappings,
the default mapping could needlessly be added multiple times. This
commit ensures that the default mapping is added at most once while
preparing to update mappings.
This commit addresses an issues introduced in #14899 to apply mapping
updates in batches. The issue is that an existing mapping for a type
could be lost if that type came in a batch that already contained a
mapping update for another type on the same index. The underlying issue
was that the existing mapping would not be merged in because the merging
logic was only tripped once per index, rather than for all types seeing
updates for each index. Resolving this issue is simply a matter of
ensuring that all existing types seeing updates are merged in.
Closes#15129
The REST bulk API rejects use of `refresh` at the item level. But the Java API lets the user setting it.
We need to have the same behavior and don't let think the user he can define `refresh` per bulk item.
Note that the user can still define `refresh` on the bulk itself.
Also a user can create with Java API an IndexRequest without any source which is causing a NPE when evaluating the bulk item size.
Closes#7361.
Closes#15120.
I will followup with ITs and other modules. By fixing this, these tests become more reliable (will never sporatically
fail due to other stuff on your machine: ports are assigned by the OS), and it allows us to move forward with
gradle parallel builds, in my tests this is a nice speedup, but we can't do it until tests are cleaned up
This commit splits cluster state update tasks into roles. Those roles
are:
- task info
- task configuration
- task executor
- task listener
All tasks that have the same executor will be executed in batches. This
removes the need for local batching as was previously in
MetaDataMappingService.
Additionally, this commit reintroduces batching on mapping update calls.
Relates #13627
Do not to load fields from _source when using the `fields` option.
Non stored (non existing) fields are ignored by the fields visitor when using the `fields` option.
Fixes#10783
Support * wildcard to retrieve stored fields when using the `fields` option.
Supported pattern styles are "xxx*", "*xxx", "*xxx*" and "xxx*yyy".
Removed check that two query builder that are different according
to equals() have different hashCode since that is not required
by the contract of hashCode.
We used to check on several places if we are still open but non of these
places did the check under the lock which leaves a small window where we
potentially get closed but still access an already closed channel or another
IO resource.
This is a pretty trivial change that moves most of the monitor service related
object creation from guice into the monitor service. This is a babystep towards removing
guice on the node level as well. Instead of opening huge PRs I try to do this in baby-steps
that are easier to digest.
We handle AlreadyClosedExceptions gracefully wherever IndexShard / Engine
is used. In some cases, instead of throwing the appropriate exception we
bubble up ChannelClosedException instead which causes shard failures etc.
Today, it seems like that this can only happen if the engine is closed without
acquireing the lock which means that the shard has failed already so the impact is really
just a confusing log message. Yet, this change enforces throwing the right exception
if the translog is already closed.
Closes#14866
This moves the registration of field mappers from the index level to the node
level and also ensures that mappers coming from plugins are treated no
differently from core mappers.
This commit fixes some leniency in the parsing of CIDRs. The leniency
that existed before includes not validating the octet values nor
validating the network mask; in some cases issues with these values were
silently ignored. Parsing is now done according to the guidelines in RFC
4632, page 6.
Closes#14862
For example: if a node left the cluster and an async store fetch was triggered. In that time no shard is marked as delayed (and strictly speaking it's not yet delayed). This caused test for shard delays post node left to fail. see : http://build-us-00.elastic.co/job/es_core_master_windows-2012-r2/2074/testReport/
To fix this, the delay update is now done by the Allocation Service, based of a fixed time stamp that is determined at the beginning of the reroute.
Also, this commit fixes a bug where unassigned info instances were reused across shard routings, causing calculated delays to be leaked.
Closes#14890
This switches query parsing from manual field parsing to using ParseField.
Also adds unit tests for each query that check original json can be parsed
into query builders.
Relates to #8964
We recently refactored the queries to make them parsable on the
coordinating note and adding serialization and equals/hashCode
capability to them. So far ShapeBuilders nested inside queries
were still transported as a byte array that needs to be parsed
later on the shard receiving the query. To be able to also
serialize geo shapes this way, we also need to make all the
implementations of ShapeBuilder implement Writable.
This PR adds this to PointBuilder and also adds tests for
serialization, equality and hashCode.
The work for #10708 requires tighter integration with the current shard routing of a shard. As such, we need to make sure it is set before the IndexService exposes the shard to external operations.
Closes#14918
Index constraints should remove indices in the response if the field to evaluate if empty. Index constraints can't work with that and it is the same as if the field doesn't match.
Currently we use the "gradle project attachment plugin" to support
building elasticsearch as part of another project. However, this plugin
has a number of issues, a large part of which is requiring consistent
use of the projectsPrefix.
This change removes projectsPrefix, and adds support for a special
extra-plugins directory in the root of elasticsearch. Any projects
checked out within this directory will be automatically added to
elasticsearch.
* Forbid System.setProperties & co in forbidden APIs.
* Ban property write access at runtime with security manager.
Plugins that need to modify system properties will need to request permission in their plugin-security.policy
This commit adds an acquired flag to BulkProcessor#execute that is set
only after successful acquisition of a permit on the semaphore
there. This flag is used to ensure that we do not release a permit on
the semaphore when we did not obtain a permit on the semaphore.
Closes#14908
This makes AvgTests use a mock plugin engine. I also removed the
textScriptExplicit* methods for the base class since they only make sense for
a groovy script, not a mock script.
After the removal of some internal shape builders in #14482 the
BaseLineStringBuilder has only one implementation, the LineStringBuilder.
Same for the BasePolygonBuilder. This PR removes the abstract classes
and merges them with their concrete implementation to simplify the
inheritance hierarchy.
The Ring subclass is just a LineStringBuilder that has an additional
close() method and keeps a reference to a parent shape builder so
builders can be chained. This PR removes it and replaces it by
using LineStringBuilder instead. The close() method is moved there
and tests are adapted.
This is a first step in reducing the number of ShapeBuilders since
before we start making the remaining implement Writable for the
search request refactoring. This shape builder seems to have been
only used in tests, and those tests didn't do much to begin with,
so this removed them.
Relates to #14416
- moves calculation of the delay to a single place (ReplicaShardAllocator)
- reduces coupling between GatewayAllocator and RoutingService
- in master failover situations, elapsed delay time is forgotten
Closes#14808
At the time of geo_shape query conception, CONTAINS was not yet a supported spatial operation in Lucene. Since it is now available this commit adds ShapeRelation.CONTAINS to GeoShapeQuery. Randomized testing is included and documentation is updated.
This commit sets ReplicationRequest.internalShardId to null when the
stream indicates that no ShardId is present in the stream.
Additionally, the use of StreamOutput#writeOptionalStreamable is
changed to be explicit for clarity since the use of
StreamInput#readOptionalStreamable is not possible due to the
no-argument constructor on ShardId being private.
This commit replaces all occurrences of Thread.interrupted() with
Thread.currentThread().interrupt(). While the former checks and clears the current
thread's interrupt flag the latter sets it, which is actually intended.
Closes#14798
This commit adds a unit test for LinkedHashMap serialization that tests
that the method of serialization writes the entries in the LinkedHashMap
in iteration order and that the reconstructed LinkedHashMap preserves
that order. This test is randomized and tests iteration order is
preserved whether the LinkedHashMap is ordered by insertion order or
access order.
Closes#14743
This commit adds a timeout mechanism for sending shard failures. The
requesting thread can attach a listener to the timeout event so that
handling it is part of the event chain.
Relates #14252
This commit changes the signature of StreamInput#readOptionalStreamable
to accept a Supplier to create new streamables rather than requiring
callers to construct new instances. This has the advantage of avoiding
an allocation in cases when the stream indicates the resulting
streamable is null
If you build elasticsearch without a git repository it was creating a null
shortHash which was causing Elasticsearch not to be able to form transport
connections.
Closes#14748
This commit adds a method of encoding longs using a variable-length
representation. This encoding is an implementation of the zig-zag
encoding from protocol buffers. Numbers that have a small absolute value
will use a small number of bytes. This is achieved by zig-zagging
through the space of longs in order of increasing absolute value (0, -1,
1, -2, 2, …, Long.MAX_VALUE, Long.MIN_VALUE) -> (0, 1, 2, 3, 4, …, -2,
-1). The resulting values are then encoded as if they represent unsigned
numbers.
ClusterStatsIT#testClusterStatus() contained a race where the
test cluster might still be initializing while test already checks
for a green health status.
With this commit the test waits until the cluster status changed and
checks health afterwards.
Checked with @bleskes.
This issue occurs if the center latitude of the GeoPointDistance query is set to one of the poles. Since this issue is set to be fixed in LUCENE-6897 this commit temporarily limits the random latitudinal location to not include the poles.
_type should have got doc values with the change to default doc values.
However, due to how metadata fields have separate builders and special
constructors, it was not picking it up. This change updates the field
type for _type to have doc values.
closes#14781
If we run out of disk while recoverying the transaction log
we repeatedly fail since we expect the latest tranlog to be uncommitted.
This change adds 2 safety levels:
* uncommitted checkpoints are first written to a temp file and then atomically
renamed into a committed (recovered) checkpoint
* if the latest uncommitted checkpoints generation is already recovered it has to be
identical, if not the recovery fails
This allows to fail in between recovering the latest uncommitted checkpoint and moving
the checkpoint generation to N+1 which can for instance happen in a situation where
we can run out of disk. If we run out of disk while recovering the uncommitted checkpoint
either the temp file writing or the atomic rename will fail such that we never have a
half written or corrupted recovered checkpoint.
Close#14695
Currently the abstract ShapeBuilder class serves too many different
purposes, making it hard to refactor and maintain the code. In order
to reduce the size and responsibilities, this PR moved all the
static factory methods used as a shortcut to create new shape builders
out to a new ShapeBuilders class, similar to how QueryBuilders is
used already.
With this commit the cluster health status changes are logged
on INFO level. The change is only logged on master and actively
triggered in AllocationService in order to minimize the impact of
constantly reevaluating ClusterState in a ClusterStateListener
although we know that no health-relevant change happened.
Closes#11657
In AbstractQueryTestCase we randomly add the `_name` property to
some of the queries. While this generally works, there are exceptional
cases where we assign the same name to two queries in the setup which
leads to test failures later. This PR adds an increasing counter value
to the base tests that gets appended to all random query names to
avoid this name clashes.
ClusterRebalanceAllocationDecider did not take unassigned shards into account
that are temporarily marked as ingored. This can cause unexpected behavior
when gateway allocator is still fetching shards or has marked shareds as ignored
since their quorum is not met yet.
Closes#14670Closes#14678
Currently the next delay is calculated based on System.currentTimeMillis() but the actual shards to delay based on the last time the GatewayAllocator tried to assign/delay the shard.
This introduces an inconsistency for the case where shards should have been delay-allocated between the GatewayAllocator-based timestamp and System.currentTimeMillis().
Closes#14765
DateHistogramTests had some dependency on groovy scripts and
were moved to the lang-groovy module. This PR moves it back
and replaces use of groovy scripts by a mock script engine.
Removing three test cases that were testing doing some date
manipulation using script, since these are more groovy script
tests than testing the DateHistogram aggregation.
This commit changes TransportRequestOptions and TransportResponseOptions
to be immutable. This is to address an issue where the empty options
were being mutated permanently altering their state. Making these
objects immutable is just good, clean coding.
This fixes an issue where if the field for the aggregation was unmapped the extended bounds would get dropped and the resulting buckets would not cover the extended bounds requested.
Closes#14735
This fixes an issue where if the field for the aggregation was unmapped the extended bounds would get dropped and the resulting buckets would not cover the extended bounds requested.
Closes#14735
closes#14726
Squashed commit of the following:
commit 5b591e98570e3fa481b2816a44063b98bff36ddf
Author: Robert Muir <rmuir@apache.org>
Date: Fri Nov 13 00:54:08 2015 -0500
add assumption for self-signing in PluginManagerTests
commit ed11e5371b6f71591dc41c6f60d033502cfcf029
Author: Robert Muir <rmuir@apache.org>
Date: Fri Nov 13 00:20:59 2015 -0500
show error output from integ test startup
commit d8b187a10e95d89a0e775333dcbe1aaa903fb376
Author: Robert Muir <rmuir@apache.org>
Date: Thu Nov 12 22:14:11 2015 -0500
fix gradle check under jigsaw
This commit removes all noreleases and cuts over to Lucene 5.4 GeoPointField type. Included are randomized testing updates to unit and integration test suites for ensuring full backward compatability with existing geo_point indexes.
Just suck in the system policy, so its compatible with any version of java.
It means it also respects configuration (e.g. for monitoring agents)
Closes#14704
The disruption rules are changed to work on all transport addresses that are bound by a node (not only publish address).
This is important as UnicastZenPing creates fake DiscoveryNode instances which match one of the bound addresses and not necessarily the publish address.
Closes#14625Closes#14653
After a delayed reroute of a shard, RoutingService misses to schedule a new delayed reroute of other delayed shards.
Closes#14494Closes#14010Closes#14445
Transitive dependencies can be confusing and hard to deal with when
conflicts arise between them. This change removes transitive
dependencies from elasticsearch, and forces any dependency conflicts to
be resolved manually, instead of automatically by gradle.
closes#14627
This allows different circuit breakers to have different logging levels.
It's useful when diagnosing problems (say for instance with the
fielddata breaker) and not seeing the enormous amount of logging from
the request breaker.
The log messages use the breaker name for logging, so example logging
will look like:
```
[2015-11-10 09:51:52,993][TRACE][indices.breaker.fielddata] [fielddata] Adding [27b][body] to used bytes [new used: [27b], limit: 623326003 [594.4mb], estimate: 27 [27b]]
[2015-11-10 09:51:53,000][TRACE][indices.breaker.fielddata] [fielddata] Adjusted breaker by [453] bytes, now [480]
[2015-11-10 09:51:53,016][TRACE][indices.breaker.request ] [request] Adjusted breaker by [16440] bytes, now [16440]
[2015-11-10 09:51:53,018][TRACE][indices.breaker.request ] [request] Adjusted breaker by [-16440] bytes, now [0]
```
`AbstractLegacyBlobContainer` was kept for historical reasons (see #13434).
We can migrate Azure and S3 repositories to use the new methods added in #13434 so we can remove `AbstractLegacyBlobContainer` class.
We only passed the engine config since we had no chance to get the cache
etc. from the IndexSearcher. Now that we have these getters we can just pass the
searcher instead.
Some dependencies must be specified in a couple places in the build.
e.g. randomized runner is specified both in buildSrc (for the gradle
wrapper plugin), as well as in the test-framework.
This change creates buildSrc/versions.properties which acts similar to
the set of shared version properties we used to have in the maven parent
pom.
This commit adds some basic javadocs to IndexingStats.Stats.
A user on the forum asked for what getIndexingCurrent is so I figured
i can just fix it in the code.
This adds the `cluster.routing.allocation.total_shards_per_node`
setting, which limits the total number of shards across all indices on
each node. It defaults to -1 and can be dynamically configured.
Resolves#14456
GeoDistanceRangeQueryTest was using an inconsistent tolerance when validating from=0 with include_lower/upper set to false. This commit changes the assertion such that validation is consistent with builder logic.
- Same as for TransportInstanceSingleOperationAction and TransportReplicationAction, onClusterServiceClose consistently throws a NodeClosedException now.
- Added retry logic if master could not publish cluster state or stepped down before publishing (ZenDiscovery). The test IndexingMasterFailoverIT shows the issue.
- Simplified retry logic by moving bits from different places into shared retry method.
- Removed boolean flag retrying that aborted retrying after a single master node change (now we retry until timeout).
- Two existing predicates that deal with master node changes unified in a single predicate masterNodeChangedPredicate
Closes#14222
We used to test only json parsing as we relied on QueryBuilder#toString which uses the json format. This commit makes sure that we now output the randomly generated queries using a random format, and that we are always able to parse them correctly.
This revealed a couple of issues with binary objects that haven't been migrated yet to be structured Writeable objects. We used to keep them in the format they were sent while parsing, which led to problems when printing them out as we expected them to always be in json format. Also we can't compare different BytesReference objects that hold the same content but in different formats (unless we want to parse them as part of equal and hashcode, doesn't seem like a good idea) and verify that we have parsed the right objects if they can be different formats. The fix is to always keep binary objects in json format. Best fix would be not to have binary objects, which we'll get to once we are done with the search refactoring.
Closes#14415
Latest version of lucene deprecated Query#setBoost and Query#getBoost which made queries effectively immutable. Those methods need to be replaced with `BoostQuery` that wraps any query that needs boosting.
This commit replaces usages of setBoost with BoostQuery and adds it to forbidden-apis for prod code.
Usages of `getBoost` are only partially removed, as some will have to stay for backwards compatibility.
Closes#14264
If you run tests under a 32-bit jvm, you will get a test failure in IndexStoreTests,
the logic there is wrong in the case of 32-bit (its NIOFSDirectory on linux).
Also if mlockall fails, you'll see huge bogus values (because of use of `long` instead of `NativeLong`)
finally add seccomp support for 32 bit too, and clean up all its `long` usage as well.
run.sh and run.bat were calling out to the old maven build system.
This is no longer in place, so we've created new gradle tasks to
start an elasticsearch node from the current codebase.
fixed#14423
The completion suggester provides auto-complete/search-as-you-type functionality.
This is a navigational feature to guide users to relevant results as they are typing, improving search precision.
It is not meant for spell correction or did-you-mean functionality like the term or phrase suggesters.
The completions are indexed as a weighted FST (finite state transducer) to provide fast Top N prefix-based
searches suitable for serving relevant results as a user types.
closes#10746
Random code shouldn't be listening on sockets elsewhere.
Today its the wild west, but we only need to grant access to what the user configured.
This means e.g. multicast plugin has to declare its intentions in its security.policy
Closes#14549
Closes#14595
Squashed commit of the following:
commit d0b2b262e9dcdbc2aee163b9a84db082c8b5b96b
Author: Robert Muir <rmuir@apache.org>
Date: Fri Nov 6 22:36:54 2015 -0500
Switch to JarInputStream, to contain suppressforbidden. Also add a test that fails if path is not accessible (regardless of whether its a jar)
commit f99c1d240db23ceb2a06987b3bd69eae0229550b
Author: Robert Muir <rmuir@apache.org>
Date: Fri Nov 6 22:16:16 2015 -0500
remove leniency in i/o here
commit b160d4303ee81a8c9298729596ecbc893f5f8894
Author: Robert Muir <rmuir@apache.org>
Date: Fri Nov 6 21:58:21 2015 -0500
Fix Build to correctly treat URLs and to not leak a file handle
Adrien Grand
Jason Tedor
Jim Ferenczi
Daniel Mitterdorfer
Simon Willnauer
Simon Willnauer
Isabel Drost-Fromm
Yannick Welsch
Ryan Ernst
Robert Muir
Robert Muir
Boaz Leskes
Jun Ohtani
Christoph Büscher
Areek Zillur
Jack Conradson
David Pilato
Colin Goodheart-Smithe
Paulo Martins
Myll
Britta Weber
Jim Ferenczi
Michael McCandless
Lee Hinman
javanna
Martijn van Groningen
Britta Weber
Clinton Gormley
Isabel Drost-Fromm
Dominik Stadler
Camilo Díaz Repka
markharwood
William Bowling
Xu Zhang
Nicholas Knize
Nik Everett
Michael McCandless
Tanguy Leroux
andrejserafim