This adds HTTP pipelining support to netty. Previously pipelining was not
supported due to the asynchronous nature of elasticsearch. The first request
that was returned by Elasticsearch, was returned as first response,
regardless of the correct order.
The solution to this problem is to add a handler to the netty pipeline
that maintains an ordered list and thus orders the responses before
returning them to the client. This means, we will always have some state
on the server side and also requires some memory in order to keep the
responses there.
Pipelining is enabled by default, but can be configured by setting the
http.pipelining property to true|false. In addition the maximum size of
the event queue can be configured.
The initial netty handler is copied from this repo
https://github.com/typesafehub/netty-http-pipeliningCloses#2665
Today any call to the current randomized context modifies the random
sequence such that cluster initialization is context dependent. If due
to an error for instance a static util method is used like `randomLong`
inside the TestCluster instead of the provided Random instance all
reproducibility guarantees are gone. This commit adds a safe mechanism
to initialize these clusters even if a static helper is used. All none
test scope clusters are now initialized in a private randomized context.
Using the Groovy jar with the indy (short for `invokedynamic`) classifier enables usage of the `invokedynamic` instruction available in Java 7+. Due to buggy JVMs, it should only be used with Java 7u60 or later.
This commit makes the lookup structures that are used for mappings immutable.
When changes are required, a new instance is created while the current instance
is left unmodified. This is done efficiently thanks to a hash table
implementation based on a array hash trie, see
org.elasticsearch.common.collect.CopyOnWriteHashMap.
ManyMappingsBenchmark returns indexing times that are similar to the ones that
can be observed in current master.
Ultimately, I would like to see if we can make mappings completely immutable as
well and updated atomically. This is not trivial however, eg. because of dynamic
mappings. So here is a first baby step that should help move towards that
direction.
Close#7486
Lucene will soon release official 4.10.1, but by upgrading sooner we can 1) sidestep the false failures due to the 1.8.0_20 JVM hotspot bug (has caused a number of false failures in recent Jenkins tests), 2) make sure none of the Lucene changes in 4.10.1 are problematic.
Closes#7844
Before the heapdump was either written in a file with the
directory name if the heapdump path ended without / or
not written at all if the path ended with /
closes#7645
Lucene's experimental codecs (from the codecs module) do not provide
backwards compatibility and are free to change from release to
release. When they do change, they typically cannot in general read
older indices and the resulting exceptions look like index corruption.
So, we are removing built-in support for them to prevent applications
from choosing one and then seeing strange exceptions on upgrade.
Closes#7566Closes#7604
Merging the accumulated work from the feautre/improve_zen branch. Here are the highlights of the changes:
__Testing infra__
- Networking:
- all symmetric partitioning
- dropping packets
- hard disconnects
- Jepsen Tests
- Single node service disruptions:
- Long GC / Halt
- Slow cluster state updates
- Discovery settings
- Easy to setup unicast with partial host list
__Zen Discovery__
- Pinging after master loss (no local elects)
- Fixes the split brain issue: #2488
- Batching join requests
- More resilient joining process (wait on a publish from master)
Closes#7493
Per default the heap dump is written to target/JX/pidXYZ.hprof
In order to keep them when a new test is is started, they
should be written to log folder which is not cleared in a new
test run.
Heap dump location can be set with -Dtests.heapdump.path=/path/to/heapdump
closes#7452
internal cluster communication.
See CorruptedCompressorTests for details on how this bug can be hit.
This change also removes the ability to use the unsafe variant of
ChunkedEncoder, removing support for the compress.lzf.decoder setting.
CliTool is a base class for command-line interface tools (such as the plugin manager and potentially others). It supports the following:
- single or multi command tool
- help printing infrastructure (based on help files)
- consistent mechanism of parsing arguments (based on commons-cli lib)
- separation of argument parsing and command execution (for easier unit testing)
- terminal abstraction (will use System.console() when available)
This commit adds a profile that skips all validation ie.
- nocommit / tabs checking
- forbidden API checks
- license headers
It's not active by default but can easily be activated with
`mvn -Pdev` or in the `~/.m2/settings.xml`
for reference see:
http://maven.apache.org/guides/introduction/introduction-to-profiles.html
We parse the version that is shipped with the Lucene segments in order
to find the version of lucene that wrote a particular segment. Yet, some lucene
version ie:
* 4.3.1 (Elasticsearch 0.90.2)
* 4.5.1 (Elasticsearch 0.90.7)
* 3.6.1 (pre Elasticsearch 0.90.0)
wrote illegal strings containing the minor version which causes IAE exceptions
being thrown from lucenes parsing method.
Closes#7055
These are javascript expressions, which can only access numeric
fielddata, parameters, and _score. They can only be used for searches (not document updates).
closes#6818
In order to have access to all codecs and handlers by netty, they
need to be exposed during shading, otherwise only the classes, which
are used by the built are exposed.
`-Dtests.filter` allows to pass filter expressions to the elasticsearch
tests. This allows to filter test annotaged with TestGroup annotations
like @Slow, @Nightly, @Backwards, @Integration with a boolean expresssion like:
* to run only backwards tests run:
`mvn -Dtests.bwc.version=X.Y.Z -Dtests.filter="@backwards"`
* to run all integration tests but skip slow tests run:
`mvn -Dtests.filter="@integration and not @slow"
* to take defaults into account ie run all test as well as backwards:
`mvn -Dtests.filter="default and @backwards"
This feature is a more powerful alternative to flags like
`-Dtests.nighly=true|false` etc.
Closes#6703
Some IO api can return after writing & reading only a part of the requested data. On these rare occasions, we should call the methods again to read/write the rest of the data. This has cause rare translog corruption while writing huge documents on Windows.
Noteful parts of the commit:
- A new Channels class with utility methods for reading and writing to channels
- Writing or reading to channels is added to the forbidden API list
- Added locking to SimpleFsTranslogFile
- Removed FileChannelInputStream which was not used
Closes#6441 , #6576
Sandboxes the groovy scripting language with multiple configurable
whitelists:
`script.groovy.sandbox.receiver_whitelist`: comma-separated list of string
classes for objects that may have methods invoked.
`script.groovy.sandbox.package_whitelist`: comma-separated list of
packages under which new objects may be constructed.
`script.groovy.sandbox.class_whitelist` comma-separated list of classes
that are allowed to be constructed.
As well as a method blacklist:
`script.groovy.sandbox.method_blacklist`: comma-separated list of
methods that are never allowed to be invoked, regardless of target
object.
The sandbox can be entirely disabled by setting:
`script.groovy.sandbox.enabled: false`
This commit add a basic infrastructure as well as primitive tests
to ensure version backwards compatibility between the current
development trunk and an arbitrary previous version. The compatibility
tests are simple unit tests derived from a base class that starts
and manages nodes from a provided elasticsearch release package.
Use the following commandline executes all backwards compatiblity tests
in isolation:
```
mvn test -Dtests.bwc=true -Dtests.bwc.version=1.2.1 -Dtests.class=org.elasticsearch.bwcompat.*
```
These tests run basic checks like rolling upgrades and
routing/searching/get etc. against the specified version. The version
must be present in the `./backwards` folder as
`./backwards/elasticsearch-x.y.z`
Alexander Reelsen
Michael McCandless
Simon Willnauer
Chris Earle
Lukasz Dywicki
Shay Banon
Adrien Grand
Ryan Ernst
Britta Weber
Robert Muir
Boaz Leskes
Uwe Schindler
uboness
Lee Hinman