Commit Graph

183 Commits

Author SHA1 Message Date
Ryan Ernst a8017ff020 Tests: Move cluster dependencies from runner to cluster (#24142)
After splitting integ tests into cluster configuration and the test
runner task, we still have dependencies of the test runner added as deps
of the cluster. This commit adds dependencies directly to the cluster,
so that the runner can have other dependencies independent of what is
needed for the cluster.
2017-04-17 16:02:46 -07:00
Ryan Ernst cc1addeac2 Build: Find bwc version during build (#23801)
We currently have the last minor version of the previous major hardcoded
in tests like rolling upgrade. This change programatically finds this
during gradle initialization by parsing versions from Version.java.
2017-03-29 12:11:38 -07:00
Ryan Ernst 175bda64a0 Build: Rework integ test setup and shutdown to ensure stop runs when desired (#23304)
Gradle's finalizedBy on tasks only ensures one task runs after another,
but not immediately after. This is problematic for our integration tests
since it allows multiple project's integ test clusters to be
simultaneously. While this has not been a problem thus far (gradle 2.13
happened to keep the finalizedBy tasks close enough that no clusters
were running in parallel), with gradle 3.3 the task graph generation has
changed, and numerous clusters may be running simultaneously, causing
memory pressure, and thus generally slower tests, or even failure if the
system has a limited amount of memory (eg in a vagrant host).

This commit reworks how integ tests are configured. It adds an
`integTestCluster` extension to gradle which is equivalent to the current
`integTest.cluster` and moves the rest test runner task to
`integTestRunner`.  The `integTest` task is then just a dummy task,
which depends on the cluster runner task, as well as the cluster stop
task. This means running `integTest` in one project will both run the
rest tests, and shut down the cluster, before running `integTest` in
another project.
2017-02-22 12:43:15 -08:00
Jay Modi b234644035 Enforce Content-Type requirement on the rest layer and remove deprecated methods (#23146)
This commit enforces the requirement of Content-Type for the REST layer and removes the deprecated methods in transport
requests and their usages.

While doing this, it turns out that there are many places where *Entity classes are used from the apache http client
libraries and many of these usages did not specify the content type. The methods that do not specify a content type
explicitly have been added to forbidden apis to prevent more of these from entering our code base.

Relates #19388
2017-02-17 14:45:41 -05:00
Ali Beyad a6389a30f2 [TEST] bumps rolling upgrade test version to 5.4.0-snapshot 2017-02-15 10:59:24 -05:00
Ali Beyad 22e64bdf4f [TEST] remove stored scripts rolling upgrade test as it doesn't apply to 6.0 2017-02-03 12:30:42 -05:00
Ali Beyad 43aadef23a [TEST] upgrade backward compatibility version of rolling upgrade
tests to 5.3.0-SNAPSHOT
2017-02-02 09:56:38 -05:00
Ali Beyad 3c1637f413 Changes the rolling upgrade version test to test against
5.2.0 instead of 5.2.0-SNAPSHOT
2017-02-01 22:20:11 -05:00
Jack Conradson 3d2626c4c6 Change Namespace for Stored Script to Only Use Id (#22206)
Currently, stored scripts use a namespace of (lang, id) to be put, get, deleted, and executed. This is not necessary since the lang is stored with the stored script. A user should only have to specify an id to use a stored script. This change makes that possible while keeping backwards compatibility with the previous namespace of (lang, id). Anywhere the previous namespace is used will log deprecation warnings.

The new behavior is the following:

When a user specifies a stored script, that script will be stored under both the new namespace and old namespace.

Take for example script 'A' with lang 'L0' and data 'D0'. If we add script 'A' to the empty set, the scripts map will be ["A" -- D0, "A#L0" -- D0]. If a script 'A' with lang 'L1' and data 'D1' is then added, the scripts map will be ["A" -- D1, "A#L1" -- D1, "A#L0" -- D0].

When a user deletes a stored script, that script will be deleted from both the new namespace (if it exists) and the old namespace.

Take for example a scripts map with {"A" -- D1, "A#L1" -- D1, "A#L0" -- D0}. If a script is removed specified by an id 'A' and lang null then the scripts map will be {"A#L0" -- D0}. To remove the final script, the deprecated namespace must be used, so an id 'A' and lang 'L0' would need to be specified.

When a user gets/executes a stored script, if the new namespace is used then the script will be retrieved/executed using only 'id', and if the old namespace is used then the script will be retrieved/executed using 'id' and 'lang'
2017-01-31 13:27:02 -08:00
Tim Brooks 719e75bb3f Add repository-url module and move URLRepository (#22752)
This is related to #22116. URLRepository requires SocketPermission
connect. This commit introduces a new module called "repository-url"
where URLRepository will reside. With the new module, permissions can
be removed from core.
2017-01-25 17:09:25 -06:00
Lee Hinman fed2a1a822 Fix Translog.Delete serialization for sequence numbers (#22543)
* Fix Translog.Delete serialization for sequence numbers

Translog.Delete used `.writeVLong` instead of `.writeLong` for the sequence
number and primary term (and their respective "read" variants). This could lead
to issues where a 5.x node sent a translog operation with a negative sequence
number (-2 for unassigned seq no) that tripped an assertion serializing a
negative number and causing ES to exit.

Adds a unit test for serialization and a mixed-cluster REST test, since that was
how this was originally caught.

* Use more realistic values for random seqNum and primary term

* Add comment with TODO for removal in 7.0

* Change comment into an assert
2017-01-11 10:08:04 -07:00
Igor Motov ca90d9ea82 Remove PROTO-based custom cluster state components
Switches custom cluster state components from PROTO-based de-serialization to named objects based de-serialization
2016-12-28 13:32:35 -05:00
Nik Everett f5f2149ff2 Remove much ceremony from parsing client yaml test suites (#22311)
* Remove a checked exception, replacing it with `ParsingException`.
* Remove all Parser classes for the yaml sections, replacing them with static methods.
* Remove `ClientYamlTestFragmentParser`. Isn't used any more.
* Remove `ClientYamlTestSuiteParseContext`, replacing it with some static utility methods.

I did not rewrite the parsers using `ObjectParser` because I don't think it is worth it right now.
2016-12-22 11:00:34 -05:00
Boaz Leskes b857b316b6 Add BWC layer to seq no infra and enable BWC tests (#22185)
Sequence BWC logic consists of two elements:

1) Wire level BWC using stream versions.
2) A changed to the global checkpoint maintenance semantics.

For the sequence number infra to work with a mixed version clusters, we have to consider situation where the primary is on an old node and replicas are on new ones (i.e., the replicas will receive operations without seq#) and also the reverse (i.e., the primary sends operations to a replica but the replica can't process the seq# and respond with local checkpoint). An new primary with an old replica is a rare because we do not allow a replica to recover from a new primary. However, it can occur if the old primary failed and a new replica was promoted or during primary relocation where the source primary is treated as a replica until the master starts the target.

1) Old Primary & New Replica - this case is easy as is taken care of by the wire level BWC. All incoming requests will have their seq# set to `UNASSIGNED_SEQ_NO`, which doesn't confuse the local checkpoint logic (keeping it at `NO_OPS_PERFORMED`) 
2) New Primary & Old replica - this one is trickier as the global checkpoint service currently takes all in sync replicas into consideration for the global checkpoint calculation. In order to deal with old replicas, we change the semantics to say all *new node* in sync replicas. That means the replicas on old nodes don't count for the global checkpointing. In this state the seq# infra is not fully operational (you can't search on it, because copies may miss it) but it is maintained on shards that can support it. The old replicas will have to go through a file based recovery at some point and will get the seq# information at that point. There is still an edge case where a new primary fails and an old replica takes over. I'lll discuss this one with @ywelsch as I prefer to avoid it completely.

This PR also re-enables the BWC tests which were disabled. As such it had to fix any BWC issue that had crept in. Most notably an issue with the removal of the `timestamp` field in #21670.

The commit also includes a fix for the default value of the seq number field in replicated write requests (it was 0 but should be -2), that surface some other minor bugs which are fixed as well.

Last - I added some debugging tools like more sane node names and forcing replication request to implement a `toString`
2016-12-19 13:08:24 +01:00
jaymode ced638bcda
test: use the correct number of bwc nodes for old cluster 2016-11-22 14:56:34 -05:00
Jason Tedor b2b7595fa7 Temporarily set BWC version to 6.0.0 for seq. no
There is not yet a BWC layer in sequence numbers. This commit sets the
BWC version to 6.0.0 for the BWC and rolling upgrade tests until this
BWC layer is built.
2016-11-16 09:09:38 -05:00
Simon Willnauer bdc942fa72 Enable 5.x to 6.x BWC tests
This commit enables real BWC testing against a 5.1 snapshot. All
REST tests plus rolling upgrade test now run against a mixed version
cross major version cluster.
2016-11-14 14:26:49 +01:00
Ryan Ernst 7a2c984bcc Test: Remove multi process support from rest test runner (#21391)
At one point in the past when moving out the rest tests from core to
their own subproject, we had multiple test classes which evenly split up
the tests to run. However, we simplified this and went back to a single
test runner to have better reproduceability in tests. This change
removes the remnants of that multiplexing support.
2016-11-07 15:07:34 -08:00
Ali Beyad 8afc83047f Change the timeout of the rolling upgrades test from 40 mins to 5 mins
to still allow accounting for slow VMs
2016-09-19 15:45:41 -04:00
Ali Beyad 98230d035a Adds a preserveIndicesUponCompletion method to ESRestTestCase
that can be overridden by subclasses if the test must not
delete indices it created after exiting.
2016-09-16 19:21:26 -04:00
Ali Beyad 83adc87015 Removes stopNodeUponCompletion in favor of moving the stop
nodes task to the final part of the cluster test task execution
graph.
2016-09-16 16:32:43 -04:00
Ali Beyad 22e6bc8359 Disables unit tests for rolling upgrades, as there are only
rest integration tests.
2016-09-16 11:32:28 -04:00
Ali Beyad 56f97500c6 In the rolling upgrades tests, we do not want to stop nodes
automatically between tasks, as we want some of the nodes from
the previous task to continue running in the next task. This
commit enables a cluster configuration setting to not stop
nodes automatically after a task runs, but instead the creator
of the test task must stop the running nodes explicitly in a
cleanup phase.
2016-09-16 10:36:55 -04:00
Ali Beyad ba072ec18e When Elasticsearch nodes are started in gradle to form a
cluster, we wait for the cluster health to indicate the
necessary nodes have formed a cluster.  This check was an
exact value (equality) check.  However, if we are trying to
connect the nodes in the cluster to nodes from a previously
formed cluster (of the same name), then we will have more
nodes returned by the cluster health check than the current
task's configured number of nodes.  Hence, this check needs
to be a >= check.  This commit fixes it.
2016-09-16 09:41:26 -04:00
Ali Beyad ec7e383783 Better checks for the cluster being up in the rolling upgrades tests. 2016-09-15 15:08:11 -04:00
Ali Beyad 3f79874042 Prevent the rolling upgrades rest tests from cleaning up indices
after finishing if a the tests.rest.preserve_indices system property
is set
2016-09-14 23:34:19 -04:00
Ali Beyad 513ed58d17 Added a indices flush to the rolling upgrades rest tests to ensure
they are on disk for the upgraded nodes to pick up.
2016-09-13 12:36:20 -04:00
Ali Beyad 0c7dd1865c Reworking yaml tests for rolling upgrades 2016-09-12 18:46:06 -04:00
Ali Beyad 0b3eb11712 Changed rest-api-spec tests folder structure 2016-09-12 16:05:36 -04:00
Ryan Ernst bd2de367cc Use 5.0 alpha5 for bwc version, so we have transport.ports
Also fixed bug to ensure unicast host is writtent in yaml quotes
2016-09-06 16:08:27 -07:00
Ryan Ernst a844b085f1 Made node config always have a unicast transport uri closure 2016-09-06 15:51:14 -07:00
Ali Beyad 5d8aa6b4fe Adds tests for rolling upgrades to execute 2016-09-03 01:34:39 -04:00
Ryan Ernst ecaf6ef001 Add rolling upgrade test project 2016-08-31 16:45:03 -07:00