Commit Graph

669 Commits

Author SHA1 Message Date
Simon Willnauer 43dbf9c7b6 Use all available hosts in REST tests and allow for real master election (#21161)
Today we only use a single node to send requests to when we run REST tests.
In some cases we have more than one node (ie. in the BWC case) where we should
send requests to all nodes in a round-robin fashion. This change passes all
available node endpoints to the rest test.

Additionally, this change adds the setting of `discovery.zen.minimum_master_nodes`
to the cluster formation forcing the nodes to wait for all other nodes until the cluster
is formed. This allows for a more realistic master election and allows all master eligable
nodes to become master while before always the first node in the cluster became the master.

This also adds logging to each test run to log the master nodes version and the minimum node
version in the cluster to help debugging BWC test failures.
2016-10-28 12:18:47 +02:00
Simon Willnauer 97cc426a89 Fix bwc cluster formation in order to run BWC tests against a mixed version cluster (#21145)
This fixes our cluster formation task to run REST tests against a mixed version cluster.
Yet, due to some limitations in our test framework `indices.rollover` tests are currently
disabled for the BWC case since they select the current master as the merge node which
happens to be a BWC node and we can't relocate all shards to it since the primaries are on
a higher version node. This will be fixed in a followup.

Closes #21142

Note: This has been cherry-picked from 5.0 and fixes several rest tests
as well as a BWC break in `OsStats.java`
2016-10-27 17:03:53 +02:00
Christoph Büscher aaf3477327 Remove checkstyle LineLenght check suppression from 23 more classes 2016-10-26 15:24:21 +02:00
Simon Willnauer 0a410d3916 Pass executor name to request interceptor to support async intercept calls (#21089)
Today the request interceptor can't support async calls since the response
of the async call would execute on a different thread ie. a client or listener
thread. This means in-turn that the intercepted handler is not executed with the
thread it was supposed to run and therefor can, if it's executing blocking
operations, potentially deadlock an entire server.
2016-10-24 13:57:07 +02:00
Ryan Ernst 53cff0f00f Move all zen discovery classes into o.e.discovery.zen (#21032)
* Move all zen discovery classes into o.e.discovery.zen

This collapses sub packages of zen into zen. These all had just a couple
classes each, and there is really no reason to have the subpackages.

* fix checkstyle
2016-10-20 00:44:48 -07:00
Ryan Ernst dca614aa3b Build: Change `gradle run` to use zip distribution (#21001)
When running `gradle run`, a developer usually intends to get a running
instance as if they had run elasticsearch from the command line. This is
different than the isolated environment we use for integration testing
plugins. This change switches the run task to use the zip distribution,
so that all modules included in the normal distribution are included.
2016-10-18 11:48:58 -07:00
Christoph Büscher 274d5d1fdb Remove org.junit.rules.ExpectedException from forbidden-apis again 2016-10-18 11:13:09 +02:00
Christoph Büscher abff485d68 Use Lucenes expectThrows() when testing exceptions
Cleaning up a few remaining occurences of using junits ExpectedException rule in
favor of using LuceneTestCase#expectThrows() which is more concise and versatile.
2016-10-18 10:34:48 +02:00
Tanguy Leroux 3b578db365 MultiGet should not fail entirely if alias resolves to many indices (#20858)
MultiGet should not fail entirely when one of the items of a multi get request refers to an alias that points to multiple indices.

closes #20845
2016-10-11 18:07:39 +02:00
Simon Willnauer c98e3f60f7 Ensure source filtering automatons are only compiled once (#20857)
This change adds a overloaded `XContentMapValues#filter` method that returns
a function enclosing the compiled automatons that can be reused across filter
calls. This for instance prevents compiling automatons over and over again when
hits are filtered or in the SourceFieldMapper for each document.

Closes #20839
2016-10-11 15:30:38 +02:00
Boaz Leskes fc4645bae6 Cluster Settings Updates should not trigger circuit breakers. (#20827)
Settings updates are important to be able to help and administer a cluster in distress. We shouldn't block it due to circuit breakers. An extreme example is where we are actually trying to increase and unreasonable low setting for the circuit breaker itself.

See https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+g1gc/242/
2016-10-11 10:36:55 +02:00
Lee Hinman bd8f546381 Enable incremental compilation in gradle (#20847) 2016-10-10 15:47:35 -06:00
Nik Everett 6418f89feb Remove NodeServicesProvider
Instead provide services where they are needed. The class worked
well as a temporary measure to easy removal of guice from the index
level but now we can remove it entirely.

-1 @Inject annotation
2016-10-08 07:35:58 -04:00
Jason Tedor 535bf92280 Upgrade Log4j 2 to version 2.7
This commit upgrades the Log4j 2 dependency to version 2.7 and removes
some hacks that we had in place to work around bugs in Log4j 2 version
2.6.2.

Relates #20805
2016-10-07 13:05:30 -05:00
Ryan Ernst 6da0f0dcc0 Build: Fix plugin poms to have correct artifact id (#20764)
We already override the name in plugin pom files to be that configured
for the plugin but we also need to explicitly set the artifactId.
2016-10-05 08:10:27 -07:00
Daniel Mitterdorfer aa308fe159 Upgrade microbenchmarks to JMH 1.15 2016-10-04 10:20:45 +02:00
Ryan Ernst a82bf7a851 Build: Use more general archivesBaseName setting, to ensure pom uses
matching file name
2016-10-03 21:40:43 -07:00
Nik Everett 7cfc2f126c CONSOLEify the first half of getting-started
Adds a `// TESTRESPONSE[_cat]` syntax that sets up some substitutions
useful for asserting that _cat APIs match.
2016-10-01 12:56:23 +02:00
Simon Willnauer fe1803c957 Remove AnalysisService and reduce it to a simple name to analyzer mapping (#20627)
Today we hold on to all possible tokenizers, tokenfilters etc. when we create
an index service on a node. This was mainly done to allow the `_analyze` API to
directly access all these primitive. We fixed this in #19827 and can now get rid of
the AnalysisService entirely and replace it with a simple map like class. This
ensures we don't create a gazillion long living objects that are entirely useless since
they are never used in most of the indices. Also those objects might consume a considerable
amount of memory since they might load stopwords or synonyms etc.

Closes #19828
2016-09-23 08:53:50 +02:00
Jason Tedor 78ceb9301b Fix cluster wait condition
Today we define a cluster wait condition to try to wait at least a
certain number of nodes when running integration tests. Alas, the wait
condition is incorrect because wait_for_nodes>=${numNodes} will be split
by parameter parsing on the equals sign so the request looks like it has
a parameter named wait_for_nodes>. The fact that REST param parsing is
lenient leads to this being undiscovered. This commit fixes this issue.

Relates #20601
2016-09-21 16:33:15 -04:00
Ali Beyad 5031824291 File-based discovery plugin integration tests (#20492)
Adds an integration test for the file-based discovery plugin
to test the plugin operates correctly and uses the hosts
configured in `unicast_hosts.txt` with a real cluster

Closes #20459
2016-09-21 15:48:18 -04:00
Nik Everett 6705c6aa2f Further document the pom hack
We have a "HUGE HACK" that allows us to publish zip artifacts to
Sonatype's OSS repository without javadoc and source jars. We don't
include those jars because the zip is just a repackaging of the
core and module jars for which we already publish the javadoc and
source jars. So we have a hack to publish the zip artifact when the
pom says the project is of type 'pom'.
2016-09-21 11:25:19 -04:00
Yannick Welsch 47088cc206 Allow build to run even if there is no Git remote named origin (#20608)
The build currently depends on the presence of a Git remote named origin to determine the URL that is used in the generated POM file. As this is best-effort anyhow and only required by Maven Central, this commit allows the build to run even if a Git remote with the name "origin" is missing.
2016-09-21 17:22:06 +02:00
Yannick Welsch 191fadafcc Fix logger usage checker for Log4j 2
With the switch to Log4j 2 throughout our code base, the logger usage checker was temporarily disabled. This commit
adapts the checks to work with Log4j 2 and re-enables the Gradle checks.

Closes #20243
2016-09-21 14:44:14 +02:00
Nik Everett af285f1aa4 Convert more docs to CONSOLE
* reference/api-conventions.asciidoc
* reference/redirects.asciidoc

Relates to #18160
2016-09-20 13:12:12 -04:00
Nik Everett 3771d80830 Fix up nebulaRealPom
I'm not sure why we need this pom instead of the pom generated by
nebula, but if we are going to have it then we need to populate it
with appropriate stuff like project name, description, and url.
2016-09-20 11:05:23 -04:00
Ryan Ernst 6921b4a66b Build: Add back hack for client copying client jar pom (#20577)
Gradle appears to have a bug in maven publshing which will not match the
artifactId of a generated pom with the artifact id it puts in the file.
This adds back a copy hack from the original pom file name to the client
pom file name (which we had before #20403 inadvertently
removed it).
2016-09-20 01:37:18 -07:00
Tanguy Leroux 38dce6384f Improve TribeIT tests
This commit adds a new test TribeIT#testClusterStateNodes() to verify that the tribe node correctly reflects the nodes of the remote clusters it is connected to.

It also changes the existing tests so that they really use two remote clusters now.
2016-09-20 10:31:32 +02:00
Nik Everett 9fd42d1d48 Fix allowed non-CONSOLE docs on windows
`\` vs `/` strikes again. This time I just convert all the `\` into
`/`. It isn't perfect but anyone who sees it'll figure out what is
up.
2016-09-19 22:43:17 -04:00
Ryan Ernst 85b8f29415 Build: Remove old maven deploy support (#20403)
* Build: Remove old maven deploy support

This change removes the old maven deploy that we have in parallel to
maven-publish, and makes maven-publish fully work with publishing to
maven local. Using `gradle publishToMavenLocal` should be used to
publish to .m2.

Note that there is an unfortunate hack that means for
zip artifacts we must first create/publish a dummy pom file, and then
follow that with the real pom file. It would be nice to have the pom
file contains packaging=zip, but maven central then requires sources and
javadocs. But our zips are really just attached artifacts, so we already
set the packaging type to pom for our zip files. This change just works
around a limitation of the underlying maven publishing library which
silently skips attached artifacts when the packaging type is set to pom.

relates #20164
closes #20375

* Remove unnecessary extra spacing
2016-09-19 15:10:41 -07:00
Ryan Ernst e2854bb3a4 Build: Remove maven central snapshots from repositories (#20567)
We were using maven snapshots during heavy development, but this should
not be something generally available (we should never release depending
on a snapshot version in maven). This change removes the snapshot repo.
If we ever need it temporarily for some reason, we can add it if/when
it is necessary.

relates #20559
2016-09-19 15:06:05 -07:00
Nik Everett 156393be0e Fail build if new doc snippets aren't `// CONSOLE`
This tracks the snippets that probably should be converted to
`// CONSOLE` or `// TESTRESPONSE` and fails the build if the list
of files with such snippets doesn't match the list in `docs/build.gradle`.
Setting the file looks like
```
/* List of files that have snippets that probably should be converted to
 * `// CONSOLE` and `// TESTRESPONSE` but have yet to be converted. Try and
 * only remove entries from this list. When it is empty we'll remove it
 * entirely and have a party! There will be cake and everything.... */
buildRestTests.expectedUnconvertedCandidates = [
  'plugins/discovery-azure-classic.asciidoc',
...
  'reference/search/suggesters/completion-suggest.asciidoc',
]
```

This list is in `build.gradle` because we expect it to be fairly
temporary. In a few months we'll have converted all of the docs and won't
ned it any more.

From now on if you add now docs that contain a snippet that shows an
interaction with elasticsearch you have three choices:
1. Stick `// CONSOLE` on the interactions and `// TESTRESPONSE` on the
responses. The build (specifically (`gradle docs:check`) will test that
these interactions "work". If there isn't a `// TESTRESPONSE` snippet
then "work" just means "Elasticsearch responds with a 200-level response
code and no `WARNING` headers. This is way better than nothing.

2. Add `// NOTCONSOLE` if the snippet isn't actually interacting with
Elasticsearch. This should only be required for stuff like javascript
source code or `curl` against an external service like AWS or GCE. The
snippet will not get "OPEN IN CONSOLE" or "COPY AS CURL" buttons or be
tested.

3. Add `// TEST[skip:reason]` under the snippet. This will just skip the
snippet in the test phase. This should really be reserved for snippets
where we can't test them because they require an external service that
we don't have at testing time.

Please, please, please, please don't add more things to the list. After
all, it sais there'll be cake when we remove it entirely!

Relates to #18160
2016-09-19 16:43:43 -04:00
Ali Beyad 50584c4103 Merge pull request #20532 from rjernst/rolling_upgrades
This PR introduces backward compatibility index tests to test the rolling upgrade process amongst Elasticsearch instances within the same major version. The test executes in three phases. In the first phase, we form a cluster of 2 ES instances on an old version. In the second phase, we keep one of the nodes from the old cluster, kill the other node, but preserve its data directory and start an instance of the current version of ES using the same data directory as the killed instance. In the third phase, we kill the other old version ES instance from the first phase and launch a new instance, using the same data directory as the killed instance. Therefore, during phase 3, we have fully migrated and have all current versions of ES running. In each phase, we run REST tests that index documents and search them, ensuring at each stage that the documents from the previous phase are still there.

Note that because we haven't released a GA yet of 5.0, the tests currently don't start an old version cluster in the first phase. Once GA is released, this will be changed to make the backward compatibility version 5.0, while the current version in the cluster will be 5.x.
2016-09-19 16:14:38 -04:00
David Pilato a530399ae5 Merge branch 'pr/remove-mapper-attachments' 2016-09-19 14:47:14 +02:00
Daniel Mitterdorfer 6f628557b9 Upgrade from JMH 1.12 to JMH 1.14 2016-09-19 14:28:16 +02:00
David Pilato dfd1eebdd0 Remove mapper attachments plugin
We now have in 5.0.0 `ingest-attachment` plugin.
We can remove `mapper-attachments` plugin for 6.0.

Closes #18837.
2016-09-19 09:01:16 +02:00
Ali Beyad 98230d035a Adds a preserveIndicesUponCompletion method to ESRestTestCase
that can be overridden by subclasses if the test must not
delete indices it created after exiting.
2016-09-16 19:21:26 -04:00
Ali Beyad 83adc87015 Removes stopNodeUponCompletion in favor of moving the stop
nodes task to the final part of the cluster test task execution
graph.
2016-09-16 16:32:43 -04:00
Ali Beyad ce86ed1fdd Merge remote-tracking branch 'upstream/master' into rolling_upgrades 2016-09-16 10:43:38 -04:00
Ali Beyad 56f97500c6 In the rolling upgrades tests, we do not want to stop nodes
automatically between tasks, as we want some of the nodes from
the previous task to continue running in the next task. This
commit enables a cluster configuration setting to not stop
nodes automatically after a task runs, but instead the creator
of the test task must stop the running nodes explicitly in a
cleanup phase.
2016-09-16 10:36:55 -04:00
Ali Beyad ba072ec18e When Elasticsearch nodes are started in gradle to form a
cluster, we wait for the cluster health to indicate the
necessary nodes have formed a cluster.  This check was an
exact value (equality) check.  However, if we are trying to
connect the nodes in the cluster to nodes from a previously
formed cluster (of the same name), then we will have more
nodes returned by the cluster health check than the current
task's configured number of nodes.  Hence, this check needs
to be a >= check.  This commit fixes it.
2016-09-16 09:41:26 -04:00
Boaz Leskes 4ac3b416f2 Don't generate html outputs from checkstyle (#20481)
We run our builds with `gradle --info` and that ends up generating horrific long dumps of xslt.
2016-09-14 17:23:40 +02:00
Ali Beyad 0c7dd1865c Reworking yaml tests for rolling upgrades 2016-09-12 18:46:06 -04:00
chico chen 4cf4683a64 Build: can not find JAVA_HOME in eclipse gradle plugin. (#17645) 2016-09-12 18:38:24 -04:00
Jason Tedor 3b90906f4f Adapt release flag to new CLI arguments
The JDK project is in the process of modifying the command-line flags
for various JDK tools (http://openjdk.java.net/jeps/293). In particular,
the release flag on javac has changed from -release to --release. This
commit adapts the build process to this change.

Relates #20420
2016-09-12 10:53:32 -04:00
javanna 90ab460fcc move parsing of search ext sections to the coordinating node 2016-09-09 19:10:42 +02:00
javanna dc2ba90f48 clarify that SearchParseElement is only used for custom fetch sub phases and clean up extension point
SearchParseElement is renamed to FetchSubPhaseParser and moved to the search.fetch package. Its parse method doesn't get the SearchContext as argument anymore, only the XContentParser, and the return type is what gets parsed (the fetch sub phase context which we may as well rename later).

It is the parser that initializes the FetchSubPhaseContext then. SearchService retrieves the parser by name, calls parse against it and stores the result of parsing by name. No need for FetchSubPhase.ContextFactory anymore, which can be removed.
2016-09-09 18:05:49 +02:00
javanna 4b57a0fd97 resolve some line length problems and remove some entry from checkstyle suppressions (deleted classes) 2016-09-09 18:05:49 +02:00
Simon Willnauer f319545814 Prepare master branch to be 6.0.0-alpha1 2016-09-08 12:55:30 +02:00
Ryan Ernst bd2de367cc Use 5.0 alpha5 for bwc version, so we have transport.ports
Also fixed bug to ensure unicast host is writtent in yaml quotes
2016-09-06 16:08:27 -07:00
Ryan Ernst e6ac27fcfa Merge branch 'master' into rolling_upgrades 2016-09-06 15:52:03 -07:00
Ryan Ernst a844b085f1 Made node config always have a unicast transport uri closure 2016-09-06 15:51:14 -07:00
Jason Tedor 0d7dfcd798 Merge pull request #20338 from jasontedor/remove-plugin
Print message when removing plugin with config
2016-09-06 11:43:51 -04:00
Nik Everett 5cff2a046d Remove most of the need for `// NOTCONSOLE`
and be much more stingy about what we consider a console candidate.

* Add `// CONSOLE` to check-running
* Fix version in some snippets
* Mark groovy snippets as groovy
* Fix versions in plugins
* Fix language marker errors
* Fix language parsing in snippets

  This adds support for snippets who's language is written like
  `[source, txt]` and `["source","js",subs="attributes,callouts"]`.

  This also makes language required for snippets which is nice because
  then we can be sure we can grep for snippets in a particular language.
2016-09-06 10:32:54 -04:00
Jason Tedor e081b2b2e8 Remove length violation in RemovePluginCommand
This commit removes a line-length violation in RemovePluginCommand.java
and removes this file from the list of files for which the line-length
check is suppressed.
2016-09-06 07:28:05 -04:00
Ali Beyad 5d8aa6b4fe Adds tests for rolling upgrades to execute 2016-09-03 01:34:39 -04:00
Jason Tedor b9966fed36 Hack around Log4j bug rendering exceptions
Log4j has a bug where it does not handle a security exception that can
be thrown when it is rendering a stack trace. This commit intentionally
introduces jar hell with the ThrowableProxy class to work around this
bug until a fix is a released.

Relates #20306
2016-09-02 20:26:32 -04:00
javanna e5a741ab67 fix line length in some touched classes 2016-09-02 10:23:49 +02:00
Martijn van Groningen a110498ad8 settings: Make `action.auto_create_index` setting a dynamic cluster setting.
Closes #7513
2016-09-01 12:33:30 +02:00
Simon Willnauer a0becd26b1 Optimize indexing for the autogenerated ID append-only case (#20211)
If elasticsearch controls the ID values as well as the documents
version we can optimize the code that adds / appends the documents
to the index. Essentially we an skip the version lookup for all
documents unless the same document is delivered more than once.

On the lucene level we can simply call IndexWriter#addDocument instead
of #updateDocument but on the Engine level we need to ensure that we deoptimize
the case once we see the same document more than once.

This is done as follows:

1. Mark every request with a timestamp. This is done once on the first node that
receives a request and is fixed for this request. This can be even the
machine local time (see why later). The important part is that retry
requests will have the same value as the original one.

2. In the engine we make sure we keep the highest seen time stamp of "retry" requests.
This is updated while the retry request has its doc id lock. Call this `maxUnsafeAutoIdTimestamp`

3. When the engine runs an "optimized" request comes, it compares it's timestamp with the
current `maxUnsafeAutoIdTimestamp` (but doesn't update it). If the the request
timestamp is higher it is safe to execute it as optimized (no retry request with the same
timestamp has been run before). If not we fall back to "non-optimzed" mode and run the request as a retry one
and update the `maxUnsafeAutoIdTimestamp` unless it's been updated already to a higher value

Relates to #19813
2016-09-01 10:39:40 +02:00
Ryan Ernst ecaf6ef001 Add rolling upgrade test project 2016-08-31 16:45:03 -07:00
Jason Tedor e166459bbe Merge branch 'master' into log4j2
* master:
  Increase visibility of deprecation logger
  Skip transport client plugin installed on JDK 9
  Explicitly disable Netty key set replacement
  percolator: Fail indexing percolator queries containing either a has_child or has_parent query.
  Make it possible for Ingest Processors to access AnalysisRegistry
  Allow RestClient to send array-based headers
  Silence rest util tests until the bogusness can be simplified
  Remove unknown HttpContext-based test as it fails unpredictably on different JVMs
  Tests: Improve rest suite names and generated test names for docs tests
  Add support for a RestClient base path
2016-08-31 10:59:27 -04:00
Jason Tedor 4e69ac0272 Add link to open logger usage issue
This commit adds comments linking to an open issue regarding updating
the logger usage check for the Log4j 2 API.
2016-08-30 21:13:17 -04:00
Ryan Ernst e19f2b6348 Tests: Improve rest suite names and generated test names for docs tests
Rest test suites are currently only the directory above the yaml test
file. That is confusing when there are more than one directory level
which contain yaml tests, as there are in generated docs tests. This
change makes rest tests use the full relative path to the rest test root
as the suite name, and also makes the test names for docs tests a little
clearer (that they are testing an example from a specific line number,
instead of just the line number as an opaque test name).
2016-08-30 13:55:44 -07:00
Jason Tedor 7da0cdec42 Introduce Log4j 2
This commit introduces Log4j 2 to the stack.
2016-08-30 13:31:24 -04:00
Jason Tedor 5a8f2d7fb3 Disable logger usage checks
This commit disables the logger usage checks as they will not be
compatible with Log4j 2. This disabling is temporary, they will return.
2016-08-30 13:28:07 -04:00
Nik Everett df73292256 Add an alias action to delete an index
While removing an index isn't actually an alias action, if we add
an alias action that deletes an index then we can delete and index
and add an alias with the same name as the index atomically, in
the same cluster state update.

Closes #20064
2016-08-30 10:15:21 -04:00
Nik Everett 9c3f6d58ac Support downgrading keyword/text into string
This changes Elasticsearch to automatically downgrade `text` and
`keyword` fields into appropriate `string` fields when changing the
mapping of indexes imported from 2.x. This allows users to use the
modern, documented syntax against 2.x indexes. It also makes it clear
that reindexing in order to recreate the index in 5.0 is required for
any long lived indexes. This change is useful for the times when you
can't (cluster is just starting, not stable enough for reindex) or
shouldn't (index will only live 90 days or something).
2016-08-29 11:27:37 -04:00
Yannick Welsch 1b75cb63a2 Add recovery source to ShardRouting (#19516)
Adds an explicit recoverySource field to ShardRouting that characterizes the type of recovery to perform:

- fresh empty shard copy
- existing local shard copy
- recover from peer (primary)
- recover from snapshot
- recover from other local shards on same node (shrink index action)
2016-08-27 16:11:10 +02:00
Mike McCandless 0ccfe69789 Upgrade to Lucene 6.2.0 2016-08-24 17:26:28 -04:00
Ryan Ernst acbece5b55 Merge pull request #20134 from rjernst/plugin_run_config
Build: Allow plugin to set run configuration distro to zip
2016-08-24 08:49:13 -07:00
Ryan Ernst 81aa67f9d5 Build: Allow plugin to set run configuration distro to zip
This was previously broken because run and integTest used the same
configuration name. This change makes the configuration name prefixed by
the task the cluster is created for.
2016-08-23 16:12:35 -07:00
Daniel Mitterdorfer c13513ed61 Allow to enable annotation processing explicitly (#20117)
In 1e91f3b we disabled annotation processors globally. However, some
project like JMH need annotation processing, so we add an ability to
selectively enabled annotation processing for certain projects by
setting an external property in the corresponding Gradle build script.

Note that `javac` would allow to set a specific annotation processor
with the command line option `-processor`. However, due to a bug in
Gradle we we cannot use this option and need to enable all annotation
processors.
2016-08-23 15:15:22 +02:00
Jason Tedor 13e20e3320 Set external nodes to default to 512m of heap
This commit sets external nodes for integration tests to default to
using 512m of heap. This can be overridden using tests.heap.size (a
system property that we already use elesewhere for setting the size of
the heap for the test runner) or using tests.jvm.argline (this last one
takes precedence).
2016-08-19 12:55:48 -04:00
Ryan Ernst 8c60455ed6 Fix checkstyle line length violations in allocation tests 2016-08-17 16:28:31 -07:00
Ryan Ernst 8d2770cd98 Merge pull request #19876 from rjernst/gradle_pom_runtime_deps
Build: Fix compile time deps in poms and simplify transitive exclusions
2016-08-15 21:53:53 -07:00
Nik Everett 1452ab4b9f Squash the rest of o.e.rest.action
Squashes all the subpackages of `org.elasticsearch.rest.action` down to
the following:
* `o.e.rest.action.admin` - Administrative actions
* `o.e.rest.action.cat` - Actions that make tables for `grep`ing
* `o.e.rest.action.document` - Actions that act on documents
* `o.e.rest.action.ingest` - Actions that act on ingest pipelines
* `o.e.rest.action.search` - Actions that search

I'm tempted to merge `search` into `document` but the `document`
package feels fairly complete as is and `Suggest` isn't actually always
about documents either....

I'm also tempted to merge `ingest` into `admin.cluster` because the
latter contains the actions for dealing with stored scripts.

I've moved the `o.e.rest.action.support` into `o.e.rest.action`.

I've also added `package-info.java`s to all packges in `o.e.rest`. I
figure if the package is too small to deserve a `package-info.java` file
then it is too small to deserve to be a package....

Also fixes checkstyle in all moved classes.
2016-08-15 21:06:32 -04:00
Nik Everett 153b2ae180 Checkstyle 2016-08-12 18:21:15 -04:00
Nik Everett 7542ef3173 [docs] Don't allow `[source,javascript]`
The syntax highlighter doesn't support it. Just use `js`.
2016-08-12 17:08:43 -04:00
Nik Everett ffd226efa0 Add `// NOTCONSOLE` to docs
We have 1074 snippets that look like they should be converted to
`// CONSOLE`. At least that is what `gradle docs:listConsoleCandidates`
says. This adds `// NOTCONSOLE` to explicitly mark snippets that
*shouldn't* be converted to `// CONSOLE`. After marking the blindingly
obvious ones this cuts the remaining snippet count to 1032.
2016-08-12 16:49:57 -04:00
Jason Tedor 1f0673c9bd Default max local storage nodes to one
This commit defaults the max local storage nodes to one. The motivation
for this change is that a default value greather than one is dangerous
as users sometimes end up unknowingly starting a second node and start
thinking that they have encountered data loss.

Relates #19964
2016-08-12 09:26:20 -04:00
Adrien Grand 0d6ac57acf Collapse o.e.index.mapper packages. #19921
I also reduced the visibility of a couple classes and renamed/consolidated some
test classes for consistency, eg. removing the `Simple` prefix or using the
`<Type>FieldMapperTests` convention for testing field mappers.
2016-08-10 17:51:11 +02:00
Jason Tedor 1e91f3b779 Disable all annotation processors
Some unused annotation processors caused build-time failures. Instead,
we should just be explicit about which annotation processors we will use
(if any) with additional command-line flags.

Relates #19919
2016-08-10 10:42:28 -04:00
Clinton Gormley 0dd6f63c49 Fixed missing changes in the version bump to alpha6 2016-08-09 18:52:03 +02:00
Ryan Ernst 576aaac976 Build: Fix compile time deps in poms and simplify transitive exclusions
This change works around a known issue with using the maven-publish
gralde plugin. All deps are marked in the generated pom as runtime. With
this change, they are set back to compile time. This also simplified the
transitive dependencies exclusion to work the same as how it was fixed in
gradle 2.14 (wildcard exclusions).

closes #19835
2016-08-08 11:55:33 -07:00
Ryan Ernst 6c34a8f4ee Add back norelease check when building a release
This was lost around 2.1. This change adds it back.

closes #19246
2016-08-08 11:20:36 -07:00
Christoph Büscher 10d64eb43a Remove unneeded 140 character line suppresions 2016-08-05 19:42:52 +02:00
Tanguy Leroux 841d5a210e Update to Jackson 2.8.1
This commit updates Jackson to the 2.8.1 version, which is more strict when it comes to build objects. It also adds the snakeyaml dependency that was previously shaded in jackson libs.

It also closes #18076
2016-08-05 12:26:06 +02:00
Nik Everett 1e587406d8 Fail yaml tests and docs snippets that get unexpected warnings
Adds `warnings` syntax to the yaml test that allows you to expect
a `Warning` header that looks like:
```
    - do:
        warnings:
            - '[index] is deprecated'
            - quotes are not required because yaml
            - but this argument is always a list, never a single string
            - no matter how many warnings you expect
        get:
            index:    test
            type:    test
            id:        1
```

These are accessible from the docs with:
```
// TEST[warning:some warning]
```

This should help to force you to update the docs if you deprecate
something. You *must* add the warnings marker to the docs or the build
will fail. While you are there you *should* update the docs to add
deprecation warnings visible in the rendered results.
2016-08-04 15:23:05 -04:00
javanna 146f02183d [TEST] remove unused methods and fix some warnings in AbstractQueryTestCase
Also fix line length issues
2016-08-04 10:06:25 +02:00
Nik Everett e249ad8dfe Fix loggerUsageCheck after clean
The `loggerUsageCheck` can only run on directories that exist. It was
checking whether or not the directories exists before they were built
built and then deciding to do no work. But only if you are building in
a cleaned environment which CI does, but people rarely do locally.
2016-08-03 15:36:11 -04:00
Ali Beyad 3d2a105825 Merge pull request #19454 from abeyad/remove-write-consistency-level
Removes write consistency level across replication action APIs in favor of wait_for_active_shards
2016-08-02 09:01:11 -04:00
Daniel Mitterdorfer 88cbfbaabd Ban dangerous methods of java.lang.Thread (#19677)
Ban dangerous methods of java.lang.Thread

With this commit we disallow usage of the method Thread.stop().
2016-08-02 14:11:42 +02:00
Ali Beyad 25d8eca62d Removes the notion of write consistency level across all APIs in
favor of waiting for active shard copy count (wait_for_active_shards).
2016-08-01 13:35:29 -04:00
Tanguy Leroux 386902903e [TEST] Kill remaining lang-groovy messy tests
After #13834 many tests that used Groovy scripts (for good or bad reason) in their tests have been moved in the lang-groovy module and the issue #13837 has been created to track these messy tests in order to clean them up.

The work started with #19280, #19302 and #19336 and this PR moves the remaining messy tests back in core, removes the dependency on Groovy, changes the scripts in order to use the mocked script engine, and change the tests to integration tests.

It also moves IndexLookupIT test back (even if it has good chance to be removed soon) and fixes its tests.

It also changes AbstractQueryTestCase to use custom script plugins in tests.

closes #13837
2016-08-01 16:59:47 +02:00
Nik Everett 303c9faca5 Squash o.e.rest.action.admin.cluster
In an effort to reduce the number of tiny packages we have in the
code base this moves all the files that were in subdirectories of
`org.elasticsearch.rest.action.admin.cluster` into
`org.elasticsearch.rest.action.admin.cluster`.

Also fixes line length in these packages.
2016-07-29 20:31:24 -04:00
Ryan Ernst efa37b8b7d Merge pull request #19667 from rjernst/client_jar_id
Build: Fix client jars for plugins/modules to have the correct artifactId
2016-07-29 11:27:28 -07:00
Alexander Reelsen 30b497622b Dependencies: Upgrade to netty 4.1.4 (#19689)
Removes our internal snapshot repo we used until 4.1.4 was released.
2016-07-29 17:53:12 +02:00
Nik Everett ad028f3f9c Squash o.e.rest.action.admin.indices
In an effort to reduce the number of tiny packages we have in the
code base this moves all the files that were in subdirectories of
`org.elasticsearch.rest.action.admin.indices` into
`org.elasticsearch.rest.action.admin.indices`.

It also adds a `package-info.java` file explaining what the files in
the package *do*.

Also fixes line length in these packages. It makes a single non-checkstyle
change: implementing `ToXContent` on `GetIndexTemplatesResponse`. I did
this because it was the right thing to do and it fixed a line length
violation.
2016-07-29 10:08:03 -04:00
Ryan Ernst c414a6cedd Build: Fix client jars for plugins/modules to have the correct artifactId
For transport client plugins, the jars and poms are renamed with the
-client suffix. But we need the artifactId to match the id in the
filename.
2016-07-28 12:19:10 -07:00
Ryan Ernst 0be363d611 Merge pull request #19589 from rjernst/license_header_generic
Allow license header check to be customized
2016-07-25 17:40:41 -07:00
Ryan Ernst 7bf6676d21 Use string concat instead of gstring 2016-07-25 17:30:15 -07:00
Ryan Ernst c2c9b51732 Add javadocs for adding additional license types to the license headers
check
2016-07-25 17:22:08 -07:00
Ryan Ernst 0e6bdd025d Construct hashmap directly instead of using crazy groovy syntax 2016-07-25 17:17:12 -07:00
Ryan Ernst 0ecaa6ec3c Build: Allow license header check to be customized
This change allows setting which license families are approved, as well
as adding matchers for additional license types.
2016-07-25 17:05:40 -07:00
Nik Everett a95d4f4ee7 Add Location header and improve REST testing
This adds a header that looks like `Location: /test/test/1` to the
response for the index/create/update API. The requirement for the header
comes from https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

https://tools.ietf.org/html/rfc7231#section-7.1.2 claims that relative
URIs are OK. So we use an absolute path which should resolve to the
appropriate location.

Closes #19079

This makes large changes to our rest test infrastructure, allowing us
to write junit tests that test a running cluster via the rest client.
It does this by splitting ESRestTestCase into two classes:
* ESRestTestCase is the superclass of all tests that use the rest client
to interact with a running cluster.
* ESClientYamlSuiteTestCase is the superclass of all tests that use the
rest client to run the yaml tests. These tests are shared across all
official clients, thus the `ClientYamlSuite` part of the name.
2016-07-25 17:02:40 -04:00
Martijn van Groningen a9ab095b8c Moved all mustache classes into one package.
No need for multiple packages inside a small module.
2016-07-25 18:50:12 +02:00
Tanguy Leroux f745c96949 Clean up more messy tests
After #13834 many tests that used Groovy scripts (for good or bad reason) in their tests have been moved in the lang-groovy module and the issue #13837 has been created to track these messy tests in order to clean them up.

This commit moves more tests back in core, removes the dependency on Groovy, changes the scripts in order to use the mocked script engine, and change the tests  to integration tests.
2016-07-25 17:02:49 +02:00
Jason Tedor 2d1b0587dd Introduce Netty 4
This commit adds transport-netty4, a transport and HTTP implementation
based on Netty 4.

Relates #19526
2016-07-22 22:26:35 -04:00
javanna 118a14fbe3 Build: upgrade httpcore version to 4.4.5
Closes #19127
2016-07-19 15:11:40 +02:00
Ryan Ernst 3f6c0feee3 Merge pull request #19461 from rjernst/plugin_default_config
Simplify plugin configuration for rest tests
2016-07-18 14:07:15 -07:00
Ryan Ernst 27d15e81c8 Merge pull request #19467 from rjernst/pom_url
Build: Ensure poms for plugin zips have url
2016-07-18 08:23:31 -07:00
Ryan Ernst b508afee1e Build: Ensure poms for plugin zips have url
Maven central requires a project url. The recent change to make poms for
plugin client jars broke that because we no longer use nebula publishing
for plugin pom generation. This change adds back the url to the pom.
2016-07-18 02:05:12 -07:00
Ryan Ernst 2fb3cdceff Build: Simplify plugin configuration for rest tests
This change removes the multiple ways that plugins can be added to the
integ test cluster. It also removes the use of the default
configuration, and instead adds a zip configuration to all plugins. This
will enable using project substitutions with plugins, which must be done
with the default configuration.
2016-07-15 14:34:21 -07:00
Nik Everett 7aeea764ba Remove wait_for_status=yellow from the docs
It is no longer required after 687e2e12b3.
2016-07-15 16:02:07 -04:00
Ali Beyad 687e2e12b3 Merge pull request #19450 from elastic/feature/friendly-index-creation
Makes index creation more friendly
2016-07-15 11:48:21 -04:00
Jason Tedor 1d9302b6a7 Copy client jars instead of moving them
Moving the dependent jars instead of copying breaks downstream builds
that rely on the jars existing for compilation. This commit modifies
these moves to be copies.
2016-07-15 07:19:32 -04:00
Ryan Ernst 20b038d976 Build: Add flag for plugins/modules which can be used in the transport client
This changes adds a flag which can be set in the esplugin closure in
build.gradle for plugins and modules which contain pieces that must be
published to maven, for use in the transport client. The jar/pom and
source/javadoc jars are moved to a new name that has the suffix
"-client".

I enabled this for the two modules that I know definitely need this;
there may be more. One open question is which groupId to use for the
generated pom.

closes #19411
2016-07-14 02:07:52 -07:00
Nik Everett 88d3527178 Migrate derivative pipeline aggregation to NamedWriteable
This is another step in the effort to remove AggregationStreams and
instead use NamedWriteableRegistry like the rest of the code base.
2016-07-13 07:12:22 -04:00
Simon Willnauer 814c7224f9 Merge pull request #19392 from elastic/modularize_netty
This moves all netty related code into modules/transport-netty the module is build as a zip file as well as a JAR to serve as a dependency for transport client. For the time being this is required otherwise we have no network based impl. for transport client users. This might be subject to change given that we move forward http client.
2016-07-13 09:52:03 +02:00
Nik Everett 06bd896ce0 Migrate geohash_grid and geo_bounds to NamedWriteable
Just another small step in removing Aggregation's custom streams
implementation in favor of NamedWriteable.
2016-07-12 22:22:51 -04:00
Nik Everett f479219ca7 Clean up significant terms aggregation results
* Clean up the generics around significant terms aggregation results
* Reduce code duplicated between `SignificantLongTerms` and
`SignificantStringTerms` by creating `InternalMappedSignificantTerms`
and moving common things there where possible.
* Migrate to `NamedWriteable`
* Line length fixes while I was there
2016-07-12 22:08:09 -04:00
Simon Willnauer 048e4416e7 Move netty transport and http into a module
This moves all netty code and it's dependency into a module.
2016-07-11 22:21:29 +02:00
Ali Beyad 7759c23272 Fix line length formatting for ClusterStateHealthTests 2016-07-11 15:32:13 -04:00
Nik Everett 89614586e9 Migrate range, date_range, and geo_distance aggregations to NamedWriteable 2016-07-11 13:00:36 -04:00
Nik Everett 81fcdfcee9 Expose task information from NodeClient
This exposes a method to start an action and return a task from
`NodeClient`. This allows reindex to use the injected `Client` rather
than require injecting `TransportAction`s
2016-07-07 18:02:09 -04:00
Nik Everett fe0f28965a Clean up serialization of terms aggregation results
Move to NamedWriteable and remove a lot of duplication.
2016-07-07 17:01:09 -04:00
Tanguy Leroux 3267fc4e0c Clean up more messy tests
After #13834 many tests that used Groovy scripts (for good or bad reason) in their tests have been moved in the lang-groovy module and the issue #13837 has been created to track these messy tests in order to clean them up.

This commit moves more tests back in core, removes the dependency on Groovy, changes the scripts in order to use the mocked script engine, and change the tests  to integration tests.
2016-07-07 17:50:23 +02:00
Tanguy Leroux b58f2eb5c2 Move back some messy tests from Groovy plugin to core
This commit moves back some messy tests that have been placed in lang-groovy module in https://github.com/elastic/elasticsearch/pull/13834. It removes the dependency on Groovy plugin as well as change back the tests to integration tests (IT suffix).

It also changes the current MockScriptEngine and MockScriptPlugin to make it easier to use.
2016-07-07 15:26:36 +02:00
Jim Ferenczi dcf6a96725 Add doc values support to the _size field in the mapper-size plugin
This change activates the doc_values on the _size field for indices created after 5.0.0-alpha4.
It also adds a note in the breaking changes that explain the situation and how to get around it.

Closes #18334
2016-07-05 14:47:58 +02:00
Adrien Grand 4b0d317e63 Bump version to 5.0.0-alpha5. 2016-07-05 14:34:23 +02:00
Boaz Leskes 6861d3571e Persistent Node Ids (#19140)
Node IDs are currently randomly generated during node startup. That means they change every time the node is restarted. While this doesn't matter for ES proper, it makes it hard for external services to track nodes. Another, more minor, side effect is that indexing the output of, say, the node stats API results in creating new fields due to node ID being used as keys.

The first approach I considered was to use the node's published address as the base for the id. We already [treat nodes with the same address as the same](https://github.com/elastic/elasticsearch/blob/master/core/src/main/java/org/elasticsearch/discovery/zen/NodeJoinController.java#L387) so this is a simple change (see [here](https://github.com/elastic/elasticsearch/compare/master...bleskes:node_persistent_id_based_on_address)). While this is simple and it works for probably most cases, it is not perfect. For example, if after a node restart, the node is not able to bind to the same port (because it's not yet freed by the OS), it will cause the node to still change identity. Also in environments where the host IP can change due to a host restart, identity will not be the same. 

Due to those limitation, I opted to go with a different approach where the node id will be persisted in the node's data folder. This has the upside of connecting the id to the nodes data. It also means that the host can be adapted in any way (replace network cards, attach storage to a new VM). I

It does however also have downsides - we now run the risk of two nodes having the same id, if someone copies clones a data folder from one node to another. To mitigate this I changed the semantics of the protection against multiple nodes with the same address to be stricter - it will now reject the incoming join if a node exists with the same id but a different address. Note that if the existing node doesn't respond to pings (i.e., it's not alive) it will be removed and the new node will be accepted when it tries another join.

Last, and most importantly, this change requires that *all* nodes persist data to disk. This is a change from current behavior where only data & master nodes store local files. This is the main reason for marking this PR as breaking.

Other less important notes:
- DummyTransportAddress is removed as we need a unique network address per node. Use `LocalTransportAddress.buildUnique()` instead.
- I renamed `node.add_lid_to_custom_path` to `node.add_lock_id_to_custom_path` to avoid confusion with the node ID which is now part of the `NodeEnvironment` logic.
- I removed the `version` paramater from `MetaDataStateFormat#write` , it wasn't really used and was just in the way :)
- TribeNodes are special in the sense that they do start multiple sub-nodes (previously known as client nodes). Those sub-nodes do not store local files but derive their ID from the parent node id, so they are generated consistently.
2016-07-04 21:09:25 +02:00
Tanguy Leroux 0e7faf1005 Enable Checkstyle RedundantModifier 2016-07-04 15:22:12 +02:00
Nik Everett c02de9227c Migrate remaining calc aggs to NamedWriteable
Once all of these are migrated we'll be able to remove aggregation's
custom "streams" which function that same as NamedWriteable. It also
allows us to make most of the fields on aggregations final which is
rather nice.
2016-07-04 08:46:00 -04:00
Tanguy Leroux 8c40b2b54e Fix order of modifiers 2016-07-01 16:57:14 +02:00
Tanguy Leroux 93b42b8e69 Enable checkstyle ModifierOrder module 2016-07-01 16:57:14 +02:00
javanna dd781d410a fix line length problems in all classes under o.e.test.rest package 2016-07-01 11:13:10 +02:00
javanna 44dc801e90 [TEST] make JsonPath independent of data format, rename to ObjectPath
The internal representation of the object that JsonPath gives access to is a map. That is independent of the initial input format, which is json but could also be yaml etc.
This commit renames JsonPath to ObjectPath and adds a static method to create an ObjectPath from an XContent
2016-07-01 11:13:10 +02:00
javanna 62462f5d9b [TEST] replace ResponseBodyAssertion with existing MatchAssertion
We introduced a special response_body assertion to test our docs snippets. The match assertion does the same job though and can be reused and adapted where needed. ResponseBodyAssertion contains provides much better and accurate errors though, which can be now utilized in MatchAssertion so that many more REST tests can benefit from readable error messages.

 Each response body gets always stashed and can be retrieved for later evaluations already. Instead of providing the response body as strings that get parsed to json objects separately, then converted to maps as ResponseBodyAssertion did, we parse everything once, the json is part of the yaml test, which is supported. The only downside is that json comments cannot be used, rather yaml comments should be used (// C style vs # ). There were only two docs tests that were using comments in ingest-node.asciidoc where I went ahead and remove the comments which didn't seem that useful anyways.
2016-07-01 11:13:10 +02:00
David Pilato 535157474e Merge branch 'pr/19144-discovery-azure-classic' 2016-06-30 15:44:28 +02:00
Nik Everett e359be7632 Don't inject TransportPercolateAction into RestPercolateAction
Instead use the client. This will help us build the actions more
easily in the future.
2016-06-30 09:36:31 -04:00
David Pilato 527a9c7f48 Deprecate discovery-azure and rename it to discovery-azure-classic
As discussed at https://github.com/elastic/elasticsearch-cloud-azure/issues/91#issuecomment-229113595, we know that the current `discovery-azure` plugin only works with Azure Classic VMs / Services (which is somehow Legacy now).

The proposal here is to rename `discovery-azure` to `discovery-azure-classic` in case some users are using it.
And deprecate it for 5.0.

Closes #19144.
2016-06-30 14:42:40 +02:00
Jason Tedor 2f638b5a23 Keep input time unit when parsing TimeValues
This commit modifies TimeValue parsing to keep the input time unit. This
enables round-trip parsing from instances of String to instances of
TimeValue and vice-versa. With this, this commit removes support for the
unit "w" representing weeks, and also removes support for fractional
values of units (e.g., 0.5s).

Relates #19102
2016-06-27 18:41:18 -04:00
Lee Hinman aec033386e Add the integ test node's configuration directory as a system property 2016-06-27 13:49:10 -06:00
Nik Everett 71b95fb63c Switch analysis from push to pull
Instead of plugins calling `registerTokenizer` to extend the analyzer
they now instead have to implement `AnalysisPlugin` and override
`getTokenizer`. This lines up extending plugins in with extending
scripts. This allows `AnalysisModule` to construct the `AnalysisRegistry`
immediately as part of its constructor which makes testing anslysis
much simpler.

This also moves the default analysis configuration into `AnalysisModule`
which is how search is setup.

Like `ScriptModule`, `AnalysisModule` no longer extends `AbstractModule`.
Instead it is only responsible for building `AnslysisRegistry`. We still
bind `AnalysisRegistry` but we only do so in `Node`. This is means it
is available at module construction time so we slowly remove the need to
bind it in guice.
2016-06-26 07:15:42 -04:00
Jason Tedor 7c87d39f0c Revert slow tests heartbeat
This commit reverts the slow tests heartbeat added in
b6fbd18e09. The heartbeat has not served
its stated purpose of drawing attention to slow tests, and the heartbeat
can kill builds with SELinux enforcing enabled. While the heartbeats can
be disabled via the environment variable PULSE_SERVER, and SELinux
policy files can be changed, since the heartbeats are not accomplishing
their intended purpose they should be removed.

Relates #19071
2016-06-24 14:32:45 -04:00
Jason Tedor 7f10174362 Upgrade JNA to 4.2.2 and remove optionality
This commit upgrades JNA from version 4.1.0 to 4.2.2. Additionally, this
dependency is now non-optional as JNA is dual-licensed with Apache
License 2.0 since JNA 4.0.0.

Relates #19045
2016-06-23 09:21:40 -04:00
javanna 490d9c8cf7 Merge branch 'master' into feature/http_client 2016-06-22 09:50:07 +02:00
Ryan Ernst 21fff51b28 Merge pull request #18955 from rjernst/gradle_req
Build: Require exactly gradle 2.13
2016-06-21 12:24:20 -07:00
javanna 886cb37efb Merge branch 'master' into feature/http_client 2016-06-21 15:53:37 +02:00