Commit Graph

358 Commits

Author SHA1 Message Date
Jason Tedor 52e80a9292 Reenable BWC tests after disabling for backport
This commit reenables the BWC tests after they were disabled for
backporting the change to track global checkpoints of shard copies on
the primary.

Relates #26666
2017-09-18 06:28:50 -04:00
Jason Tedor c238b79cf4 Add global checkpoint tracking on the primary
This commit adds local tracking of the global checkpoints on all shard
copies when a global checkpoint tracker is operating in primary
mode. With this, we relay the global checkpoint on a shard copy back to
the primary shard during replication operations. This serves as another
step towards adding a background sync of the global checkpoint to the
shard copies.

Relates #26666
2017-09-18 06:04:44 -04:00
Boaz Leskes a99803f89a enable bwc testing 2017-09-15 00:20:19 +03:00
Boaz Leskes 1ca0b5e9e4 Introduce a History UUID as a requirement for ops based recovery (#26577)
The new ops based recovery, introduce as part of  #10708, is based on the assumption that all operations below the global checkpoint known to the replica do not need to be synced with the primary. This is based on the guarantee that all ops below it are available on primary and they are equal. Under normal operations this guarantee holds. Sadly, it can be violated when a primary is restored from an old snapshot. At the point the restore primary can miss operations below the replica's global checkpoint, or even worse may have total different operations at the same spot. This PR introduces the notion of a history uuid to be able to capture the difference with the restored primary (in a follow up PR).

The History UUID is generated by a primary when it is first created and is synced to the replicas which are recovered via a file based recovery. The PR adds a requirement to ops based recovery to make sure that the history uuid of the source and the target are equal. Under normal operations, all shard copies will stay with that history uuid for the rest of the index lifetime and thus this is a noop. However, it gives us a place to guarantee we fall back to file base syncing in special events like a restore from snapshot (to be done as a follow up) and when someone calls the truncate translog command which can go wrong when combined with primary recovery (this is done in this PR).

We considered in the past to use the translog uuid for this function (i.e., sync it across copies) and thus avoid adding an extra identifier. This idea was rejected as it removes the ability to verify that a specific translog really belongs to a specific lucene index. We also feel that having a history uuid will serve us well in the future.
2017-09-14 21:25:02 +03:00
Ryan Ernst 8ba4ff3be0 Build: Move javadoc linking to root build.gradle (#26529)
Javadoc linking between projects currently relies on
projectSubstitutions. However, that is an extension variable that is not
part of BuildPlugin. This commit moves the javadoc linking into the root
build.gradle, alongside where projectSubstitutions are defined.
2017-09-11 15:43:34 -07:00
Jason Tedor 53848877c8 Reenable BWC tests after backport
This commit reenables the BWC tests after they were disabled for
backporting a change that had broad BWC implications.
2017-08-31 22:13:50 -04:00
Tim Vernum eb87df9ff9 Allow abort of bulk items before processing (#26434)
Adds support for bulk items to be aborted before they are processed by the TransportShardBulkAction.
This can be used by an ActionFilter to reject a subset of the items in a bulk action without rejecting the whole action (or all the items for a shard).
2017-08-31 21:23:14 +10:00
Ryan Ernst 35a2ee38e1 Build: Add git hashes used as build metadata (#26397)
This commit adds files to the build output called build_metadata which
contain key/value pairs of metadata associated with the build. The first
use of this metadata are the git hashes associated with bwc checkouts.
These metadata files will be picked up by CI intake jobs and stored
along with last-good-commit, and then passed back in throug the
BUILD_METADATA env var on periodic jobs.
2017-08-28 14:10:06 -07:00
Michael Basnight cfd14cd2b8 Revert shading for the low level rest client (#26367)
At current, we do not feel there is enough of a reason to shade the low
level rest client. It caused problems with commons logging and IDE's
during the brief time it was used. We did not know exactly how many
users will need this, and decided that leaving shading out until we
gather more information is best. Users can still shade the jar
themselves. For information and feeback, see issue #26366.

Closes #26328

This reverts commit 3a20922046.
This reverts commit 2c271f0f22.
This reverts commit 9d10dbea39.
This reverts commit e816ef89a2.
2017-08-25 14:13:12 -05:00
Ryan Ernst 661648b3aa Build: Allow build to configure which license/notice to embed in jars (#26373)
We currently add the apache license/notice for elasticsearch to any
plugin that uses our ES plugin gradle plugin. However, each plugin
should be able to use their own license. This commit adds a licenseFile
and noticeFile property to the root of project's using BuildPlugin,
which is added to jar files for that project.
2017-08-24 22:46:30 -07:00
Nik Everett 99ac7beb8e Teach the build about betas and rcs (#26066)
The build was ignoring suffixes like "beta1" and "rc1" on the version numbers which was causing the backwards compatibility packaging tests to fail because they expected to be upgrading from 6.0.0 even though they were actually upgrading from 6.0.0-beta1. This adds the suffixes to the information that the build scrapes from Version.java. It then uses those suffixes when it resolves artifacts build from the bwc branch and for testing.

Closes #26017
2017-08-10 14:30:00 -04:00
Ryan Ernst 072281d5aa Update version to 7.0.0-alpha1 (#25876)
This commit updates the version for master to 7.0.0-alpha1. It also adds
the 6.1 version constant, and fixes many tests, as well as marking some
as awaits fix.

Closes #25893
Closes #25870
2017-08-01 15:47:48 -04:00
Michael Basnight e816ef89a2 Shade external dependencies in the rest client jar
This commit removes all external dependencies from the rest client jar
and shades them in an 'org.elasticsearch.client' package within the jar
using shadowJar gradle plugin. All projects that depended on the
existing jar have been converted to using the 'org.elasticsearch.client'
package prefixes to interact with the rest client.

Closes #25208
2017-07-24 12:55:43 -05:00
Yannick Welsch 49279d26da Reenable BWC tests
after merging #25822 & #25824
2017-07-21 16:36:50 +02:00
Yannick Welsch a2624dfcef Move primary term from ReplicationRequest to ConcreteShardRequest (#25822)
Removes the primary term from the replication request and pushes it into the transport envelope. This makes it possible to remove the term from the ReplicationOperation universe. The primary term that is to be used for a replication operation is now determined in the reroute phase when the node decides to execute a primary action (and validated once the primary action gets to execute). This makes it possible to validate that the primary action was sent to the correct primary shard instance that it was meant to be sent to (currently we only validate primary actions using the allocation id, which can be reused for failed and reallocated primaries).
2017-07-21 15:57:42 +02:00
Luca Cavanna ec66d655b5 Rename client artifacts (#25693)
It was brought up that our current client artifacts have generic names like 'rest' that may cause conflicts with other artifacts.

This commit renames:

- rest -> elasticsearch-rest-client
- sniffer -> elasticsearch-rest-client-sniffer
- rest-high-level -> elasticsearch-rest-high-level-client

A couple of small changes are also preparing the high level client for its first release.

Closes #20248
2017-07-13 09:44:25 +02:00
Boaz Leskes 215bffb08b Enable bwc testing
#25512 & #25511 have been merged
2017-07-08 11:57:22 +02:00
Boaz Leskes b40b002c1d Disable bwc tests in preparation of merging #25512 & #25511 2017-07-08 09:25:43 +02:00
Nik Everett 21b1db2965 Remove assemble from build task when assemble removed
Removes the `assemble` task from the `build` task when we have
removed `assemble` from the project. We removed `assemble` from
projects that aren't published so our releases will be faster. But
That broke CI because CI builds with `gradle precommit build` and,
it turns out, that `build` includes `check` and `assemble`. With
this change CI will only run `check` for projects without an
`assemble`.
2017-06-16 17:19:14 -04:00
Nik Everett 7b358190d6 Remove assemble task when not used for publishing (#25228)
Removes the `assemble` task from projects that are not published.
This should speed up `gradle assemble` by skipping projects that
don't need to be built. Which is useful because `gradle assemble`
is how we cut releases.
2017-06-16 11:46:34 -04:00
Ryan Ernst 106e373412 Build: Add master flag for disabling bwc tests (#25230)
This commit adds a gradle project, set inside the root build.gradle,
which controls all our bwc tests. This allows for seamless (ie no errant
CI failures) backporting of behavior.
2017-06-14 22:01:49 -07:00
Ryan Ernst 160a049930 Build: Move verifyVersions to new branchConsistency task (#25009)
This commit adds a new `branchConsistency` task which will run in CI
once a day, instead of on every commit. This allows `verifyVersions` to
not break immediately once a new version is released in maven.
2017-06-01 10:29:51 -07:00
Nik Everett 6a167a7ba8 Build: improve verifyVersions error message (#25006)
The error message was confusing because it doesn't
include unreleased versions like CURRENT.
2017-06-01 11:55:23 -04:00
Nik Everett bc4df47a76 Rework bwc snapshot projects to build up to two bwc versions (#24870)
Removes the `distribution:bwc` project in favor of
`distribution:bwc-release-snapshot` and
`distribution:bwc-stable-snapshot`.
`distribution:bwc-release-snapshot` builds a snapshot of the
latest release branch (5.4 now) if needed for backwards
compatibility. `distribution:bwc-stable-snapshot` builds a
snapshot of the latest stable branch (5.x now) if needed for
backwards compatibility.
2017-05-29 10:22:32 -04:00
Nik Everett 5da8ce8318 Remove the need for _UNRELEASED suffix in versions (#24798)
Removes the need for the `_UNRELEASED` suffix on versions by detecting if a version should be unreleased or not based on the versions around it. This should make it simpler to automate the task of adding a new version label.
2017-05-26 18:36:32 -04:00
Jason Tedor 95ebdbaddb Add BWC packaging distributions
Some packaging tests depend on snapshot versions of packaging
distributions yet the build does not use a repository that includes such
distributions. While we could add such a repository, a better strategy
is to follow our approach for other BWC tests where we depend on a
locally-compiled archive distribution. This commit adds a local
compilation of packaging artifacts and substitutes these anywhere that
we would otherwise depend on a snapshot of these artifacts.

Relates #24861
2017-05-24 11:55:32 -04:00
Luca Cavanna 747fa721e4 Build: add client jar for aggs-matrix-stats (#24827)
This will be useful for the high level client to add support for the matrix stats aggregation, as we will ship with this jar by default like we do for parent-join-client which is aligned with distributing core with the modules already included.

Relates to #24796
2017-05-23 13:33:54 +02:00
Nik Everett 82d2c7a142 Remove vagrant testing versions (#24754)
Now that we generate the versions list from Versions.java we can
drop the list of versions maintained for vagrant testing. One nice
thing that the vagrant testing did was to check if the list of
versions was out of date. This moves that test to the core
project.
2017-05-18 09:33:13 -04:00
Ryan Ernst 0353bd1fb6 Test: Convert rolling upgrade test to have task per wire compat version (#24758)
This commit changes the rolling upgrade test to create a set of rest
test tasks per wire compat version. The most recent wire compat version
is always tested with the `integTest` task, and all versions can be
tested with `bwcTest`.
2017-05-18 01:14:24 -07:00
Ryan Ernst ff34434bba Build: Extract all ES versions into gradle properties (#24748)
This commit expands the logic for version extraction from Version.java
to include a list of all versions for backcompat purposes. The tests
using bwcVersion are converted to use this list, but those tests
(rolling upgrade and backwards-5.0) are still not randomized; that will
happen in another followup.
2017-05-17 12:58:37 -07:00
Jim Ferenczi 279a18a527 Add parent-join module (#24638)
* Add parent-join module

This change adds a new module named `parent-join`.
The goal of this module is to provide a replacement for the `_parent` field but as a first step this change only moves the `has_child`, `has_parent` queries and the `children` aggregation to this module.
These queries and aggregations are no longer in core but they are deployed by default as a module.

Relates #20257
2017-05-12 15:58:06 +02:00
Ryan Ernst b3466818ac Remove outdated comment 2017-03-29 14:37:32 -07:00
Ryan Ernst cc1addeac2 Build: Find bwc version during build (#23801)
We currently have the last minor version of the previous major hardcoded
in tests like rolling upgrade. This change programatically finds this
during gradle initialization by parsing versions from Version.java.
2017-03-29 12:11:38 -07:00
Ryan Ernst 8c53555b28 Tests: Use local clone build of 5.x with bwc tests (#22946)
The current rest backcompat tests, which run against a mixed cluster of
5.x and 6.0 nodes, depend on snapshot builds of 5.x. However, this has
the potential for inconsistency that results in CI failures, and happens
quite often, whenever some backcompat logic is added to 5.x, but the bwc
test on master fails because the 5.x code has not yet been published as
a snapshot.

This change creates a git clone of the 5.x branch,
builds the zip distribution, and ties that into gradle substitutions for
the 5.x version.
2017-03-23 22:32:13 -07:00
Ryan Ernst 621643a5c3 Build: Only add ASL license to pom for elasticsearch project (#22664)
The extra plugins that may be attached to the elasticsearch build
contain their own license. In the past, the ASL license elasticsearch
uses was avoided by specially checking for the gradle project prefix of
`:x-plugins`. However, since refactoring to the elasticsearch-extra dir
structure, this mechanism was broken. This change fixes the pom license
adding to only be applied to projects that fall under the root project
(ie elasticsearch).
2017-01-17 13:38:32 -08:00
Tim Brooks 864e3128e7 Add task to clean idea build directory. Make cleanIdea task invoke it. 2016-12-22 11:24:13 -06:00
Ryan Ernst a30f649013 Build: Apply license section in poms only to elasticsearch artifacts (#21757) 2016-11-24 00:03:43 -08:00
Tanguy Leroux b9bee8bca3 Remove transport-netty3-client mention (#21650)
Now netty3 is gone, this mention must also be removed.
2016-11-18 11:54:22 +01:00
Ryan Ernst 85b8f29415 Build: Remove old maven deploy support (#20403)
* Build: Remove old maven deploy support

This change removes the old maven deploy that we have in parallel to
maven-publish, and makes maven-publish fully work with publishing to
maven local. Using `gradle publishToMavenLocal` should be used to
publish to .m2.

Note that there is an unfortunate hack that means for
zip artifacts we must first create/publish a dummy pom file, and then
follow that with the real pom file. It would be nice to have the pom
file contains packaging=zip, but maven central then requires sources and
javadocs. But our zips are really just attached artifacts, so we already
set the packaging type to pom for our zip files. This change just works
around a limitation of the underlying maven publishing library which
silently skips attached artifacts when the packaging type is set to pom.

relates #20164
closes #20375

* Remove unnecessary extra spacing
2016-09-19 15:10:41 -07:00
Yannick Welsch 95ae58b531 Set up project substitution for client:transport
This ensures that projects depending on client:transport (such as client:benchmark) properly resolve this dependency.
2016-07-26 17:58:39 +02:00
Jason Tedor 2d1b0587dd Introduce Netty 4
This commit adds transport-netty4, a transport and HTTP implementation
based on Netty 4.

Relates #19526
2016-07-22 22:26:35 -04:00
Ryan Ernst 2fb3cdceff Build: Simplify plugin configuration for rest tests
This change removes the multiple ways that plugins can be added to the
integ test cluster. It also removes the use of the default
configuration, and instead adds a zip configuration to all plugins. This
will enable using project substitutions with plugins, which must be done
with the default configuration.
2016-07-15 14:34:21 -07:00
Nik Everett 0bf447c697 Group client projects under :client
:client ---------> :client:rest
:client-sniffer -> :client:sniffer
:client-test ----> :client:test

This lines the client up with how we do things like modules and
plugins.
2016-06-22 14:26:41 -04:00
javanna c2839c1577 [TEST] Add client-test module and make client tests use randomized runner directly
The lucene-test dependency caused issues with IDEs as they would always load the lucene 5 jar although they shouldn't have, which caused jarhell in es core tests.

If we depend directly on randomized runner we don't have this problem. It is luckily still compatible with java 1.7. This requires though adding a thin module that includes the base test class which can be shared between client and client-sniffer.
2016-06-22 19:37:08 +02:00
javanna ace3a7b146 Merge branch 'master' into feature/http_client 2016-06-15 11:44:46 +02:00
Nik Everett d0e4485d42 Move NamingConventionsCheck into buildSrc
This will let things that don't depend on :test:framework like the
client use it.

Also skip initializing the classes we check because we don't care
about their initialization behavior because we're not executing them.
This makes the naming conventions check pretty close to instant
from a "human eye" perspective.
2016-06-14 18:30:34 -04:00
javanna 044a97c740 move client sniffer to its own project
Create a new subproject called client-sniffer that contains the o.e.client.sniff package. Since it is going to go to a separate jar, due to its additional functionalities and dependency on jackson, it makes sense to have it as a separate project that depends on client. This way we make sure that client doesn't depend on it etc.
2016-06-03 16:01:07 +02:00
javanna 3890842fd4 add project substitution for org.elasticsearch:client so that we can add it as a dep in other projects 2016-06-03 16:01:07 +02:00
Ryan Ernst 9bfa613901 Build: Rework eclipse settings copy so it does not get automatically cleaned
Gradle has "rules" for certain task names, and clean is one of these.
When you run clean, it searches for any tasks named cleanX, and tries to
reverse engineer the X task. For eclipse, this means it finds
cleanEclipse, and internally runs it (but this does not show up as a
dependency of clean in tasks list!!). Since we added .settings as an
additional file to delete with cleanEclipse, this gets deleted when
running just "clean". It doesn't delete the other files because those
have their own clean methods, and dependencies are not followed in this
insanity. This change simply makes a separate task for cleaning eclipse
settings.
2016-05-31 14:20:21 -07:00
javanna 1a298c2e6f Revert "Revert "Update the minimum required version of gradle to 2.13""
This reverts commit 0b965167fd.
All the CI machines have been updated to gradle 2.13, we should be ok upgrading the required minimum version.
2016-05-27 14:40:38 +02:00
javanna 0b965167fd Revert "Update the minimum required version of gradle to 2.13"
This reverts commit 8c6cbf0305. We need to update all our CI machines first, then get this change back in.
2016-05-27 13:16:43 +02:00
javanna 8c6cbf0305 Update the minimum required version of gradle to 2.13
It came out with improvements around idea integration and language levels. This will make it possible to have the upcoming java client as a new project compiled against java 7 and have idea working on the right language level.
2016-05-27 12:26:06 +02:00
Ryan Ernst 6dcfcf0bf1 Build: Remove buildSrc as a project in IDEs
We have have a virtual build-tools project which pulls in all the same
IDE settings we have for other projects, so there is no longer a need
for hacking buildSrc into our IDEs.
2016-05-25 12:08:57 -07:00
Ryan Ernst ceff71fa98 Build: Use -release with java 9 instead of -source/-target
This uses the new -release flag for javac which does not require setting
the bootclasspath as -source/-target did.
2016-05-23 12:51:01 -07:00
Ryan Ernst 7cf758943e Fix git info lookup to use correct dir where git exists 2016-05-21 22:58:55 -07:00
Ryan Ernst 5aea24137c Build: More pom generation improvements
This adds a non empty description to all generated poms, as well as
fixing distributions so they actually have a generated pom.
2016-05-13 16:32:35 -07:00
Ryan Ernst 81a521fcb5 Build: Add developer info to generated pom files 2016-05-13 13:57:08 -07:00
Ryan Ernst e16af604bf Build: Add pom generation to assemble task
In preparation for a unified release process, we need to be able to
generate the pom files independently of trying to actually publish. This
change adds back the maven-publish plugin just for that purpose. The
nexus plugin still exists for now, so that we do not break snapshots,
but that can be removed at a later time once snapshots are happenign
through the unified tools. Note I also changed the dir jars are written
into so that all our artifacts are under build/distributions.
2016-05-05 17:57:44 -07:00
Gabriel Moskovicz 48d26d8886 Fix indentation 2016-04-28 11:08:21 -03:00
Gabriel Moskovicz efabaa7c19 Allow import of all projects in Eclipse on Windows 2016-04-28 11:02:11 -03:00
Nik Everett c966d14f8b [build] Don't resolve the revision hash unless you need it
This fixes the build for folks that build without git installed locally
and should speed up the general case because we aren't trying to resolve
git information when it isn't really needed.
2016-04-15 11:35:05 -04:00
Alexander Reelsen 95579ca95a Build: Allow for file based deploy, sign packages
This allows for a local file based deploy without needed nexus
auth information.

Also signing of packages has been added, either via gradle.properties
or using system properties as a fallback.

The property build.repository allows to configure another endpoint if no
snapshot build is done.

Fix creation of .asc file for tar.gz distribution

Closes #17405
2016-04-15 17:14:05 +02:00
Ryan Ernst ba2abbf37c Build: Exclude gradle and eclipse build dirs from intellij
Intellij has a model of "everything is a source dir unless you say
otherwise". This change fixes the intellij configuration to not think
the gradle or eclipse build dirs are source dirs.
2016-04-14 14:54:52 -07:00
Ryan Ernst f71f0d6010 Revert "Build: Switch to maven-publish plugin"
This reverts commit a90a2b34fc.
2016-03-18 17:22:25 -07:00
Ryan Ernst cf8c9ffabb Always set version, even for x-plugins 2016-03-17 12:21:31 -07:00
Ryan Ernst a90a2b34fc Build: Switch to maven-publish plugin
The build currently uses the old maven support in gradle. This commit
switches to use the newer maven-publish plugin. This will allow future
changes, for example, easily publishing to artifactory.

An additional part of this change makes publishing of build-tools part
of the normal publishing, instead of requiring a separate upload step
from within buildSrc. That also sets us up for a follow up to enable
precomit checks on the buildSrc code itself.
2016-03-15 19:16:37 -07:00
Yannick Welsch b17f4b40ba Check that number of placeholders in log message matches number of parameters 2016-03-11 10:30:59 +01:00
Nik Everett 15a9da4d84 Remove accidental println 2016-02-11 17:21:21 -05:00
Nik Everett 903d4d48ab Merge pull request #16300 from nik9000/eclipse_project_names
Name eclipse projects to gradle path
2016-02-11 12:53:10 -05:00
Nik Everett bb61cdb330 Name eclipse projects to gradle path
This groups like projects together which is nice. It creates two weirdly
named projects:

1. buildSrc - its still just called buildSrc and it doesn't match. I don't
see why we import it into Eclipse anyway. Its groovy and easier to just edit
in vim or whatever.

2. elasticsearch - this is the name of the root project. It's also not
particularly useful to import into eclipse but we've always named it this way
and the name ':' was even more confusing so we just kept the name.
2016-02-10 13:30:18 -05:00
Ryan Ernst b9342b305b Also use separte buildSrc build dir for intellij 2016-02-04 15:58:05 -08:00
Ryan Ernst 2fef5dee96 Build: use separate build directories for eclipse and intellij
This fixes the problem where doing a gradle clean from the command line
completely breaks the IDE.
2016-02-04 15:44:48 -08:00
Ryan Ernst 9f1dfdbaea Build: Add AntTask to simplify controlling logging when running ant from gradle
This new task allows setting code, similar to a doLast or doFirst,
except it is specifically geared at running ant (and thus called doAnt).
It adjusts the ant logging while running the ant so that the log
level/behavior can be tweaked, and automatically buffers based on gradle
logging level, and dumps the ant output upon failure.
2015-12-18 11:45:52 -08:00
Robert Muir 7e53076112 Merge pull request #15545 from rmuir/rat
add gradle licenseHeaders to precommit
2015-12-18 13:27:37 -05:00
Robert Muir 94d6b22137 add gradle licenseHeaders to precommit
This is a port of the logic from apache lucene that uses Rat
2015-12-18 11:43:58 -05:00
Yannick Welsch 887789d1a9 Gradle idea plugin does not properly mark resources directories 2015-12-18 12:28:07 +01:00
Ryan Ernst 6a99796b02 Build: Move test framework under a "test" top level dir
This allows adding more test projects, eg integ test fixtures that will
be coming soon.
2015-12-17 21:25:06 -08:00
Robert Muir ee79d46583 Add gradle thirdPartyAudit to precommit tasks 2015-12-16 16:38:16 -05:00
Costin Leau 7bca97bba6 HDFS Snapshot/Restore plugin
Migrated from ES-Hadoop. Contains several improvements regarding:

* Security
Takes advantage of the pluggable security in ES 2.2 and uses that in order
to grant the necessary permissions to the Hadoop libs. It relies on a
dedicated DomainCombiner to grant permissions only when needed only to the
libraries installed in the plugin folder
Add security checks for SpecialPermission/scripting and provides out of
the box permissions for the latest Hadoop 1.x (1.2.1) and 2.x (2.7.1)

* Testing
Uses a customized Local FS to perform actual integration testing of the
Hadoop stack (and thus to make sure the proper permissions and ACC blocks
are in place) however without requiring extra permissions for testing.
If needed, a MiniDFS cluster is provided (though it requires extra
permissions to bind ports)
Provides a RestIT test

* Build system
Picks the build system used in ES (still Gradle)
2015-12-14 21:50:09 +02:00
Ryan Ernst fa1c708ccd Merge branch 'master' into standalone_tests_intellij 2015-12-07 16:01:32 -08:00
Ryan Ernst caf77f7eea Add integ-test-zip distribution 2015-12-03 14:52:51 -08:00
Ryan Ernst 30fe74b5df Build: Setup standalone tests to compile in intellij
This adds the standalone tests so they will compile (and thus can be
modified with import completion) within IntelliJ. It also explicitly
sets up buildSrc as a module.

Note that this does *not* mean eg evil-tests can be run from intellij.
These are special tests that require special settings (eg disabling
security manager). They need to be run from the command line.

closes #15075
2015-11-28 16:57:09 -08:00
Nik Everett 45e71329ee [build] Fix deb and rpm tests
Gradle is causing so much trouble here! Its too cute for its own good.
2015-11-28 19:07:06 -05:00
Ryan Ernst 3071de2d33 Build: Enforce that gradle idea was run before importing in intellij
Due to how intellij imports gradle projects, the tasks which setup
resources are not run. This means we must be sure to run gradle idea
from the command line before importing into elasticsearch. This change
adds a simple marker file to indicate we have run from the command line
before importing. It won't help for new projects that add plugin
metadata, but it will at least make sure the initial project is set up
correctly.
2015-11-22 16:07:39 -08:00
Ryan Ernst 6b2f3a9ad2 Fix bad merge 2015-11-22 08:51:34 -08:00
Ryan Ernst b8e462b934 Build: Change project attachment into special extra-plugins dir
Currently we use the "gradle project attachment plugin" to support
building elasticsearch as part of another project. However, this plugin
has a number of issues, a large part of which is requiring consistent
use of the projectsPrefix.

This change removes projectsPrefix, and adds support for a special
extra-plugins directory in the root of elasticsearch. Any projects
checked out within this directory will be automatically added to
elasticsearch.
2015-11-22 08:44:33 -08:00
Ryan Ernst f9351b3e0e Build: Enforce similar tasks run for dependencies first
Gradle ensures task dependencies are executed in the correct order.
However, project dependencies only build what is needed for the
dependency. This means the order of higher level tasks are not
guaranteed. This change adds task ordering between test and integTest
for a project and its dependencies.
2015-11-22 01:17:49 -08:00
Ryan Ernst 88ee2e23a6 Merge branch 'master' into javadocs 2015-11-20 12:20:59 -08:00
Ryan Ernst 167ccefba6 Add reference to gradle discussion on javadoc options 2015-11-20 12:18:30 -08:00
Ryan Ernst 28cba31447 Build: Remove cleaning of buildSrc on clean of ES, it doens't work as intended 2015-11-20 09:47:48 -08:00
Ryan Ernst 1952b06cfe Merge pull request #14854 from rjernst/trick_ant
Fix integTest output if the elasticsearch script fails
2015-11-19 12:03:03 -08:00
Ryan Ernst eb09d0b680 Add a comment explaining the -quiet parameter for adding javadoc options 2015-11-19 11:15:00 -08:00
Ryan Ernst c3cb1fd08c Merge branch 'master' into javadocs 2015-11-19 10:43:43 -08:00
Jason Tedor cac281424c Stop Gradle from destroying IntelliJ project settings
This commit stops Gradle from destroying the IntelliJ project settings
when a Gradle refresh is executed in IntelliJ.

Closes #14809
2015-11-19 07:42:19 -05:00
Ryan Ernst d44dbd4757 Build: Fix integTest output if the elasticsearch script fails
If there is a failure in the elasticsearch start script, we currently
completely lose the failure. This is due to how spawning works with ant.
This change avoids the issue by introducing an intermediate script,
built dynamically before running ant exec, which runs elasticsearch and
redirects the output to a log file. This essentially causes us to run
elasticsearch in the foreground and capture the output, but at the same
time keep a running script which ant can pump streams from (which will
always be empty).
2015-11-18 23:30:33 -08:00
Ryan Ernst 0e8d605e2d Build: Add --debug-jvm option to run and integTest tasks
Sometimes when running elasticsearch, it is useful to attach a remote
debugger. This change adds a --debug-jvm option (the same name gradle
uses for its tests debug option), which adds java agent config for a
remote debugger. The configuration is set to hava java suspend until the
remove debugger is attached.

closes #14772
2015-11-17 16:29:01 -08:00
Ryan Ernst 50b48e7d8c Build: Make run command work within plugins
We recently got a run command with gradle, but it is sometimes useful to
run ES with a specific plugin. This is a start, by making each esplugin
have a run command which installs the plugin and runs elasticsearch in
the foreground.
2015-11-16 17:29:01 -08:00
Ryan Ernst ee16614911 Build: Also clean buildSrc when cleaning
This forces buildSrc to be cleaned when running gradle clean. It's what
someone would expect, even though buildSrc is a pseudo-independent
project.
2015-11-16 12:41:21 -08:00
Ryan Ernst 4b17492456 Build: Add javadocs jars
This change adds javadoc jars to core, test-framework and plugins. There
were a couple issues which javadoc found, but doclint did not already
find.
2015-11-15 01:44:42 -08:00
Ryan Ernst 837c593ec2 Build: Effectively remove transitive deps from generated maven poms
With gradle, deploying to maven means first generating poms. These are
filled in based on dependencies of the project. Recently, we started
disallowing transitive dependencies. However, this configuration does
not translate to maven poms because maven has no concept of excluding
all transitive dependencies.

This change adds exclusions for each of the transitive deps of each
dependency being added to the maven pom. It does so by creating dummy
configurations for each direct dependency (which does not have
transitive deps excluded), so that we can iterate the transitive deps
when building the pom.

Note, this should be simpler (just modifying maven's pom model), but
gradle tries to hide that from their api, causing us to need to
manipulate the xml directly.
https://discuss.gradle.org/t/modifying-maven-pom-generation-to-add-excludes/12744
2015-11-13 14:10:17 -08:00
Ryan Ernst f327beac49 Build: Enforce JAVA_HOME is set
If we use JAVA_HOME consistently for tests, we can run tests with a
different version of java than gradle runs with. For example, this
enables running tests with jigsaw, but building with java 8. The only
caveat is intellij does not set JAVA_HOME. This change enforces
JAVA_HOME is set, but ignores for intellij.
2015-11-12 10:22:56 -08:00
Ryan Ernst 505a4d9e09 Build: Simplify where min java version is specified
This moves the min java version used by elasticsearch to one place, a
constant in BuildPlugin. For me on java 9, this fixed my jar to have the
correct target/source versions.

closes #14702
2015-11-11 23:09:57 -08:00
Ryan Ernst 4b5f87cb7d Build: Remove transitive dependencies
Transitive dependencies can be confusing and hard to deal with when
conflicts arise between them. This change removes transitive
dependencies from elasticsearch, and forces any dependency conflicts to
be resolved manually, instead of automatically by gradle.

closes #14627
2015-11-10 15:01:41 -08:00
Ryan Ernst 7a6155e12f Build: Consolidate dependencies specified in multiple places
Some dependencies must be specified in a couple places in the build.
e.g. randomized runner is specified both in buildSrc (for the gradle
wrapper plugin), as well as in the test-framework.

This change creates buildSrc/versions.properties which acts similar to
the set of shared version properties we used to have in the maven parent
pom.
2015-11-09 23:59:56 -08:00
andrejserafim 2bd353d72d replacing run.bat and run.sh with gradle run
run.sh and run.bat were calling out to the old maven build system.
This is no longer in place, so we've created new gradle tasks to
start an elasticsearch node from the current codebase.

fixed #14423
2015-11-08 17:07:19 +00:00
Robert Muir 866a514785 add groovy build code (buildSrc/) in 'gradle eclipse'
This makes it a groovy project that works in eclipse.
You will have to install a plugin for groovy language support
(I used a snapshot build from https://github.com/groovy/groovy-eclipse/wiki)
2015-11-07 15:14:00 -05:00
Ryan Ernst 548430072f Build: Fix eclipse generation to add a core-tests projects
Eclipse does not have the ability to differentiate test dependencies
from main dependencies. This causes what looks like a circular
dependency through test-framework. This change sets up an additional
core-tests project for eclipse only, which removes this problem.
2015-11-06 11:41:18 -08:00
Robert Muir 1a0a910d1e Merge pull request #14526 from rjernst/standalone_test
Add qa/evil-tests
2015-11-04 13:49:55 -05:00
Ryan Ernst a534ace896 Fix standalone test to work with check, and comment out mavenLocal 2015-11-04 10:28:31 -08:00
Robert Muir 3d8b4dae33 remove the ability for code to change file permissions, this was
only needed for pluginmanager's test, and pluginmanager doesn't
even run with securitymanager yet.
2015-11-04 00:31:15 -05:00
Ryan Ernst 3e4f87d4d6 Fix tar issues and build distro dependency for integ tests 2015-11-02 22:19:29 -08:00
Ryan Ernst b7f8e5c1df Build: Make idea/eclipse project generation build generated resources for plugins
This adds a generated-resources dir that the plugin properties are
generated into. This must be outside of the build dir, since intellij
has build as "excluded".

closes #14392
2015-11-02 11:35:02 -08:00
Robert Muir 4587a94fcf generate complete eclipse configuration from 'gradle eclipse' 2015-11-02 10:44:51 -05:00
Robert Muir 9056ebb20d gradle cleanEclipse should completely nuke .settings.
Today this will only remove `.settings/org.eclipse.jdt.core.prefs`,
leaving a bunch of stale eclipse configuration everywhere.
2015-11-02 09:39:14 -05:00
Robert Muir 3961c9b58b fix eclipse (again) 2015-10-31 02:39:35 -04:00
Ryan Ernst 0dca49fae4 fix eclipse config for qa projects 2015-10-30 21:32:16 -07:00
Robert Muir 373dee4b5e make gradle eclipse always run cleanEclipse
Otherwise the 'merging' gets really trappy. it basically never works without a clean.

See 38.4.1.1. Disabling merging with a complete rewrite:
https://docs.gradle.org/current/userguide/eclipse_plugin.html
2015-10-30 23:00:05 -04:00
Robert Muir 10d4f17aa8 make eclipse work without specifying a sysprop 2015-10-30 14:21:06 -04:00
Robert Muir 9bdc10802e Get eclipse working with gradle 2015-10-30 11:39:31 -04:00
Ryan Ernst 561436eba3 Build: Add back manifest info to jars
closes #14365
2015-10-29 13:59:52 -07:00
Ryan Ernst c86100f636 Switch build system to Gradle
See #13930
2015-10-29 11:40:19 -07:00
Shay Banon bf0caafa04 remove old modules and gradle 2011-12-06 01:01:04 +02:00
Shay Banon 49a97f6fcf move to 0.19.0 snap 2011-10-26 21:04:10 +02:00
Shay Banon 3544b07330 release 0.18.0 2011-10-26 20:58:24 +02:00
Nicolas Huray d7f7f77d81 Build ElasticSearch as Debian package 2011-09-28 01:08:35 +03:00
kimchy d4e18f5f45 move to 0.18 release 2011-07-19 01:34:55 +03:00
kimchy eacdacb410 0.17 release 2011-07-19 01:32:43 +03:00
kimchy 2dd6075171 maybe we will do this later... 2011-07-15 10:19:36 +03:00
kimchy 1f70be7de2 fix testng deps 2011-07-08 00:56:52 +03:00
Igor Motov e5f2dfbf0e Add missing testng dependencies 2011-07-08 00:50:22 +03:00
kimchy 420ed9daa6 upgrade to testng 6.1.1 2011-07-06 23:15:03 +03:00
kimchy 67dc32100d upgrade gradle to 1.0 m3 2011-06-13 14:38:00 +03:00
kimchy 84c5d07726 move to 0.17.0-SNAPSHOT 2011-04-23 23:56:12 +03:00
kimchy 0ebd6db7b9 release 0.16.0 2011-04-23 23:48:15 +03:00
kimchy bcf8bc5993 update to next 0.16.0 version 2011-02-18 19:52:17 +02:00
kimchy dac2a888fb update to version 0.15 2011-02-18 19:49:30 +02:00
kimchy 0e595532f7 add maxMem to the heapsize when executing tests 2011-02-09 20:45:21 +02:00
kimchy 00cba66eb8 move to use mvel 2.1.RC1 2011-02-03 11:29:24 +02:00
kimchy c7add5f377 upgrade to mvel 2.1 snapshot to gain performance improvements 2011-01-31 21:41:23 +02:00
kimchy 2820988171 upgrade to gradle 0.9.2 2011-01-30 16:05:05 +02:00
kimchy 9ff8006d37 have compilation and javadoc use UTF8 encoding 2011-01-20 14:24:12 +02:00
kimchy 84fa13a5e0 upgrade to gradle 0.9.1 2011-01-03 11:45:59 +02:00
kimchy 8e8a3a894f upgrade testing framework jars, improve build scripts to share the testing jars 2010-12-30 16:08:38 +02:00
kimchy bb4017769e update to 0.15.0 SNAPSHOT 2010-12-27 22:25:34 +02:00
kimchy 956e775a4e release 0.14 2010-12-27 22:23:18 +02:00
kimchy ec943b5160 fix urlRoot for gradle to point to the 0.9 dist 2010-12-20 17:58:16 +02:00
kimchy 09bc3b90e7 upgrade to gradle 0.9 2010-12-19 19:53:50 +02:00
kimchy 29c7fc39ba upgrade to gradle 0.9 rc3 2010-12-19 05:51:15 +02:00
kimchy 5d43e6ff30 update to 0.14.0 SNAPSHOT 2010-11-18 17:00:18 +02:00
kimchy 7ec6e00b6f release 0.13 2010-11-18 16:40:21 +02:00
kimchy 9bddd6770c Add a tar.gz distribution file as well as the zip one 2010-11-06 01:28:10 +02:00
kimchy be96e9495a upgrade to gradle 0.9 rc2 2010-10-30 19:57:50 +02:00
kimchy 96a1ad6335 When on linux, force it not to swap elasticsearch process (jvm), closes #464. 2010-10-30 19:40:24 +02:00
kimchy c38f07a713 update to 0.13.0 SNAPSHOT 2010-10-18 23:21:29 +02:00
kimchy 27ee635933 update to release 0.12 2010-10-18 23:17:34 +02:00
kimchy 64662d8449 update to 0.12.0 SNAPSHOT version 2010-09-28 22:20:22 +02:00
kimchy 67c205722d release 0.11 2010-09-28 22:14:34 +02:00
kimchy 6a79a16e5b move to 0.11 snapshot 2010-08-27 17:48:51 +03:00
kimchy 153523cf7d update to version 0.10.0 2010-08-27 17:28:27 +03:00
kimchy 80c71350cd change from 0.9.1 to version 0.10.0 2010-08-24 01:29:17 +03:00
kimchy b78597934f expose the maven params in the build files as sys envs 2010-08-20 17:50:04 +03:00
kimchy c7f765d025 add (commented) playing with trying to execute from gradle 2010-08-16 12:56:45 +03:00
kimchy 12bacbbc38 upgrade to gradle 0.9-rc-1 2010-08-16 12:43:14 +03:00
kimchy cb92d22e84 move to 0.9.1 snapshot release 2010-07-26 23:05:02 +03:00
kimchy 2cded30f4c release 0.9.0 2010-07-26 23:02:25 +03:00
kimchy b1c0a78c8e add snapshot repo 2010-07-22 23:33:38 +03:00
kimchy 97b5b41522 enable eager cache eviction based on readers, requires a nasty hack to Lucene, basically overriding two classes to hook into when a reader actually gets closed (and only enabled when elasticsearch is before lucene in the classpath, which we take for in the startup scripts) 2010-07-07 17:15:26 +03:00
kimchy 0d7cebbaf0 upgrade to async http client 1.0.0 2010-06-23 21:51:29 +03:00
kimchy 636bfe8466 big refactoring thanks to proper jarjar built from source with asm 3.3 allowing to jarjar guice and others, includes package relocations 2010-06-15 16:51:38 +03:00
kimchy 751c56f31e Query DSL: custom score (scripted) query, closes #220. 2010-06-14 03:15:23 +03:00
kimchy 288aa36a39 upgrade to gradle 0.9 preview3 2010-06-11 01:02:27 +03:00
kimchy 8cc2638be8 update to 0.9.0 SNAPSHOT 2010-05-27 19:30:48 +03:00
kimchy 5be8ef9fd2 release 0.8.0 2010-05-27 18:59:51 +03:00
kimchy b7d11f1303 Gateway Storage: Improve it to support non breaking changes in the future, closes #191. 2010-05-24 18:27:48 +03:00
kimchy 6185e439c7 next version will be 0.7.2 2010-05-18 23:37:46 +03:00
kimchy d3881947ce change to 0.8.0-SNAPSHOT 2010-05-17 01:55:14 +03:00
kimchy bb3a3785f3 releasing version 0.7.1 2010-05-17 01:38:50 +03:00
kimchy be3c4b0196 version 0.7.1 will be released soon, not 0.8.0 2010-05-16 17:56:24 +03:00
kimchy d22ffa8c21 move to 0.8.0 SNAPSHOT 2010-05-14 23:29:04 +03:00
kimchy 797cb66dbe release 0.7.0 2010-05-14 23:15:53 +03:00
kimchy 2642eca9af fix the jboss repo 2010-05-14 18:23:10 +03:00
kimchy f6a1a2fefb add back old jboss repo for jarjar 2010-05-13 21:26:42 +03:00
kimchy f6777f1e03 jarjar netty as well, so it won't be exposed to users using elasticsearch embedded 2010-05-13 14:41:50 +03:00
kimchy ece1395b57 update repository 2010-05-12 17:43:34 +03:00
kimchy f6509930c7 upgrade to netty 3.2.0 CR1 2010-05-12 04:03:20 +03:00
kimchy 371bfebd16 initial support for process monitoring using sigar 2010-05-06 21:07:24 +03:00
kimchy b0e1c58474 Add online plugin repository and a 'plugin' command to download them, closes #157 2010-05-03 10:49:50 +03:00
kimchy 9651789870 jarjar snakeyaml and remove it from our sources now that it is in maven repo 2010-05-02 18:01:58 +03:00
kimchy 97958c3a66 Service Wrapper: A Java Service Wrapper integration, closes #154. 2010-05-01 01:42:30 +03:00
kimchy 70a2a3ef52 jarjar jackson and joda 2010-04-30 19:00:17 +03:00
kimchy dfd002bf98 add custom elasticsearch repo on google code 2010-04-27 14:47:32 +03:00
kimchy 10f0eaad68 refactor plugins into sub groups 2010-04-25 23:01:00 +03:00
kimchy bda476eee8 Remove dependency on slf4j for logging, create own internal abstraction, closes #146. 2010-04-21 00:29:42 +03:00
kimchy cb7e92b0f8 start work on the groovy plugin 2010-04-14 08:34:47 +03:00
kimchy f10a2d428d move to 0.7.0 2010-04-09 18:19:37 +03:00
kimchy 9549b9c2d3 fix wrong handling of doc ids to load when request is not serialized 2010-04-09 17:06:09 +03:00
kimchy dad0686fa9 fix chmod of elasticsearch in dist zip 2010-04-02 20:39:27 +03:00
kimchy 6ed23744fc upgrade to gradle 0.9 preview 1 2010-04-02 15:47:27 +03:00
kimchy 691a336b9a bring out exploded 2010-03-29 13:53:09 +03:00
kimchy 7c402507fe bring back the exploded plugin 2010-03-29 13:52:36 +03:00