Commit Graph

127 Commits

Author SHA1 Message Date
Michael McCandless 868778c5b6 Core: improve build_release.py
Don't insist on log file removal until after usage is printed.

Some simple Python code improvements (x.find(y) != -1 --> y in x)

Make sure the git area is "clean" (has no unpushed changes, has pulled
all changes, has no untracked files)

Add label color detail when creating next github version label.

Closes #7913
2014-10-02 06:31:45 -04:00
Ryan Ernst df22e54baf Move forbidden api signature files to dev-tools.
This avoids the files showing up in the binary release, since .txt files
are copied.

closes #7917
closes #7921
2014-09-29 15:27:43 -07:00
mikemccand 997b94b427 fix typo: sonartype -> Sonatype 2014-09-29 04:12:08 -04:00
Michael McCandless 5e9e2cf50c Core: try again to upgrade to Lucene 4.10.1-snapshot 2014-09-24 13:48:49 -04:00
Michael McCandless ab3be76644 Revert Lucene upgrade 2014-09-24 13:25:55 -04:00
Michael McCandless 15c75b1967 Core: upgrade to Lucene 4.10.1 snapshot
Lucene will soon release official 4.10.1, but by upgrading sooner we can 1) sidestep the false failures due to the 1.8.0_20 JVM hotspot bug (has caused a number of false failures in recent Jenkins tests), 2) make sure none of the Lucene changes in 4.10.1 are problematic.

Closes #7844
2014-09-24 13:13:07 -04:00
Ryan Ernst 3397908454 Tools: Add script to grab ES version for BWC tests.
closes #7653
2014-09-15 09:21:55 -07:00
Clinton Gormley da963ee1f7 Added support for github access tokens to es_release_notes 2014-09-12 10:59:11 +02:00
Clinton Gormley ea94535557 Fixed bug in es_release_notes script 2014-09-12 10:38:58 +02:00
Clinton Gormley 1455988d91 Changed dev-tools/es_release_notes to:
* remove markdown support
* prefer PRs over issues
* include related issues in the list
* always use section headers if there is a header
2014-09-11 16:46:41 +02:00
Simon Willnauer 64dce61233 [BUILD] Randomly enable AggressiveOpts in builds 2014-09-09 21:58:37 +02:00
Simon Willnauer a9f42e7176 [BUILD] Randomize JVM heapsize between 512 and 1024m 2014-08-22 09:40:09 +02:00
Alexander Reelsen a207f8c7b1 Build: Changed license header definition file xml encoding to UTF-8 2014-08-18 10:55:32 +02:00
Simon Willnauer dc696e6fc8 [TEST] Pick random version from backwards directory
The upgrade tests required to specify the lower and the upper version
that it should run against. This commit adds support for random picks
if either the lower or the upper or both are not specified.
2014-07-29 14:29:56 +02:00
Simon Willnauer 7343ae1911 [TEST] Add basic upgrade test script
This commit adds a basic full cluster restart upgrade script that tests
upgrades of elasticsearch across major version. In contrast to the BWC
tests we are running using the java test framework this test uses a python
script as well as the REST APIs to ensure upgrades across major version work and
indices are compatible.

The upgrade test starts 2 or more nodes of an old elasticserach version, indexes
a random number of documents into the running nodes and executes a full cluster restart.
After the nodes are recovered a small set of basic checks are executed to ensure all
documents are still searchable and field data can be loaded etc.

Closes #7074
2014-07-29 13:00:59 +02:00
Clinton Gormley e543a4dc60 Updated es_release_notes script to replace backticks with code tags 2014-07-28 16:32:15 +02:00
Simon Willnauer 0a1701d416 [BUILD] skip bwc version check if directory doesn't exists or is not a directory 2014-07-23 14:54:28 +02:00
Simon Willnauer b923f9556f [BUILD] Add BWC tests to release script
This commit adds the ability to run bwc tests during the release
process to ensure the current release is backwards compatible with
the latest installed previous version. Installed means available
in the configured bwc test path.

Closes #6953
2014-07-22 11:11:44 +02:00
Bill Hwang bdf77fec65 [BUILD] Randomization script refactoring
1) Refactored randomization logic to the top
2) Add logics to support window platforms
2014-07-16 14:03:59 -07:00
Lee Hinman c70f6d0171 Add Groovy as a scripting language, add sandboxing for Groovy
Sandboxes the groovy scripting language with multiple configurable
whitelists:

`script.groovy.sandbox.receiver_whitelist`: comma-separated list of string
classes for objects that may have methods invoked.
`script.groovy.sandbox.package_whitelist`: comma-separated list of
packages under which new objects may be constructed.
`script.groovy.sandbox.class_whitelist` comma-separated list of classes
that are allowed to be constructed.

As well as a method blacklist:

`script.groovy.sandbox.method_blacklist`: comma-separated list of
methods that are never allowed to be invoked, regardless of target
object.

The sandbox can be entirely disabled by setting:

`script.groovy.sandbox.enabled: false`
2014-06-20 10:20:16 +02:00
Simon Willnauer 5575ba1a12 [BUILD] Check for tabs and nocommits in the code on validate
This commit adds checks for nocommit and tabs in the source code.
The task is executed during the validate phase and can be disabled via
`-Dvalidate.skip`
2014-06-12 11:11:23 +02:00
Simon Willnauer 93552fe8f4 Revert "[BUILD] Calculate the full path ahead of time"
Revert "[BUILD] Promote artifacts from strings to their own type"

This reverts commit dcd4ba0654eb6780235718092969c2f9e6b38775.
This reverts commit 00d7eb3c0a6eefdb5947d07b18cf071ba538d696.
2014-06-03 16:27:10 +02:00
Simon Willnauer a4de19efdd [BUILD] Calculate the full path ahead of time 2014-06-03 16:11:31 +02:00
Andrew Raines fc9c6e33c3 [BUILD] Promote artifacts from strings to their own type 2014-06-02 13:23:08 -05:00
Simon Willnauer 5b13ef55f5 [BUILD] Check for nocommit in source files before building release 2014-06-02 19:29:42 +02:00
Simon Willnauer 15515f0f05 [TOOLS] Added simple S3 Download tool based on boto 2014-06-02 13:35:21 +02:00
Simon Willnauer 60ba9d1e69 [BUILD] pass java exe to plugin install command 2014-05-22 11:44:27 +02:00
Alexander Reelsen 814451f499 [BUILD] Fixed release notes script to work with github API again 2014-05-21 11:14:28 +02:00
javanna 4331021361 enable dynamic scripts in release node, otherwise REST tests will fail
Relates to #5853
2014-05-10 11:16:25 +02:00
javanna 6678da8c28 [TEST] randomly added node.bench=true to client node in test cluster and re-enabled REST benchmark tests based on number of bench nodes available
In our REST tests we already have support for features and skip sections that allow to skip tests if a feature is not supported.
We can then add a skip section based on the benchmark feature to the benchmark tests and execute them only when they are supported, knowing that they need at least a node with node.bench settings within the cluster. We can check that this requirement is met by calling the nodes info api.

This way we can dynamically decide whether to execute those tests or not and we don't need to have a node.bench around all the time. In fact, given that the REST tests use the GLOBAL cluster, we want to be able to randomize settings as much as possible and run tests against default settings as well. Also, this mechanism can be easily supported by the external cluster implementation that is used during the release process.

Introduced ability to disable benchmark nodes which is needed by BenchmarkNegativeTest.
2014-05-09 23:36:00 +02:00
Simon Willnauer eaf04fc828 [BUILD] Fail release if test have AwaitsFix annotations
Note this commit upgrades Forbidden APIs to version 1.5

Closes #5807
2014-04-17 16:45:46 +02:00
Simon Willnauer 27d4d76769 [BUILD] remove leftover print statement 2014-04-14 22:23:10 +02:00
Simon Willnauer 11bf13c363 Check for no open issues before build release 2014-04-14 18:54:04 +02:00
javanna 1ec4f8f04b [TEST] Replaced RestTestSuiteRunner with parametrized test that uses RandomizedRunner directly
ElasticsearchRestTests extends now ElasticsearchIntegrationTest and makes use of our ordinary test infrastructure, in particular all randomized aspects now come for free instead of having to maintain a separate (custom) tests runner

We previously parsed only the tests that needed to be run given the version of the cluster the tests are running against. This doesn't happen anymore as it didn't buy much and it would be harder to support as the tests get now parsed before the test cluster gets started. Thus all the tests are now parsed regardless of their skip sections, afterwards the ones that don't need to be run will be skipped through assume directives.

Fixed REST tests that rely on a specific number of shards as this change introduces also random number of shards and replicas (through randomIndexTemplate)

Closes #5654
2014-04-07 17:08:05 +02:00
Bill Hwang fac4da822d [BUILD] Switch 1x to JDK7 download
Changed download URL for 1x build to point to JDK7 latest
2014-03-31 19:28:27 -07:00
Simon Willnauer 13109d3547 [Build] splitted release commits into version and feature flag commits
Elasticsearch is release from release-branches but the modifications
to the documentation must be cherry-picked into the current development
branch. To make this easier this commit splits the commits of the
Version and the documenation into seperate commits.
2014-03-25 09:16:34 +01:00
Bill Hwang 703f374a8a Add 1.1 branch url
Added S3 access URL for elasticsearch 1.1 snapshot
2014-03-24 10:16:41 -07:00
Bill Hwang 2e56253293 Added static analysis profile to pom.xml
Added pmd, findbug as well as site generation logic to top pom.xml file
Created customized pmd ruleset
2014-03-13 12:23:07 -07:00
Simon Willnauer 8ceb98752d Move master to Java 1.7
Closes #5267
2014-02-27 15:12:02 +01:00
Bill Hwang 7a48d079d3 Moved es download master url from JDK7 to JDK6 2014-02-26 14:18:44 -08:00
Alexander Reelsen d4c506ec65 Added script to extract release notes
Just call ./es_release_notes.pl <issue-tag> to get all release notes.

By default html output is returned, but you can switch to markdown by calling
./es_release_notes.pl <issue-tag> markdown
2014-02-19 13:43:40 +01:00
David Pilato a5001440bf Fix typo 2014-02-17 18:46:51 +01:00
mrsolo aadcfa7b51 Updated URLs to include 1.x and 1.0 snapshots
This file is used by elasticsearch client jerkin projects
2014-01-28 10:59:50 -08:00
mrsolo 9a37922e57 fix if hook failure when running under jenkins
Local mode modification done previously faulty.  env[‘WORKSPACE’ is not
the sufficient discriminator to see if script is running under Jenkins.
  This fails on the Jenkins parent jobs since those type of jobs don’t
have WORKSPACE set.
2014-01-24 15:59:48 -08:00
Simon Willnauer 25b49dd50b Install SecurityManager inside ElasticsearchTestCase for easier randomization
We currently run always with SecurityManager installed. To make sure we
work also without we should randomly swap it out ie. run without the
security manager.
2014-01-24 22:18:05 +01:00
mrsolo c6769bcf7a modify how local mode is run
1) create dummy directory structure with JDK6 and JDK7 when running
local mode
2) move prop.txt from ‘/var/tmp’ to ENV[‘PWD’] when running local mode
2014-01-17 12:52:22 -08:00
Simon Willnauer 7f51fbc5ab Add SecurityManger / policy when running tests.
This commit adds a security manager to the test JVMs
that prevents mainly writing files outside of the JVMs
current test directory.
2014-01-17 15:15:10 +01:00
b.hwang 471853c649 Merge pull request #4714 from mrsolo/features/thirdparty
third party library license information extractor for logstash
2014-01-16 09:38:38 -08:00
b.hwang 52e640dbd9 Merge pull request #4747 from mrsolo/features/test_randomization
Add randomization logic for assert.  Tests will have 10% chance running with assertion disabled.
2014-01-16 09:35:51 -08:00
Alexander Reelsen bcfba8bc46 Added note to release script to announce on ML 2014-01-16 09:43:56 +01:00
mrsolo 661e9b91cf assert randomization
add randomization to disable assert, 10% chance that test will run with
tests.assertion.disabled=org.elasticsearch
2014-01-15 16:27:46 -08:00
mrsolo 1dd218ca71 add url, email 2014-01-14 14:46:22 -08:00
Simon Willnauer 99eae50f9c run mvn clean before smoketesting 2014-01-14 17:00:50 +01:00
Simon Willnauer 6829b67ff0 Fix Release tool to run smoke tests off tags
this commit allows to run the release tool for smoke
testing without being on the actually released branch.
This commit also added a list of plugins that will be installed
for smoke testing to see if the plugin startup mechanism works
correctly.
2014-01-14 16:31:25 +01:00
mrsolo 3e28f417e7 license extractor
extract dependency license information
2014-01-13 16:04:31 -08:00
mrsolo 2df42e4460 Added licene-maven-plugin to build
The plugin adds basic checks for license headers. At
this stage we only enable checks for java source files.
Other files are omitted at this point.
2014-01-10 22:04:48 +01:00
Simon Willnauer ed7daada2d Open files with encoding='utf-8' in release script 2014-01-10 10:14:08 +01:00
mrsolo 3cffe334ba change url to latest snapshops 2014-01-07 12:18:19 -08:00
Simon Willnauer 10ec2e948a Fix ASL Header in source files to reflect s/ElasticSearch/Elasticsearch
This commit also removes the license to Shay Banon in favor of soley
Elasticsearch. Thanks Shay for this awesome product you took it far!

Closes #4636
2014-01-07 11:22:01 +01:00
Simon Willnauer 8d9af7e7a5 Start elasticsearch as deamon when running smoketests
In previous version daemonized was default now we need to set it explicitly.
2014-01-07 10:52:06 +01:00
Simon Willnauer af10b65fe1 Add smoketest support for published packages
the build_release.py tool now also downloads and verfyfies the
released packages from S3. It checks integrity based on the sha1
checksums and runs the smoketest against the specs in the current branch.
2013-12-23 10:34:48 +01:00
Simon Willnauer 7872ef401c Run REST test-suite against release packages during release process 2013-12-18 12:14:08 +01:00
Simon Willnauer 65b49bfbf6 Allow setting release log file via env 2013-12-02 16:16:38 +01:00
Simon Willnauer cf3ba7c51c Only use -Dtests.jvm.argline instead of numbered options 2013-11-26 11:11:19 +01:00
mrsolo 5cd780846b jenkin scripts modification
1) client_tests_url, switch to JDK6
2) build_ranodmization, add jvm options randomization
2013-11-21 12:08:04 -08:00
mrsolo 0d1bebb32c add nightly es_node_mode radomization 2013-11-21 11:03:22 -08:00
mrsolo 2a0a1d3cec property file for client build projects
this property file contains url information for s3 elasticsearch server
bits retrieval on master, 0.90 branch
2013-11-21 11:02:19 -08:00
colings86 76c5f53dfa Changed pom to allow import and running from eclipse
Currently when importing projects into eclipse you need to run 'mvn
eclipse:eclipse' on the command line to generate the poject files. This
means that when the pom changes you need to re-run the command on the
command line to reflect those changes in the project in eclipse.  This
commit allows the developer to import the project as an existing maven
project (can be shared using git after import) and then allows the
application to be run inside eclipse using the .launch file in
/dev-tools enabling easy debugging of the application within eclipse
without requiring a maven build.
2013-11-21 10:56:52 +01:00
mrsolo 054844ca09 jenkin hookup modification
1) add build_desc property variable
2) use build_id plus build_number as an unique file name
3) tools subdirectory should be under current PWD when the script is
executing
2013-11-19 19:04:44 -08:00
mrsolo 731d84322d build randomization script
this script genereates envrionment property file necessary to do
jenkins randomization testing
2013-11-19 15:21:00 -08:00
Simon Willnauer 098cbef6d0 use dedicated release log in order to build lucene & es releases in parallel 2013-11-13 12:41:23 +01:00
Simon Willnauer 71f5b16137 Make RPM building mandatory
If RPM tools are not installed the release tool now fails with an
appropriate message. The tool now also fails if any of the required
artifacts is not present.
2013-11-05 23:55:13 +01:00
Simon Willnauer 97c8fe6801 Add optional RPM building, signing and uploading to release script 2013-11-05 16:11:36 +01:00
Simon Willnauer f2840c5ccf Added doc file processing to update coming/added references
We use 'coming[x.y.z]' in our ref docs which needs to be updated
when doing a release to 'added[x.y.z]' This commit adds support
for replacing the references where applicable during preparing the release.
2013-11-04 21:23:22 +01:00
Simon Willnauer 555ececaa9 Improve top level documentation for building releases 2013-11-04 16:10:58 +01:00
Simon Willnauer 1f3c7be851 Fix typo in mvn command for running with multiple JVMs 2013-11-04 14:30:31 +01:00
Simon Willnauer 35f33c4bbc Added tool to build a complete Elasticsearch release.
This tool builds a release and runs several checks to make sure the
release is in a reasonable shape (smoke test). From a top level
perspective it runs the following steps:

 * clean the build environment `mvn clean`
 * check if a Java 6 JDK is available
 * run the tests with network and local
 * generates the checksums for the binary packages
 * uploads the binary packages to Amazon S3
 * runs a 'mvn deploy' to publish the maven artifacts

The script will create an intermediate branch from a given 'release
branch' updates all versions based on the version we are currently
releasing. Updates the 'pom.xml' file as well as the 'Version.java'
class. Once this is done it commits the changes and rebase with the
branch we want to release from, merges the changes from the intermediate
branch and pushes to the given remote repository including the release
tag.
2013-11-04 14:08:59 +01:00