lucene/dev-tools/scripts
Nicholas Knize 78655239c5 LUCENE-8369: Remove obsolete spatial module 2020-01-16 11:22:05 -06:00
..
LUCENE-3753.patch.hack.pl LUCENE-3753: Restructure the Lucene build system 2012-02-07 19:59:05 +00:00
README.md LUCENE-8852 ReleaseWizard tool (#710) 2019-06-20 14:45:17 +02:00
SOLR-2452.patch.hack.pl SOLR-11694: Remove outdated UIMA module 2018-07-07 09:58:57 -04:00
addBackcompatIndexes.py LUCENE-8852 ReleaseWizard tool (#710) 2019-06-20 14:45:17 +02:00
addVersion.py LUCENE-8883: addVersion.py now adds categories to CHANGES.txt 2019-08-06 13:53:22 +02:00
archive-solr-ref-guide.sh LUCENE-6134: fix typos: wether->whether, neccessary->necessary, initalize->initialize, specifed->specified, etc. (thanks Ahmet Arslan\!) 2014-12-24 14:33:59 +00:00
buildAndPushRelease.py LUCENE-8852 ReleaseWizard tool (#710) 2019-06-20 14:45:17 +02:00
checkJavaDocs.py LUCENE-8738: Move to Java 11 as minimum Java version (merged branch: jira/LUCENE-8738) 2019-04-16 14:00:09 +02:00
checkJavadocLinks.py LUCENE-8738: Move to Java 11 as minimum Java version (merged branch: jira/LUCENE-8738) 2019-04-16 14:00:09 +02:00
crawl.maven.release.dist.sh LUCENE-7935: Publish .sha512 hash files with the release artifacts 2018-03-28 09:14:03 +02:00
createPatch.py don't recurse into dist when diffing sources 2015-06-01 08:17:30 +00:00
githubPRs.py LUCENE-8861: Script to find open PRs that needs attention (#719) 2019-06-14 13:30:04 +02:00
gitignore-gen.sh LUCENE-3775: A shell script to generate .gitignore from svn:ignore properties. 2012-02-13 10:27:44 +00:00
jenkins.build.ref.guide.sh SOLR-13782: Remove PDF Ref Guide build (#999) 2019-11-19 14:54:30 -07:00
poll-mirrors.py LUCENE-8827: Speed up poll-mirrors.py 2019-06-05 21:22:29 +02:00
prep-solr-ref-guide-rc.sh LUCENE-8493: Stop publishing insecure .sha1 files with releases 2018-09-26 15:31:26 +02:00
publish-solr-ref-guide.sh SOLR-5257: new scripts for use with solr ref guide RCs and publishing 2013-09-25 01:28:19 +00:00
releaseWizard.py Change 72h voting rules for release wizard (#1118) 2019-12-24 23:13:01 +01:00
releaseWizard.yaml Minor fixes to the release wizard. (#1127) 2019-12-30 17:21:18 +01:00
releasedJirasRegex.py LUCENE-8852 ReleaseWizard tool (#710) 2019-06-20 14:45:17 +02:00
reproduceJenkinsFailures.py LUCENE-9054: fix stupid nocommit comment 2019-11-22 08:22:39 -07:00
requirements.txt LUCENE-8852 ReleaseWizard tool (#710) 2019-06-20 14:45:17 +02:00
scriptutil.py LUCENE-8837 smokeTestRelease.py option --download-only (#702) 2019-06-11 10:17:10 +02:00
smokeTestRelease.py LUCENE-8369: Remove obsolete spatial module 2020-01-16 11:22:05 -06:00
svnBranchToGit.py LUCENE-6922: more improvements in the svn to git mirror workaround tool 2016-01-08 15:53:36 +00:00
write.stage.maven.build.xml.pl LUCENE-7995: 'ant stage-maven-artifacts' should work from the top-level project directory, and should provide a better error message when its 'maven.dist.dir' param points to a non-existent directory 2017-10-16 16:00:14 -04:00

README.md

Developer Scripts

This folder contains various useful scripts for developers, mostly related to releasing new versions of Lucene/Solr and testing those.

Python scripts require Python 3. To install necessary python modules, please run:

pip3 install -r requirements.txt

Scripts description

smokeTestRelease.py

Used to validate new release candidates (RC). The script downloads an RC from a URL or local folder, then runs a number of sanity checks on the artifacts, and then runs the full tests.

usage: smokeTestRelease.py [-h] [--tmp-dir PATH] [--not-signed]
                           [--local-keys PATH] [--revision REVISION]
                           [--version X.Y.Z(-ALPHA|-BETA)?]
                           [--test-java12 JAVA12_HOME] [--download-only]
                           url ...

Utility to test a release.

positional arguments:
  url                   Url pointing to release to test
  test_args             Arguments to pass to ant for testing, e.g.
                        -Dwhat=ever.

optional arguments:
  -h, --help            show this help message and exit
  --tmp-dir PATH        Temporary directory to test inside, defaults to
                        /tmp/smoke_lucene_$version_$revision
  --not-signed          Indicates the release is not signed
  --local-keys PATH     Uses local KEYS file instead of fetching from
                        https://archive.apache.org/dist/lucene/KEYS
  --revision REVISION   GIT revision number that release was built with,
                        defaults to that in URL
  --version X.Y.Z(-ALPHA|-BETA)?
                        Version of the release, defaults to that in URL
  --test-java12 JAVA12_HOME
                        Path to Java12 home directory, to run tests with if
                        specified
  --download-only       Only perform download and sha hash check steps

Example usage:
python3 -u dev-tools/scripts/smokeTestRelease.py https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-6.0.1-RC2-revc7510a0...

releaseWizard.py

The Release Wizard guides the Release Manager through the release process step by step, helping you to to run the right commands in the right order, generating e-mail templates with the correct texts, versions, paths etc, obeying the voting rules and much more. It also serves as a documentation of all the steps, with timestamps, preserving log files from each command etc, showing only the steps and commands required for a major/minor/bugfix release. It also lets you generate a full Asciidoc guide for the release. The wizard will execute many of the other tools in this folder.

usage: releaseWizard.py [-h] [--dry-run] [--init]

Script to guide a RM through the whole release process

optional arguments:
  -h, --help  show this help message and exit
  --dry-run   Do not execute any commands, but echo them instead. Display
              extra debug info
  --init      Re-initialize root and version

Go push that release!

buildAndPushRelease.py

usage: buildAndPushRelease.py [-h] [--no-prepare] [--local-keys PATH]
                              [--push-local PATH] [--sign KEYID]
                              [--rc-num NUM] [--root PATH] [--logfile PATH]

Utility to build, push, and test a release.

optional arguments:
  -h, --help         show this help message and exit
  --no-prepare       Use the already built release in the provided checkout
  --local-keys PATH  Uses local KEYS file to validate presence of RM's gpg key
  --push-local PATH  Push the release to the local path
  --sign KEYID       Sign the release with the given gpg key
  --rc-num NUM       Release Candidate number. Default: 1
  --root PATH        Root of Git working tree for lucene-solr. Default: "."
                     (the current directory)
  --logfile PATH     Specify log file path (default /tmp/release.log)

Example usage for a Release Manager:
python3 -u dev-tools/scripts/buildAndPushRelease.py --push-local /tmp/releases/6.0.1 --sign 6E68DA61 --rc-num 1

addBackcompatIndexes.py

usage: addBackcompatIndexes.py [-h] [--force] [--no-cleanup] [--temp-dir DIR]
                               version

Add backcompat index and test for new version.  See:
http://wiki.apache.org/lucene-java/ReleaseTodo#Generate_Backcompat_Indexes

positional arguments:
  version         Version to add, of the form X.Y.Z

optional arguments:
  -h, --help      show this help message and exit
  --force         Redownload the version and rebuild, even if it already
                  exists
  --no-cleanup    Do not cleanup the built indexes, so that they can be reused
                  for adding to another branch
  --temp-dir DIR  Temp directory to build backcompat indexes within

addVersion.py

usage: addVersion.py [-h] version

Add a new version to CHANGES, to Version.java, lucene/version.properties and
solrconfig.xml files

positional arguments:
  version

optional arguments:
  -h, --help  show this help message and exit

releasedJirasRegex.py

Pulls out all JIRAs mentioned at the beginning of bullet items under the given version in the given CHANGES.txt file and prints a regular expression that will match all of them

usage: releasedJirasRegex.py [-h] version changes

Prints a regex matching JIRAs fixed in the given version by parsing the given
CHANGES.txt file

positional arguments:
  version     Version of the form X.Y.Z
  changes     CHANGES.txt file to parse

optional arguments:
  -h, --help  show this help message and exit

reproduceJenkinsFailures.py

usage: reproduceJenkinsFailures.py [-h] [--no-git] [--iters N] URL

Must be run from a Lucene/Solr git workspace. Downloads the Jenkins
log pointed to by the given URL, parses it for Git revision and failed
Lucene/Solr tests, checks out the Git revision in the local workspace,
groups the failed tests by module, then runs
'ant test -Dtest.dups=%d -Dtests.class="*.test1[|*.test2[...]]" ...'
in each module of interest, failing at the end if any of the runs fails.
To control the maximum number of concurrent JVMs used for each module's
test run, set 'tests.jvms', e.g. in ~/lucene.build.properties

positional arguments:
  URL         Points to the Jenkins log to parse

optional arguments:
  -h, --help  show this help message and exit
  --no-git    Do not run "git" at all
  --iters N   Number of iterations per test suite (default: 5)

poll-mirrors.py

usage: poll-mirrors.py [-h] [-version VERSION] [-path PATH]
                       [-interval INTERVAL] [-details] [-once]

Periodically checks that all Lucene/Solr mirrors contain either a copy of a
release or a specified path

optional arguments:
  -h, --help            show this help message and exit
  -version VERSION, -v VERSION
                        Lucene/Solr version to check
  -path PATH, -p PATH   instead of a versioned release, check for
                        some/explicit/path
  -interval INTERVAL, -i INTERVAL
                        seconds to wait before re-querying mirrors
  -details, -d          print missing mirror URLs
  -once, -o             run only once

githubPRs.py

usage: githubPRs.py [-h] [--json] [--token TOKEN]

Find open Pull Requests that need attention

optional arguments:
  -h, --help     show this help message and exit
  --json         Output as json
  --token TOKEN  Github access token in case you query too often anonymously

gitignore-gen.sh

TBD

publish-solr-ref-guide.sh

TBD

prep-solr-ref-guide-rc.sh

TBD

Tools to be removed

  • svnBranchToGit.py
  • createPatch.py (svn)