Commit Graph

138 Commits

Author SHA1 Message Date
Robert Muir 945b1cb872
LUCENE-9856: fail precommit on unused local variables, take two (#37)
Enable ecj unused local variable, private instance and method detection. Allow SuppressWarnings("unused") to disable unused checks (e.g. for generated code or very special tests). Fix gradlew regenerate for python 3.9 SuppressWarnings("unused") for generated javacc and jflex code. Enable a few other easy ecj checks such as Deprecated annotation, hashcode/equals, equals across different types.

Co-authored-by: Mike McCandless <mikemccand@apache.org>
2021-03-23 13:59:00 -04:00
Robert Muir e6c4956cf6
Revert "LUCENE-9856: fail precommit on unused local variables (#34)"
This reverts commit 20dba278bb.
2021-03-23 12:46:36 -04:00
Robert Muir 20dba278bb
LUCENE-9856: fail precommit on unused local variables (#34)
Enable ecj unused local variable, private instance and method detection. Allow SuppressWarnings("unused") to disable unused checks (e.g. for generated code or very special tests). Fix gradlew regenerate for python 3.9 SuppressWarnings("unused") for generated javacc and jflex code. Enable a few other easy ecj checks such as Deprecated annotation, hashcode/equals, equals across different types.

Co-authored-by: Mike McCandless <mikemccand@apache.org>
2021-03-23 11:09:24 -04:00
Dawid Weiss 078d0079d1
LUCENE-9861: pull tuned vm options into a separate aspect. (#33) 2021-03-23 10:39:09 +01:00
Dawid Weiss 246c4beb22
LUCENE-9854: Clean up utilities to download and extract test/ benchmark data sets. (#27) 2021-03-22 12:22:39 +01:00
Dawid Weiss 53bea54669
LUCENE-9375: cleaning up post-split conditional build logic and solr refs. (#22) 2021-03-18 11:04:45 +01:00
Dawid Weiss 7f5e660395
LUCENE-9375: some build file cleanups. (#10) 2021-03-10 21:47:37 +01:00
Dawid Weiss fdf486ba54 LUCENE-9375: post-repo-split removal of solr counterpart. 2021-03-10 11:20:08 +01:00
Dawid Weiss 409bc37c13
SOLR-14759: a few initial changes so that Lucene can be built independently while Solr code is still in place. (#2448) 2021-03-08 14:59:08 +01:00
Dawid Weiss d2fb89c22f LUCENE-9793: Add task time aggregation utility (enabled with -Ptask.times=true). 2021-02-20 20:18:16 +01:00
Dawid Weiss 8f56ae0a4b
LUCENE-9767: infrastructure for icu regeneration in place. (#2362) 2021-02-14 21:07:39 +01:00
Houston Putman 2b8d7bcd6a
SOLR-15075: Solr docker gradle improvements (#2197)
* Removed docker plugin from gradle builds.
* Removed package docker image.
* Tasks now have correct inputs/outputs/dependencies.
* Move gradle help text to docker folder.
* Reduce duplicated Docker layer by doing file removal and chmod in another stage.

Co-authored-by: David Smiley <dsmiley@apache.org>
2021-01-26 10:22:50 -05:00
Dawid Weiss c94b035df8
LUCENE-9564: add spotless and gjf (automatic code formatter) 2020-12-17 13:11:54 +01:00
zacharymorn ebc87a8a27
LUCENE-8982: Separate out native code to another module to allow cpp build with gradle (#2068)
* LUCENE-8982: Separate out native code to another module to allow cpp build with gradle
2020-11-16 09:40:03 +01:00
Dawid Weiss a29d7c70d5
LUCENE-9597: checkWorkingCopyClean shouldn't complain about untracked empty folders (similar to git status). Piggybacking jgit update. (#2061) 2020-11-03 12:50:05 +01:00
Mike Drob 08e38d3452
LUCENE-9488 Create Release Artifacts with Gradle (#1905)
* Build Lucene binary distribution using Gradle
* Generate SHA-512 checksums for all release artifacts
* Update documentation artifacts included in binaries
* Delete some additional Ant relics

Co-authored-by: Dawid Weiss <dawid.weiss@carrotsearch.com>
Co-authored-by: Uwe Schindler <uschindler@apache.org>
2020-10-08 14:25:51 -05:00
Namgyu Kim 00d7f5ea68
LUCENE-9544: Port Nori dictionary compilation (#1926) 2020-09-28 20:28:21 +09:00
Dawid Weiss 6b0149ec1a Revert "add regenerate gradle script for nori dictionary (#1924)"
This reverts commit e28e8c0e0c.
2020-09-28 09:52:34 +02:00
Namgyu Kim e28e8c0e0c
add regenerate gradle script for nori dictionary (#1924) 2020-09-28 08:54:27 +02:00
Uwe Schindler e19239d96b Upgrade forbiddenapis to version 3.1 2020-09-23 14:56:46 +02:00
Dawid Weiss 6c9d7adf79
LUCENE-9527: upgrade javacc to 7.0.4 (#1884) 2020-09-17 13:29:18 +02:00
Houston Putman 485d5fb41a
SOLR-14789: Absorb the docker-solr repo. (#1769) 2020-09-11 12:29:29 -04:00
Dawid Weiss a05a055d9d
LUCENE-9518: Add sanity to gradle archiving tasks (predictable order, default permissions, fail on duplicates) (#1851) 2020-09-10 09:48:04 +02:00
Varun Thacker 121b262389
LUCENE-9497: Integerate Error Prone ( Static Analysis Tool ) during compilation (#1816)
LUCENE-9497:  Integrate Error Prone, a static analysis tool during compilation
2020-09-07 14:42:48 -07:00
Robert Muir 784ede4eda
LUCENE-9215: replace checkJavaDocs.py with doclet (#1802)
This has the same logic as the previous python, but no longer relies
upon parsing HTML output, instead using java's doclet processor.

The errors are reported like "normal" javadoc errors with source file
name and line number and happen when running "gradlew javadoc"

Although the "rules" are the same as the previous python, the python had
some bugs where the checker didn't quite do exactly what we wanted, so
some fixes were applied throughout.

Co-authored-by: Dawid Weiss <dawid.weiss@carrotsearch.com>
Co-authored-by: Uwe Schindler <uschindler@apache.org>
2020-09-02 08:29:17 -04:00
Dawid Weiss 9445d2c10e Move Solr-specific config. 2020-08-31 15:43:56 +02:00
Dawid Weiss 4a2a2da462 LUCENE-9435: revert Solr's packaging stuff for now. 2020-08-31 14:55:59 +02:00
Dawid Weiss 5e8bb7922b
LUCENE-9491: Consolidate test options and improve support for dynamic… (#1807)
* LUCENE-9491: Consolidate test options and improve support for dynamically computed values.
2020-08-31 12:20:30 +02:00
Dawid Weiss d847f40237 LUCENE-9474: make externalTool a function and add a build-stopping message on Windows for snowball generator. 2020-08-30 17:10:18 +02:00
Dawid Weiss def82ab556
LUCENE-9435: Clean up ant compatability remnants in Gradle build
* Removing ant-only unused tasks.
* Correct message in TestVersion.java
* Remove unused file.
* Removing forbidden API rules for ant.
* Remove 'resolve' emulation.
* Move ecj-lint to task-definition-relative folder.
* Remove 'packaging' specification. It'll have to wait until proper new packaging is implemented for Solr distribution.
* Move render-javadoc tasks's files to task-relative location.
* Moved security manager policies and default JDK logging file to gradle's task relative locations.
* Removing obsolete ant tools. Moving check source patterns under gradle's folder.
* Correct paths.
* Correct property name in task selector.
2020-08-30 14:01:04 +02:00
Uwe Schindler 494a8a8e04 LUCENE-9474: Make external tools configurable like in ant through those sysprops: perl.exe, python3.exe, python2.exe 2020-08-23 20:16:22 +02:00
Uwe Schindler ee6cf4c418 LUCENE-9474: Make the version number configurable from outside. This mainly allows Jenkins to set a different suffix than "SNAPSHOT" 2020-08-23 19:26:40 +02:00
Dawid Weiss b1e2d0c890
LUCENE-9438: Eclipse IDE support with gradle build system (#1761) 2020-08-21 21:47:11 +02:00
Dawid Weiss 581e836569
LUCENE-9471: clean up task and global temporary junk files (workaround). (#1767) 2020-08-20 13:49:24 +02:00
Dawid Weiss 83ed210fd0
LUCENE-9465: 'beast' task from within gradle (#1757) 2020-08-18 09:28:50 +02:00
Dawid Weiss 8ebf2d0b21 LUCENE-9312: Allow builds against arbitrary JVMs (squashed
jira/LUCENE-9312)
2020-07-21 09:19:38 +02:00
Erick Erickson 9c1772f094 LUCENE-9411: Fail complation on warnings, 9x gradle-only 2020-06-23 16:21:10 -04:00
Uwe Schindler 08a13ce589 Upgrade forbiddenapis to hotfix release 3.0.1 (allows upgrade to commons-io 2.7 in Solr) 2020-06-04 01:01:42 +02:00
Dawid Weiss da3dbb1921 LUCENE-9301: include build time and user name only in non-snapshot builds so that jars are not recompiled on each build in development. 2020-06-01 10:09:44 +02:00
Tomoko Uchida 84ea0cb87d
LUCENE-9374: Add checkBrokenLinks gradle task (#1522) 2020-05-20 23:23:24 +09:00
Uwe Schindler 06df50e759
LUCENE-9321: Port markdown task to Gradle (#1477) 2020-05-17 14:46:26 +02:00
Uwe Schindler 010168c57b
LUCENE-9321, LUCENE-9278: Refactor renderJavadoc to allow relative links with multiple Gradle tasks (#1488)
This also automatically collects linked projects by its dependencies, so we don't need to maintain all inter-project javadocs links.

Co-authored-by: Dawid Weiss <dweiss@apache.org>
2020-05-14 13:16:16 +02:00
Tomoko Uchida 5354f7e88e
LUCENE-9333: Add gradle task to compile changes.txt to a html (#1468) 2020-04-30 17:21:53 +09:00
Uwe Schindler 64eed9a1a6
LUCENE-9347: Add support for forbiddenapis 3.0 (#1459)
LUCENE-9347: Add support for forbiddenapis 3.0
2020-04-27 11:54:59 +02:00
Erick Erickson 3af165b32a LUCENE-7788: fail precommit on unparameterised log messages and examine for wasted work/objects 2020-04-17 20:40:32 -04:00
Dawid Weiss 9244558752 LUCENE-9201: remove javadoc task remnants. Make javadoc depend on renderJavadoc and skip the default gradle's implementation. 2020-04-12 12:55:56 +02:00
Dawid Weiss 2437f3f56c
LUCENE-9311: detect intellij reimport and modify sourceset to exclude solr-ref-guide/tools (#1422) 2020-04-09 13:55:42 +02:00
Dawid Weiss d32858b127
LUCENE-9301: add manifest entries to JARs (gradle build). 2020-04-04 20:56:35 +02:00
Tomoko Uchida d4a137d2b6
LUCENE-9242: generate javadocs by calling Ant javadoc task (#1304) 2020-03-12 00:09:26 +09:00
Tomoko Uchida 312d6b2a0d LUCENE-9201: Add an equivalent to "check missing javadocs" task to gradle build
Co-Authored-By: Dawid Weiss <dawid.weiss@carrotsearch.com>
2020-02-24 11:05:35 +09:00
Dawid Weiss cb68d7d2c5 LUCENE-9232: add a script-hack check so that in case somebody upgrades the scripts automatically they'll know they need to add the hack. 2020-02-21 10:40:27 +01:00
Dawid Weiss 62662e477a LUCENE-9155: Port Kuromoji dictionary compilation (regenerate). 2020-02-20 19:00:56 +01:00
Dawid Weiss 7604639b59 Move jgit version declaration to scriptDepVersions. 2020-02-20 13:54:07 +01:00
Dawid Weiss 2a88aa9d0f LUCENE-9219: Port ECJ-based linter to gradle
Co-authored-by: Tomoko Uchida <tomoko@apache.org>
2020-02-19 02:43:47 +09:00
Robert Muir 0203815ab2
LUCENE-9220: regenerate all stemmers/stopwords/test data from snowball 2.0 (#1262)
Previous situation:

* The snowball base classes (Among, SnowballProgram, etc) had accumulated local performance-related changes. There was a task that would also "patch" generated classes (e.g. GermanStemmer) after-the-fact.
* Snowball classes had many "non-changes" from the original such as removal of tabs addition of javadocs, license headers, etc.
* Snowball test data (inputs and expected stems) was incorporated into lucene testing, but this was maintained manually. Also files had become large, making the test too slow (Nightly).
* Snowball stopwords lists from their website were manually maintained. In some cases encoding fixes were manually applied.
* Some generated stemmers (such as Estonian and Armenian) exist in lucene, but have no corresponding `.sbl` file in snowball sources at all.

Besides this mess, snowball project is "moving along" and acquiring new languages, adding non-BSD-licensed test data, huge test data, and other complexity. So it is time to automate the integration better.

New situation:

* Lucene has a `gradle snowball` regeneration task. It works on Linux or Mac only. It checks out their repos, applies the `snowball.patch` in our repository, compiles snowball stemmers, regenerates all java code, applies any adjustments so that our build is happy.
* Tests data is automatically regenerated from the commit hash of the snowball test data repository. Not all languages are tested from their data: only where the license is simple BSD. Test data is also (deterministically) sampled, so that we don't have huge files. We just want to make sure our integration works.
* Randomized tests are still set to test every language with generated fake words. The regeneration task ensures all languages get tested (it writes a simple text file list of them).
* Stopword files are automatically regenerated from the commit hash of the snowball website repository.
* The regeneration procedure is idempotent. This way when stuff does change, you know exactly what happened. For example if test data changes to a different license, you may see a git deletion. Or if a new language/stopwords/test data gets added, you will see git additions.
2020-02-17 12:38:01 -05:00
Erick Erickson f9357ab0d2
LUCENE-9134: Port ant-regenerate tasks to Gradle build (util and packed) (#1251)
* LUCENE-9134: Port ant-regenerate tasks to Gradle build
2020-02-11 18:56:11 -05:00
Dawid Weiss e25dac085f LUCENE-9134: this adds initial javacc support (without follow-up tweaks required to make the sources identical as those generated by ant). 2020-01-29 17:02:59 +01:00
Robert Muir e504798a44
LUCENE-9185: add "tests.profile" to gradle build to aid fixing slow tests
Run test(s) with -Ptests.profile=true to print a histogram at the end of
the build.
2020-01-28 11:27:18 -05:00
Robert Muir 975df9ddd3
LUCENE-9182: add apache license headers to all .gradle files and enforce in rat task 2020-01-27 12:05:34 -05:00
Dawid Weiss ae95f0ab68 LUCENE-9134: lucene:core:jflexStandardTokenizerImpl 2020-01-27 09:03:19 +01:00
Dawid Weiss 5ab59f59ac SOLR-11207: minor changes:
- added 'owasp' task to the root project. This depends on
dependencyCheckAggregate which seems to be a better fit for multi-module
projects than dependencyCheckAnalyze (the difference is vague to me
from plugin's documentation).

- you can run the "gradlew owasp" task explicitly and it'll run the
validation without any flags.

- the owasp task is only added to check if validation.owasp property
is true. I think this should stay as the default on non-CI systems
(developer defaults) because it's a significant chunk of time it takes
to download and validate dependencies.

- I'm not sure *all* configurations should be included in the check...
perhaps we should only limit ourselves to actual runtime dependencies
 not build dependencies, solr-ref-guide, etc.
2020-01-26 10:45:05 +01:00
Jan Høydahl 74a8d6d5ac SOLR-11207: Add OWASP dependency checker to gradle build (#1121)
* SOLR-11207: Add OWASP dependency checker to gradle build
2020-01-26 10:01:51 +01:00
Dawid Weiss 44c203d72f Add workaround for https://github.com/palantir/gradle-consistent-versions/issues/383 2020-01-15 11:44:21 +01:00
Mike c9e7eebe28 Add RAT check using Gradle (#1157)
Merging Apache rat checks.
2020-01-15 09:55:41 +01:00
Dawid Weiss 3beb1cfd1e Add initial support for rendering javadocs. 2020-01-10 16:43:52 +01:00
Dawid Weiss 39a5323999 Add config file sanity check for precommit. 2020-01-10 12:49:04 +01:00
Dawid Weiss 109444fc5b Add an equivalent of validate-source-patterns task, delegating to the same groovy script. 2020-01-10 12:02:30 +01:00
Dawid Weiss 4599c51f0d LUCENE-9122: add support for running tests against alternate jvms. 2020-01-09 19:00:32 +01:00
Dawid Weiss 10baa68b60 Revert "Disable checkUnusedConstraints in palantir's plugin (bug)."
This reverts commit b32db8ee6a.
2020-01-09 17:40:51 +01:00
Dawid Weiss b32db8ee6a Disable checkUnusedConstraints in palantir's plugin (bug). 2020-01-09 17:05:02 +01:00
Dawid Weiss 0166e89dba Upgrading palantir's plugin to 1.14.0 2020-01-09 14:23:35 +01:00
Dawid Weiss 322a8cb995 Remove buildscan plugin. 2020-01-09 14:02:14 +01:00
Dawid Weiss e713ca62b9 Remove buildscan configuration. 2020-01-08 20:17:20 +01:00
Dawid Weiss e87228982c Remove travis support for now. 2020-01-08 15:14:11 +01:00
Dawid Weiss 7808dd5c3a Add minimum repro line at the end of the build. 2020-01-08 12:20:09 +01:00
Dawid Weiss c9c0bab2eb Ensure versions.props contains sorted entries (like check-lib-versions did for ant). 2020-01-03 16:04:12 +01:00
Dawid Weiss 8b03a7104e Add a precommit placeholder task and working copy's git status check. 2020-01-03 15:22:36 +01:00
Dawid Weiss 363f2e3654 Move intiial environment checks to buildSrc so that they happen before any other script has a chance to evaluate. 2020-01-03 14:46:29 +01:00
Dawid Weiss 401ddc6dd1 Upgrade gradlew. Add environment sanity check. 2019-12-16 15:23:06 +01:00
Dawid Weiss 73e8b49f0d Align versions with ant build. 2019-12-13 12:01:26 +01:00
Dawid Weiss 453eee3987 Initial work on jar checksums/ license file validation. 2019-12-11 18:41:27 +01:00
Dawid Weiss cd7fd6d750 Clean up test property passing and move a number of properties and randomizations from common.build (ant counterpart) 2019-12-06 11:55:53 +01:00
Dawid Weiss 62a810cda7 Fail the build if --tests filter is applied and no tests execute during the entire build (this allows for an empty set of filtered tests at single project level). 2019-12-05 13:23:43 +01:00
Dawid Weiss bf7d115414 Generate hardware-specific defaults for gradle parallelism on the first build run (any task). Add some explanations on how to tweak local settings even further (gradlew :helpLocalSettings 2019-12-05 11:14:09 +01:00
Dawid Weiss 64e1499bc7 Add verification check that gradle and ant rules are in sync. 2019-12-03 23:08:57 +01:00
Dawid Weiss 6461909129 Port forbidden APIs. See gradlew :helpForbiddenApis to see how rules are applied automatically based on the set of dependencies of a project. 2019-12-03 14:40:35 +01:00
Dawid Weiss 0d7336db9d Moved gradle fragments under ci/ and maven/ for clarity. 2019-12-03 12:10:13 +01:00
Dawid Weiss d4a9842375 Initial gradle build layer. 2019-12-02 15:34:57 +01:00