51 Commits

Author SHA1 Message Date
Tomoko Uchida
d4a137d2b6
LUCENE-9242: generate javadocs by calling Ant javadoc task (#1304) 2020-03-12 00:09:26 +09:00
Tomoko Uchida
312d6b2a0d LUCENE-9201: Add an equivalent to "check missing javadocs" task to gradle build
Co-Authored-By: Dawid Weiss <dawid.weiss@carrotsearch.com>
2020-02-24 11:05:35 +09:00
Dawid Weiss
cb68d7d2c5 LUCENE-9232: add a script-hack check so that in case somebody upgrades the scripts automatically they'll know they need to add the hack. 2020-02-21 10:40:27 +01:00
Dawid Weiss
7604639b59 Move jgit version declaration to scriptDepVersions. 2020-02-20 13:54:07 +01:00
Dawid Weiss
2a88aa9d0f LUCENE-9219: Port ECJ-based linter to gradle
Co-authored-by: Tomoko Uchida <tomoko@apache.org>
2020-02-19 02:43:47 +09:00
Robert Muir
0203815ab2
LUCENE-9220: regenerate all stemmers/stopwords/test data from snowball 2.0 (#1262)
Previous situation:

* The snowball base classes (Among, SnowballProgram, etc) had accumulated local performance-related changes. There was a task that would also "patch" generated classes (e.g. GermanStemmer) after-the-fact.
* Snowball classes had many "non-changes" from the original such as removal of tabs addition of javadocs, license headers, etc.
* Snowball test data (inputs and expected stems) was incorporated into lucene testing, but this was maintained manually. Also files had become large, making the test too slow (Nightly).
* Snowball stopwords lists from their website were manually maintained. In some cases encoding fixes were manually applied.
* Some generated stemmers (such as Estonian and Armenian) exist in lucene, but have no corresponding `.sbl` file in snowball sources at all.

Besides this mess, snowball project is "moving along" and acquiring new languages, adding non-BSD-licensed test data, huge test data, and other complexity. So it is time to automate the integration better.

New situation:

* Lucene has a `gradle snowball` regeneration task. It works on Linux or Mac only. It checks out their repos, applies the `snowball.patch` in our repository, compiles snowball stemmers, regenerates all java code, applies any adjustments so that our build is happy.
* Tests data is automatically regenerated from the commit hash of the snowball test data repository. Not all languages are tested from their data: only where the license is simple BSD. Test data is also (deterministically) sampled, so that we don't have huge files. We just want to make sure our integration works.
* Randomized tests are still set to test every language with generated fake words. The regeneration task ensures all languages get tested (it writes a simple text file list of them).
* Stopword files are automatically regenerated from the commit hash of the snowball website repository.
* The regeneration procedure is idempotent. This way when stuff does change, you know exactly what happened. For example if test data changes to a different license, you may see a git deletion. Or if a new language/stopwords/test data gets added, you will see git additions.
2020-02-17 12:38:01 -05:00
Jan Høydahl
53f7b394e4 SOLR-11207: Mute warnings for owasp false positives 2020-01-27 21:03:20 +01:00
Dawid Weiss
ff635cf701 LUCENE-9184, LUCENE-9183: allow skipping git status check in precommit with -Pvalidation.git.failOnModified=false (or place this in gradle.properties to make it permanent). 2020-01-27 20:47:02 +01:00
Uwe Schindler
7dc35e3a62 Let precommit depend on generic forbiddenApis task 2020-01-27 19:47:54 +01:00
Robert Muir
fd5a0ce7c2
LUCENE-9182: the rat-sources.gradle was the one .gradle file already with a license header, we don't need it twice 2020-01-27 12:11:44 -05:00
Robert Muir
975df9ddd3
LUCENE-9182: add apache license headers to all .gradle files and enforce in rat task 2020-01-27 12:05:34 -05:00
Robert Muir
8e357b167b
LUCENE-9180: dos2unix files that don't need dos line endings 2020-01-27 11:29:59 -05:00
Jan Høydahl
39df74de37 SOLR-11207: Exclude configuration 'unifiedClasspath'
It is generated by consistent-versions plugin and triggers owasp warnings for deps even for excluded projects
2020-01-27 12:17:31 +01:00
Jan Høydahl
9ddd05cd14 SOLR-11207: Exclude solr-ref-guide from owasp check
It picked up log4j1 dependency only used during build
2020-01-27 09:55:12 +01:00
Dawid Weiss
5ab59f59ac SOLR-11207: minor changes:
- added 'owasp' task to the root project. This depends on
dependencyCheckAggregate which seems to be a better fit for multi-module
projects than dependencyCheckAnalyze (the difference is vague to me
from plugin's documentation).

- you can run the "gradlew owasp" task explicitly and it'll run the
validation without any flags.

- the owasp task is only added to check if validation.owasp property
is true. I think this should stay as the default on non-CI systems
(developer defaults) because it's a significant chunk of time it takes
to download and validate dependencies.

- I'm not sure *all* configurations should be included in the check...
perhaps we should only limit ourselves to actual runtime dependencies
 not build dependencies, solr-ref-guide, etc.
2020-01-26 10:45:05 +01:00
Jan Høydahl
74a8d6d5ac SOLR-11207: Add OWASP dependency checker to gradle build (#1121)
* SOLR-11207: Add OWASP dependency checker to gradle build
2020-01-26 10:01:51 +01:00
Dawid Weiss
1ad6bc9361 LUCENE-9077: Allow locally staged files in git status precommit check. 2020-01-20 09:36:14 +01:00
Dawid Weiss
44c203d72f Add workaround for https://github.com/palantir/gradle-consistent-versions/issues/383 2020-01-15 11:44:21 +01:00
Dawid Weiss
e6d85cd4bc Cleaning up minor things in rat task. 2020-01-15 10:07:24 +01:00
Mike
c9e7eebe28 Add RAT check using Gradle (#1157)
Merging Apache rat checks.
2020-01-15 09:55:41 +01:00
Dawid Weiss
4a8762cc2c Add javadoc generation/linter to precommit. 2020-01-13 19:11:43 +01:00
Dawid Weiss
34aa8714d8 Correct class->classname. 2020-01-10 12:53:30 +01:00
Dawid Weiss
b4d26f94d3 Don't load all of groovy's tasks, just groovy. 2020-01-10 12:51:46 +01:00
Dawid Weiss
39a5323999 Add config file sanity check for precommit. 2020-01-10 12:49:04 +01:00
Dawid Weiss
109444fc5b Add an equivalent of validate-source-patterns task, delegating to the same groovy script. 2020-01-10 12:02:30 +01:00
Dawid Weiss
c7ed133910 LUCENE-9122: upgrade gradle wrapper to 6.0.1. Relax JVM requirement to require at least Java 11. We can't even check for higher bound because gradle itself breaks before it can execute the check script. I verified locally and it works with 11-13. 2020-01-09 14:13:32 +01:00
Dawid Weiss
7a12c89ce6 Move precommit dependencies to precommit for clarity. 2020-01-08 14:20:16 +01:00
Dawid Weiss
c9c0bab2eb Ensure versions.props contains sorted entries (like check-lib-versions did for ant). 2020-01-03 16:04:12 +01:00
Dawid Weiss
37fb4a5f49 Verify lock state on precommit. 2020-01-03 15:53:29 +01:00
Dawid Weiss
ae4a2e381d Hook up license checks to precommit. 2020-01-03 15:50:04 +01:00
Dawid Weiss
797f571fc3 Hook up forbidden apis to precommit. 2020-01-03 15:35:15 +01:00
Dawid Weiss
8b03a7104e Add a precommit placeholder task and working copy's git status check. 2020-01-03 15:22:36 +01:00
Dawid Weiss
ca8661bc3a Reworked dependency resolution for license checks to work around a problem with gradle. Consolidated licenses with the ant build (excluding some jars from the ref-guide). 2019-12-30 14:05:08 +01:00
Dawid Weiss
7c762c969a Allow simultaneous call to sha regeneration and validation by introducing soft ordering constraint. 2019-12-18 14:54:13 +01:00
Dawid Weiss
d2d28329ef Changed license checksum regeneration task name to updateLicenses. 2019-12-18 14:14:39 +01:00
Dawid Weiss
0e2a493446 Add transitive Lucene dependencies so that Solr licenses/ folder is (more) consistent with ant. This is an insane hack at the build-level. Mark it for removal once we get rid of ant. 2019-12-17 15:02:08 +01:00
Dawid Weiss
faadb65202 Regenerate checksum for a single dependency once. Add trailing newline for consistency with ant code. 2019-12-17 14:27:25 +01:00
Dawid Weiss
8906c2ddbe Merge forbidden APIs rules. 2019-12-17 13:39:10 +01:00
Dawid Weiss
401ddc6dd1 Upgrade gradlew. Add environment sanity check. 2019-12-16 15:23:06 +01:00
Dawid Weiss
208d094262 Remove checksum files from ref guide. 2019-12-13 17:09:25 +01:00
Dawid Weiss
3aff1664e5 updateChecksums, validation of dangling unreferenced files under licenses/. Separated licenses-gradle for Solr for now (doesn't include transitive Lucene dependencies). 2019-12-13 15:07:59 +01:00
Dawid Weiss
d8cac07d2a Sort output of dangling license files. 2019-12-13 14:03:06 +01:00
Dawid Weiss
4500f0e327 Consolidating versions between gradle and ant. 2019-12-13 13:31:23 +01:00
Dawid Weiss
25fc0487a1 Working jar checksums and aligned with ant build. 2019-12-13 12:12:29 +01:00
Dawid Weiss
73e8b49f0d Align versions with ant build. 2019-12-13 12:01:26 +01:00
Dawid Weiss
a392a83558 Add support for validating the presence of licenses and notices. 2019-12-12 19:25:46 +01:00
Dawid Weiss
453eee3987 Initial work on jar checksums/ license file validation. 2019-12-11 18:41:27 +01:00
Dawid Weiss
85e0e4fb75 Add a workaround for the problem of forbiddenApis not running upon changing just the rules/ rulesets. 2019-12-03 18:41:11 +01:00
Dawid Weiss
0247f02a70 Only apply log4j rules to Solr. 2019-12-03 15:18:10 +01:00
Dawid Weiss
a6d6d633d5 Apply servlet APIs to just Solr. 2019-12-03 14:43:50 +01:00