Commit Graph

107 Commits

Author SHA1 Message Date
Uwe Schindler f668cfd1cd
Fix forUtil.gradle to actually execute python script and also fix type error in script (#12411) 2023-07-03 16:22:03 +02:00
Chris Hegarty 1090928c14
Implement VectorUtilProvider with Java 21 Project Pamana Vector API (#12363)
This commit enables the Panama Vector API for Java 21. The version of
VectorUtilPanamaProvider for Java 21 is identical to that of Java 20.
As such, there is no specific 21 version - the Java 20 version will be
loaded from the MRJAR.
2023-06-13 09:44:58 +01:00
Uwe Schindler c8e05c8cd6
Implement MMapDirectory with Java 21 Project Panama Preview API (#12294) 2023-06-12 21:07:04 +02:00
Uwe Schindler c188d47a8b
Handle jdk.internal classes mentioned in vector superclass or interfaces during extraction (#12329) 2023-05-25 17:21:03 +02:00
Uwe Schindler cf7245e38e Refactor loop to not addAll set to itsself on initial round (followup of #12311) 2023-05-25 09:06:45 +02:00
Chris Hegarty f756f90644
Integrate the Incubating Panama Vector API (#12311)
Leverage accelerated vector hardware instructions in Vector Search.

Lucene already has a mechanism that enables the use of non-final JDK APIs, currently used for the Previewing Pamana Foreign API. This change expands this mechanism to include the Incubating Pamana Vector API. When the jdk.incubator.vector module is present at run time the Panamaized version of the low-level primitives used by Vector Search is enabled. If not present, the default scalar version of these low-level primitives is used (as it was previously).

Currently, we're only targeting support for JDK 20. A subsequent PR should evaluate JDK 21.
---------

Co-authored-by: Uwe Schindler <uschindler@apache.org>
Co-authored-by: Robert Muir <rmuir@apache.org>
2023-05-25 07:59:50 +01:00
Uwe Schindler 84e2e3afc3
Make sure APIJAR reproduces with different timezone (unfortunately java encodes the date using local timezone) (#12315) 2023-05-19 18:42:55 +02:00
Uwe Schindler e4d8a5c5cb
Implement MMapDirectory with Java 20 Project Panama Preview API (#12188) 2023-03-09 21:27:31 +01:00
Uwe Schindler 8564da434d
Generate gradle.properties from gradlew (#12131)
* SOLR-16641 - Generate gradle.properties from gradlew (#1320)
* Adapt for Lucene
* Remove localSettings from smoker; thanks @colvinco
* Print properties at end for debugging
* Add CHANGES.txt entry

---------

Co-authored-by: Colvin Cowie <colvin.cowie.dev@gmail.com>
Co-authored-by: Colvin Cowie <51863265+colvinco@users.noreply.github.com>
2023-02-06 19:47:15 +01:00
Uwe Schindler c9401bf064
Patch class files for Java 19 code to no longer have the "preview" flag (this enables Java 19 memory segments by default) (#12033) 2022-12-26 10:07:44 +01:00
Dawid Weiss 3f6410b738
Implement source code regeneration for test-framework perl scripts (#11952) 2022-11-19 23:40:45 +01:00
Robert Muir 99aa6dd0eb
Lower gradle heap: 3GB is unnecessary (#11936) 2022-11-18 10:55:32 -05:00
Robert Muir b292151f21
reland: 11925 javac options (#11937)
* pass jvm args to javac  #11925

* Update net.ltgt.errorprone to the latest version so that jvm args are not overwritten, add -XX:+PrintFlagsFinal for debugging

* speed up the pure javac case too

It does not help to fork additional VMs (although error-prone will do
this since it messes with bootstrap classpath), so we avoid forking.

Instead it is best to tune org.gradle.jvmargs.

* use the flags consistently everywhere (tests and doc)

* handle the different possible alt toolchain cases

The difference is invoking 'java' versus invoking 'javac', so the args must be fed differently.

Co-authored-by: Dawid Weiss <dawid.weiss@carrotsearch.com>
2022-11-15 19:28:52 -05:00
Robert Muir 0753f48a80
Revert "speed up javac and error-prone tasks by using less resources (#11927)"
This reverts commit deab699188.

Having a shitty day: git lost changes :(
2022-11-15 19:08:43 -05:00
Robert Muir deab699188
speed up javac and error-prone tasks by using less resources (#11927)
* pass jvm args to javac  #11925
* Update net.ltgt.errorprone to the latest version so that jvm args are not overwritten, add -XX:+PrintFlagsFinal for debugging
* speed up the pure javac case too

It does not help to fork additional VMs (although error-prone will do
this since it messes with bootstrap classpath), so we avoid forking.

Instead it is best to tune org.gradle.jvmargs.

* use the flags consistently everywhere (tests and doc)

Co-authored-by: Dawid Weiss <dweiss@apache.org>
2022-11-15 19:04:17 -05:00
Robert Muir a02db83235
Revert "speed up javac and error-prone tasks by using less resources (#11927)"
This reverts commit b9c51d9a59.
2022-11-15 18:19:09 -05:00
Dawid Weiss b9c51d9a59
speed up javac and error-prone tasks by using less resources (#11927)
* pass jvm args to javac  #11925
* Update net.ltgt.errorprone to the latest version so that jvm args are not overwritten, add -XX:+PrintFlagsFinal for debugging
* speed up the pure javac case too

It does not help to fork additional VMs (although error-prone will do
this since it messes with bootstrap classpath), so we avoid forking.

Instead it is best to tune org.gradle.jvmargs.

* use the flags consistently everywhere (tests and doc)

Co-authored-by: Robert Muir <rmuir@apache.org>
2022-11-15 16:06:38 -05:00
Uwe Schindler 3b9c728ab5
MR-JAR rewrite of MMapDirectory with JDK-19 preview Panama APIs (>= JDK-19-ea+23) (#912)
This uses Gradle's auto-provisioning to compile Java 19 classes and build a multi-release JAR from them. Please make sure to regenerate gradle.properties (delete it) or change "org.gradle.java.installations.auto-download" to "true"
2022-09-26 15:22:04 +02:00
Dawid Weiss 6b82be5f11
Regenerate sources after dependency updates. (#11817) 2022-09-25 18:09:30 +02:00
tang donghai b08e34722d
LUCENE-10646: Add some comment on LevenshteinAutomata (#1016)
* add Comment on Lev & pretty the toDot

* use auto generate scripts to add comment

* update checksum

* update checksum

* restore toDot

* add removeDeadStates in levAutomata

Co-authored-by: tangdonghai <tangdonghai@meituan.com>
2022-08-07 10:01:30 -04:00
Dawid Weiss f93e52e5bb
LUCENE-10669: The build should be more helpful when generated resources are touched (#1053) 2022-07-30 20:45:32 +02:00
Dawid Weiss ba1062620c
LUCENE-10510: Check module access prior to running gjf/spotless tasks (#802) 2022-04-10 20:35:45 +02:00
Tomoko Uchida 2a3e5ca07f
LUCENE-10475: Merge o.a.l.a.[ja|ko].util into o.a.l.a.[ja|ko].dict (#772) 2022-03-29 21:09:26 +09:00
Tomoko Uchida 76c9fd4e38 LUCENE-10416: Update Korean Dictionary to mecab-ko-dic-2.1.1-20180720 for Nori 2022-02-20 21:39:03 +09:00
Dawid Weiss b48cac0206
LUCENE-10285: try to force ordering of internal tasks, in spite of making top-level wrapper dependencies. (#549) 2021-12-17 19:12:09 +01:00
Robert Muir c8f5b9127d
LUCENE-10243: increase unicode versions of tokenizers to 12.1 (#465)
* Bump %unicode 9 -> %unicode 12.1 for the 3 unicode grammars
* regenerate emoji conformance tests for unicode 12.1
* modify wordbreak conformance tests to use emoji data (which replaces old crazy E_base etc properties)
* regenerate wordbreak conformance tests
* Simplify grammar files and word-break conformance test generator, now that full-width numbers are WordBreak=Numeric
* Use jflex emoji properties rather than ICU-generated ones
2021-12-03 20:20:57 -05:00
Robert Muir af831d2810
LUCENE-10239: upgrade jflex (1.7.0 -> 1.8.2) (#452)
Upgrade jflex.

Change doesn't alter the behavior of any of the analyzers (unicode
version or grammar refactorings), just the minimal to get new tooling
working.
2021-11-19 09:24:27 -05:00
Dawid Weiss bae095ae48
LUCENE-10240: gradle regenerate fails on java 17 (#449) 2021-11-17 18:36:34 +01:00
Dawid Weiss 0eeba8d37c
LUCENE-10238: Update icu4j to 70.1. (#447) 2021-11-17 18:13:40 +01:00
Dawid Weiss 1a38cac68e LUCENE-10195: add commented-out org.gradle.caching=true to the generated local settings. 2021-11-02 12:18:51 +01:00
Dawid Weiss 0544819b78
LUCENE-10200: store git revision in the release folder and read it back from buildAndPushRelease (#419) 2021-11-01 09:29:06 +01:00
Dawid Weiss e470535072
LUCENE-9654: Expressions module gramar antlr code regeneration (#269) 2021-08-27 12:47:19 +02:00
Dawid Weiss 39a2fc62d4
LUCENE-10066: Build does not work with JDK16 as gradle's runtime (#259) 2021-08-26 10:08:37 +02:00
David Smiley 8ac2673791
LUCENE-10003: No C style array declaration (#206)
Most cases of C-style array declarations have been switched.  The Google Java Format, that which we adhere to, disallows C-style array declarations: https://google.github.io/styleguide/javaguide.html#s4.8.3-arrays
Some cases (esp. Snowball) can't be updated.
2021-08-25 17:06:41 -04:00
Dawid Weiss 0cbafa4879 Fix gradle error hints. 2021-08-25 10:03:59 +02:00
Dawid Weiss 2b0378cd4a Use JavaInfo instead of toolchains. Internal but works and is free of toolchain's quirks. 2021-08-25 10:03:59 +02:00
Dawid Weiss aac6581f6e
LUCENE-9915: Add generation/ checksumming task for gen_ForUtil.py (#126) 2021-05-05 22:03:06 +02:00
Dawid Weiss 8eb4eb2611
LUCENE-9909: add checksums of included files for some jflex generation tasks. Fix a task ordering issue with spotless. (#121)
* LUCENE-9909: Some jflex regeneration tasks should have proper dependencies and also check the checksums of included files.

* Force a dependency on low-level spotless tasks so that they're always properly ordered (hell!). Update ASCIITLD and regenerate the remaining code. Add cross-dependencies between generation tasks that take includes as input.
2021-05-02 19:17:18 +02:00
Robert Muir 044d152d95
LUCENE-9928: speed up analysis/icu regeneration (#82)
The compilation of the library is slow, disable optimization as it doesn't speed up our usage of the gennorm2 tool.
Use better heuristic for make parallelism (tests.jvms rather than just hardcoded value of four).
2021-04-22 07:24:44 -04:00
Ignacio Vera e0436872c4
LUCENE-9907: Move PackedInts#getReaderNoHeader() to backwards codec 2021-04-20 09:09:38 +02:00
Ignacio Vera b0662c807c
LUCENE-9907: Remove unused method PackedInts.Mutable#save 2021-04-19 14:52:21 +02:00
Dawid Weiss bd8f182b13
LUCENE-9933: Add non-file properties to wrapped regenerate checksums (#95) 2021-04-19 13:37:47 +02:00
Dawid Weiss beafd113de
LUCENE-9931: Rename checksummed regen. tasks FooInternal and generated wrappers Foo (#88) 2021-04-16 22:35:51 +02:00
Dawid Weiss 0b1d8ccba6
LUCENE-9925: add checksums to snowball-generated files (#80) 2021-04-13 08:59:31 +02:00
Dawid Weiss 3f3917d504 LUCENE-9914: remove stale file. 2021-04-12 20:19:14 +02:00
Dawid Weiss f91700a713
LUCENE-9914: Modernize Emoji regeneration scripts (#78) 2021-04-12 20:16:43 +02:00
Robert Muir 9d15435b15
LUCENE-9916: add a simple regeneration help doc (#73)
Add a simple regeneration help doc

Improve task help and checksum failure message (include corresponding regeneration task). Sorry for being verbose. Maybe somebody will read it. :)

Co-authored-by: Dawid Weiss <dawid.weiss@carrotsearch.com>
2021-04-11 11:28:41 -04:00
Robert Muir b0bd64c620
LUCENE-9924: generate TLD list from IANA TLD db, rather than root zone db (#77)
This adds a bit of simplicity as the file is a simple domain list,
rather than a DNS zone. So the regexes parsing DNS can be removed.

Also the file may change less often as it contains JUST the list of
TLDs, and not any additional DNS metadata.
2021-04-11 11:25:15 -04:00
Robert Muir 15bfb28d7f
LUCENE-9922: checksum files should use a deterministic sort order (#75)
This way the files don't unnecessarily change, depending on filesystem
order or anything else.
2021-04-10 16:00:55 -04:00
Uwe Schindler 779e00542c Make the character printout code uniform (always print at least 4 hex chars) 2021-04-08 16:38:31 +02:00