lucene/help
Michael McCandless 2ca906d99c Bound the RAM used by the NodeHash (sharing common suffixes) during FST compilation (#12633)
* tweak comments; change if to switch

* remove old SOPs, minor comment styling, fixed silly performance bug on rehash using the wrong bitsRequired (count vs node)

* first raw cut; some nocommits added; some tests fail

* tests pass!

* fix silly fallback hash bug

* remove SOPs; add some temporary debugging metrics

* add temporary tool to test FST performance across differing NodeHash sizes

* remove (now deleted) shouldShareNonSingletonNodes call from Lucene90BlockTreeTermsWriter

* add simple tool to render results table to GitHub MD

* add simple temporary tool to iterate all terms from a provided luceneutil wikipedia index and build an FST from them

* first cut at using packed ints for hash t able again

* add some nocommits; tweak test_all_sizes.py to new RAM usage approach; when half of the double barrel is full, allocate new primary hash at full size to save cost of continuously rehashing for a large FST

* switch to limit suffix hash by RAM usage not count (more intuitive for users); clean up some stale nocommits

* switch to more intuitive approximate RAM (mb) limit for allowed size of NodeHash

* nuke a few nocommits; a few more remain

* remove DO_PRINT_HASH_RAM

* no more FST pruning

* remove final nocommit: randomly change allowed NodeHash suffix RAM size in TestFSTs.testRealTerms

* remove SOP

* tidy

* delete temp utility tools

* remove dead (FST pruning) code

* add CHANGES entry; fix one missed fst.addNode -> fstCompiler.addNode during merge conflict resolution

* remove a mal-formed nocommit

* fold PR feedback

* fold feedback

* add gradle help test details on how to specify heap size for the test JVM; fix bogus assert (uncovered by Test2BFST); add TODO to Test2BFST anticipating building massive FSTs in small bounded RAM

* suppress sysout checks for Test2BFSTs; add helpful comment showing how to run it directly

* tidy
2023-11-20 13:32:42 -05:00
..
IDEs.txt LUCENE-9438: Eclipse IDE support with gradle build system (#1761) 2020-08-21 21:47:11 +02:00
dependencies.txt Nuke more unused/ obsolete refs. 2021-03-19 13:11:37 +01:00
forbiddenApis.txt Revert "SOLR-12930: move Gradle docs from ./help/ to new ./dev-docs/ directory" 2020-01-24 15:56:00 -06:00
formatting.txt LUCENE-9564: add spotless and gjf (automatic code formatter) 2020-12-17 13:11:54 +01:00
git.txt LUCENE-9570: add parial reformatting commit reference to .git-blame-ignore-revs [thanks rmuir] and add a note to git.txt aiding in making this a persistent local option. 2020-12-18 13:10:47 +01:00
jvms.txt LUCENE-9312: Allow builds against arbitrary JVMs (squashed 2020-07-21 09:19:38 +02:00
localSettings.txt Generate gradle.properties from gradlew (#12131) 2023-02-06 19:51:03 +01:00
publishing.txt Fix typo 2021-10-14 13:28:10 -05:00
regeneration.txt LUCENE-9933: Add non-file properties to wrapped regenerate checksums (#95) 2021-04-19 13:37:47 +02:00
tests.txt Bound the RAM used by the NodeHash (sharing common suffixes) during FST compilation (#12633) 2023-11-20 13:32:42 -05:00
workflow.txt Nuke more unused/ obsolete refs. 2021-03-19 13:11:37 +01:00