Apache
/
lucene
mirror of https://github.com/apache/lucene.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

SOLR-8542, SOLR-9746: prefix solr/contrib/ltr's search and response.transform packages with ltr

This commit is contained in:
Christine Poerschke 2016-11-11 18:38:36 +00:00
parent 2c752b04cb
commit 86a515789f
11 changed files with 30 additions and 23 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

10
solr/contrib/ltr/README.md
View File

@ -90,7 +90,7 @@ BONUS: Train an actual machine learning model
... ...
<!-- Query parser used to rerank top docs with a provided model --> <!-- Query parser used to rerank top docs with a provided model -->
<queryParser name="ltr" class="org.apache.solr.search.LTRQParserPlugin" /> <queryParser name="ltr" class="org.apache.solr.ltr.search.LTRQParserPlugin" />
<!-- Transformer that will encode the document features in the response. <!-- Transformer that will encode the document features in the response.
For each document the transformer will add the features as an extra field For each document the transformer will add the features as an extra field
@ -99,7 +99,7 @@ BONUS: Train an actual machine learning model
In order to get the feature vector you will have to In order to get the feature vector you will have to
specify that you want the field (e.g., fl="*,[features]) --> specify that you want the field (e.g., fl="*,[features]) -->
<transformer name="features" class="org.apache.solr.response.transform.LTRFeatureLoggerTransformerFactory" /> <transformer name="features" class="org.apache.solr.ltr.response.transform.LTRFeatureLoggerTransformerFactory" />
<query> <query>
... ...
@ -373,7 +373,7 @@ At this point you'll need to collect feature vectors for each query document pai
from the Extract features section above to do this. An example script has been included in example/train_and_upload_demo_model.py. from the Extract features section above to do this. An example script has been included in example/train_and_upload_demo_model.py.
# Explanation of the core reranking logic # Explanation of the core reranking logic
An LTR model is plugged into the ranking through the [LTRQParserPlugin](/solr/contrib/ltr/src/java/org/apache/solr/search/LTRQParserPlugin.java). The plugin will An LTR model is plugged into the ranking through the [LTRQParserPlugin](/solr/contrib/ltr/src/java/org/apache/solr/ltr/search/LTRQParserPlugin.java). The plugin will
read from the request the model, an instance of [LTRScoringModel](/solr/contrib/ltr/src/java/org/apache/solr/ltr/model/LTRScoringModel.java), read from the request the model, an instance of [LTRScoringModel](/solr/contrib/ltr/src/java/org/apache/solr/ltr/model/LTRScoringModel.java),
plus other parameters. The plugin will generate an LTRQuery, a particular [ReRankQuery](/solr/core/src/java/org/apache/solr/search/AbstractReRankQuery.java). plus other parameters. The plugin will generate an LTRQuery, a particular [ReRankQuery](/solr/core/src/java/org/apache/solr/search/AbstractReRankQuery.java).
It wraps the original solr query for the first pass ranking, and uses the provided model in an It wraps the original solr query for the first pass ranking, and uses the provided model in an
@ -388,13 +388,13 @@ About half the time for ranking is spent in the creation of weights for each fea
<config> <config>
<!-- Query parser used to rerank top docs with a provided model --> <!-- Query parser used to rerank top docs with a provided model -->
<queryParser name="ltr" class="org.apache.solr.search.LTRQParserPlugin"> <queryParser name="ltr" class="org.apache.solr.ltr.search.LTRQParserPlugin">
<int name="threadModule.totalPoolThreads">10</int> <!-- Maximum threads to share for all requests --> <int name="threadModule.totalPoolThreads">10</int> <!-- Maximum threads to share for all requests -->
<int name="threadModule.numThreadsPerRequest">5</int> <!-- Maximum threads to use for a single requests--> <int name="threadModule.numThreadsPerRequest">5</int> <!-- Maximum threads to use for a single requests-->
</queryParser> </queryParser>
<!-- Transformer for extracting features --> <!-- Transformer for extracting features -->
<transformer name="features" class="org.apache.solr.response.transform.LTRFeatureLoggerTransformerFactory"> <transformer name="features" class="org.apache.solr.ltr.response.transform.LTRFeatureLoggerTransformerFactory">
<int name="threadModule.totalPoolThreads">10</int> <!-- Maximum threads to share for all requests --> <int name="threadModule.totalPoolThreads">10</int> <!-- Maximum threads to share for all requests -->
<int name="threadModule.numThreadsPerRequest">5</int> <!-- Maximum threads to use for a single requests--> <int name="threadModule.numThreadsPerRequest">5</int> <!-- Maximum threads to use for a single requests-->
</transformer> </transformer>

4
solr/contrib/ltr/example/solrconfig.xml
View File

@ -839,7 +839,7 @@
</requestHandler> </requestHandler>
<!-- Query parser used to rerank top docs with a provided model --> <!-- Query parser used to rerank top docs with a provided model -->
<queryParser name="ltr" class="org.apache.solr.search.LTRQParserPlugin" > <queryParser name="ltr" class="org.apache.solr.ltr.search.LTRQParserPlugin" >
<int name="threadModule.totalPoolThreads">10</int> <!-- Maximum threads to use for all queries --> <int name="threadModule.totalPoolThreads">10</int> <!-- Maximum threads to use for all queries -->
<int name="threadModule.numThreadsPerRequest">10</int> <!-- Maximum threads to use for a single query--> <int name="threadModule.numThreadsPerRequest">10</int> <!-- Maximum threads to use for a single query-->
</queryParser> </queryParser>
@ -848,7 +848,7 @@
will add the features as an extra field in the response. The name of the field we will be the the name of the will add the features as an extra field in the response. The name of the field we will be the the name of the
transformer enclosed between brackets (in this case [features]). In order to get the feature vector you will have to transformer enclosed between brackets (in this case [features]). In order to get the feature vector you will have to
specify that you want the field (e.g., fl="*,[features]) --> specify that you want the field (e.g., fl="*,[features]) -->
<transformer name="features" class="org.apache.solr.response.transform.LTRFeatureLoggerTransformerFactory" /> <transformer name="features" class="org.apache.solr.ltr.response.transform.LTRFeatureLoggerTransformerFactory" />
<!-- A request handler that returns indented JSON by default --> <!-- A request handler that returns indented JSON by default -->

6
solr/contrib/ltr/src/java/org/apache/solr/response/transform/LTRFeatureLoggerTransformerFactory.java → solr/contrib/ltr/src/java/org/apache/solr/ltr/response/transform/LTRFeatureLoggerTransformerFactory.java
View File

@ -14,7 +14,7 @@
* See the License for the specific language governing permissions and * See the License for the specific language governing permissions and
* limitations under the License. * limitations under the License.
*/ */
package org.apache.solr.response.transform; package org.apache.solr.ltr.response.transform;
import java.io.IOException; import java.io.IOException;
import java.util.Collections; import java.util.Collections;
@ -35,11 +35,13 @@ import org.apache.solr.ltr.SolrQueryRequestContextUtils;
import org.apache.solr.ltr.feature.Feature; import org.apache.solr.ltr.feature.Feature;
import org.apache.solr.ltr.model.LTRScoringModel; import org.apache.solr.ltr.model.LTRScoringModel;
import org.apache.solr.ltr.norm.Normalizer; import org.apache.solr.ltr.norm.Normalizer;
import org.apache.solr.ltr.search.LTRQParserPlugin;
import org.apache.solr.ltr.store.FeatureStore; import org.apache.solr.ltr.store.FeatureStore;
import org.apache.solr.ltr.store.rest.ManagedFeatureStore; import org.apache.solr.ltr.store.rest.ManagedFeatureStore;
import org.apache.solr.request.SolrQueryRequest; import org.apache.solr.request.SolrQueryRequest;
import org.apache.solr.response.ResultContext; import org.apache.solr.response.ResultContext;
import org.apache.solr.search.LTRQParserPlugin; import org.apache.solr.response.transform.DocTransformer;
import org.apache.solr.response.transform.TransformerFactory;
import org.apache.solr.search.SolrIndexSearcher; import org.apache.solr.search.SolrIndexSearcher;
import org.apache.solr.util.SolrPluginUtils; import org.apache.solr.util.SolrPluginUtils;

2
solr/contrib/ltr/src/java/org/apache/solr/response/transform/package-info.java → solr/contrib/ltr/src/java/org/apache/solr/ltr/response/transform/package-info.java
View File

@ -18,6 +18,6 @@
/** /**
* APIs and implementations of {@link org.apache.solr.response.transform.DocTransformer} for modifying documents in Solr request responses * APIs and implementations of {@link org.apache.solr.response.transform.DocTransformer} for modifying documents in Solr request responses
*/ */
package org.apache.solr.response.transform; package org.apache.solr.ltr.response.transform;

7
solr/contrib/ltr/src/java/org/apache/solr/search/LTRQParserPlugin.java → solr/contrib/ltr/src/java/org/apache/solr/ltr/search/LTRQParserPlugin.java
View File

@ -14,7 +14,7 @@
* See the License for the specific language governing permissions and * See the License for the specific language governing permissions and
* limitations under the License. * limitations under the License.
*/ */
package org.apache.solr.search; package org.apache.solr.ltr.search;
import java.io.IOException; import java.io.IOException;
import java.lang.invoke.MethodHandles; import java.lang.invoke.MethodHandles;
@ -40,6 +40,11 @@ import org.apache.solr.ltr.store.rest.ManagedModelStore;
import org.apache.solr.request.SolrQueryRequest; import org.apache.solr.request.SolrQueryRequest;
import org.apache.solr.rest.ManagedResource; import org.apache.solr.rest.ManagedResource;
import org.apache.solr.rest.ManagedResourceObserver; import org.apache.solr.rest.ManagedResourceObserver;
import org.apache.solr.search.AbstractReRankQuery;
import org.apache.solr.search.QParser;
import org.apache.solr.search.QParserPlugin;
import org.apache.solr.search.RankQuery;
import org.apache.solr.search.SyntaxError;
import org.apache.solr.util.SolrPluginUtils; import org.apache.solr.util.SolrPluginUtils;
import org.slf4j.Logger; import org.slf4j.Logger;
import org.slf4j.LoggerFactory; import org.slf4j.LoggerFactory;

2
solr/contrib/ltr/src/java/org/apache/solr/search/package-info.java → solr/contrib/ltr/src/java/org/apache/solr/ltr/search/package-info.java
View File

@ -18,6 +18,6 @@
/** /**
* APIs and classes for {@linkplain org.apache.solr.search.QParserPlugin parsing} and {@linkplain org.apache.solr.search.SolrIndexSearcher processing} search requests * APIs and classes for {@linkplain org.apache.solr.search.QParserPlugin parsing} and {@linkplain org.apache.solr.search.SolrIndexSearcher processing} search requests
*/ */
package org.apache.solr.search; package org.apache.solr.ltr.search;

6
solr/contrib/ltr/src/java/overview.html
View File

@ -35,11 +35,11 @@ feature engineering and feature extraction.
</p> </p>
<h2> Code structure </h2> <h2> Code structure </h2>
<p> <p>
A Learning to Rank model is plugged into the ranking through the {@link org.apache.solr.search.LTRQParserPlugin}, A Learning to Rank model is plugged into the ranking through the {@link org.apache.solr.ltr.search.LTRQParserPlugin},
a {@link org.apache.solr.search.QParserPlugin}. The plugin will a {@link org.apache.solr.search.QParserPlugin}. The plugin will
read from the request the model (instance of {@link org.apache.solr.ltr.model.LTRScoringModel}) read from the request the model (instance of {@link org.apache.solr.ltr.model.LTRScoringModel})
used to perform the request plus other used to perform the request plus other
parameters. The plugin will generate a {@link org.apache.solr.search.LTRQParserPlugin.LTRQuery LTRQuery}: parameters. The plugin will generate a {@link org.apache.solr.ltr.search.LTRQParserPlugin.LTRQuery LTRQuery}:
a particular {@link org.apache.solr.search.RankQuery} a particular {@link org.apache.solr.search.RankQuery}
that will encapsulate the given model and use it to that will encapsulate the given model and use it to
rescore and rerank the document (by using an {@link org.apache.solr.ltr.LTRRescorer}). rescore and rerank the document (by using an {@link org.apache.solr.ltr.LTRRescorer}).
@ -73,7 +73,7 @@ of features. One benefit of extending the Query object is that we can reuse
Query as a feature, see for example {@link org.apache.solr.ltr.feature.SolrFeature}. Query as a feature, see for example {@link org.apache.solr.ltr.feature.SolrFeature}.
Features for a document can also be returned in the response by Features for a document can also be returned in the response by
using the FeatureTransformer (a {@link org.apache.solr.response.transform.DocTransformer DocTransformer}) using the FeatureTransformer (a {@link org.apache.solr.response.transform.DocTransformer DocTransformer})
provided by {@link org.apache.solr.response.transform.LTRFeatureLoggerTransformerFactory}. provided by {@link org.apache.solr.ltr.response.transform.LTRFeatureLoggerTransformerFactory}.
</p> </p>
<p> <p>
{@link org.apache.solr.ltr.store} contains all the logic to store all the features and the models. {@link org.apache.solr.ltr.store} contains all the logic to store all the features and the models.

4
solr/contrib/ltr/src/test-files/solr/collection1/conf/solrconfig-ltr.xml
View File

@ -21,7 +21,7 @@
<!-- Query parser used to rerank top docs with a provided model --> <!-- Query parser used to rerank top docs with a provided model -->
<queryParser name="ltr" <queryParser name="ltr"
class="org.apache.solr.search.LTRQParserPlugin" /> class="org.apache.solr.ltr.search.LTRQParserPlugin" />
<query> <query>
<filterCache class="solr.FastLRUCache" size="4096" <filterCache class="solr.FastLRUCache" size="4096"
@ -36,7 +36,7 @@
enclosed between brackets (in this case [fv]). In order to get the feature enclosed between brackets (in this case [fv]). In order to get the feature
vector you will have to specify that you want the field (e.g., fl="*,[fv]) --> vector you will have to specify that you want the field (e.g., fl="*,[fv]) -->
<transformer name="fv" <transformer name="fv"
class="org.apache.solr.response.transform.LTRFeatureLoggerTransformerFactory" /> class="org.apache.solr.ltr.response.transform.LTRFeatureLoggerTransformerFactory" />
<updateHandler class="solr.DirectUpdateHandler2"> <updateHandler class="solr.DirectUpdateHandler2">
<autoCommit> <autoCommit>

4
solr/contrib/ltr/src/test-files/solr/collection1/conf/solrconfig-ltr_Th10_10.xml
View File

@ -20,7 +20,7 @@
<!-- Query parser used to rerank top docs with a provided model --> <!-- Query parser used to rerank top docs with a provided model -->
<queryParser name="ltr" class="org.apache.solr.search.LTRQParserPlugin" > <queryParser name="ltr" class="org.apache.solr.ltr.search.LTRQParserPlugin" >
<int name="threadModule.totalPoolThreads">10</int> <!-- Maximum threads to use for all queries --> <int name="threadModule.totalPoolThreads">10</int> <!-- Maximum threads to use for all queries -->
<int name="threadModule.numThreadsPerRequest">10</int> <!-- Maximum threads to use for a single query--> <int name="threadModule.numThreadsPerRequest">10</int> <!-- Maximum threads to use for a single query-->
</queryParser> </queryParser>
@ -40,7 +40,7 @@
enclosed between brackets (in this case [fv]). In order to get the feature enclosed between brackets (in this case [fv]). In order to get the feature
vector you will have to specify that you want the field (e.g., fl="*,[fv]) --> vector you will have to specify that you want the field (e.g., fl="*,[fv]) -->
<transformer name="fv" <transformer name="fv"
class="org.apache.solr.response.transform.LTRFeatureLoggerTransformerFactory" /> class="org.apache.solr.ltr.response.transform.LTRFeatureLoggerTransformerFactory" />
<updateHandler class="solr.DirectUpdateHandler2"> <updateHandler class="solr.DirectUpdateHandler2">
<autoCommit> <autoCommit>

6
solr/contrib/ltr/src/test-files/solr/collection1/conf/solrconfig-multiseg.xml
View File

@ -20,7 +20,7 @@
<!-- Query parser used to rerank top docs with a provided model --> <!-- Query parser used to rerank top docs with a provided model -->
<queryParser name="ltr" class="org.apache.solr.search.LTRQParserPlugin" /> <queryParser name="ltr" class="org.apache.solr.ltr.search.LTRQParserPlugin" />
<maxBufferedDocs>1</maxBufferedDocs> <maxBufferedDocs>1</maxBufferedDocs>
<mergePolicyFactory class="org.apache.solr.index.TieredMergePolicyFactory"> <mergePolicyFactory class="org.apache.solr.index.TieredMergePolicyFactory">
@ -33,7 +33,7 @@
enclosed between brackets (in this case [fv]). In order to get the feature enclosed between brackets (in this case [fv]). In order to get the feature
vector you will have to specify that you want the field (e.g., fl="*,[fv]) --> vector you will have to specify that you want the field (e.g., fl="*,[fv]) -->
<transformer name="features" <transformer name="features"
class="org.apache.solr.response.transform.LTRFeatureLoggerTransformerFactory" /> class="org.apache.solr.ltr.response.transform.LTRFeatureLoggerTransformerFactory" />
<updateHandler class="solr.DirectUpdateHandler2"> <updateHandler class="solr.DirectUpdateHandler2">
<autoCommit> <autoCommit>
@ -59,4 +59,4 @@
</lst> </lst>
</requestHandler> </requestHandler>
</config> </config>

2
solr/contrib/ltr/src/test/org/apache/solr/ltr/store/rest/TestModelManager.java
View File

@ -22,10 +22,10 @@ import org.apache.solr.ltr.TestRerankBase;
import org.apache.solr.ltr.feature.FieldValueFeature; import org.apache.solr.ltr.feature.FieldValueFeature;
import org.apache.solr.ltr.feature.ValueFeature; import org.apache.solr.ltr.feature.ValueFeature;
import org.apache.solr.ltr.model.LinearModel; import org.apache.solr.ltr.model.LinearModel;
import org.apache.solr.ltr.search.LTRQParserPlugin;
import org.apache.solr.rest.ManagedResource; import org.apache.solr.rest.ManagedResource;
import org.apache.solr.rest.ManagedResourceStorage; import org.apache.solr.rest.ManagedResourceStorage;
import org.apache.solr.rest.RestManager; import org.apache.solr.rest.RestManager;
import org.apache.solr.search.LTRQParserPlugin;
import org.junit.Before; import org.junit.Before;
import org.junit.BeforeClass; import org.junit.BeforeClass;
import org.junit.Test; import org.junit.Test;