SOLR-15121: Move XSLT (tr param) response writer and update request handler to scripting contrib (#2306)

* relocate xslt related classes into scripting contrib
* relocating files to scripting and seperating out unit tests
* relocate files under test-files/scripting/solr, similar to how we do it in other contribs.  deals with some issues in finding files
* Reformatting using the Google Java Format...
* use actual param name, not the variable to properly test api!
* Clean up references to paths, and deal with the mish mash of Xslt and XSLT in class names.
* Move XSLT processing out of XMLLoader
* Move TransformerProvider.Dedupe getTransformer logic.


Co-authored-by: epugh@opensourceconnections.com <>
Co-authored-by: David Smiley <dsmiley@apache.org>
This commit is contained in:
Eric Pugh 2021-02-15 13:16:18 -05:00 committed by GitHub
parent 5856c0f176
commit e6d9eaaf00
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
46 changed files with 1028 additions and 287 deletions

View File

@ -0,0 +1,21 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
/**
* Classes related to applying run time scripting changes to Solr.
*/
package org.apache.solr.scripting;

View File

@ -14,38 +14,39 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.solr.util.xslt;
package org.apache.solr.scripting.xslt;
import static org.apache.solr.scripting.xslt.XSLTConstants.CONTEXT_TRANSFORMER_KEY;
import java.io.IOException;
import java.lang.invoke.MethodHandles;
import java.util.Map;
import java.util.concurrent.TimeUnit;
import org.apache.solr.common.util.TimeSource;
import org.apache.solr.util.TimeOut;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.commons.io.IOUtils;
import javax.xml.transform.Templates;
import javax.xml.transform.Transformer;
import javax.xml.transform.TransformerConfigurationException;
import javax.xml.transform.TransformerFactory;
import javax.xml.transform.stream.StreamSource;
import org.apache.commons.io.IOUtils;
import org.apache.lucene.util.ResourceLoader;
import org.apache.solr.util.SystemIdResolver;
import org.apache.solr.common.util.TimeSource;
import org.apache.solr.common.util.XMLErrorLogger;
import org.apache.solr.core.SolrConfig;
import org.apache.solr.request.SolrQueryRequest;
import org.apache.solr.util.SystemIdResolver;
import org.apache.solr.util.TimeOut;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/** Singleton that creates a Transformer for the XSLTServletFilter.
/**
* Singleton that creates a Transformer for XSLT
* For now, only caches the last created Transformer, but
* could evolve to use an LRU cache of Transformers.
*
* See http://www.javaworld.com/javaworld/jw-05-2003/jw-0502-xsl_p.html for
* one possible way of improving caching.
*/
public class TransformerProvider {
class TransformerProvider {
private String lastFilename;
private Templates lastTemplates = null;
private TimeOut cacheExpiresTimeout;
@ -64,6 +65,27 @@ public class TransformerProvider {
+ " and xsltCacheLifetimeSeconds is set to a sufficiently high value.");
}
/**
* Get Transformer from request context, or from TransformerProvider. This allows either
* getContentType(...) or write(...) to instantiate the Transformer, depending on which one is
* called first, then the other one reuses the same Transformer
*/
static Transformer getTransformer(
SolrQueryRequest request, String xslt, int xsltCacheLifetimeSeconds) throws IOException {
// not the cleanest way to achieve this
// no need to synchronize access to context, right?
// Nothing else happens with it at the same time
final Map<Object, Object> ctx = request.getContext();
Transformer result = (Transformer) ctx.get(CONTEXT_TRANSFORMER_KEY);
if (result == null) {
SolrConfig solrConfig = request.getCore().getSolrConfig();
result = instance.getTransformer(solrConfig, xslt, xsltCacheLifetimeSeconds);
result.setErrorListener(xmllog);
ctx.put(CONTEXT_TRANSFORMER_KEY, result);
}
return result;
}
/** Return a new Transformer, possibly created from our cached Templates object
* @throws IOException If there is a low-level I/O error.
*/

View File

@ -0,0 +1,26 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.solr.scripting.xslt;
class XSLTConstants {
/** Transformer param */
static final String TR = "tr";
static final String CONTEXT_TRANSFORMER_KEY = "xsltwriter.transformer";
static final int XSLT_CACHE_DEFAULT = 60;
static final String XSLT_CACHE_PARAM = "xsltCacheLifetimeSeconds";
}

View File

@ -14,7 +14,11 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.solr.response;
package org.apache.solr.scripting.xslt;
import static org.apache.solr.scripting.xslt.XSLTConstants.TR;
import static org.apache.solr.scripting.xslt.XSLTConstants.XSLT_CACHE_DEFAULT;
import static org.apache.solr.scripting.xslt.XSLTConstants.XSLT_CACHE_PARAM;
import java.io.BufferedReader;
import java.io.CharArrayReader;
@ -22,48 +26,36 @@ import java.io.CharArrayWriter;
import java.io.IOException;
import java.io.Reader;
import java.io.Writer;
import java.lang.invoke.MethodHandles;
import java.util.Map;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import javax.xml.transform.Transformer;
import javax.xml.transform.TransformerException;
import javax.xml.transform.stream.StreamResult;
import javax.xml.transform.stream.StreamSource;
import org.apache.solr.core.SolrConfig;
import org.apache.solr.common.params.CommonParams;
import org.apache.solr.common.params.SolrParams;
import org.apache.solr.common.util.NamedList;
import org.apache.solr.common.util.XMLErrorLogger;
import org.apache.solr.request.SolrQueryRequest;
import org.apache.solr.util.xslt.TransformerProvider;
import org.apache.solr.response.QueryResponseWriter;
import org.apache.solr.response.SolrQueryResponse;
import org.apache.solr.response.XMLWriter;
/** QueryResponseWriter which captures the output of the XMLWriter
* (in memory for now, not optimal performancewise), and applies an XSLT transform
/**
* Customize the format of your search results via XSL stylesheet applied to the default
* XML response format.
*
* QueryResponseWriter captures the output of the XMLWriter
* (in memory for now, not optimal performance-wise), and applies an XSLT transform
* to it.
*/
public class XSLTResponseWriter implements QueryResponseWriter {
public static final String DEFAULT_CONTENT_TYPE = "application/xml";
public static final String CONTEXT_TRANSFORMER_KEY = "xsltwriter.transformer";
private Integer xsltCacheLifetimeSeconds = null;
public static final int XSLT_CACHE_DEFAULT = 60;
private static final String XSLT_CACHE_PARAM = "xsltCacheLifetimeSeconds";
private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
private static final XMLErrorLogger xmllog = new XMLErrorLogger(log);
@Override
public void init(@SuppressWarnings({"rawtypes"})NamedList n) {
final SolrParams p = n.toSolrParams();
xsltCacheLifetimeSeconds = p.getInt(XSLT_CACHE_PARAM,XSLT_CACHE_DEFAULT);
log.info("xsltCacheLifetimeSeconds={}", xsltCacheLifetimeSeconds);
xsltCacheLifetimeSeconds = p.getInt(XSLT_CACHE_PARAM, XSLT_CACHE_DEFAULT);
}
@Override
public String getContentType(SolrQueryRequest request, SolrQueryResponse response) {
Transformer t = null;
@ -112,26 +104,9 @@ public class XSLTResponseWriter implements QueryResponseWriter {
}
}
/** Get Transformer from request context, or from TransformerProvider.
* This allows either getContentType(...) or write(...) to instantiate the Transformer,
* depending on which one is called first, then the other one reuses the same Transformer
*/
protected Transformer getTransformer(SolrQueryRequest request) throws IOException {
final String xslt = request.getParams().get(CommonParams.TR,null);
if(xslt==null) {
throw new IOException("'" + CommonParams.TR + "' request parameter is required to use the XSLTResponseWriter");
}
// not the cleanest way to achieve this
SolrConfig solrConfig = request.getCore().getSolrConfig();
// no need to synchronize access to context, right?
// Nothing else happens with it at the same time
final Map<Object,Object> ctx = request.getContext();
Transformer result = (Transformer)ctx.get(CONTEXT_TRANSFORMER_KEY);
if(result==null) {
result = TransformerProvider.instance.getTransformer(solrConfig, xslt,xsltCacheLifetimeSeconds.intValue());
result.setErrorListener(xmllog);
ctx.put(CONTEXT_TRANSFORMER_KEY,result);
}
return result;
final String xslt = request.getParams().required().get(TR);
return TransformerProvider.getTransformer(request, xslt, xsltCacheLifetimeSeconds);
}
}

View File

@ -0,0 +1,140 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.solr.scripting.xslt;
import static org.apache.solr.scripting.xslt.XSLTConstants.*;
import java.util.Map;
import javax.xml.stream.XMLStreamException;
import javax.xml.stream.XMLStreamReader;
import javax.xml.transform.Transformer;
import javax.xml.transform.TransformerException;
import javax.xml.transform.dom.DOMResult;
import javax.xml.transform.dom.DOMSource;
import javax.xml.transform.sax.SAXSource;
import com.google.common.annotations.VisibleForTesting;
import org.apache.solr.common.EmptyEntityResolver;
import org.apache.solr.common.SolrException;
import org.apache.solr.common.params.SolrParams;
import org.apache.solr.common.util.ContentStream;
import org.apache.solr.common.util.ContentStreamBase;
import org.apache.solr.common.util.NamedList;
import org.apache.solr.handler.UpdateRequestHandler;
import org.apache.solr.handler.loader.XMLLoader;
import org.apache.solr.request.SolrQueryRequest;
import org.apache.solr.response.SolrQueryResponse;
import org.apache.solr.update.processor.UpdateRequestProcessor;
import org.xml.sax.InputSource;
import org.xml.sax.XMLReader;
/**
* Send XML formatted documents to Solr, transforming them from the original XML
* format to the Solr XML format using an XSLT stylesheet via the 'tr' parameter.
*/
public class XSLTUpdateRequestHandler extends UpdateRequestHandler {
@Override
public void init(@SuppressWarnings({"rawtypes"})NamedList args) {
super.init(args);
setAssumeContentType("application/xml");
SolrParams p = null;
if (args != null) {
p = args.toSolrParams();
}
final XsltXMLLoader loader = new XsltXMLLoader().init(p);
loaders = Map.of("application/xml", loader, "text/xml", loader);
}
@VisibleForTesting
static class XsltXMLLoader extends XMLLoader {
int xsltCacheLifetimeSeconds;
@Override
public XsltXMLLoader init(SolrParams args) {
super.init(args);
xsltCacheLifetimeSeconds = XSLT_CACHE_DEFAULT;
if (args != null) {
xsltCacheLifetimeSeconds = args.getInt(XSLT_CACHE_PARAM, XSLT_CACHE_DEFAULT);
}
return this;
}
@Override
public void load(
SolrQueryRequest req,
SolrQueryResponse rsp,
ContentStream stream,
UpdateRequestProcessor processor)
throws Exception {
String tr = req.getParams().get(TR, null);
if (tr == null) {
super.load(req, rsp, stream, processor); // no XSLT; do standard processing
return;
}
if (req.getCore().getCoreDescriptor().isConfigSetTrusted() == false) {
throw new SolrException(
SolrException.ErrorCode.UNAUTHORIZED,
"The configset for this collection was uploaded without any authentication in place,"
+ " and this operation is not available for collections with untrusted configsets. To use this feature, re-upload the configset"
+ " after enabling authentication and authorization.");
}
final Transformer t = TransformerProvider.getTransformer(req, tr, xsltCacheLifetimeSeconds);
final DOMResult result = new DOMResult();
// first step: read XML and build DOM using Transformer (this is no overhead, as XSL always
// produces
// an internal result DOM tree, we just access it directly as input for StAX):
try (var is = stream.getStream()) {
final XMLReader xmlr = saxFactory.newSAXParser().getXMLReader();
xmlr.setErrorHandler(xmllog);
xmlr.setEntityResolver(EmptyEntityResolver.SAX_INSTANCE);
final InputSource isrc = new InputSource(is);
isrc.setEncoding(ContentStreamBase.getCharsetFromContentType(stream.getContentType()));
final SAXSource source = new SAXSource(xmlr, isrc);
t.transform(source, result);
} catch (TransformerException e) {
throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, e.toString(), e);
}
// second step: feed the intermediate DOM tree into StAX parser:
XMLStreamReader parser = null; // does not implement AutoCloseable!
try {
parser = inputFactory.createXMLStreamReader(new DOMSource(result.getNode()));
this.processUpdate(req, processor, parser);
} catch (XMLStreamException e) {
throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, e.toString(), e);
} finally {
if (parser != null) parser.close();
}
}
}
//////////////////////// SolrInfoMBeans methods //////////////////////
@Override
public String getDescription() {
return "Add documents with XML, transforming with XSLT first";
}
}

View File

@ -16,8 +16,8 @@
*/
/**
* XSLT related utilities (deprecated package, do not add new classes)
* XSLT related classes.
*/
package org.apache.solr.util.xslt;
package org.apache.solr.scripting.xslt;

View File

@ -0,0 +1,382 @@
%PDF-1.3
%ª«¬­
4 0 obj
<< /Type /Info
/Producer (FOP 0.20.5) >>
endobj
5 0 obj
<< /Length 425 /Filter [ /ASCII85Decode /FlateDecode ]
>>
stream
Gb!$BYuAO_'ZTnF'lQbNnGsdiUK'C#3dAWc3lI>k\P#:a@Qja<+itJa;R]7&ni\$9pOi?T._;3m?jT+q7>,P^70oB=!nr]%k%\U^KVqaF4*Z`$VJ7Gs`T5OO`(tY]Q1`-5*m;!--h%?*_0SbIU\BV=OFg<#%YcH_YI$(sDCIJts'M2*drjRrJE!OM7HP!^-&EW>B\:RYFnaY.m[$s5f"XG0>^fduHe6/++D0fY3@AWR@HYabmQ5jDQ.c0>I.uQX&(lA@VLm_s_9XnBh7%"*/%^]AO3eTI!BTo'pF?%''A*PDU*NW%d`2@p'@:D@U??4PP08m[K4N,8,(e`N+\7n+a>ac%q#,D8DRQ*3l]MS>'gn3lWNGmRAtQ7n]eDnLPrD!?DEdB/hNarb_7$B7U-H7!['nXLkV_no5AHq`>6~>
endstream
endobj
6 0 obj
<< /Type /Page
/Parent 1 0 R
/MediaBox [ 0 0 612 792 ]
/Resources 3 0 R
/Contents 5 0 R
/Annots 7 0 R
>>
endobj
7 0 obj
[
8 0 R
10 0 R
12 0 R
]
endobj
8 0 obj
<< /Type /Annot
/Subtype /Link
/Rect [ 102.0 559.666 137.324 547.666 ]
/C [ 0 0 0 ]
/Border [ 0 0 0 ]
/A 9 0 R
/H /I
>>
endobj
10 0 obj
<< /Type /Annot
/Subtype /Link
/Rect [ 102.0 541.466 164.648 529.466 ]
/C [ 0 0 0 ]
/Border [ 0 0 0 ]
/A 11 0 R
/H /I
>>
endobj
12 0 obj
<< /Type /Annot
/Subtype /Link
/Rect [ 102.0 523.266 154.016 511.266 ]
/C [ 0 0 0 ]
/Border [ 0 0 0 ]
/A 13 0 R
/H /I
>>
endobj
14 0 obj
<< /Length 2197 /Filter [ /ASCII85Decode /FlateDecode ]
>>
stream
Gb"/)9lo&I&A@C2n5a2!7YkueV^?,ABrC@*[F.^sK-J\u-^*\VZ9A3]?'#&sU^3,]d[;/F9HjMs^A"j:!rHNC?7rs!0)f1q`$?\lOaRt/g/f.>-Am[t'`RUrGL7Uk8K90.i-up;qeIYfjWZ2&ki:[3`TuXFj]`a&Hbo8r&P(RZ+M_>&eY.T4jXOI%UHbq1GnF>g$KgW%R24nBkc\[qA$(koU$isG(W7`PE,nMam;U4(ZC8,Ca!_P2VYf>\V0gK0g;-.E[Y(&s=+&g6ms""'Ip>0b/D!>a&PX9eo_tuueR:b=r@6Q5LM],XbK;&L$0WubNX9c"=FM$543G_>rAQ_%2/dW<)/"U1&]l:AZ&\Mif8sF`r5>b<$lqK"2t]maZ*oDb!^$Zn6OC'%XkI];&*rkLP1BMGI@$,0fK(=gC-3q7n7d4EQ4DepBc'^Q^A%e?19a(`S*FHTN*RNjP&P%2`6%jpOU\DBUN)cnMYa3PQ!sYETiGJi'q>>m*e;[,.1l\rZo3K;>$K"a1:s3pU>o+:'7fND!+6GV@2G;qf`\`=J#WkOjSke<1f>VfbcUtXM"1jGN:@Ptec8Mc-hmS5S>q/nAY%[4%7BCI![NA:We(41]ld_`pU80;+e`1DbG.RQ:'#GQJAL2![aIWY'A*Y_>mF7>2S0IWM%nLg3%%;7r5=;3!7]05r?Ft-6I]9n9<Lb+:B=E*!8^]$9`UiKf`\`M;-o%O.g0)F=(=?[qc+HJhq0BO9jfLD<O'9]*n,ZrK>C\fUUF6R\9bPEVSutd9LFTpaoaP7Iuus-S#S.3;sVu-*T/:&2Ld]&g0oHoo`TmR'b]ps6hq9s&f+6_5c(k"m96-f:YA!:)K:q+(Hl=t`:+"<<V9NSKW!;_M1J^/lHH8on!\i;gmF)[%KTE$#LU#%U:#tmP/L6uEKiNP):=VPM<Lt!NI>lQm6B=K&/r/Ep6Y]EG.T/34(fT0=6_m5PA-7PVo:"r)W'.mX>1A8Yg9kfa?"Qp+ta7Hb$FM`*OP^>3Sg)P[?jIXd]i]"h)Tdjnm[6@=kmEBkP1/K[bg`"7U:BWk^=!+3\ANTnN75*Rh_<-UA*!&rr#KW/7EXkeJU9GF5RA,#kqJ5aC9Ra5,PsiI`uF23/B"nkPHe2Q;B@pBXGM-i;<'oOM,dc3'qL)Ne,OV2.*f^Bt;0P#roPn?h]@<RnIe!7?\#tM[!.2blSX<d1X(%ZY!g88S_/L=t5jCWooeWL(tk*\hmj-5JMtK[V#5J>-63,-9lQSF!dic13Ag\_]m=7Llb\*&C+>\+o6)Y,C._?+X1Qok%j>f[#T!,CD2T4cL'.Nb_Vit&M]!j7j6LHB.g9AQre&be$gJ<c;+UTKUa!5E#pM8dJ%N!d5/ucVUFL(+oBhWGlK\([L#B;k>hbAg68kDJf@XZ7'2791RD*qAP]u")(lEjX)\-#O$aK(E<RN^:9L@8L6]YUAt1>]jq*3XbL:3q:o&9gcZLl?:E-l'-dHf;;_hhH3m/Q<FPY/g[]/::WS>3]9jJRn>Z8]1Gt6PAVJ[r2gsg=4$!6I$RQ@Y6;H(U>,LWdW>Z5iTYZ'tAcSfoN,U=/fIoA::l8X^fXIa4m3-]9$Zc\E0H^!pmfeMjW3#p1J)pbH^VZML"NZ$U,Yg;f[AVrZRhlRCC[)D*>K0IRWR98A=<>dPSd)@Ec)OXGjK01hM%!FhVR[I<5Va3V,I"YuQZb-,XEM!Gk_-r<9T0W#M!!;RX!]MtBdJ0ah'FCoNF1r"gmU>Rb4aE:Z'I)d-f_1:B0gfmnM?K9ljY>R%*Fc9oYiohHndi(!dK+]ElID:<Q"lXKqqTE&r':_6UaFT`(mD'NPMh:t7!_)^B.dso/\CW_RVQ?:#W"(WA76('o9qt$^4krFT!Y_XaP"`3>'g:PKq6fKKHdO>bmG-2]ZmVcqs+ef-EWR(1Da)F&CoL[['3)UZ^!fo+Ua2NSC7m5oIXlLoF)+cWUr/MaMP@shSN$gD*jB=:/ru]MF>3-m'j6_-'>(Uq'PN4Fl*XC8ABmg\b`kmI@<0Sh)bkNopK]E6S7,V*o!<)infW?).%mtC2S8!kqh$BpiWu=4)><W_fs1`$.q$[DXh*J8JH8r!Z".=W&OPNF-1>.Wm+Mt.YPC"ZlO^Ge*Y5)8QlX2<So:7j.#u?pp;NtbZel#<_#-;#n#GSg-N\`U2aA6`FHJFIG6tJeCP`~>
endstream
endobj
15 0 obj
<< /Type /Page
/Parent 1 0 R
/MediaBox [ 0 0 612 792 ]
/Resources 3 0 R
/Contents 14 0 R
/Annots 16 0 R
>>
endobj
16 0 obj
[
17 0 R
18 0 R
19 0 R
20 0 R
21 0 R
22 0 R
23 0 R
24 0 R
25 0 R
26 0 R
27 0 R
28 0 R
29 0 R
]
endobj
17 0 obj
<< /Type /Annot
/Subtype /Link
/Rect [ 232.344 608.466 372.012 596.466 ]
/C [ 0 0 0 ]
/Border [ 0 0 0 ]
/A << /URI (mailto:solr-user@lucene.apache.org)
/S /URI >>
/H /I
>>
endobj
18 0 obj
<< /Type /Annot
/Subtype /Link
/Rect [ 108.0 591.266 189.336 579.266 ]
/C [ 0 0 0 ]
/Border [ 0 0 0 ]
/A << /URI (mailto:solr-user-subscribe@lucene.apache.org)
/S /URI >>
/H /I
>>
endobj
19 0 obj
<< /Type /Annot
/Subtype /Link
/Rect [ 108.0 578.066 215.988 566.066 ]
/C [ 0 0 0 ]
/Border [ 0 0 0 ]
/A << /URI (mailto:solr-user-unsubscribe@lucene.apache.org)
/S /URI >>
/H /I
>>
endobj
20 0 obj
<< /Type /Annot
/Subtype /Link
/Rect [ 108.0 564.866 197.316 552.866 ]
/C [ 0 0 0 ]
/Border [ 0 0 0 ]
/A << /URI (http://mail-archives.apache.org/mod_mbox/lucene-solr-user/)
/S /URI >>
/H /I
>>
endobj
21 0 obj
<< /Type /Annot
/Subtype /Link
/Rect [ 453.924 564.866 475.26 552.866 ]
/C [ 0 0 0 ]
/Border [ 0 0 0 ]
/A << /URI (http://wiki.apache.org/solr/SolrResources)
/S /URI >>
/H /I
>>
endobj
22 0 obj
<< /Type /Annot
/Subtype /Link
/Rect [ 259.668 441.722 396.672 429.722 ]
/C [ 0 0 0 ]
/Border [ 0 0 0 ]
/A << /URI (mailto:solr-dev@lucene.apache.org)
/S /URI >>
/H /I
>>
endobj
23 0 obj
<< /Type /Annot
/Subtype /Link
/Rect [ 108.0 424.522 189.336 412.522 ]
/C [ 0 0 0 ]
/Border [ 0 0 0 ]
/A << /URI (mailto:solr-dev-subscribe@lucene.apache.org)
/S /URI >>
/H /I
>>
endobj
24 0 obj
<< /Type /Annot
/Subtype /Link
/Rect [ 108.0 411.322 215.988 399.322 ]
/C [ 0 0 0 ]
/Border [ 0 0 0 ]
/A << /URI (mailto:solr-dev-unsubscribe@lucene.apache.org)
/S /URI >>
/H /I
>>
endobj
25 0 obj
<< /Type /Annot
/Subtype /Link
/Rect [ 108.0 398.122 197.316 386.122 ]
/C [ 0 0 0 ]
/Border [ 0 0 0 ]
/A << /URI (http://mail-archives.apache.org/mod_mbox/lucene-solr-dev/)
/S /URI >>
/H /I
>>
endobj
26 0 obj
<< /Type /Annot
/Subtype /Link
/Rect [ 453.924 398.122 475.26 386.122 ]
/C [ 0 0 0 ]
/Border [ 0 0 0 ]
/A << /URI (http://wiki.apache.org/solr/SolrResources)
/S /URI >>
/H /I
>>
endobj
27 0 obj
<< /Type /Annot
/Subtype /Link
/Rect [ 294.624 296.178 403.284 284.178 ]
/C [ 0 0 0 ]
/Border [ 0 0 0 ]
/A << /URI (version_control.html)
/S /URI >>
/H /I
>>
endobj
28 0 obj
<< /Type /Annot
/Subtype /Link
/Rect [ 108.0 265.778 189.336 253.778 ]
/C [ 0 0 0 ]
/Border [ 0 0 0 ]
/A << /URI (mailto:solr-commits-subscribe@lucene.apache.org)
/S /URI >>
/H /I
>>
endobj
29 0 obj
<< /Type /Annot
/Subtype /Link
/Rect [ 108.0 252.578 215.988 240.578 ]
/C [ 0 0 0 ]
/Border [ 0 0 0 ]
/A << /URI (mailto:solr-commits-unsubscribe@lucene.apache.org)
/S /URI >>
/H /I
>>
endobj
31 0 obj
<<
/Title (\376\377\0\61\0\40\0\125\0\163\0\145\0\162\0\163)
/Parent 30 0 R
/Next 32 0 R
/A 9 0 R
>> endobj
32 0 obj
<<
/Title (\376\377\0\62\0\40\0\104\0\145\0\166\0\145\0\154\0\157\0\160\0\145\0\162\0\163)
/Parent 30 0 R
/Prev 31 0 R
/Next 33 0 R
/A 11 0 R
>> endobj
33 0 obj
<<
/Title (\376\377\0\63\0\40\0\103\0\157\0\155\0\155\0\151\0\164\0\163)
/Parent 30 0 R
/Prev 32 0 R
/A 13 0 R
>> endobj
34 0 obj
<< /Type /Font
/Subtype /Type1
/Name /F3
/BaseFont /Helvetica-Bold
/Encoding /WinAnsiEncoding >>
endobj
35 0 obj
<< /Type /Font
/Subtype /Type1
/Name /F5
/BaseFont /Times-Roman
/Encoding /WinAnsiEncoding >>
endobj
36 0 obj
<< /Type /Font
/Subtype /Type1
/Name /F1
/BaseFont /Helvetica
/Encoding /WinAnsiEncoding >>
endobj
37 0 obj
<< /Type /Font
/Subtype /Type1
/Name /F2
/BaseFont /Helvetica-Oblique
/Encoding /WinAnsiEncoding >>
endobj
38 0 obj
<< /Type /Font
/Subtype /Type1
/Name /F7
/BaseFont /Times-Bold
/Encoding /WinAnsiEncoding >>
endobj
1 0 obj
<< /Type /Pages
/Count 2
/Kids [6 0 R 15 0 R ] >>
endobj
2 0 obj
<< /Type /Catalog
/Pages 1 0 R
/Outlines 30 0 R
/PageMode /UseOutlines
>>
endobj
3 0 obj
<<
/Font << /F3 34 0 R /F5 35 0 R /F1 36 0 R /F2 37 0 R /F7 38 0 R >>
/ProcSet [ /PDF /ImageC /Text ] >>
endobj
9 0 obj
<<
/S /GoTo
/D [15 0 R /XYZ 85.0 659.0 null]
>>
endobj
11 0 obj
<<
/S /GoTo
/D [15 0 R /XYZ 85.0 492.256 null]
>>
endobj
13 0 obj
<<
/S /GoTo
/D [15 0 R /XYZ 85.0 325.512 null]
>>
endobj
30 0 obj
<<
/First 31 0 R
/Last 33 0 R
>> endobj
xref
0 39
0000000000 65535 f
0000007198 00000 n
0000007263 00000 n
0000007355 00000 n
0000000015 00000 n
0000000071 00000 n
0000000587 00000 n
0000000707 00000 n
0000000746 00000 n
0000007478 00000 n
0000000881 00000 n
0000007541 00000 n
0000001018 00000 n
0000007607 00000 n
0000001155 00000 n
0000003445 00000 n
0000003568 00000 n
0000003679 00000 n
0000003867 00000 n
0000004063 00000 n
0000004261 00000 n
0000004471 00000 n
0000004665 00000 n
0000004852 00000 n
0000005047 00000 n
0000005244 00000 n
0000005453 00000 n
0000005647 00000 n
0000005821 00000 n
0000006020 00000 n
0000007673 00000 n
0000006221 00000 n
0000006342 00000 n
0000006508 00000 n
0000006642 00000 n
0000006755 00000 n
0000006865 00000 n
0000006973 00000 n
0000007089 00000 n
trailer
<<
/Size 39
/Root 2 0 R
/Info 4 0 R
>>
startxref
7724
%%EOF

View File

@ -29,6 +29,7 @@
<fieldType name="int" class="${solr.tests.IntegerFieldType}" docValues="${solr.tests.numeric.dv}" precisionStep="0" omitNorms="true" positionIncrementGap="0"/>
<fieldType name="double" class="${solr.tests.DoubleFieldType}" docValues="${solr.tests.numeric.dv}" precisionStep="0" omitNorms="true" positionIncrementGap="0"/>
<fieldType name="date" class="${solr.tests.DateFieldType}" precisionStep="0" docValues="${solr.tests.numeric.dv}"/>
<fieldType name="string" class="solr.StrField" sortMissingLast="true"/>
<!-- solr.TextField allows the specification of custom
@ -52,6 +53,9 @@
<field name="id" type="string" indexed="true" stored="true" multiValued="false" required="false"/>
<field name="name" type="nametext" indexed="true" stored="true"/>
<field name="subject" type="text" indexed="true" stored="true"/>
<field name="text" type="text" indexed="true" stored="true"/>
<field name="title" type="text" indexed="true" stored="true"/>
<field name="timestamp" type="date" indexed="true" stored="true"/>
<!-- Dynamic field definitions. If a field name is not found, dynamicFields

View File

@ -0,0 +1,73 @@
<?xml version="1.0" ?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<!-- This is a "kitchen sink" config file that tests can use.
When writting a new test, feel free to add *new* items (plugins,
config options, etc...) as long as they don't break any existing
tests. if you need to test something esoteric please add a new
"solrconfig-your-esoteric-purpose.xml" config file.
Note in particular that this test is used by MinimalSchemaTest so
Anything added to this file needs to work correctly even if there
is now uniqueKey or defaultSearch Field.
-->
<config>
<dataDir>${solr.data.dir:}</dataDir>
<directoryFactory name="DirectoryFactory"
class="${solr.directoryFactory:solr.NRTCachingDirectoryFactory}"/>
<schemaFactory class="ClassicIndexSchemaFactory"/>
<luceneMatchVersion>${tests.luceneMatchVersion:LATEST}</luceneMatchVersion>
<xi:include href="solrconfig.snippet.randomindexconfig.xml" xmlns:xi="http://www.w3.org/2001/XInclude"/>
<updateHandler class="solr.DirectUpdateHandler2">
<commitWithin>
<softCommit>${solr.commitwithin.softcommit:true}</softCommit>
</commitWithin>
</updateHandler>
<queryResponseWriter name="xslt" class="solr.scripting.xslt.XSLTResponseWriter" />
<requestHandler name="/update" class="solr.UpdateRequestHandler" />
<requestHandler name="/update/xslt" class="solr.scripting.xslt.XSLTUpdateRequestHandler" >
<int name="xsltCacheLifetimeSeconds">5</int>
</requestHandler>
<requestHandler name="/select" class="solr.SearchHandler">
<lst name="defaults">
<str name="echoParams">explicit</str>
<str name="indent">true</str>
<str name="df">text</str>
</lst>
</requestHandler>
<initParams path="/select">
<lst name="defaults">
<str name="df">text</str>
</lst>
</initParams>
</config>

View File

@ -40,7 +40,7 @@ public class ScriptUpdateProcessorFactoryTest extends UpdateProcessorTestBase {
@BeforeClass
public static void beforeClass() throws Exception {
Assume.assumeNotNull((new ScriptEngineManager()).getEngineByExtension("js"));
initCore("solrconfig-script-updateprocessor.xml", "schema.xml");
initCore("solrconfig-script-updateprocessor.xml", "schema.xml", getFile("scripting/solr").getAbsolutePath());
}
/**

View File

@ -16,12 +16,16 @@
*/
package org.apache.solr.scripting.update;
import java.util.Map;
import java.util.regex.Pattern;
import javax.script.ScriptEngineManager;
import org.apache.solr.core.AbstractBadConfigTestBase;
import org.apache.solr.SolrTestCaseJ4;
import org.apache.solr.core.CoreContainer;
import org.junit.Assume;
public class TestBadScriptingUpdateProcessorConfig extends AbstractBadConfigTestBase {
public class TestBadScriptingUpdateProcessorConfig extends SolrTestCaseJ4 {
public void testBogusScriptEngine() throws Exception {
@ -29,21 +33,67 @@ public class TestBadScriptingUpdateProcessorConfig extends AbstractBadConfigTest
Assume.assumeTrue(null == (new ScriptEngineManager()).getEngineByName("giberish"));
assertConfigs("bad-solrconfig-bogus-scriptengine-name.xml",
"schema.xml","giberish");
"schema.xml",getFile("scripting/solr/collection1").getParent(),"giberish");
}
public void testMissingScriptFile() throws Exception {
// sanity check
Assume.assumeNotNull((new ScriptEngineManager()).getEngineByExtension("js"));
assertConfigs("bad-solrconfig-missing-scriptfile.xml",
"schema.xml","a-file-name-that-does-not-exist.js");
"schema.xml",getFile("scripting/solr/collection1").getParent(),"a-file-name-that-does-not-exist.js");
}
public void testInvalidScriptFile() throws Exception {
// sanity check
Assume.assumeNotNull((new ScriptEngineManager()).getEngineByName("javascript"));
assertConfigs("bad-solrconfig-invalid-scriptfile.xml",
"schema.xml","invalid.script.xml");
"schema.xml",getFile("scripting/solr/collection1").getParent(),"invalid.script.xml");
}
/**
* Given a solrconfig.xml file name, a schema file name, a solr home directory,
* and an expected errString, asserts that initializing a core with these
* files causes an error matching the specified errString ot be thrown.
*/
protected final void assertConfigs(final String solrconfigFile,
final String schemaFile,
final String solrHome,
final String errString)
throws Exception {
ignoreException(Pattern.quote(errString));
try {
if (null == solrHome) {
initCore( solrconfigFile, schemaFile );
} else {
initCore( solrconfigFile, schemaFile, solrHome );
}
CoreContainer cc = h.getCoreContainer();
for (Map.Entry<String, CoreContainer.CoreLoadFailure> entry : cc.getCoreInitFailures().entrySet()) {
if (matches(entry.getValue().exception, errString))
return;
}
}
catch (Exception e) {
if (matches(e, errString))
return;
throw e;
}
finally {
deleteCore();
resetExceptionIgnores();
}
fail("Did not encounter any exception from: " + solrconfigFile + " using " + schemaFile);
}
private static boolean matches(Exception e, String errString) {
for (Throwable t = e; t != null; t = t.getCause()) {
if (t.getMessage() != null && -1 != t.getMessage().indexOf(errString))
return true;
}
return false;
}
}

View File

@ -0,0 +1,53 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.solr.scripting.xslt;
import org.apache.solr.SolrTestCaseJ4;
import org.junit.BeforeClass;
import org.junit.Test;
/**
* Tests the ability to configure multiple query output writers, and select those at query time. This test
* is specific to the XSLT writer, which isn't part of the core.
*
* See the related unit test OutputWriterTest.
*
*/
public class XSLTOutputWriterTest extends SolrTestCaseJ4 {
@BeforeClass
public static void beforeClass() throws Exception {
initCore("solrconfig.xml", "schema.xml", getFile("scripting/solr").getAbsolutePath());
}
@Test
public void testTrivialXsltWriter() throws Exception {
lrf.args.put("wt", "xslt");
lrf.args.put("tr", "dummy.xsl");
String out = h.query(req("*:*"));
assertTrue(out.contains("DUMMY"));
}
@Test
public void testTrivialXsltWriterInclude() throws Exception {
lrf.args.put("wt", "xslt");
lrf.args.put("tr", "dummy-using-include.xsl");
String out = h.query(req("*:*"));
assertTrue(out.contains("DUMMY"));
}
}

View File

@ -14,24 +14,22 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.solr.handler;
package org.apache.solr.scripting.xslt;
import org.apache.solr.SolrTestCaseJ4;
import java.io.StringWriter;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.Map;
import org.apache.solr.common.params.CommonParams;
import org.apache.solr.SolrTestCaseJ4;
import org.apache.solr.common.params.MapSolrParams;
import org.apache.solr.common.util.ContentStream;
import org.apache.solr.common.util.ContentStreamBase;
import org.apache.solr.common.util.NamedList;
import org.apache.solr.core.SolrCore;
import org.apache.solr.handler.loader.XMLLoader;
import org.apache.solr.handler.loader.ContentStreamLoader;
import org.apache.solr.request.LocalSolrQueryRequest;
import org.apache.solr.response.QueryResponseWriter;
import org.apache.solr.request.SolrQueryRequest;
import org.apache.solr.response.QueryResponseWriter;
import org.apache.solr.response.SolrQueryResponse;
import org.apache.solr.update.AddUpdateCommand;
import org.apache.solr.update.processor.BufferingRequestProcessor;
@ -39,11 +37,17 @@ import org.junit.Before;
import org.junit.BeforeClass;
import org.junit.Test;
public class XsltUpdateRequestHandlerTest extends SolrTestCaseJ4 {
/**
* <p>
* This tests the XSLTUpdateRequestHandler ability to work with XSLT stylesheet and xml content.
* </p>
*/
public class XSLTUpdateRequestHandlerTest extends SolrTestCaseJ4 {
@BeforeClass
public static void beforeTests() throws Exception {
initCore("solrconfig.xml","schema.xml");
initCore("solrconfig.xml", "schema.xml", getFile("scripting/solr").getAbsolutePath());
}
@Override
@ -69,7 +73,7 @@ public class XsltUpdateRequestHandlerTest extends SolrTestCaseJ4 {
"</random>";
Map<String,String> args = new HashMap<>();
args.put(CommonParams.TR, "xsl-update-handler-test.xsl");
args.put("tr", "xsl-update-handler-test.xsl");
SolrCore core = h.getCore();
LocalSolrQueryRequest req = new LocalSolrQueryRequest( core, new MapSolrParams( args) );
@ -77,7 +81,8 @@ public class XsltUpdateRequestHandlerTest extends SolrTestCaseJ4 {
streams.add(new ContentStreamBase.StringStream(xml));
req.setContentStreams(streams);
SolrQueryResponse rsp = new SolrQueryResponse();
try (UpdateRequestHandler handler = new UpdateRequestHandler()) {
//try (UpdateRequestHandler handler = new UpdateRequestHandler()) {
try (XSLTUpdateRequestHandler handler = new XSLTUpdateRequestHandler()) {
handler.init(new NamedList<String>());
handler.handleRequestBody(req, rsp);
}
@ -98,7 +103,7 @@ public class XsltUpdateRequestHandlerTest extends SolrTestCaseJ4 {
@Test
public void testEntities() throws Exception
{
// use a binary file, so when it's loaded fail with XML eror:
// use a binary file, so when it's loaded fail with XML error:
String file = getFile("mailing_lists.pdf").toURI().toASCIIString();
String xml =
"<?xml version=\"1.0\"?>" +
@ -115,10 +120,10 @@ public class XsltUpdateRequestHandlerTest extends SolrTestCaseJ4 {
" <node name=\"foo_s\" value=\"&wacky;\"/>" +
" </document>" +
"</random>";
SolrQueryRequest req = req(CommonParams.TR, "xsl-update-handler-test.xsl");
SolrQueryRequest req = req("tr", "xsl-update-handler-test.xsl");
SolrQueryResponse rsp = new SolrQueryResponse();
BufferingRequestProcessor p = new BufferingRequestProcessor(null);
XMLLoader loader = new XMLLoader().init(null);
ContentStreamLoader loader = new XSLTUpdateRequestHandler.XsltXMLLoader().init(null);
loader.load(req, rsp, new ContentStreamBase.StringStream(xml), p);
AddUpdateCommand add = p.addCommands.get(0);

View File

@ -64,7 +64,7 @@ public class UpdateRequestHandler extends ContentStreamHandlerBase implements Pe
// NOTE: This constant is for use with the <add> XML tag, not the HTTP param with same name
public static final String COMMIT_WITHIN = "commitWithin";
Map<String,ContentStreamLoader> loaders = null;
protected Map<String,ContentStreamLoader> loaders = null;
ContentStreamLoader instance = new ContentStreamLoader() {
@Override
@ -168,7 +168,7 @@ public class UpdateRequestHandler extends ContentStreamHandlerBase implements Pe
@Override
public String getDescription() {
return "Add documents using XML (with XSLT), CSV, JSON, or javabin";
return "Add documents using XML, CSV, JSON, or javabin.";
}
@Override
@ -182,6 +182,3 @@ public class UpdateRequestHandler extends ContentStreamHandlerBase implements Pe
public static final String BIN_PATH = "/update/bin";
}

View File

@ -16,17 +16,9 @@
*/
package org.apache.solr.handler.loader;
import javax.xml.parsers.SAXParserFactory;
import javax.xml.stream.FactoryConfigurationError;
import javax.xml.stream.XMLInputFactory;
import javax.xml.stream.XMLStreamConstants;
import javax.xml.stream.XMLStreamException;
import javax.xml.stream.XMLStreamReader;
import javax.xml.transform.Transformer;
import javax.xml.transform.TransformerException;
import javax.xml.transform.dom.DOMResult;
import javax.xml.transform.dom.DOMSource;
import javax.xml.transform.sax.SAXSource;
import static org.apache.solr.common.params.CommonParams.ID;
import static org.apache.solr.common.params.CommonParams.NAME;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
@ -37,14 +29,17 @@ import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.atomic.AtomicBoolean;
import javax.xml.parsers.SAXParserFactory;
import javax.xml.stream.FactoryConfigurationError;
import javax.xml.stream.XMLInputFactory;
import javax.xml.stream.XMLStreamConstants;
import javax.xml.stream.XMLStreamException;
import javax.xml.stream.XMLStreamReader;
import com.google.common.collect.Lists;
import org.apache.commons.io.IOUtils;
import org.apache.solr.common.EmptyEntityResolver;
import org.apache.solr.common.SolrException;
import org.apache.solr.common.SolrException.ErrorCode;
import org.apache.solr.common.SolrInputDocument;
import org.apache.solr.common.params.CommonParams;
import org.apache.solr.common.params.ModifiableSolrParams;
import org.apache.solr.common.params.ShardParams;
import org.apache.solr.common.params.SolrParams;
@ -53,7 +48,6 @@ import org.apache.solr.common.util.ContentStream;
import org.apache.solr.common.util.ContentStreamBase;
import org.apache.solr.common.util.StrUtils;
import org.apache.solr.common.util.XMLErrorLogger;
import org.apache.solr.core.SolrConfig;
import org.apache.solr.handler.RequestHandlerUtils;
import org.apache.solr.handler.UpdateRequestHandler;
import org.apache.solr.request.SolrQueryRequest;
@ -63,30 +57,16 @@ import org.apache.solr.update.CommitUpdateCommand;
import org.apache.solr.update.DeleteUpdateCommand;
import org.apache.solr.update.RollbackUpdateCommand;
import org.apache.solr.update.processor.UpdateRequestProcessor;
import org.apache.solr.util.xslt.TransformerProvider;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.xml.sax.InputSource;
import org.xml.sax.XMLReader;
import static org.apache.solr.common.params.CommonParams.ID;
import static org.apache.solr.common.params.CommonParams.NAME;
public class XMLLoader extends ContentStreamLoader {
private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
private static final AtomicBoolean WARNED_ABOUT_INDEX_TIME_BOOSTS = new AtomicBoolean();
static final XMLErrorLogger xmllog = new XMLErrorLogger(log);
protected static final XMLErrorLogger xmllog = new XMLErrorLogger(log);
public static final String CONTEXT_TRANSFORMER_KEY = "xsltupdater.transformer";
private static final String XSLT_CACHE_PARAM = "xsltCacheLifetimeSeconds";
public static final int XSLT_CACHE_DEFAULT = 60;
int xsltCacheLifetimeSeconds;
XMLInputFactory inputFactory;
SAXParserFactory saxFactory;
protected XMLInputFactory inputFactory;
protected SAXParserFactory saxFactory;
@Override
public XMLLoader init(SolrParams args) {
@ -113,11 +93,6 @@ public class XMLLoader extends ContentStreamLoader {
saxFactory.setNamespaceAware(true); // XSL needs this!
EmptyEntityResolver.configureSAXParserFactory(saxFactory);
xsltCacheLifetimeSeconds = XSLT_CACHE_DEFAULT;
if(args != null) {
xsltCacheLifetimeSeconds = args.getInt(XSLT_CACHE_PARAM,XSLT_CACHE_DEFAULT);
log.debug("xsltCacheLifetimeSeconds={}", xsltCacheLifetimeSeconds);
}
return this;
}
@ -133,45 +108,7 @@ public class XMLLoader extends ContentStreamLoader {
InputStream is = null;
XMLStreamReader parser = null;
String tr = req.getParams().get(CommonParams.TR,null);
if(tr!=null) {
if (req.getCore().getCoreDescriptor().isConfigSetTrusted() == false) {
throw new SolrException(ErrorCode.UNAUTHORIZED, "The configset for this collection was uploaded without any authentication in place,"
+ " and this operation is not available for collections with untrusted configsets. To use this feature, re-upload the configset"
+ " after enabling authentication and authorization.");
}
final Transformer t = getTransformer(tr,req);
final DOMResult result = new DOMResult();
// first step: read XML and build DOM using Transformer (this is no overhead, as XSL always produces
// an internal result DOM tree, we just access it directly as input for StAX):
try {
is = stream.getStream();
final InputSource isrc = new InputSource(is);
isrc.setEncoding(charset);
final XMLReader xmlr = saxFactory.newSAXParser().getXMLReader();
xmlr.setErrorHandler(xmllog);
xmlr.setEntityResolver(EmptyEntityResolver.SAX_INSTANCE);
final SAXSource source = new SAXSource(xmlr, isrc);
t.transform(source, result);
} catch(TransformerException te) {
throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, te.getMessage(), te);
} finally {
IOUtils.closeQuietly(is);
}
// second step: feed the intermediate DOM tree into StAX parser:
try {
parser = inputFactory.createXMLStreamReader(new DOMSource(result.getNode()));
this.processUpdate(req, processor, parser);
} catch (XMLStreamException e) {
throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, e.getMessage(), e);
} finally {
if (parser != null) parser.close();
}
}
// Normal XML Loader
else {
try {
is = stream.getStream();
if (log.isTraceEnabled()) {
@ -195,33 +132,11 @@ public class XMLLoader extends ContentStreamLoader {
IOUtils.closeQuietly(is);
}
}
}
/** Get Transformer from request context, or from TransformerProvider.
* This allows either getContentType(...) or write(...) to instantiate the Transformer,
* depending on which one is called first, then the other one reuses the same Transformer
*/
Transformer getTransformer(String xslt, SolrQueryRequest request) throws IOException {
// not the cleanest way to achieve this
// no need to synchronize access to context, right?
// Nothing else happens with it at the same time
final Map<Object,Object> ctx = request.getContext();
Transformer result = (Transformer)ctx.get(CONTEXT_TRANSFORMER_KEY);
if(result==null) {
SolrConfig solrConfig = request.getCore().getSolrConfig();
result = TransformerProvider.instance.getTransformer(solrConfig, xslt, xsltCacheLifetimeSeconds);
result.setErrorListener(xmllog);
ctx.put(CONTEXT_TRANSFORMER_KEY,result);
}
return result;
}
/**
* @since solr 1.2
*/
void processUpdate(SolrQueryRequest req, UpdateRequestProcessor processor, XMLStreamReader parser)
protected void processUpdate(SolrQueryRequest req, UpdateRequestProcessor processor, XMLStreamReader parser)
throws XMLStreamException, IOException, FactoryConfigurationError {
AddUpdateCommand addCmd = null;
SolrParams params = req.getParams();

View File

@ -0,0 +1 @@
Placeholder file to test replication of subdirs.

View File

@ -35,10 +35,10 @@
<requestHandler name="/replication" class="solr.ReplicationHandler">
<lst name="leader">
<str name="replicateAfter">commit</str>
<!-- we don't really need dummy.xsl, but we want to be sure subdir
<!-- we don't really need foo/bar.txt, but we want to be sure subdir
files replicate (see SOLR-3809)
-->
<str name="confFiles">schema.xml,xslt/dummy.xsl</str>
<str name="confFiles">schema.xml,foo/bar.txt</str>
</lst>
</requestHandler>

View File

@ -51,7 +51,6 @@
<queryResponseWriter name="standard" class="solr.XMLResponseWriter"/>
<queryResponseWriter name="useless" class="org.apache.solr.OutputWriterTest$UselessOutputWriter" startup="lazy"/>
<queryResponseWriter name="xslt" class="solr.XSLTResponseWriter"/>
<admin>
<defaultQuery>solr</defaultQuery>

View File

@ -71,24 +71,6 @@ public class OutputWriterTest extends SolrTestCaseJ4 {
assertEquals(USELESS_OUTPUT, out);
}
@Test
public void testTrivialXsltWriter() throws Exception {
lrf.args.put("wt", "xslt");
lrf.args.put("tr", "dummy.xsl");
String out = h.query(req("foo"));
// System.out.println(out);
assertTrue(out.contains("DUMMY"));
}
@Test
public void testTrivialXsltWriterInclude() throws Exception {
lrf.args.put("wt", "xslt");
lrf.args.put("tr", "dummy-using-include.xsl");
String out = h.query(req("foo"));
// System.out.println(out);
assertTrue(out.contains("DUMMY"));
}
public void testLazy() {
PluginBag.PluginHolder<QueryResponseWriter> qrw = h.getCore().getResponseWriters().getRegistry().get("useless");
assertTrue("Should be a lazy class", qrw instanceof PluginBag.LazyPluginHolder);

View File

@ -587,15 +587,15 @@ public class TestReplicationHandler extends SolrTestCaseJ4 {
followerJetty.stop();
// setup an xslt dir to force subdir file replication
File leaderXsltDir = new File(leader.getConfDir() + File.separator + "xslt");
File leaderXsl = new File(leaderXsltDir, "dummy.xsl");
assertTrue("could not make dir " + leaderXsltDir, leaderXsltDir.mkdirs());
assertTrue(leaderXsl.createNewFile());
// setup an sub directory /foo/ in order to force subdir file replication
File leaderFooDir = new File(leader.getConfDir() + File.separator + "foo");
File leaderBarFile = new File(leaderFooDir, "bar.txt");
assertTrue("could not make dir " + leaderFooDir, leaderFooDir.mkdirs());
assertTrue(leaderBarFile.createNewFile());
File followerXsltDir = new File(follower.getConfDir() + File.separator + "xslt");
File followerXsl = new File(followerXsltDir, "dummy.xsl");
assertFalse(followerXsltDir.exists());
File followerFooDir = new File(follower.getConfDir() + File.separator + "foo");
File followerBarFile = new File(followerFooDir, "bar.txt");
assertFalse(followerFooDir.exists());
followerJetty = createAndStartJetty(follower);
followerClient.close();
@ -611,8 +611,8 @@ public class TestReplicationHandler extends SolrTestCaseJ4 {
SolrDocument d = ((SolrDocumentList) followerQueryRsp.get("response")).get(0);
assertEquals("newname = 2000", (String) d.getFieldValue("newname"));
assertTrue(followerXsltDir.isDirectory());
assertTrue(followerXsl.exists());
assertTrue(followerFooDir.isDirectory());
assertTrue(followerBarFile.exists());
checkForSingleIndex(leaderJetty);
checkForSingleIndex(followerJetty, true);

View File

@ -39,6 +39,9 @@ import java.util.LinkedList;
import java.util.Objects;
import java.util.Queue;
/**
* Tests the UpdateRequestHandler support for XML updates.
*/
public class XmlUpdateRequestHandlerTest extends SolrTestCaseJ4 {
private static XMLInputFactory inputFactory;
protected static UpdateRequestHandler handler;

View File

@ -828,6 +828,17 @@
</lst>
</requestHandler>
<!-- XSLT Update Request Handler
https://lucene.apache.org/solr/guide/uploading-data-with-index-handlers.html#using-xslt-to-transform-xml-index-updates
-->
<requestHandler name="/update/xslt"
startup="lazy"
class="solr.scripting.xslt.XSLTUpdateRequestHandler" >
<int name="xsltCacheLifetimeSeconds">5</int>
</requestHandler>
<!-- Spell Check
The spell check component can return a list of alternative spelling
@ -1331,10 +1342,14 @@
</queryResponseWriter>
<!-- XSLT response writer transforms the XML output by any xslt file found
in Solr's conf/xslt directory. Changes to xslt files are checked for
every xsltCacheLifetimeSeconds.
in Solr's conf/xslt directory. Changes to xslt files are checked
every xsltCacheLifetimeSeconds. This is part of the Scripting contrib module.
See more about this response writer at https://lucene.apache.org/solr/guide/response-writers.html#xslt-response-writer
-->
<queryResponseWriter name="xslt" class="solr.XSLTResponseWriter">
<queryResponseWriter name="xslt"
startup="lazy"
class="solr.scripting.xslt.XSLTResponseWriter">
<int name="xsltCacheLifetimeSeconds">5</int>
</queryResponseWriter>

View File

@ -17,10 +17,14 @@
<!--
Simple transform of Solr query response into Solr Update XML compliant XML.
When used in the xslt response writer you will get UpdaateXML as output.
When used in the xslt response writer you will get UpdateXML as output.
But you can also store a query response XML to disk and feed this XML to
the XSLTUpdateRequestHandler to index the content. Provided as example only.
See http://wiki.apache.org/solr/XsltUpdateRequestHandler for more info
This is part of the Scripting contrib module.
See more about this response writer at https://lucene.apache.org/solr/guide/response-writers.html#xslt-response-writer
-->
<xsl:stylesheet version='1.0' xmlns:xsl='http://www.w3.org/1999/XSL/Transform'>
<xsl:output media-type="text/xml" method="xml" indent="yes"/>

View File

@ -144,8 +144,11 @@ _(raw; not yet edited)_
* SOLR-14972: The default port of prometheus exporter has changed from 9983 to 8989, so you may need to adjust your configuration after upgrade.
* SOLR-14067: StatelessScriptUpdateProcessorFactory moved to it's own /contrib/scripting/ package instead
of shipping as part of Solr due to security concerns. Renamed to ScriptUpdateProcessorFactory for simpler name.
* SOLR-14067: StatelessScriptUpdateProcessorFactory moved to /contrib/scripting/ package instead
of shipping as part of Solr, due to security concerns. Renamed to ScriptUpdateProcessorFactory for simpler name.
* SOLR-15121: XSLTResponseWriter moved to /contrib/scripting/ package instead
of shipping as part of Solr, due to security concerns.
=== Upgrade Prerequisites in Solr 9

View File

@ -162,6 +162,10 @@ The default behavior is not to indent.
The XSLT Response Writer applies an XML stylesheet to output. It can be used for tasks such as formatting results for an RSS feed.
This response writer has been added as part of the Scripting contrib module.
Learn more about adding the `dist/solr-scripting-*.jar` file into Solr's <<libs.adoc#lib-directories,Lib Directories>>.
=== tr Parameter
The XSLT Response Writer accepts one parameter: the `tr` parameter, which identifies the XML transformation to use. The transformation must be found in the Solr `conf/xslt` directory.
@ -170,7 +174,7 @@ The Content-Type of the response is set according to the `<xsl:output>` statemen
=== XSLT Configuration
The example below, from the `sample_techproducts_configs` <<response-writers.adoc#,configset>> in the Solr distribution, shows how the XSLT Response Writer is configured.
The example below, from the `sample_techproducts_configs` <<config-sets.adoc#,configset>> in the Solr distribution, shows how the XSLT Response Writer is configured.
[source,xml]
----
@ -179,13 +183,59 @@ The example below, from the `sample_techproducts_configs` <<response-writers.ado
every xsltCacheLifetimeSeconds at most.
-->
<queryResponseWriter name="xslt"
class="org.apache.solr.request.XSLTResponseWriter">
class="solr.scripting.xslt.XSLTResponseWriter">
<int name="xsltCacheLifetimeSeconds">5</int>
</queryResponseWriter>
----
A value of 5 for `xsltCacheLifetimeSeconds` is good for development, to see XSLT changes quickly. For production you probably want a much higher value.
=== XSLT Writer Example
`\http://localhost:8983/solr/techproducts/select?q=ipod&fl=id,cat,name,popularity,price,score&wt=xslt&tr=example_rss.xsl` transforms the results into a RSS feed:
[source,xml]
----
<rss version="2.0">
<channel>
<title>Example Solr RSS 2.0 Feed</title>
<link>http://localhost:8983/solr</link>
<description>
This has been formatted by the sample "example_rss.xsl" transform - use your own XSLT to get a nicer RSS feed.
</description>
<language>en-us</language>
<docs>http://localhost:8983/solr</docs>
<item>
<title>iPod & iPod Mini USB 2.0 Cable</title>
<link>
http://localhost:8983/solr/select?q=id:IW-02
</link>
<description/>
<pubDate/>
<guid>
http://localhost:8983/solr/select?q=id:IW-02
</guid>
</item>
----
The `sample_techproducts_configs` also includes `example.xsl` which generates a simplistic HTML page
and `example_atom.xsl` that outputs in the Atom format.
`updateXml.xsl` can be used to convert the standard Solr XML output into the Solr XML add docs format! Indeed you
could round trip your data via:
[source,bash]
----
curl -o docs_formatted_as_solr_add.xml "http://localhost:8983/solr/techproducts/select?q=ipod&fl=id,cat,name,popularity,price,score&wt=xslt&tr=updateXml.xsl"
curl -X POST -H "Content-Type: text/xml" -d @docs_formatted_as_solr_add.xml "http://localhost:8983/solr/techproducts/update?commitWithin=1000&overwrite=true"
----
Lastly, the `luke.xsl` transformation demonstrates that you can apply very sophisticated transformations: `\http://localhost:8983/solr/techproducts/admin/luke?wt=xslt&tr=luke.xsl`
== Binary Response Writer
This is a custom binary format used by Solr for inter-node communication as well as client-server communication. SolrJ uses this as the default for indexing as well as querying. See <<client-apis.adoc#,Client APIs>> for more details.
@ -310,7 +360,7 @@ This response writer has been added as part of the extraction library, and will
[source,bash]
----
cp contrib/extraction/lib/*.jar server/solr-webapp/webapp/WEB-INF/lib/
cp dist/solr-cell-6.3.0.jar server/solr-webapp/webapp/WEB-INF/lib/
cp dist/solr-extraction-*.jar server/solr-webapp/webapp/WEB-INF/lib/
----
Once the libraries are in place, you can add `wt=xlsx` to your request, and results will be returned as an XLSX sheet.

View File

@ -220,9 +220,36 @@ The status field will be non-zero in case of failure.
=== Using XSLT to Transform XML Index Updates
The UpdateRequestHandler allows you to index any arbitrary XML using the `<tr>` parameter to apply an https://en.wikipedia.org/wiki/XSLT[XSL transformation]. You must have an XSLT stylesheet in the `conf/xslt` directory of your <<config-sets.adoc#,configset>> that can transform the incoming data to the expected `<add><doc/></add>` format, and use the `tr` parameter to specify the name of that stylesheet.
The Scripting contrib module provides a separate XSLT Update Request Handler that allows you to index any arbitrary XML by using the `<tr>` parameter to apply an https://en.wikipedia.org/wiki/XSLT[XSL transformation]. You must have an XSLT stylesheet in the `conf/xslt` directory of your <<config-sets.adoc#,configset>> that can transform the incoming data to the expected `<add><doc/></add>` format, and use the `tr` parameter to specify the name of that stylesheet.
Here is an example XSLT stylesheet:
Learn more about adding the `dist/solr-scripting-*.jar` file into Solr's <<libs.adoc#lib-directories,Lib Directories>>.
=== tr Parameter
The XSLT Update Request Handler accepts one parameter: the `tr` parameter, which identifies the XML transformation to use. The transformation must be found in the Solr `conf/xslt` directory.
=== XSLT Configuration
The example below, from the `sample_techproducts_configs` <<config-sets.adoc#,configset>> in the Solr distribution, shows how the XSLT Update Request Handler is configured.
[source,xml]
----
<!--
Changes to XSLT transforms are taken into account
every xsltCacheLifetimeSeconds at most.
-->
<requestHandler name="/update/xslt"
class="solr.scripting.xslt.XSLTUpdateRequestHandler">
<int name="xsltCacheLifetimeSeconds">5</int>
</requestHandler>
----
A value of 5 for `xsltCacheLifetimeSeconds` is good for development, to see XSLT changes quickly. For production you probably want a much higher value.
=== XSLT Update Example
Here is the `sample_techproducts_configs/conf/xslt/updateXml.xsl` XSL file for converting standard Solr XML output to the Solr expected `<add><doc/></add>` format:
[source,xml]
----
@ -266,18 +293,15 @@ Here is an example XSLT stylesheet:
This stylesheet transforms Solr's XML search result format into Solr's Update XML syntax. One example usage would be to copy a Solr 1.3 index (which does not have CSV response writer) into a format which can be indexed into another Solr file (provided that all fields are stored):
[source,plain]
----
http://localhost:8983/solr/my_collection/select?q=*:*&wt=xslt&tr=updateXml.xsl&rows=1000
----
You can also use the stylesheet in `XsltUpdateRequestHandler` to transform an index when updating:
[source,bash]
----
curl "http://localhost:8983/solr/my_collection/update?commit=true&tr=updateXml.xsl" -H "Content-Type: text/xml" --data-binary @myexporteddata.xml
curl -o standard_solr_xml_format.xml "http://localhost:8983/solr/techproducts/select?q=ipod&fl=id,cat,name,popularity,price,score&wt=xml"
curl -X POST -H "Content-Type: text/xml" -d @standard_solr_xml_format.xml "http://localhost:8983/solr/techproducts/update/xslt?commit=true&tr=updateXml.xsl"
----
NOTE: You can see the opposite export/import cycle using the `tr` parameter in <<response-writers.adoc#xslt-writer-example,Response Writer XSLT example>>.
== JSON Formatted Index Updates
Solr can accept JSON that conforms to a defined structure, or can accept arbitrary JSON-formatted documents. If sending arbitrarily formatted JSON, there are some additional parameters that need to be sent with the update request, described below in the section <<transforming-and-indexing-custom-json.adoc#,Transforming and Indexing Custom JSON>>.

View File

@ -97,9 +97,6 @@ public interface CommonParams {
/** default query field */
String DF = "df";
/** Transformer param -- used with XSLT */
String TR = "tr";
/** whether to include debug data for all components pieces, including doing explains*/
String DEBUG_QUERY = "debugQuery";