Apache Solr Release Notes Introduction ------------ Apache Solr is an open source enterprise search server based on the Apache Lucene Java search library, with XML/HTTP and JSON APIs, hit highlighting, faceted search, caching, replication, and a web administration interface. It runs in a Java servlet container such as Tomcat. See http://lucene.apache.org/solr for more information. Getting Started --------------- You need a Java 1.7 VM or later installed. In this release, there is an example Solr server including a bundled servlet container in the directory named "example". See the tutorial at http://lucene.apache.org/solr/tutorial.html $Id$ ================== 5.0.0 ================== Versions of Major Components --------------------- Apache Tika 1.4 Carrot2 3.8.0 Velocity 1.7 and Velocity Tools 2.0 Apache UIMA 2.3.1 Apache ZooKeeper 3.4.5 Upgrading from Solr 4.x ---------------------- * The "file" attribute of infoStream in solrconfig.xml is removed. Control this via your logging configuration (org.apache.solr.update.LoggingInfoStream) instead. * UniqFieldsUpdateProcessorFactory no longer supports the init param style that was deprecated in Solr 4.5. If you are still using this syntax, update your configs to use instead. See SOLR-4249 for more details. Detailed Change List ---------------------- Other Changes ---------------------- * SOLR-4622: Hardcoded SolrCloud defaults for hostContext and hostPort that were deprecated in 4.3 have been removed completely. (hossman) * SOLR-4792: Stop shipping a .war. (Robert Muir) ================== 4.7.0 ================== Versions of Major Components --------------------- Apache Tika 1.4 Carrot2 3.8.0 Velocity 1.7 and Velocity Tools 2.0 Apache UIMA 2.3.1 Apache ZooKeeper 3.4.5 Upgrading from Solr 4.6.0 ---------------------- Detailed Change List ---------------------- New Features ---------------------- * SOLR-5308: A new 'migrate' collection API to split all documents with a route key into another collection (shalin) * SOLR-5441: Expose number of transaction log files and their size via JMX. (Rafał Kuć via shalin) * SOLR-5320: Added support for tri-level compositeId routing. (Anshum Gupta via shalin) * SOLR-5287: You can edit files in the conf directory from the admin UI (Erick Erickson, Stefan Matheis) * SOLR-5447, SOLR-5490: Add a QParserPlugin for Lucene's SimpleQueryParser. (Jack Conradson via shalin) * SOLR-5446: Admin UI - Allow changing Schema and Config (steffkes) * SOLR-5456: Admin UI - Allow creating new Files (steffkes) * SOLR-5208: Support for the setting of core.properties key/values at create-time on Collections API (Erick Erickson) * SOLR-5428: New 'stats.calcdistinct' parameter in StatsComponent returns set of distinct values and their count. This can also be specified per field e.g. 'f.field.stats.calcdistinct'. (Elran Dvir via shalin) * SOLR-5378: A new SuggestComponent that fully utilizes the Lucene suggester module and adds pluggable dictionaries, payloads and better distributed support. This is intended to eventually replace the Suggester support through the SpellCheckComponent. (Areek Zillur, Varun Thacker via shalin) * SOLR-5494: CoreContainer#remove throws NPE rather than returning null when a SolrCore does not exist in core discovery mode. (Mark Miller) Bug Fixes ---------------------- * SOLR-5438: DebugComponent throws NPE when used with grouping. (Tomás Fernández Löbbe via shalin) * SOLR-5442: Python client cannot parse proxied response when served by Tomcat. (Patrick Hunt, Gregory Chanan, Vamsee Yarlagadda, Romain Rigaux, Mark Miller) * SOLR-5445: Proxied responses should propagate all headers rather than the first one for each key. (Patrick Hunt, Mark Miller) * SOLR-4612: Admin UI - Analysis Screen contains empty table-columns (steffkes) * SOLR-5451: SyncStrategy closes it's http connection manager before the executor that uses it in it's close method. (Mark Miller) * SOLR-5460: SolrDispatchFilter#sendError can get a SolrCore that it does not close. (Mark Miller) * SOLR-5461: Request proxying should only set con.setDoOutput(true) if the request is a post. (Mark Miller) * SOLR-5479: SolrCmdDistributor retry logic stops if a leader for the request cannot be found in 1 second. (Mark Miller) * SOLR-5481: SolrCmdDistributor should not let the http client do it's own retries. (Mark Miller) * SOLR-4709: The core reload after replication if config files have changed can fail due to a race condition. (Mark Miller, Hossman)) * LUCENE-5347: Fixed Solr's Zookeeper Client to copy files to Zookeeper using binary transfer. Previously data was read with default encoding and stored in zookeeper as UTF-8. This bug was found after upgrading to forbidden-apis 1.4. (Uwe Schindler) * SOLR-4376: DataImportHandler uses wrong date format for last_index_time if a delta-import is run first before any full-imports. (Sebastien Lorber, Arcadius Ahouansou via shalin) * SOLR-5496: We should share an http connection manager across non search HttpClients and ensure all http connection managers get shutdown. (Mark Miller) Optimizations ---------------------- * SOLR-5458: Admin UI - Remove separated Pages for Config & Schema (steffkes) * SOLR-5436: Eliminate the 1500ms wait in overseer loop as well as polling the ZK distributed queue. (Noble Paul, Mark Miller) Other Changes --------------------- * SOLR-5399: Add distributed request tracking information to DebugComponent (Tomás Fernández Löbbe via Ryan Ernst) * SOLR-5421: Remove double set of distrib.from param in processAdd method of DistributedUpdateProcessor. (Anshum Gupta via shalin) * SOLR-5404: The example config references deprecated classes. (Uwe Schindler, Rafał Kuć via Mark Miller) * SOLR-5487: Replication factor error message doesn't match constraint. (Patrick Hunt via shalin) ================== 4.6.0 ================== Versions of Major Components --------------------- Apache Tika 1.4 Carrot2 3.8.0 Velocity 1.7 and Velocity Tools 2.0 Apache UIMA 2.3.1 Apache ZooKeeper 3.4.5 Upgrading from Solr 4.5.0 ---------------------- * If you are using methods from FieldMutatingUpdateProcessorFactory for getting configuration information (oneOrMany or getBooleanArg), those methods have been moved to NamedList and renamed to removeConfigArgs and removeBooleanArg, respectively. The original methods are deprecated, to be removed in 5.0. See SOLR-5264. Detailed Change List ---------------------- New Features ---------------------- * SOLR-5167: Add support for AnalyzingInfixSuggester (AnalyzingInfixLookupFactory). (Areek Zillur, Varun Thacker via Robert Muir) * SOLR-5246: Shard splitting now supports collections configured with router.field. (shalin) * SOLR-5274: Allow JettySolrRunner SSL config to be specified via a constructor. (Mark Miller) * SOLR-5300: Shards can be split by specifying arbitrary number of hash ranges within the shard's hash range. (shalin) * SOLR-5226: Add Lucene index heap usage to the Solr admin UI. (Areek Zillur via Robert Muir) * SOLR-5324: Make sub shard replica recovery and shard state switch asynchronous. (Yago Riveiro, shalin) * SOLR-5338: Split shards by a route key using split.key parameter. (shalin) * SOLR-5353: Enhance CoreAdmin api to split a route key's documents from an index and leave behind all other documents. (shalin) * SOLR-5027: CollapsingQParserPlugin for high performance field collapsing on high cardinality fields. (Joel Bernstein) * SOLR-5395: Added a RunAlways marker interface for UpdateRequestProcessorFactory implementations indicating that they should not be removed in later stages of distributed updates (usually signalled by the update.distrib parameter) (yonik) * SOLR-5310: Add a collection admin command to remove a replica (noble) * SOLR-5311: Avoid registering replicas which are removed (noble) * SOLR-5406: CloudSolrServer failed to propagate request parameters along with delete updates. (yonik) * SOLR-5374: Support user configured doc-centric versioning rules via the optional DocBasedVersionConstraintsProcessorFactory update processor (Hossman, yonik) * SOLR-5392: Extend solrj apis to cover collection management. (Roman Shaposhnik via Mark Miller) * SOLR-5084: new field type EnumField. (Elran Dvir via Erick Erickson) * SOLR-5464: Add option to ConcurrentSolrServer to stream pure delete requests. (Mark Miller) Bug Fixes ---------------------- * SOLR-5216: Document updates to SolrCloud can cause a distributed deadlock. (Mark Miller) * SOLR-5367: Unmarshalling delete by id commands with JavaBin can lead to class cast exception. (Mark Miller) * SOLR-5359: ZooKeeper client is not closed when it fails to connect to an ensemble. (Mark Miller, Klaus Herrmann) * SOLR-5042: MoreLikeThisComponent was using the rows/count value in place of flags, which caused a number of very strange issues, including NPEs and ignoring requests for the results to include the score. (Anshum Gupta, Mark Miller, Shawn Heisey) * SOLR-5371: Solr should consistently call SolrServer#shutdown (Mark Miller) * SOLR-5363: Solr doesn't start up properly with Log4J2 (Petar Tahchiev via Alan Woodward) * SOLR-5380: Using cloudSolrServer.setDefaultCollection(collectionId) does not work as intended for an alias spanning more than 1 collection. (Thomas Egense, Shawn Heisey, Mark Miller) * SOLR-5418: Background merge after field removed from solr.xml causes error. (Reported on user's list, Robert M's patch via Erick Erickson) * SOLR-5318: Creating a core via the admin API doesn't respect transient property (Olivier Soyez via Erick Erickson) * SOLR-5388: Creating a new core via the HTTP API that results in a transient being unloaded results in a " Too many close [count:-1]" error. (Olivier Soyez via Erick Erickson) * SOLR-5453: Raise recovery socket read timeouts. (Mark Miller) * SOLR-5397: Replication can fail silently in some cases. (Mark Miller) * SOLR-5465: SolrCmdDistributor retry logic has a concurrency race bug. (Mark Miller) * SOLR-5452: Do not attempt to proxy internal update requests. (Mark Miller) Optimizations ---------------------- * SOLR-5232: SolrCloud should distribute updates via streaming rather than buffering. (Mark Miller) * SOLR-5223: SolrCloud should use the JavaBin binary format for communication by default. (Mark Miller) * SOLR-5370: Requests to recover when an update fails should be done in background threads. (Mark Miller) * LUCENE-5300,LUCENE-5304: Specialized faceting for fields which are declared as multi-valued in the schema but are actually single-valued. (Adrien Grand) Security ---------------------- * SOLR-4882: SolrResourceLoader was restricted to only allow access to resource files below the instance dir. The reason for this is security related: Some Solr components allow to pass in resource paths via REST parameters (e.g. XSL stylesheets, velocity templates,...) and load them via resource loader. For backwards compatibility, this security feature can be disabled by a new system property: solr.allow.unsafe.resourceloading=true (Uwe Schindler) Other Changes ---------------------- * SOLR-5237: Add indexHeapUsageBytes to LukeRequestHandler, indicating how much heap memory is being used by the underlying Lucene index structures. (Areek Zillur via Robert Muir) * SOLR-5241: Fix SimplePostToolTest performance problem - implicit DNS lookups (hossman) * SOLR-5273: Update HttpComponents to 4.2.5 and 4.2.6. (Mark Miller) * SOLR-5264: Move methods for getting config information from FieldMutatingUpdateProcessorFactory to NamedList. (Shawn Heisey) * SOLR-5319: Remove unused and incorrect router name from Collection ZK nodes. (Jessica Cheng via shalin) * SOLR-5321: Remove unnecessary code in Overseer.updateState method which tries to use router name from message where none is ever sent. (shalin) * SOLR-5401: SolrResourceLoader logs a warning if a deprecated (factory) class is used in schema or config. (Uwe Schindler) * SOLR-3397: Warn if master or slave replication is enabled in SolrCloud mode. (Erick Erickson) ================== 4.5.1 ================== Versions of Major Components --------------------- Apache Tika 1.4 Carrot2 3.8.0 Velocity 1.7 and Velocity Tools 2.0 Apache UIMA 2.3.1 Apache ZooKeeper 3.4.5 Detailed Change List ---------------------- Bug Fixes ---------------------- * SOLR-4590: Collections API should return a nice error when not in SolrCloud mode. (Anshum Gupta, Mark Miller) * SOLR-5295: The CREATESHARD collection API creates maxShardsPerNode number of replicas if replicationFactor is not specified. (Brett Hoerner, shalin) * SOLR-5296: Creating a collection with implicit router adds shard ranges to each shard. (shalin) * SOLR-5263: Fix CloudSolrServer URL cache update race. (Jessica Cheng, Mark Miller) * SOLR-5297: Admin UI - Threads Screen missing Icon (steffkes) * SOLR-5301: DELETEALIAS command prints CREATEALIAS in logs (janhoy) * SOLR-5255: Remove unnecessary call to fetch and watch live nodes in ZkStateReader cluster watcher. (Jessica Cheng via shalin) * SOLR-5305: Admin UI - Reloading System-Information on Dashboard does not work anymore (steffkes) * SOLR-5314: Shard split action should use soft commits instead of hard commits to make sub shard data visible. (Kalle Aaltonen, shalin) * SOLR-5327: SOLR-4915, "The root cause should be returned to the user when a SolrCore create call fails", was reverted. (Mark Miller) * SOLR-5317: SolrCore persistence bugs if defining SolrCores in solr.xml. (Mark Miller, Yago Riveiro) * SOLR-5306: Extra collection creation parameters like collection.configName are not being respected. (Mark Miller, Liang Tianyu, Nathan Neulinger) * SOLR-5325: ZooKeeper connection loss can cause the Overseer to stop processing commands. (Christine Poerschke, Mark Miller, Jessica Cheng) * SOLR-4327: HttpSolrServer can leak connections on errors. (Karl Wright, Mark Miller) * SOLR-5349: CloudSolrServer - ZK timeout arguments passed to ZkStateReader are flipped. (Ricardo Merizalde via shalin) * SOLR-5330: facet.method=fcs on single values fields could sometimes result in incorrect facet labels. (Michael Froh, yonik) Other Changes ---------------------- * SOLR-5323: Disable ClusteringComponent by default in collection1 example. The solr.clustering.enabled system property needs to be set to 'true' to enable the clustering contrib (reverts SOLR-4708). (Dawid Weiss) ================== 4.5.0 ================== Versions of Major Components --------------------- Apache Tika 1.4 Carrot2 3.8.0 Velocity 1.7 and Velocity Tools 2.0 Apache UIMA 2.3.1 Apache ZooKeeper 3.4.5 Upgrading from Solr 4.4.0 ---------------------- * XML configuration parsing is now more strict about situations where a single setting is allowed but multiple values are found. In the past, one value would be chosen arbitrarily and silently. Starting with 4.5, configuration parsing will fail with an error in situations like this. If you see error messages such as "solrconfig.xml contains more than one value for config path: XXXXX" or "Found Z configuration sections when at most 1 is allowed matching expression: XXXXX" check your solrconfig.xml file for multiple occurrences of XXXXX and delete the ones that you do not wish to use. See SOLR-4953 & SOLR-5108 for more details. * In the past, schema.xml parsing would silently ignore "default" or "required" options specified on declarations. Begining with 4.5, attempting to do configured these on a dynamic field will cause an init error. If you encounter one of these errors when upgrading an existing schema.xml, you can safely remove these attributes, regardless of their value, from your config and Solr will continue to bahave exactly as it did in previous versions. See SOLR-5227 for more details. * The UniqFieldsUpdateProcessorFactory has been improved to support all of the FieldMutatingUpdateProcessorFactory selector options. The init param option is now deprecated and should be replaced with the more standard . See SOLR-4249 for more details. * UpdateRequestExt has been removed as part of SOLR-4816. You should use UpdateRequest instead. * CloudSolrServer can now use multiple threads to add documents by default. This is a small change in runtime semantics when using the bulk add method - you will still end up with the same exception on a failure, but some documents beyond the one that failed may have made it in. To get the old, single threaded behavior, set parallel updates to false on the CloudSolrServer instance. Detailed Change List ---------------------- New Features ---------------------- * SOLR-5219: Rewritten selection of the default search and document clustering algorithms. (Dawid Weiss) * SOLR-5202: Support easier overrides of Carrot2 clustering attributes via XML data sets exported from the Workbench. (Dawid Weiss) * SOLR-5126: Update Carrot2 clustering to version 3.8.0, update Morfologik to version 1.7.1 (Dawid Weiss) * SOLR-2345: Enhanced geodist() to work with an RPT field, provided that the field is referenced via 'sfield' and the query point is constant. (David Smiley) * SOLR-5082: The encoding of URL-encoded query parameters can be changed with the "ie" (input encoding) parameter, e.g. "select?q=m%FCller&ie=ISO-8859-1". The default is UTF-8. To change the encoding of POSTed content, use the "Content-Type" HTTP header. (Uwe Schindler, Shawn Heisey) * SOLR-4221: Custom sharding (Noble Paul) * SOLR-4808: Persist and use router,replicationFactor and maxShardsPerNode at Collection and Shard level (Noble Paul, Shalin Mangar) * SOLR-5006: CREATESHARD command for 'implicit' shards (Noble Paul) * SOLR-5017: Allow sharding based on the value of a field (Noble Paul) * SOLR-4222: create custom sharded collection via collections API (Noble Paul) * SOLR-4718: Allow solr.xml to be stored in ZooKeeper. (Mark Miller, Erick Erickson) * SOLR-5156: Enhance ZkCLI to allow uploading of arbitrary files to ZK. (Erick Erickson) * SOLR-5165: Single-valued docValues fields no longer require a default value. Additionally they work with sortMissingFirst, sortMissingLast, facet.missing, exists() in function queries, etc. (Robert Muir) * SOLR-5182: Add NoOpRegenerator, a regenerator for custom per-segment caches where items are preserved across commits. (Robert Muir) * SOLR-4249: UniqFieldsUpdateProcessorFactory now extends FieldMutatingUpdateProcessorFactory and supports all of it's selector options. Use of the "fields" init param is now deprecated in favor of "fieldName" (hossman) * SOLR-2548: Allow multiple threads to be specified for faceting. When threading, one can specify facet.threads to parallelize loading the uninverted fields. In at least one extreme case this reduced warmup time from 20 seconds to 3 seconds. (Janne Majaranta, Gun Akkor via Erick Erickson, David Smiley) * SOLR-4816: CloudSolrServer can now route updates locally and no longer relies on inter-node update forwarding. (Joel Bernstein, Shikhar Bhushan, Stephen Riesenberg, Mark Miller) * SOLR-3249: Allow CloudSolrServer and SolrCmdDistributor to use JavaBin. (Mark Miller) Bug Fixes ---------------------- * SOLR-3633: web UI reports an error if CoreAdminHandler says there are no SolrCores (steffkes) * SOLR-4489: SpellCheckComponent can throw StringIndexOutOfBoundsException when generating collations involving multiple word-break corrections. (James Dyer) * SOLR-5107: Fixed NPE when using numTerms=0 in LukeRequestHandler (Ahmet Arslan, hossman) * SOLR-4679, SOLR-4908, SOLR-5124: Text extracted from HTML or PDF files using Solr Cell was missing ignorable whitespace, which is inserted by TIKA for convenience to support plain text extraction without using the HTML elements. This bug resulted in glued words. (hossman, Uwe Schindler) * SOLR-5121: zkcli usage help for makepath doesn't match actual command. (Daniel Collins via Mark Miller) * SOLR-5119: Managed schema problems after adding fields via Schema Rest API. (Nils Kübler, Steve Rowe) * SOLR-5133: HdfsUpdateLog can fail to close a FileSystem instance if init is called more than once. (Mark Miller) * SOLR-5135: Harden Collection API deletion of /collections/$collection ZooKeeper node. (Mark Miller) * SOLR-4764: When using NRT, just init the first reader from IndexWriter. (Robert Muir, Mark Miller) * SOLR-5122: Fixed bug in spellcheck.collateMaxCollectDocs. Eliminates risk of divide by zero, and makes estimated hit counts meaningful in non-optimized indexes. (hossman) * SOLR-3936: Fixed QueryElevationComponent sorting when used with Grouping (Michael Garski via hossman) * SOLR-5171: SOLR Admin gui works in IE9, breaks in IE10. (Joseph L Howard via steffkes) * SOLR-5174: Admin UI - Query View doesn't highlight (json) Result if it contains HTML Tags (steffkes) * SOLR-4817 Solr should not fall back to the back compat built in solr.xml in SolrCloud mode (Erick Erickson) * SOLR-5112: Show full message in Admin UI Logging View (Matthew Keeney via steffkes) * SOLR-5190: SolrEntityProcessor substitutes variables only once in child entities (Harsh Chawla, shalin) * SOLR-3852: Fixed ZookeeperInfoServlet so that the SolrCloud Admin UI pages will work even if ZK contains nodes with data which are not utf8 text. (hossman) * SOLR-5206: Fixed OpenExchangeRatesOrgProvider to use refreshInterval correctly (Catalin, hossman) * SOLR-5215: Fix possibility of deadlock in ZooKeeper ConnectionManager. (Mark Miller, Ricardo Merizalde) * SOLR-4909: Use DirectoryReader.openIfChanged in non-NRT mode. (Michael Garski via Robert Muir) * SOLR-5227: Correctly fail schema initalization if a dynamicField is configured to be required, or have a default value. (hossman) * SOLR-5231: Fixed a bug with the behavior of BoolField that caused documents w/o a value for the field to act as if the value were true in functions if no other documents in the same index segment had a value of true. (Robert Muir, hossman, yonik) * SOLR-5233: The "deleteshard" collections API doesn't wait for cluster state to update, can fail if some nodes of the deleted shard were down and had incorrect logging. (Christine Poerschke, shalin) * SOLR-5150: HdfsIndexInput may not fully read requested bytes. (Mark Miller, Patrick Hunt) * SOLR-5240: All solr cores will now be loaded in parallel (as opposed to a fixed number) in zookeeper mode to avoid deadlocks due to replicas waiting for other replicas to come up. (yonik) * SOLR-5243: Killing a shard in one collection can result in leader election in a different collection if they share the same coreNodeName. (yonik, Mark Miller) * SOLR-5281: IndexSchema log message was printing '[null]' instead of '[]' (Jun Ohtani via Steve Rowe) * SOLR-5279: Implicit properties don't seem to exist on core RELOAD (elyograg, hossman, Steve Rowe) * SOLR-5291: Solrj does not propagate the root cause to the user for many errors. (Mark Miller) Optimizations ---------------------- * SOLR-5044: Admin UI - Note on Core-Admin about directories while creating core (steffkes) * SOLR-5134: Have HdfsIndexOutput extend BufferedIndexOutput. (Mark Miller, Uwe Schindler) * SOLR-5057: QueryResultCache should not related with the order of fq's list (Feihong Huang via Erick Erickson) * SOLR-4816: CloudSolrServer now uses multiple threads to send updates by default. (Joel Bernstein via Mark Miller) * SOLR-3530: Better error messages / Content-Type validation in SolrJ. (Mark Miller, hossman) Other Changes ---------------------- * SOLR-4708: Enable ClusteringComponent by default in collection1 example. The solr.clustering.enabled system property is set to 'true' by default. (ehatcher, Dawid Weiss) * SOLR-4914, SOLR-5162: Factor out core list persistence and discovery into a new CoresLocator interface. (Alan Woodward, Shawn Heisey) * SOLR-5056: Improve type safety of ConfigSolr class. (Alan Woodward) * SOLR-4951: Better randomization of MergePolicy in Solr tests (hossman) * SOLR-4953, SOLR-5108: Make XML Configuration parsing fail if an xpath matches multiple nodes when only a single value or plugin instance is expected. (hossman) * The routing parameter "shard.keys" is deprecated as part of SOLR-5017 .The new parameter name is '_route_' . The old parameter should continue to work for another release (Noble Paul) * SOLR-5173: Solr-core's Maven configuration includes test-only Hadoop dependencies as indirect compile-time dependencies. (Chris Collins, Steve Rowe) ================== 4.4.0 ================== Versions of Major Components --------------------- Apache Tika 1.4 Carrot2 3.6.2 Velocity 1.7 and Velocity Tools 2.0 Apache UIMA 2.3.1 Apache ZooKeeper 3.4.5 Upgrading from Solr 4.3.0 ---------------------- * TieredMergePolicy and the various subtypes of LogMergePolicy no longer have an explicit "setUseCompoundFile" method. Instead the behavior of new segments is determined by the IndexWriter configuration, and the MergePolicy is only consulted to determine if merge segements should use the compound file format (based on the value of "setNoCFSRatio"). If you have explicitly configured one of these classes using and include an init arg like this... true ...this will now be treated as if you specified... true ...directly on the (overriding any value already set using that syntax) and a warning will be logged to updated your configuration. Users with an explicitly declared are encouraged to review the current javadocs for their MergePolicy subclass and review their configured options carefully. See SOLR-4941, SOLR-4934 and LUCENE-5038 for more information. * SOLR-4778: The signature of LogWatcher.registerListener has changed, from (ListenerConfig, CoreContainer) to (ListenerConfig). Users implementing their own LogWatcher classes will need to change their code accordingly. * LUCENE-5063: ByteField and ShortField have been deprecated and will be removed in 5.0. If you are still using these field types, you should migrate your fields to TrieIntField. Detailed Change List ---------------------- New Features ---------------------- * SOLR-3251: Dynamically add fields to schema. (Steve Rowe, Robert Muir, yonik) * SOLR-4761, SOLR-4976: Add option to plugin a merged segment warmer into solrconfig.xml. Info about segments warmed in the background is available via infostream. (Mark Miller, Ryan Ernst, Mike McCandless, Robert Muir) * SOLR-3240: Add "spellcheck.collateMaxCollectDocs" option so that when testing potential Collations against the index, SpellCheckComponent will only collect n documents, thereby estimating the hit-count. This is a performance optimization in cases where exact hit-counts are unnecessary. Also, when "collateExtendedResults" is false, this optimization is always made (James Dyer). * SOLR-4785: New MaxScoreQParserPlugin returning max() instead of sum() of terms (janhoy) * SOLR-4234: Add support for binary files in ZooKeeper. (Eric Pugh via Mark Miller) * SOLR-4048: Add findRecursive method to NamedList. (Shawn Heisey) * SOLR-4228: SolrJ's SolrPing object has new methods for ping, enable, and disable. (Shawn Heisey, hossman, Steve Rowe) * SOLR-4893: Extend FieldMutatingUpdateProcessor.ConfigurableFieldNameSelector to enable checking whether a field matches any schema field. To select field names that don't match any fields or dynamic fields in the schema, add false to an update processor's configuration in solrconfig.xml. (Steve Rowe, hossman) * SOLR-4921: Admin UI now supports adding documents to Solr (gsingers, steffkes) * SOLR-4916: Add support to write and read Solr index files and transaction log files to and from HDFS. (phunt, Mark Miller, Greg Chanan) * SOLR-4892: Add FieldMutatingUpdateProcessorFactory subclasses Parse{Date,Integer,Long,Float,Double,Boolean}UpdateProcessorFactory. These factories have a default selector that matches all fields that either don’t match any schema field, or are in the schema with the corresponding typeClass. If they see a value that is not a CharSequence, or can't parse the value, they leave it as is. For multi-valued fields, these processors will not convert any values unless all are first successfully parsed, or already are instances of the target class. Ordering the processors, e.g. [Boolean, Long, Double, Date] will allow e.g. values ["2", "5", "8.6"] to be left alone by the Boolean and Long processors, but then converted by the Double processor. (Steve Rowe, hossman) * SOLR-4972: Add PUT command to ZkCli tool. (Roman Shaposhnik via Mark Miller) * SOLR-4973: Adding getter method for defaultCollection on CloudSolrServer. (Furkan KAMACI via Mark Miller) * SOLR-4897: Add solr/example/example-schemaless/, an example config set for schemaless mode. (Steve Rowe) * SOLR-4655: Add option to have Overseer assign generic node names so that new addresses can host shards without naming confusion. (Mark Miller, Anshum Gupta) * SOLR-4977: Add option to send IndexWriter's infostream to the logging system. (Ryan Ernst via Robert Muir) * SOLR-4693: A "deleteshard" collections API that unloads all replicas of a given shard and then removes it from the cluster state. It will remove only those shards which are INACTIVE or have no range (created for custom sharding). (Anshum Gupta, shalin) * SOLR-5003: CSV Update Handler supports optionally adding the line number/row id to a document (gsingers) * SOLR-5010: Add support for creating copy fields to the Schema REST API (gsingers) * SOLR-4991: Register QParserPlugins as SolrInfoMBeans (ehatcher) * SOLR-4943: Add a new system wide info admin handler that exposes the system info that could previously only be retrieved using a SolrCore. (Mark Miller) * SOLR-3076: Block joins. Documents and their sub-documents must be indexed as a block. {!parent which=} takes in a query that matches child documents and results in matches on their parents. {!child of=} takes in a query that matches some parent documents and results in matches on their children. (Mikhail Khludnev, Vadim Kirilchuk, Alan Woodward, Tom Burton-West, Mike McCandless, hossman, yonik) Bug Fixes ---------------------- * SOLR-4333: edismax parser to not double-escape colons if already escaped by the client application (James Dyer, Robert J. van der Boon) * SOLR-4776: Solrj doesn't return "between" count in range facets (Philip K. Warren via shalin) * SOLR-4616: HitRatio on caches is now exposed over JMX MBeans as a float. (Greg Bowyer) * SOLR-4803: Fixed core discovery mode (ie: new style solr.xml) to treat 'collection1' as the default core name. (hossman) * SOLR-4790: Throw an error if a core has the same name as another core, both old and new style solr.xml * SOLR-4842: Fix facet.field local params from affecting other facet.field's. (ehatcher, hossman) * SOLR-4814: If a SolrCore cannot be created it should remove any information it published about itself from ZooKeeper. (Mark Miller) * SOLR-4863: Removed non-existent attribute sourceId from dynamic JMX stats to fix AttributeNotFoundException (suganuma, hossman via shalin) * SOLR-4891: JsonLoader should preserve field value types from the JSON content stream. (Steve Rowe) * SOLR-4805: SolreCore#reload should not call preRegister and publish a DOWN state to ZooKeeper. (Mark Miller, Jared Rodriguez) * SOLR-4899: When reconnecting after ZooKeeper expiration, we need to be willing to wait forever, not just for 30 seconds. (Mark Miller) * SOLR-4920: JdbcDataSource incorrectly suppresses exceptions when retrieving a connection from a JNDI context and falls back to trying to use DriverManager to obtain a connection. Additionally, if a SQLException is thrown while initializing a connection, such as in setAutoCommit(), the connection will not be closed. (Chris Eldredge via shalin) * SOLR-4915: The root cause should be returned to the user when a SolrCore create call fails. (Mark Miller) * SOLR-4925 : Collection create throws NPE when 'numShards' param is missing (Noble Paul) * SOLR-4910: persisting solr.xml is broken. More stringent testing of persistence fixed up a number of issues and several bugs with persistence. Among them are > don't persisting implicit properties > should persist zkHost in the tag (user's list) > reloading a core that has transient="true" returned an error. reload should load a transient core if it's not yet loaded. > No longer persisting loadOnStartup or transient core properties if they were not specified in the original solr.xml > Testing flushed out the fact that you couldn't swap a core marked transient=true loadOnStartup=false because it hadn't been loaded yet. > SOLR-4862, CREATE fails to persist schema, config, and dataDir > SOLR-4363, not persisting coreLoadThreads in tag > SOLR-3900, logWatcher properties not persisted > SOLR-4850, cores defined as loadOnStartup=true, transient=false can't be searched (Erick Erickson) * SOLR-4923: Commits to non leaders as part of a request that also contain updates can execute out of order. (hossman, Ricardo Merizalde, Mark Miller) * SOLR-4932: persisting solr.xml saves some parameters it shouldn't when they weren't defined in the original. Benign since the default values are saved, but still incorrect. (Erick Erickson, thanks Shawn Heisey for helping test!) * SOLR-4934, SOLR-4941: Fix handling of init arg "useCompoundFile" needed after changes in LUCENE-5038 (hossman) * SOLR-4456: Admin UI: Displays dashboard even if Solr is down (steffkes) * SOLR-4949: UI Analysis page dropping characters from input box (steffkes) * SOLR-4960: Fix race conditions in shutdown of CoreContainer and getCore that could cause a request to attempt to use a core that has shut down. (yonik) * SOLR-4926: Fixed rare replication bug that normally only manifested when using compound file format. (yonik, Mark Miller) * SOLR-4974: Outgrowth of SOLR-4960 that includes transient cores and pending cores (Erick Erickson) * SOLR-3369: shards.tolerant=true is broken for group queries (Russell Black, Martijn van Groningen, Jabouille jean Charles, Ryan McKinley via shalin) * SOLR-4452: Hunspell stemmer should not merge duplicate dictionary entries (janhoy) * SOLR-5000: ManagedIndexSchema doesn't persist uniqueKey tag after calling addFields method. (Jun Ohtani, Steve Rowe) * SOLR-4982: Creating a core while referencing system properties looks like it loses files Actually, instanceDir, config, dataDir and schema are not dereferenced properly when creating cores that reference sys vars (e.g. &dataDir=${dir}). In the dataDir case in particular this leads to the index being put in a directory literally named ${dir} but on restart the sysvar will be properly dereferenced. * SOLR-4788: Multiple Entities DIH delta import: dataimporter.[entityName].last_index_time is empty. (chakming wong, James Dyer via shalin) * SOLR-4978: Time is stripped from datetime column when imported into Solr date field if convertType=true. (Bill Au, shalin) * SOLR-5019: spurious ConcurrentModificationException when spell check component was in use with filters. (yonik) * SOLR-5018: The Overseer should avoid publishing the state for collections that do not exist under the /collections zk node. (Mark Miller) * SOLR-5028,SOLR-5029: ShardHandlerFactory was not being created properly when using new-style solr.xml, and was not being persisted properly when using old-style. (Tomás Fernández Löbbe, Ryan Ernst, Alan Woodward) * SOLR-4997: The splitshard api doesn't call commit on new sub shards before switching shard states. Multiple bugs related to sub shard recovery and replication are also fixed. (shalin) * SOLR-5034: A facet.query that parses or analyzes down to a null Query would throw a NPE. Fixed. (David Smiley) * SOLR-5039: Admin/Schema Browser displays -1 for term counts for multiValued fields. * SOLR-5037: The CSV loader now accepts field names that are not in the schema. (gsingers, ehatcher, Steve Rowe) Optimizations ---------------------- * SOLR-4923: Commit to all nodes in a collection in parallel rather than locally and then to all other nodes. (hossman, Ricardo Merizalde, Mark Miller) * SOLR-3838: Admin UI - Multiple filter queries are not supported in Query UI (steffkes) * SOLR-4719 : Admin UI - Default to wt=json on Query-Screen (steffkes) * SOLR-4611: Admin UI - Analysis-Urls with empty parameters create empty result table (steffkes) * SOLR-4955: Admin UI - Show address bar on top for Schema + Config (steffkes) * SOLR-4412: New parameter langid.lcmap to map detected language code to be placed in "language" field (janhoy) * SOLR-4815: Admin-UI - DIH: Let "commit" be checked by default (steffkes) * SOLR-5002: optimize numDocs(Query,DocSet) when filterCache is null (Robert Muir) * SOLR-5012: optimize search with filter when filterCache is null (Robert Muir) Other Changes ---------------------- * SOLR-4737: Update Guava to 14.0.1 (Mark Miller) * SOLR-2079: Add option to pass HttpServletRequest in the SolrQueryRequest context map. (Tomás Fernández Löbbe via Robert Muir) * SOLR-4738: Update Jetty to 8.1.10.v20130312 (Mark Miller, Robert Muir) * SOLR-4749: Clean up and refactor CoreContainer code around solr.xml and SolrCore management. (Mark Miller) * SOLR-4547: Move logging of filenames on commit from INFO to DEBUG. (Shawn Heisey, hossman) * SOLR-4757: Change the example to use the new solr.xml format and core discovery by directory structure. (Mark Miller) * SOLR-4759: Velocity (/browse) template cosmetic cleanup. (Mark Bennett, ehatcher) * SOLR-4778: LogWatcher init code moved out of CoreContainer (Alan Woodward) * SOLR-4784: Make class LuceneQParser public (janhoy) * SOLR-4448: Allow the solr internal load balancer to be more easily pluggable. (Philip Hoy via Robert Muir) * SOLR-4224: Refactor JavaBinCodec input stream definition to enhance reuse. (phunt via Mark Miller) * SOLR-4931: SolrDeletionPolicy onInit and onCommit methods changed to override exact signatures (with generics) from IndexDeletionPolicy (shalin) * SOLR-4942: test improvements to randomize use of compound files (hosman) * SOLR-4966: CSS, JS and other files in webapp without license (uschindler, steffkes) * SOLR-4986: Upgrade to Tika 1.4 (Markus Jelsma via janhoy) * SOLR-4948, SOLR-5009: Tidied up CoreContainer construction logic. (Alan Woodward, Uwe Schindler, Steve Rowe) * LUCENE-5107: Properties files by Solr are now written in UTF-8 encoding, Unicode is no longer escaped. Reading of legacy properties files with \u escapes is still possible. (Uwe Schindler, Robert Muir) ================== 4.3.1 ================== Versions of Major Components --------------------- Apache Tika 1.3 Carrot2 3.6.2 Velocity 1.7 and Velocity Tools 2.0 Apache UIMA 2.3.1 Apache ZooKeeper 3.4.5 Detailed Change List ---------------------- Bug Fixes ---------------------- * SOLR-4795: Sub shard leader should not accept any updates from parent after it goes active (shalin) * SOLR-4798: shard splitting does not respect the router for the collection when executing the index split. One effect of this is that documents may be placed in the wrong shard when the default compositeId router is used in conjunction with IDs containing "!". (yonik) * SOLR-4797: Shard splitting creates sub shards which have the wrong hash range in cluster state. This happens when numShards is not a power of two and router is compositeId. (shalin) * SOLR-4791: solr.xml sharedLib does not work in 4.3.0 (Ryan Ernst, Jan Høydahl via Erick Erickson) * SOLR-4806: Shard splitting does not abort if WaitForState times out (shalin) * SOLR-4807: The zkcli script now works with log4j. The zkcli.bat script was broken on Windows in 4.3.0, now it works. (Shawn Heisey) * SOLR-4813: Fix SynonymFilterFactory to allow init parameters for tokenizer factory used when parsing synonyms file. (Shingo Sasaki, hossman) * SOLR-4829: Fix transaction log leaks (a failure to clean up some old logs) on a shard leader, or when unexpected exceptions are thrown during log recovery. (Steven Bower, Mark Miller, yonik) * SOLR-4751: Fix replication problem of files in sub directory of conf directory. (Minoru Osuka via Koji) * SOLR-4741: Deleting a collection should set DELETE_DATA_DIR to true. (Mark Miller) * SOLR-4752: There are some minor bugs in the Collections API parameter validation. (Mark Miller) * SOLR-4563: RSS DIH-example not working (janhoy) * SOLR-4796: zkcli.sh should honor JAVA_HOME (Roman Shaposhnik via Mark Miller) * SOLR-4734: Leader election fails with an NPE if there is no UpdateLog. (Mark Miller, Alexander Eibner) * SOLR-4868: Setting the log level for the log4j root category results in adding a new category, the empty string. (Shawn Heisey) * SOLR-4855: DistributedUpdateProcessor doesn't check for peer sync requests (shalin) * SOLR-4867: Admin UI - setting loglevel on root throws RangeError (steffkes) * SOLR-4870: RecentUpdates.update() does not increment numUpdates loop counter (Alexey Kudinov via shalin) * SOLR-4877, LUCENE-5023: Removed SolrIndexSearcher#getDocSetNC()'s special case for handling TermQuery to prevent NullPointerException if reader does not have fields. (Bao Yang Yang, Uwe Schindler) * SOLR-4881: Fix DocumentAnalysisRequestHandler to correctly use EmptyEntityResolver to prevent loading of external entities like UpdateRequestHandler does. (Hossman, Uwe Schindler) * SOLR-4858: SolrCore reloading was broken when the UpdateLog was enabled. (Hossman, Anshum Gupta, Alexey Serba, Mark Miller, yonik) * SOLR-4853: Fixed SolrJettyTestBase so it may be reused by end users (hossman) * SOLR-4744: Update failure on sub shard is not propagated to clients by parent shard (Anshum Gupta, yonik, shalin) Other Changes ---------------------- * SOLR-4760: Include core name in logs when loading schema. (Shawn Heisey) ================== 4.3.0 ================== Versions of Major Components --------------------- Apache Tika 1.3 Carrot2 3.6.2 Velocity 1.7 and Velocity Tools 2.0 Apache UIMA 2.3.1 Apache ZooKeeper 3.4.5 Upgrading from Solr 4.2.0 ---------------------- * In the schema REST API, the output path for copyFields and dynamicFields has been changed from all lowercase "copyfields" and "dynamicfields" to camelCase "copyFields" and "dynamicFields", respectively, to align with all other schema REST API outputs, which use camelCase. The URL format remains the same: all resource names are lowercase. See SOLR-4623 for details. * Slf4j/logging jars are no longer included in the Solr webapp. All logging jars are now in example/lib/ext. Changing logging impls is now as easy as updating the jars in this folder with those necessary for the logging impl you would like. If you are using another webapp container, these jars will need to go in the corresponding location for that container. In conjunction, the dist-excl-slf4j and dist-war-excl-slf4 build targets have been removed since they are redundent. See the Slf4j documentation, SOLR-3706, and SOLR-4651 for more details. * The hardcoded SolrCloud defaults for 'hostContext="solr"' and 'hostPort="8983"' have been deprecated and will be removed in Solr 5.0. Existing solr.xml files that do not have these options explicitly specified should be updated accordingly. See SOLR-4622 for more details. Detailed Change List ---------------------- New Features ---------------------- * SOLR-4648 PreAnalyzedUpdateProcessorFactory allows using the functionality of PreAnalyzedField with other field types. See javadoc for details and examples. (Andrzej Bialecki) * SOLR-4623: Provide REST API read access to all elements of the live schema. Add a REST API request to return the entire live schema, in JSON, XML, and schema.xml formats. Move REST API methods from package org.apache.solr.rest to org.apache.solr.rest.schema, and rename base functionality REST API classes to remove the current schema focus, to prepare for other non-schema REST APIs. Change output path for copyFields and dynamicFields from "copyfields" and "dynamicfields" (all lowercase) to "copyFields" and "dynamicFields", respectively, to align with all other REST API outputs, which use camelCase. (Steve Rowe) * SOLR-4658: In preparation for REST API requests that can modify the schema, a "managed schema" is introduced. Add '' to solrconfig.xml in order to use it, and to enable schema modifications via REST API requests. (Steve Rowe, Robert Muir) * SOLR-4656: Added two new highlight parameters, hl.maxMultiValuedToMatch and hl.maxMultiValuedToExamine. maxMultiValuedToMatch stops looking for snippets after finding the specified number of matches, no matter how far into the multivalued field you've gone. maxMultiValuedToExamine stops looking for matches after the specified number of multiValued entries have been examined. If both are specified, the limit hit first stops the loop. Also this patch cuts down on the copying of the document entries during highlighting. These optimizations are probably unnoticeable unless there are a large number of entries in the multiValued field. Conspicuously, this will prevent the "best" match from being found if it appears later in the MV list than the cutoff specified by either of these params. (Erick Erickson) * SOLR-4675: Improve PostingsSolrHighlighter to support per-field/query-time overrides and add additional configuration parameters. See the javadocs for more details and examples. (Robert Muir) * SOLR-3755: A new collections api to add additional shards dynamically by splitting existing shards. (yonik, Anshum Gupta, shalin) * SOLR-4530: DIH: Provide configuration to use Tika's IdentityHtmlMapper (Alexandre Rafalovitch via shalin) * SOLR-4662: Discover SolrCores by directory structure rather than defining them in solr.xml. Also, change the format of solr.xml to be closer to that of solrconfig.xml. This version of Solr will ship the example in the old style, but you can manually try the new style. Solr 4.4 will ship with the new style, and Solr 5.0 will remove support for the old style. (Erick Erickson, Mark Miller) Additional Work: - SOLR-4347: Ensure that newly-created cores via Admin handler are persisted in solr.xml (Erick Erickson) - SOLR-1905: Cores created by the admin request handler should be persisted to solr.xml. Also fixed a problem whereby properties like solr.solr.datadir would be persisted to solr.xml. Also, cores that didn't happen to be loaded were not persisted. (Erick Erickson) * SOLR-4717/SOLR-1351: SimpleFacets now work with localParams allowing faceting on the same field multiple ways (ryan, Uri Boness) * SOLR-4671: CSVResponseWriter now supports pseudo fields. (ryan, nihed mbarek) * SOLR-4358: HttpSolrServer sends the stream name and exposes 'useMultiPartPost' (Karl Wright via ryan) Bug Fixes ---------------------- * SOLR-4543: setting shardHandlerFactory in solr.xml/solr.properties does not work. (Ryan Ernst, Robert Muir via Erick Erickson) * SOLR-4634: Fix scripting engine tests to work with Java 8's "Nashorn" Javascript implementation. (Uwe Schindler) * SOLR-4636: If opening a reader fails for some reason when opening a SolrIndexSearcher, a Directory can be left unreleased. (Mark Miller) * SOLR-4405: Admin UI - admin-extra files are not rendered into the core-menu (steffkes) * SOLR-3956: Fixed group.facet=true to work with negative facet.limit (Chris van der Merwe, hossman) * SOLR-4650: copyField doesn't work with source globs that don't match any explicit or dynamic fields. This regression was introduced in Solr 4.2. (Daniel Collins, Steve Rowe) * SOLR-4641: Schema now throws exception on illegal field parameters. (Robert Muir) * SOLR-3758: Fixed SpellCheckComponent to work consistently with distributed grouping (James Dyer) * SOLR-4652: Fix broken behavior with shared libraries in resource loader for solr.xml plugins. (Ryan Ernst, Robert Muir, Uwe Schindler) * SOLR-4664: ZkStateReader should update aliases on construction. (Mark Miller, Elodie Sannier) * SOLR-4682: CoreAdminRequest.mergeIndexes can not merge multiple cores or indexDirs. (Jason.D.Cao via shalin) * SOLR-4581: When faceting on numeric fields in Solr 4.2, negative values (constraints) were sorted incorrectly. (Alexander Buhr, shalin, yonik) * SOLR-4699: The System admin handler should not assume a file system based data directory location. (Mark Miller) * SOLR-4695: Fix core admin SPLIT action to be useful with non-cloud setups (shalin) * SOLR-4680: Correct example spellcheck configuration's queryAnalyzerFieldType and use "text" field instead of narrower "name" field (ehatcher, Mark Bennett) * SOLR-4702: Fix example /browse "Did you mean?" suggestion feature. (ehatcher, Mark Bennett) * SOLR-4710: You cannot delete a collection fully from ZooKeeper unless all nodes are up and functioning correctly. (Mark Miller) * SOLR-4487: SolrExceptions thrown by HttpSolrServer will now contain the proper HTTP status code returned by the remote server, even if that status code is not something Solr itself returned -- eg: from the Servlet Container, or an intermediate HTTP Proxy (hossman) * SOLR-4661: Admin UI Replication details now correctly displays the current replicable generation/version of the master. (hossman) * SOLR-4716,SOLR-4584: SolrCloud request proxying does not work on Tomcat and perhaps other non Jetty containers. (Po Rui, Yago Riveiro via Mark Miller) * SOLR-4746: Distributed grouping used a NamedList instead of a SimpleOrderedMap for the top level group commands, causing output formatting differences compared to non-distributed grouping. (yonik) * SOLR-4705: Fixed bug causing NPE when querying a single replica in SolrCloud using the shards param (Raintung Li, hossman) * SOLR-4729: LukeRequestHandler: Using a dynamic copyField source that is not also a dynamic field triggers error message 'undefined field: "(glob)"'. (Adam Hahn, hossman, Steve Rowe) Optimizations ---------------------- Other Changes ---------------------- * SOLR-4653: Solr configuration should log inaccessible/ non-existent relative paths in lib dir=... (Dawid Weiss) * SOLR-4317: SolrTestCaseJ4: Can't avoid "collection1" convention (Tricia Jenkins, via Erick Erickson) * SOLR-4571: SolrZkClient#setData should return Stat object. (Mark Miller) * SOLR-4603: CachingDirectoryFactory should use an IdentityHashMap for byDirectoryCache. (Mark Miller) * SOLR-4544: Refactor HttpShardHandlerFactory so load-balancing logic can be customized. (Ryan Ernst via Robert Muir) * SOLR-4607: Use noggit 0.5 release jar rather than a forked copy. (Yonik Seeley, Robert Muir) * SOLR-3706: Ship setup to log with log4j. (ryan, Mark Miller) * SOLR-4651: Remove dist-excl-slf4j build target. (Shawn Heisey) * SOLR-4622: The hardcoded SolrCloud defaults for 'hostContext="solr"' and 'hostPort="8983"' have been deprecated and will be removed in Solr 5.0. Existing solr.xml files that do not have these options explicitly specified should be updated accordingly. (hossman) * SOLR-4672: Requests attempting to use SolrCores which had init failures (that would be reported by CoreAdmin STATUS requests) now result in 500 error responses with the details about the init failure, instead of 404 error responses. (hossman) * SOLR-4730: Make the wiki link more prominent in the release documentation. (Uri Laserson via Robert Muir) ================== 4.2.1 ================== Versions of Major Components --------------------- Apache Tika 1.3 Carrot2 3.6.2 Velocity 1.7 and Velocity Tools 2.0 Apache UIMA 2.3.1 Apache ZooKeeper 3.4.5 Detailed Change List ---------------------- Bug Fixes ---------------------- * SOLR-4567: copyField source glob matching explicit field(s) stopped working in Solr 4.2. (Alexandre Rafalovitch, Steve Rowe) * SOLR-4475: Fix various places that still assume File based paths even when not using a file based DirectoryFactory. (Mark Miller) * SOLR-4551: CachingDirectoryFactory needs to create CacheEntry's with the fullpath not path. (Mark Miller) * SOLR-4555: When forceNew is used with CachingDirectoryFactory#get, the old CachValue should give up it's path as it will be used by a new Directory instance. (Mark Miller) * SOLR-4578: CoreAdminHandler#handleCreateAction gets a SolrCore and does not close it in SolrCloud mode when a core with the same name already exists. (Mark Miller) * SOLR-4574: The Collections API will silently return success on an unknown ACTION parameter. (Mark Miller) * SOLR-4576: Collections API validation errors should cause an exception on clients and otherwise act as validation errors with the Core Admin API. (Mark Miller) * SOLR-4577: The collections API should return responses (success or failure) for each node it attempts to work with. (Mark Miller) * SOLR-4568: The lastPublished state check before becoming a leader is not working correctly. (Mark Miller) * SOLR-4570: Even if an explicit shard id is used, ZkController#preRegister should still wait to see the shard id in it's current ClusterState. (Mark Miller) * SOLR-4585: The Collections API validates numShards with < 0 but should use <= 0. (Mark Miller) * SOLR-4592: DefaultSolrCoreState#doRecovery needs to check the CoreContainer shutdown flag inside the recoveryLock sync block. (Mark Miller) * SOLR-4595: CachingDirectoryFactory#close can throw a concurrent modification exception. (Mark Miller) * SOLR-4573: Accessing Admin UI files in SolrCloud mode logs warnings. (Mark Miller, Phil John) * SOLR-4594: StandardDirectoryFactory#remove accesses byDirectoryCache without a lock. (Mark Miller) * SOLR-4597: CachingDirectoryFactory#remove should not attempt to empty/remove the index right away but flag for removal after close. (Mark Miller) * SOLR-4598: The Core Admin unload command's option 'deleteDataDir', should use the DirectoryFactory API to remove the data dir. (Mark Miller) * SOLR-4599: CachingDirectoryFactory calls close(Directory) on forceNew if the Directory has a refCnt of 0, but it should call closeDirectory(CacheValue). (Mark Miller) * SOLR-4602: ZkController#unregister should cancel it's election participation before asking the Overseer to delete the SolrCore information. (Mark Miller) * SOLR-4601: A Collection that is only partially created and then deleted will leave pre allocated shard information in ZooKeeper. (Mark Miller) * SOLR-4604: UpdateLog#init is over called on SolrCore#reload. (Mark Miller) * SOLR-4605: Rollback does not work correctly. (Mark S, Mark Miller) * SOLR-4609: The Collections API should only send the reload command to ACTIVE cores. (Mark Miller) * SOLR-4297: Atomic update request containing null=true sets all subsequent fields to null (Ben Pennell, Rob, shalin) * SOLR-4371: Admin UI - Analysis Screen shows empty result (steffkes) * SOLR-4318: NPE encountered with querying with wildcards on a field that uses the DefaultAnalyzer (i.e. no analysis chain defined). (Erick Erickson) * SOLR-4361: DataImportHandler would throw UnsupportedOperationException if handler-level parameters were specified containing periods in the name (James Dyer) * SOLR-4538: Date Math expressions were being truncated to 32 characters when used in field:value queries in the lucene QParser. (hossman, yonik) * SOLR-4617: SolrCore#reload needs to pass the deletion policy to the next SolrCore through it's constructor rather than setting a field after. (Mark Miller) * SOLR-4589: Fixed CPU spikes and poor performance in lazy field loading of multivalued fields. (hossman) * SOLR-4608: Update Log replay and PeerSync replay should use the default processor chain to update the index. (Ludovic Boutros, yonik) * SOLR-4625: The solr (lucene syntax) query parser lost top-level boost values and top-level phrase slops on queries produced by nested sub-parsers. (yonik) * SOLR-4624: CachingDirectoryFactory does not need to support forceNew any longer and it appears to be causing a missing close directory bug. forceNew is no longer respected and will be removed in 4.3. (Mark Miller) * SOLR-3819: Grouped faceting (group.facet=true) did not respect filter exclusions. (Petter Remen, yonik) * SOLR-4637: Replication can sometimes wait until shutdown or core unload until removing some tmp directories. (Mark Miller) * SOLR-4638: DefaultSolrCoreState#getIndexWriter(null) is a way to avoid creating the IndexWriter earlier than necessary, but it's not implemented quite right. (Mark Miller) * SOLR-4640: CachingDirectoryFactory can fail to close directories in some race conditions. (Mark Miller) * SOLR-4642: QueryResultKey is not calculating the correct hashCode for filters. (Joel Bernstein via Mark Miller) Optimizations ---------------------- * SOLR-4569: waitForReplicasToComeUp should bail right away if it doesn't see the expected slice in the clusterstate rather than waiting. (Mark Miller) * SOLR-4311: Admin UI - Optimize Caching Behaviour (steffkes) Other Changes ---------------------- * SOLR-4537: Clean up schema information REST API. (Steve Rowe) * SOLR-4596: DistributedQueue should ensure its full path exists in the constructor. (Mark Miller) ================== 4.2.0 ================== Versions of Major Components --------------------- Apache Tika 1.3 Carrot2 3.6.2 Velocity 1.7 and Velocity Tools 2.0 Apache UIMA 2.3.1 Apache ZooKeeper 3.4.5 Upgrading from Solr 4.1.0 ---------------------- (No upgrade instructions yet) Detailed Change List ---------------------- New Features ---------------------- * SOLR-4043: Add ability to get success/failure responses from Collections API. (Raintung Li, Mark Miller) * SOLR-2827: RegexpBoost Update Processor (janhoy) * SOLR-4370: Allow configuring commitWithin to do hard commits. (Mark Miller, Senthuran Sivananthan) * SOLR-4451: SolrJ, and SolrCloud internals, now use SystemDefaultHttpClient under the covers -- allowing many HTTP connection related properties to be controlled via 'standard' java system properties. (hossman) * SOLR-3855, SOLR-4490: Doc values support. (Adrien Grand, Robert Muir) * SOLR-4417: Reopen the IndexWriter on SolrCore reload. (Mark Miller) * SOLR-4477: Add support for queries (match-only) against docvalues fields. (Robert Muir) * SOLR-4488: Return slave replication details for a master if the master has also acted like a slave. (Mark Miller) * SOLR-4498: Add list command to ZkCLI that prints out the contents of ZooKeeper. (Roman Shaposhnik via Mark Miller) * SOLR-4481: SwitchQParserPlugin registered by default as 'switch' using syntax: {!switch case=XXX case.foo=YYY case.bar=ZZZ default=QQQ}foo (hossman) * SOLR-4078: Allow custom naming of SolrCloud nodes so that a new host:port combination can take over for a previous shard. (Mark Miller) * SOLR-4210: Requests to a Collection that does not exist on the receiving node should be proxied to a suitable node. (Mark Miller, Po Rui, yonik) * SOLR-1365: New SweetSpotSimilarityFactory allows customizable TF/IDF based Similarity when you know the optimal "Sweet Spot" of values for the field length and TF scoring factors. (hossman) * SOLR-4138: CurrencyField fields can now be used in a ValueSources to get the "raw" value (using the default number of fractional digits) in the default currency of the field type. There is also a new currency(field,[CODE]) function for generating a ValueSource of the "natural" value, converted to an optionally specified currency to override the default for the field type. (hossman) * SOLR-4503: Add REST API methods, via Restlet integration, for reading schema elements, at /schema/fields/, /schema/dynamicfields/, /schema/fieldtypes/, and /schema/copyfields/. (Steve Rowe) Bug Fixes ---------------------- * SOLR-2850: Do not refine facets when minCount == 1 (Matt Smith, lundgren via Adrien Grand) * SOLR-4309: /browse: Improve JQuery autosuggest behavior (janhoy) * SOLR-4330: group.sort is ignored when using group.truncate and ex/tag local params together (koji) * SOLR-4321: Collections API will sometimes use a node more than once, even when more unused nodes are available. (Eric Falcao, Brett Hoerner, Mark Miller) * SOLR-4345 : Solr Admin UI dosent work in IE 10 (steffkes) * SOLR-4349 : Admin UI - Query Interface does not work in IE (steffkes) * SOLR-4359: The RecentUpdates#update method should treat a problem reading the next record the same as a problem parsing the record - log the exception and break. (Mark Miller) * SOLR-4225: Term info page under schema browser shows incorrect count of terms (steffkes) * SOLR-3926: Solr should support better way of finding active sorts (Eirik Lygre via Erick Erickson) * SOLR-4342: Fix DataImportHandler stats to be a proper Map (hossman) * SOLR-3967: langid.enforceSchema option checks source field instead of target field (janhoy) * SOLR-4380: Replicate after startup option would not replicate until the IndexWriter was lazily opened. (Mark Miller, Gregg Donovan) * SOLR-4400: Deadlock can occur in a rare race between committing and closing a SolrIndexWriter. (Erick Erickson, Mark Miller) * SOLR-3655: A restarted node can briefly appear live and active before it really is in some cases. (Mark Miller) * SOLR-4426: NRTCachingDirectoryFactory does not initialize maxCachedMB and maxMergeSizeMB if is not present in solrconfig.xml (Jack Krupansky via shalin) * SOLR-4463: Fix SolrCoreState reference counting. (Mark Miller) * SOLR-4459: The Replication 'index move' rather than copy optimization doesn't kick in when using NRTCachingDirectory or the rate limiting feature. (Mark Miller) * SOLR-4421,SOLR-4165: On CoreContainer shutdown, all SolrCores should publish their state as DOWN. (Mark Miller, Markus Jelsma) * SOLR-4467: Ephemeral directory implementations may not recover correctly because the code to clear the tlog files on startup is off. (Mark Miller) * SOLR-4413: Fix SolrCore#getIndexDir() to return the current index directory. (Gregg Donovan, Mark Miller) * SOLR-4469: A new IndexWriter must be opened on SolrCore reload when the index directory has changed and the previous SolrCore's state should not be propagated. (Mark Miller, Gregg Donovan) * SOLR-4471: Replication occurs even when a slave is already up to date. (Mark Miller, Andre Charton) * SOLR-4484: ReplicationHandler#loadReplicationProperties still uses Files rather than the Directory to try and read the replication properties files. (Mark Miller) * SOLR-4352: /browse pagination now supports and preserves sort context (Eric Spiegelberg, Erik Hatcher) * LUCENE-4796, SOLR-4373: Fix concurrency issue in NamedSPILoader and AnalysisSPILoader when doing concurrent core loads in multicore Solr configs. (Uwe Schindler, Hossman) * SOLR-4504: Fixed CurrencyField range queries to correctly exclude documents w/o values (hossman) * SOLR-4480: A trailing + or - caused the edismax parser to throw an exception. (Fiona Tay, Jan Høydahl, yonik) * SOLR-4507: The Cloud tab does not show up in the Admin UI if you set zkHost in solr.xml. (Alfonso Presa, Mark Miller) * SOLR-4505: Possible deadlock around SolrCoreState update lock. (Erick Erickson, Mark Miller) * SOLR-4511: When a new index is replicated into place, we need to update the most recent replicatable index point without doing a commit. This is important for repeater use cases, as well as when nodes may switch master/slave roles. (Mark Miller, Raúl Grande) * SOLR-4515: CurrencyField's OpenExchangeRatesOrgProvider now requires a ratesFileLocation init param, since the previous global default no longer works (hossman) * SOLR-4518: Improved CurrencyField error messages when attempting to use a Currency that is not supported by the current JVM. (hossman) * SOLR-3798: Fix copyField implementation in IndexSchema to handle dynamic field references that aren't string-equal to the name of the referenced dynamic field. (Steve Rowe) * SOLR-4497: Collection Aliasing. (Mark Miller) Optimizations ---------------------- * SOLR-4339: Admin UI - Display Field-Flags on Schema-Browser (steffkes) * SOLR-4340: Admin UI - Analysis's Button Spinner goes wild (steffkes) * SOLR-4341: Admin UI - Plugins/Stats Page contains loooong Values which result in horizontal Scrollbar (steffkes) * SOLR-3915: Color Legend for Cloud UI (steffkes) * SOLR-4306: Utilize indexInfo=false when gathering core names in UI (steffkes) * SOLR-4284: Admin UI - make core list scrollable separate from the rest of the UI (steffkes) * SOLR-4364: Admin UI - Locale based number formatting (steffkes) * SOLR-4521: Stop using the 'force' option for recovery replication. This will keep some less common unnecessary replications from happening. (Mark Miller, Simon Scofield) * SOLR-4529: Improve Admin UI Dashboard legibility (Felix Buenemann via steffkes) * SOLR-4526: Admin UI depends on optional system info (Felix Buenemann via steffkes) Other Changes ---------------------- * SOLR-4259: Carrot2 dependency should be declared on the mini version, not the core. (Dawid Weiss). * SOLR-4348: Make the lock type configurable by system property by default. (Mark Miller) * SOLR-4353: Renamed example jetty context file to reduce confusion (hossman) * SOLR-4384: Make post.jar report timing information (Upayavira via janhoy) * SOLR-4415: Add 'state' to shards (default to 'active') and read/write them to ZooKeeper (Anshum Gupta via shalin) * SOLR-4394: Tests and example configs demonstrating SSL with both server and client certs (hossman) * SOLR-3060: SurroundQParserPlugin highlighting tests (Ahmet Arslan via hossman) * SOLR-2470: Added more tests for VelocityResponseWriter * SOLR-4471: Improve and clean up TestReplicationHandler. (Amit Nithian via Mark Miller) * SOLR-3843: Include lucene codecs jar and enable per-field postings and docvalues support in the schema.xml (Robert Muir, Steve Rowe) * SOLR-4511: Add new test for 'repeater' replication node. (Mark Miller) * SOLR-4458: Sort directions (asc, desc) are now case insensitive (Shawn Heisey via hossman) * SOLR-2996: A bare * without a field specification is treated as *:* by the lucene and edismax query paesers. (hossman, Jan Høydahl, Alan Woodward, yonik) * SOLR-4416: Upgrade to Tika 1.3. (Markus Jelsma via Mark Miller) * SOLR-4200: Reduce INFO level logging from CachingDirectoryFactory (Shawn Heisey via hossman) ================== 4.1.0 ================== Versions of Major Components --------------------- Apache Tika 1.2 Carrot2 3.6.2 Velocity 1.7 and Velocity Tools 2.0 Apache UIMA 2.3.1 Apache ZooKeeper 3.4.5 Upgrading from Solr 4.0.0 ---------------------- Custom java parsing plugins need to migrate from throwing the internal ParseException to throwing SyntaxError. BaseDistributedSearchTestCase now randomizes the servlet context it uses when creating Jetty instances. Subclasses that assume a hard coded context of "/solr" should either be fixed to use the "String context" variable, or should take advantage of the new BaseDistributedSearchTestCase(String) constructor to explicitly specify a fixed servlet context path. See SOLR-4136 for details. Detailed Change List ---------------------- New Features ---------------------- * SOLR-2255: Enhanced pivot faceting to use local-params in the same way that regular field value faceting can. This means support for excluding a filter query, using a different output key, and specifying 'threads' to do facet.method=fcs concurrently. PivotFacetHelper now extends SimpleFacet and the getFacetImplementation() extension hook was removed. (dsmiley) * SOLR-3897: A highlighter parameter "hl.preserveMulti" to return all of the values of a multiValued field in their original order when highlighting. (Joel Bernstein via yonik) * SOLR-3929: Support configuring IndexWriter max thread count in solrconfig. (phunt via Mark Miller) * SOLR-3906: Add support for AnalyzingSuggester (LUCENE-3842), where the underlying analyzed form used for suggestions is separate from the returned text. (Robert Muir) * SOLR-3985: ExternalFileField caches can be reloaded on firstSearcher/ newSearcher events using the ExternalFileFieldReloader (Alan Woodward) * SOLR-3911: Make Directory and DirectoryFactory first class so that the majority of Solr's features work with any custom implementations. (Mark Miller) Additional Work: - SOLR-4032: Files larger than an internal buffer size fail to replicate. (Mark Miller, Markus Jelsma) - SOLR-4033: Consistently use the solrconfig.xml lockType everywhere. (Mark Miller, Markus Jelsma) - SOLR-4144: Replication using too much RAM. (yonik, Markus Jelsma) - SOLR-4187: NPE on Directory release (Mark Miller, Markus Jelsma) * SOLR-4051: Add element to DIH's data-config.xml file, allowing the user to specify the location, filename and Locale for the "data-config.properties" file. Alternatively, users can specify their own property writer implementation for greater control. This new configuration element is optional, and defaults mimic prior behavior. The one exception is that the "root" locale is default. Previously it was the machine's default locale. (James Dyer) * SOLR-4084: Add FuzzyLookupFactory, which is like AnalyzingSuggester except that it can tolerate typos in the input. (Areek Zillur via Robert Muir) * SOLR-4088: New and improved auto host detection strategy for SolrCloud. (Raintung Li via Mark Miller) * SOLR-3970: SystemInfoHandler now exposes more details about the JRE/VM/Java version in use. (hossman) * SOLR-4101: Add support for storing term offsets in the index via a 'storeOffsetsWithPositions' flag on field definitions in the schema. (Tom Winch, Alan Woodward) * SOLR-4093: Solr QParsers may now be directly invoked in the lucene query syntax without the _query_ magic field hack. Example: foo AND {!term f=myfield v=$qq} (yonik) * SOLR-4087: Add MAX_DOC_FREQ option to MoreLikeThis. (Andrew Janowczyk via Mark Miller) * SOLR-4114: Allow creating more than one shard per instance with the Collection API. (Per Steffensen, Mark Miller) * SOLR-3531: Allowing configuring maxMergeSizeMB and maxCachedMB when using NRTCachingDirectoryFactory. (Andy Laird via Mark Miller) * SOLR-4118: Fix replicationFactor to align with industry usage. replicationFactor now means the total number of copies of a document stored in the collection (or the total number of physical indexes for a single logical slice of the collection). For example if replicationFactor=3 then for a given shard there will be a total of 3 replicas (one of which will normally be designated as the leader.) (yonik) * SOLR-4124: You should be able to set the update log directory with the CoreAdmin API the same way as the data directory. (Mark Miller) * SOLR-4028: When using ZK chroot, it would be nice if Solr would create the initial path when it doesn't exist. (Tomás Fernández Löbbe via Mark Miller) * SOLR-3948: Calculate/display deleted documents in admin interface. (Shawn Heisey via Mark Miller) * SOLR-4030: Allow rate limiting Directory IO based on the IO context. (Mark Miller, Radim Kolar) * SOLR-4166: LBHttpSolrServer ignores ResponseParser passed in constructor. (Steve Molloy via Mark Miller) * SOLR-4140: Allow access to the collections API through CloudSolrServer without referencing an existing collection. (Per Steffensen via Mark Miller) * SOLR-788: Distributed search support for MLT. (Matthew Woytowitz, Mike Anderson, Jamie Johnson, Mark Miller) * SOLR-4120: Collection API: Support for specifying a list of Solr addresses to spread a new collection across. (Per Steffensen via Mark Miller) * SOLR-4110: Configurable Content-Type headers for PHPResponseWriters and PHPSerializedResponseWriter. (Dominik Siebel via Mark Miller) * SOLR-1028: The ability to specify "transient" and "loadOnStartup" as a new properties of tags in solr.xml. Can specify "transientCacheSize" in the tag. Together these allow cores to be loaded only when needed and only transientCacheSize transient cores will be loaded at a time, the rest aged out on an LRU basis. * SOLR-4246: When update.distrib is set to skip update processors before the distributed update processor, always include the log update processor so forwarded updates will still be logged. (yonik) * SOLR-4230: The new Solr 4 spatial fields now work with the {!geofilt} and {!bbox} query parsers. The score local-param works too. (David Smiley) * SOLR-1972: Add extra statistics to RequestHandlers - 5 & 15-minute reqs/sec rolling averages; median, 75th, 95th, 99th, 99.9th percentile request times (Alan Woodward, Shawn Heisey, Adrien Grand, Uwe Schindler) * SOLR-4271: Add support for PostingsHighlighter. (Robert Muir) * SOLR-4255: The new Solr 4 spatial fields now have a 'filter' boolean local-param that can be set to false to not filter. Its useful when there is already a spatial filter query but you also need to sort or boost by distance. (David Smiley) * SOLR-4265, SOLR-4283: Solr now parses request parameters (in URL or sent with POST using content-type application/x-www-form-urlencoded) in its dispatcher code. It no longer relies on special configuration settings in Tomcat or other web containers to enable UTF-8 encoding, which is mandatory for correct Solr behaviour. Query strings passed in via the URL need to be properly-%-escaped, UTF-8 encoded bytes, otherwise Solr refuses to handle the request. The maximum length of x-www-form-urlencoded POST parameters can now be configured through the requestDispatcher/requestParsers/@formdataUploadLimitInKB setting in solrconfig.xml (defaults to 2 MiB). Solr now works out of the box with e.g. Tomcat, JBoss,... (Uwe Schindler, Dawid Weiss, Alex Rocher) * SOLR-2201: DIH's "formatDate" function now supports a timezone as an optional fourth parameter (James Dyer, Mark Waddle) * SOLR-4302: New parameter 'indexInfo' (defaults to true) in CoreAdmin STATUS command can be used to omit index specific information (Shahar Davidson via shalin) * SOLR-2592: Collection specific document routing. The "compositeId" router is the default for collections with hash based routing (i.e. when numShards=N is specified on collection creation). Documents with ids sharing the same domain (prefix) will be routed to the same shard, allowing for efficient querying. Example: The following two documents will be indexed to the same shard since they share the same domain "customerB!". {"id" : "customerB!doc1" [...] } {"id" : "customerB!doc2" [...] } At query time, one can specify a "shard.keys" parameter that lists what shards the query should cover. http://.../query?q=my_query&shard.keys=customerB! Collections that do not specify numShards at collection creation time use custom sharding and default to the "implicit" router. Document updates received by a shard will be indexed to that shard, unless a "_shard_" parameter or document field names a different shard. (Michael Garski, Dan Rosher, yonik) Optimizations ---------------------- * SOLR-3788: Admin Cores UI should redirect to newly created core details (steffkes) * SOLR-3895: XML and XSLT UpdateRequestHandler should not try to resolve external entities. This improves speed of loading e.g. XSL-transformed XHTML documents. (Martin Herfurt, uschindler, hossman) * SOLR-3614: Fix XML parsing in XPathEntityProcessor to correctly expand named entities, but ignore external entities. (uschindler, hossman) * SOLR-3734: Improve Schema-Browser Handling for CopyField using dynamicField's (steffkes) * SOLR-3941: The "commitOnLeader" part of distributed recovery can use openSearcher=false. (Tomás Fernández Löbbe via Mark Miller) * SOLR-4063: Allow CoreContainer to load multiple SolrCores in parallel rather than just serially. (Mark Miller) * SOLR-4199: When doing zk retries due to connection loss, rather than just retrying for 2 minutes, retry in proportion to the session timeout. (Mark Miller) * SOLR-4262: Replication Icon on Dashboard does not reflect Master-/Slave- State (steffkes) * SOLR-4264: Missing Error-Screen on UI's Cloud-Page (steffkes) * SOLR-4261: Percentage Infos on Dashboard have a fixed width (steffkes) * SOLR-3851: create a new core/delete an existing core should also update the main/left list of cores on the admin UI (steffkes) * SOLR-3840: XML query response display is unreadable in Solr Admin Query UI (steffkes) * SOLR-3982: Admin UI: Various Dataimport Improvements (steffkes) * SOLR-4296: Admin UI: Improve Dataimport Auto-Refresh (steffkes) * SOLR-3458: Allow multiple Items to stay open on Plugins-Page (steffkes) Bug Fixes ---------------------- * SOLR-4288: Improve logging for FileDataSource (basePath, relative resources). (Dawid Weiss) * SOLR-4007: Morfologik dictionaries not available in Solr field type due to class loader lookup problems. (Lance Norskog, Dawid Weiss) * SOLR-3560: Handle different types of Exception Messages for Logging UI (steffkes) * SOLR-3637: Commit Status at Core-Admin UI is always false (steffkes) * SOLR-3917: Partial State on Schema-Browser UI is not defined for Dynamic Fields & Types (steffkes) * SOLR-3939: Consider a sync attempt from leader to replica that fails due to 404 a success. (Mark Miller, Joel Bernstein) * SOLR-3940: Rejoining the leader election incorrectly triggers the code path for a fresh cluster start rather than fail over. (Mark Miller) * SOLR-3961: Fixed error using LimitTokenCountFilterFactory (Jack Krupansky, hossman) * SOLR-3933: Distributed commits are not guaranteed to be ordered within a request. (Mark Miller) * SOLR-3939: An empty or just replicated index cannot become the leader of a shard after a leader goes down. (Joel Bernstein, yonik, Mark Miller) * SOLR-3971: A collection that is created with numShards=1 turns into a numShards=2 collection after starting up a second core and not specifying numShards. (Mark Miller) * SOLR-3988: Fixed SolrTestCaseJ4.adoc(SolrInputDocument) to respect field and document boosts (hossman) * SOLR-3981: Fixed bug that resulted in document boosts being compounded in destination fields. (hossman) * SOLR-3920: Fix server list caching in CloudSolrServer when using more than one collection list with the same instance. (Grzegorz Sobczyk, Mark Miller) * SOLR-3938: prepareCommit command omits commitData causing a failure to trigger replication to slaves. (yonik) * SOLR-3992: QuerySenderListener doesn't populate document cache. (Shotaro Kamio, yonik) * SOLR-3995: Recovery may never finish on SolrCore shutdown if the last reference to a SolrCore is closed by the recovery process. (Mark Miller) * SOLR-3998: Atomic update on uniqueKey field itself causes duplicate document. (Eric Spencer, yonik) * SOLR-4001: In CachingDirectoryFactory#close, if there are still refs for a Directory outstanding, we need to wait for them to be released before closing. (Mark Miller) * SOLR-4005: If CoreContainer fails to register a created core, it should close it. (Mark Miller) * SOLR-4009: OverseerCollectionProcessor is not resilient to many error conditions and can stop running on errors. (Raintung Li, milesli, Mark Miller) * SOLR-4019: Log stack traces for 503/Service Unavailable SolrException if not thrown by PingRequestHandler. Do not log exceptions if a user tries to view a hidden file using ShowFileRequestHandler. (Tomás Fernández Löbbe via James Dyer) * SOLR-3589: Edismax parser does not honor mm parameter if analyzer splits a token. (Tom Burton-West, Robert Muir) * SOLR-4031: Upgrade to Jetty 8.1.7 to fix a bug where in very rare occasions the content of two concurrent requests get mixed up. (Per Steffensen, yonik) * SOLR-4060: ReplicationHandler can try and do a snappull and open a new IndexWriter after shutdown has already occurred, leaving an IndexWriter that is not closed. (Mark Miller) * SOLR-4055: Fix a thread safety issue with the Collections API that could cause actions to be targeted at the wrong SolrCores. (Raintung Li, Per Steffensen via Mark Miller) * SOLR-3993: If multiple SolrCore's for a shard coexist on a node, on cluster restart, leader election would stall until timeout, waiting to see all of the replicas come up. (Mark Miller, Alexey Kudinov) * SOLR-2045: Databases that require a commit to be issued before closing the connection on a non-read-only database leak connections. Also expanded the SqlEntityProcessor test to sometimes use Derby as well as HSQLDB (Derby is one db affected by this bug). (Fenlor Sebastia, James Dyer) * SOLR-4064: When there is an unexpected exception while trying to run the new leader process, the SolrCore will not correctly rejoin the election. (Po Rui via Mark Miller) * SOLR-3989: SolrZkClient constructor dropped exception cause when throwing a new RuntimeException. (Colin Bartolome, yonik) * SOLR-4036: field aliases in fl should not cause properties of target field to be used. (Martin Koch, yonik) * SOLR-4003: The SolrZKClient clean method should not try and clear zk paths that start with /zookeeper, as this can fail and stop the removal of further nodes. (Mark Miller) * SOLR-4076: SolrQueryParser should run fuzzy terms through MultiTermAwareComponents to ensure that (for example) a fuzzy query of foobar~2 is equivalent to FooBar~2 on a field that includes lowercasing. (yonik) * SOLR-4081: QueryParsing.toString, used during debugQuery=true, did not correctly handle ExtendedQueries such as WrappedQuery (used when cache=false), spatial queries, and frange queries. (Eirik Lygre, yonik) * SOLR-3959: Ensure the internal comma separator of poly fields is escaped for CSVResponseWriter. (Areek Zillur via Robert Muir) * SOLR-4075: A logical shard that has had all of it's SolrCores unloaded should be removed from the cluster state. (Mark Miller, Gilles Comeau) * SOLR-4034: Check if a collection already exists before trying to create a new one. (Po Rui, Mark Miller) * SOLR-4097: Race can cause NPE in logging line on first cluster state update. (Mark Miller) * SOLR-4099: Allow the collection api work queue to make forward progress even when it's watcher is not fired for some reason. (Raintung Li via Mark Miller) * SOLR-3960: Fixed a bug where Distributed Grouping ignored PostFilters (Nathan Visagan, hossman) * SOLR-3842: DIH would not populate multivalued fields if the column name derives from a resolved variable (James Dyer) * SOLR-4117: Retrieving the size of the index may use the wrong index dir if you are replicating. (Mark Miller, Markus Jelsma) * SOLR-2890: Fixed a bug that prevented omitNorms and omitTermFreqAndPositions options from being respected in some declarations (hossman) * SOLR-4159: When we are starting a shard from rest, a potential leader should not consider it's last published state when deciding if it can be the new leader. (Mark Miller) * SOLR-4158: When a core is registering in ZooKeeper it may not wait long enough to find the leader due to how long the potential leader waits to see replicas. (Mark Miller, Alain Rogister) * SOLR-4162: ZkCli usage examples are not correct because the zkhost parameter is not present and it is mandatory for all commands. (Tomás Fernández Löbbe via Mark Miller) * SOLR-4071: Validate that name is pass to Collections API create, and behave the same way as on startup when collection.configName is not explicitly passed. (Po Rui, Mark Miller) * SOLR-4127: Added explicit error message if users attempt Atomic document updates with either updateLog or DistribUpdateProcessor. (hossman) * SOLR-4136: Fix SolrCloud behavior when using "hostContext" containing "_" or"/" characters. This fix also makes SolrCloud more accepting of hostContext values with leading/trailing slashes. (hossman) * SOLR-4168: Ensure we are using the absolute latest index dir when getting list of files for replication. (Mark Miller) * SOLR-4171: CachingDirectoryFactory should not return any directories after it has been closed. (Mark Miller) * SOLR-4102: Fix UI javascript error if canonical hostname can not be resolved (steffkes via hossman) * SOLR-4178: ReplicationHandler should abort any current pulls and wait for it's executor to stop during core close. (Mark Miller) * SOLR-3918: Fixed the 'dist-war-excl-slf4j' ant target to exclude all slf4j jars, so that the resulting war is usable as is provided the servlet container includes the correct slf4j api and impl jars. (Shawn Heisey, hossman) * SOLR-4198: OverseerCollectionProcessor should implement ClosableThread. (Mark Miller) * SOLR-4213: Directories that are not shutdown until DirectoryFactory#close do not have close listeners called on them. (Mark Miller) * SOLR-4134: Standard (XML) request writer cannot "set" multiple values into multivalued field with partial updates. (Luis Cappa Banda, Will Butler, shalin) * SOLR-3972: Fix ShowFileRequestHandler to not log a warning in the (expected) situation of a file not found. (hossman) * SOLR-4133: Cannot "set" field to null with partial updates when using the standard RequestWriter. (Will Butler, shalin) * SOLR-4223: "maxFormContentSize" in jetty.xml is not picked up by jetty 8 so set it via solr webapp context file. (shalin) * SOLR-4175:SearchComponent chain can't contain two components of the same class and use debugQuery. (Tomás Fernández Löbbe via ehatcher) * SOLR-4244: When coming back from session expiration we should not wait for the leader to see us in the down state if we are the node that must become the leader. (Mark Miller) * SOLR-4245: When a core is registering with ZooKeeper, the timeout to find the leader in the cluster state is 30 seconds rather than leaderVoteWait + extra time. (Mark Miller) * SOLR-4238: Fix jetty example requestLog config (jm via hossman) * SOLR-4251: Fix SynonymFilterFactory when an optional tokenizerFactory is supplied. (Chris Bleakley via rmuir) * SOLR-4253: Misleading resource loading warning from Carrot2 clustering component fixed (Stanisław Osiński) * SOLR-4257: PeerSync updates and Log Replay updates should not wait for a ZooKeeper connection in order to proceed. (yonik) * SOLR-4045: SOLR admin page returns HTTP 404 on core names containing a '.' (dot) (steffkes) * SOLR-4176: analysis ui: javascript not properly handling URL decoding of input (steffkes) * SOLR-4079: Long core names break web gui appearance and functionality (steffkes) * SOLR-4263: Incorrect Link from Schema-Browser to Query From for Top-Terms (steffkes) * SOLR-3829: Admin UI Logging events broken if schema.xml defines a catch-all dynamicField with type ignored (steffkes) * SOLR-4275: Fix TrieTokenizer to no longer throw StringIndexOutOfBoundsException in admin UI / AnalysisRequestHandler when you enter no number to tokenize. (Uwe Schindler) * SOLR-4279: Wrong exception message if _version_ field is multivalued (shalin) * SOLR-4170: The 'backup' ReplicationHandler command can sometimes use a stale index directory rather than the current one. (Mark Miller, Marcin Rzewucki) * SOLR-3876: Solr Admin UI is completely dysfunctional on IE 9 (steffkes) * SOLR-4112: Fixed DataImportHandler ZKAwarePropertiesWriter implementation so import works fine with SolrCloud clusters (Deniz Durmus, James Dyer, Erick Erickson, shalin) * SOLR-4291: Harden the Overseer work queue thread loop. (Mark Miller) * SOLR-3820: Solr Admin Query form is missing some edismax request parameters (steffkes) * SOLR-4217: post.jar no longer ignores -Dparams when -Durl is used. (Alexandre Rafalovitch, ehatcher) * SOLR-4303: On replication, if the generation of the master is lower than the slave we need to force a full copy of the index. (Mark Miller, Gregg Donovan) * SOLR-4266: HttpSolrServer does not release connection properly on exception when no response parser is used. (Steve Molloy via Mark Miller) * SOLR-2298: Updated JavaDoc for SolrDocument.addField and SolrInputDocument.addField to have more information on name and value parameters. (Siva Natarajan) Other Changes ---------------------- * SOLR-4106: Javac/ ivy path warnings with morfologik fixed by upgrading to Morfologik 1.5.5 (Robert Muir, Dawid Weiss) * SOLR-3899: SolrCore should not log at warning level when the index directory changes - it's an info event. (Tobias Bergman, Mark Miller) * SOLR-3861: Refactor SolrCoreState so that it's managed by SolrCore. (Mark Miller, hossman) * SOLR-3966: Eliminate superfluous warning from LanguageIdentifierUpdateProcessor (Markus Jelsma via hossman) * SOLR-3932: SolrCmdDistributorTest either takes 3 seconds or 3 minutes. (yonik, Mark Miller) * SOLR-3856: New tests for SqlEntityProcessor/CachedSqlEntityProcessor (James Dyer) * SOLR-4067: ZkStateReader#getLeaderProps should not return props for a leader that it does not think is live. (Mark Miller) * SOLR-4086: DIH refactor of VariableResolver and Evaluator. VariableResolver and each built-in Evaluator are separate concrete classes. DateFormatEvaluator now defaults with the ROOT Locale. However, users may specify a different Locale using an optional new third parameter. (James Dyer) * SOLR-3602: Update ZooKeeper to 3.4.5 (Mark Miller) * SOLR-4095: DIH NumberFormatTransformer & DateFormatTransformer default to the ROOT Locale if none is specified. These previously used the machine's default. (James Dyer) * SOLR-4096: DIH FileDataSource & FieldReaderDataSource default to UTF-8 encoding if none is specified. These previously used the machine's default. (James Dyer) * SOLR-1916: DIH to not use Lucene-forbidden Java APIs (default encoding, locale, etc.) (James Dyer, Robert Muir) * SOLR-4111: SpellCheckCollatorTest#testContextSensitiveCollate to test against both DirectSolrSpellChecker & IndexBasedSpellChecker (Tomás Fernández Löbbe via James Dyer) * SOLR-2141: Better test coverage for Evaluators (James Dyer) * SOLR-4119: Update Guava to 13.0.1 (Mark Miller) * SOLR-4074: Raise default ramBufferSizeMB to 100 from 32. (yonik, Mark Miller) * SOLR-4062: The update log location in solrconfig.xml should default to ${solr.ulog.dir} rather than ${solr.data.dir:} (Mark Miller) * SOLR-4155: Upgrade Jetty to 8.1.8. (Robert Muir) * SOLR-2986: Add MoreLikeThis to warning about features that require uniqueKey. Also, change the warning to warn log level. (Shawn Heisey via Mark Miller) * SOLR-4163: README improvements (Shawn Heisey via hossman) * SOLR-4248: "ant eclipse" should declare .svn directories as derived. (Shawn Heisey via Mark Miller) * SOLR-3279: Upgrade Carrot2 to 3.6.2 (Stanisław Osiński) * SOLR-4254: Harden the 'leader requests replica to recover' code path. (Mark Miller, yonik) * SOLR-4226: Extract fl parsing code out of ReturnFields constructor. (Ryan Ernst via Robert Muir) * SOLR-4208: ExtendedDismaxQParserPlugin has been refactored to make subclassing easier. (Tomás Fernández Löbbe, hossman) * SOLR-3735: Relocate the example mime-to-extension mapping, and upgrade Velocity Engine to 1.7 (ehatcher) * SOLR-4287: Removed "apache-" prefix from Solr distribution and artifact filenames. (Ryan Ernst, Robert Muir, Steve Rowe) * SOLR-4016: Deduplication does not work with atomic/partial updates so disallow atomic update requests which change signature generating fields. (Joel Nothman, yonik, shalin) * SOLR-4308: Remove the problematic and now unnecessary log4j-over-slf4j. (Mark Miller) ================== 4.0.0 ================== Versions of Major Components --------------------- Apache Tika 1.2 Carrot2 3.5.0 Velocity 1.6.4 and Velocity Tools 2.0 Apache UIMA 2.3.1 Apache ZooKeeper 3.3.6 Upgrading from Solr 4.0.0-BETA ---------------------- In order to better support distributed search mode, the TermVectorComponent's response format has been changed so that if the schema defines a uniqueKeyField, then that field value is used as the "key" for each document in it's response section, instead of the internal lucene doc id. Users w/o a uniqueKeyField will continue to see the same response format. See SOLR-3229 for more details. If you are using SolrCloud's distributed update request capabilities and a non string type id field, you must re-index. Upgrading from Solr 4.0.0-ALPHA ---------------------- Solr is now much more strict about requiring that the uniqueKeyField feature (if used) must refer to a field which is not multiValued. If you upgrade from an earlier version of Solr and see an error that your uniqueKeyField "can not be configured to be multivalued" please add 'multiValued="false"' to the declaration for your uniqueKeyField. See SOLR-3682 for more details. In addition, please review the notes above about upgrading from 4.0.0-BETA Upgrading from Solr 3.6 ---------------------- * The Lucene index format has changed and as a result, once you upgrade, previous versions of Solr will no longer be able to read your indices. In a master/slave configuration, all searchers/slaves should be upgraded before the master. If the master were to be updated first, the older searchers would not be able to read the new index format. * Setting abortOnConfigurationError=false is no longer supported (since it has never worked properly). Solr will now warn you if you attempt to set this configuration option at all. (see SOLR-1846) * The default logic for the 'mm' param of the 'dismax' QParser has been changed. If no 'mm' param is specified (either in the query, or as a default in solrconfig.xml) then the effective value of the 'q.op' param (either in the query or as a default in solrconfig.xml or from the 'defaultOperator' option in schema.xml) is used to influence the behavior. If q.op is effectively "AND" then mm=100%. If q.op is effectively "OR" then mm=0%. Users who wish to force the legacy behavior should set a default value for the 'mm' param in their solrconfig.xml file. * The VelocityResponseWriter is no longer built into the core. Its JAR and dependencies now need to be added (via or solr/home lib inclusion), and it needs to be registered in solrconfig.xml like this: * The update request parameter to choose Update Request Processor Chain is renamed from "update.processor" to "update.chain". The old parameter was deprecated but still working since Solr3.2, but is now removed entirely. * The and sections of solrconfig.xml are discontinued and replaced with the section. There are also better defaults. When migrating, if you don't know what your old settings mean, simply delete both and sections. If you have customizations, put them in section - with same syntax as before. * Two of the SolrServer subclasses in SolrJ were renamed/replaced. CommonsHttpSolrServer is now HttpSolrServer, and StreamingUpdateSolrServer is now ConcurrentUpdateSolrServer. * The PingRequestHandler no longer looks for a option in the (legacy) section of solrconfig.xml. Users who wish to take advantage of this feature should configure a "healthcheckFile" init param directly on the PingRequestHandler. As part of this change, relative file paths have been fixed to be resolved against the data dir. See the example solrconfig.xml and SOLR-1258 for more details. * Due to low level changes to support SolrCloud, the uniqueKey field can no longer be populated via or in the schema.xml. Users wishing to have Solr automatically generate a uniqueKey value when adding documents should instead use an instance of solr.UUIDUpdateProcessorFactory in their update processor chain. See SOLR-2796 for more details. In addition, please review the notes above about upgrading from 4.0.0-BETA, and 4.0.0-ALPHA Detailed Change List ---------------------- New Features ---------------------- * SOLR-3670: New CountFieldValuesUpdateProcessorFactory makes it easy to index the number of values in another field for later use at query time. (hossman) * SOLR-2768: new "mod(x,y)" function for computing the modulus of two value sources. (hossman) * SOLR-3238: Numerous small improvements to the Admin UI (steffkes) * SOLR-3597: seems like a lot of wasted whitespace at the top of the admin screens (steffkes) * SOLR-3304: Added Solr adapters for Lucene 4's new spatial module. With SpatialRecursivePrefixTreeFieldType ("location_rpt" in example schema), it is possible to index a variable number of points per document (and sort on them), index not just points but any Spatial4j supported shape such as polygons, and to query on these shapes too. Polygons requires adding JTS to the classpath. (David Smiley) * SOLR-3825: Added optional capability to log what ids are in a response (Scott Stults via gsingers) * SOLR-3821: Added 'df' to the UI Query form (steffkes) * SOLR-3822: Added hover titles to the edismax params on the UI Query form (steffkes) Optimizations ---------------------- * SOLR-3715: improve concurrency of the transaction log by removing synchronization around log record serialization. (yonik) * SOLR-3807: Currently during recovery we pause for a number of seconds after waiting for the leader to see a recovering state so that any previous updates will have finished before our commit on the leader - we don't need this wait for peersync. (Mark Miller) * SOLR-3837: When a leader is elected and asks replicas to sync back to him and that fails, we should ask those nodes to recovery asynchronously rather than synchronously. (Mark Miller) * SOLR-3709: Cache the url list created from the ClusterState in CloudSolrServer on each request. (Mark Miller) Bug Fixes ---------------------- * SOLR-3685: Solr Cloud sometimes skipped peersync attempt and replicated instead due to tlog flags not being cleared when no updates were buffered during a previous replication. (Markus Jelsma, Mark Miller, yonik) * SOLR-3229: Fixed TermVectorComponent to work with distributed search (Hang Xie, hossman) * SOLR-3725: Fixed package-local-src-tgz target to not bring in unnecessary jars and binary contents. (Michael Dodsworth via rmuir) * SOLR-3649: Fixed bug in JavabinLoader that caused deleteById(List ids) to not work in SolrJ (siren) * SOLR-3730: Rollback is not implemented quite right and can cause corner case fails in SolrCloud tests. (rmuir, Mark Miller) * SOLR-2981: Fixed StatsComponent to no longer return duplicated information when requesting multiple stats.facet fields. (Roman Kliewer via hossman) * SOLR-3743: Fixed issues with atomic updates and optimistic concurrency in conjunction with stored copyField targets by making real-time get never return copyField targets. (yonik) * SOLR-3746: Proper error reporting if updateLog is configured w/o necessary "_version_" field in schema.xml (hossman) * SOLR-3745: Proper error reporting if SolrCloud mode is used w/o necessary "_version_" field in schema.xml (hossman) * SOLR-3770: Overseer may lose updates to cluster state (siren) * SOLR-3721: Fix bug that could theoretically allow multiple recoveries to run briefly at the same time if the recovery thread join call was interrupted. (Per Steffensen, Mark Miller) * SOLR-3782: A leader going down while updates are coming in can cause shard inconsistency. (Mark Miller) * SOLR-3611: We do not show ZooKeeper data in the UI for a node that has children. (Mark Miller) * SOLR-3789: Fix bug in SnapPuller that caused "internal" compression to fail. (siren) * SOLR-3790: ConcurrentModificationException could be thrown when using hl.fl=*. Fixed in r1231606. (yonik, koji) * SOLR-3668: DataImport : Specifying Custom Parameters (steffkes) * SOLR-3793: UnInvertedField faceting cached big terms in the filter cache that ignored deletions, leading to duplicate documents in search later when a filter of the same term was specified. (Günter Hipler, hossman, yonik) * SOLR-3679: Core Admin UI gives no feedback if "Add Core" fails (steffkes, hossman) * SOLR-3795: Fixed LukeRequestHandler response to correctly return field name strings in copyDests and copySources arrays (hossman) * SOLR-3699: Fixed some Directory leaks when there were errors during SolrCore or SolrIndexWriter initialization. (hossman) * SOLR-3518: Include final 'hits' in log information when aggregating a distributed request (Markus Jelsma via hossman) * SOLR-3628: SolrInputField and SolrInputDocument are now consistently backed by Collections passed in to setValue/setField, and defensively copy values from Collections passed to addValue/addField (Tom Switzer via hossman) * SOLR-3595: CurrencyField now generates an appropriate error on schema init if it is configured as multiValued - this has never been properly supported, but previously failed silently in odd ways. (hossman) * SOLR-3823: Fix 'bq' parsing in edismax. Please note that this required reverting the negative boost support added by SOLR-3278 (hossman) * SOLR-3827: Fix shareSchema=true in solr.xml (Tomás Fernández Löbbe via hossman) * SOLR-3809: Fixed config file replication when subdirectories are used (Emmanuel Espina via hossman) * SOLR-3828: Fixed QueryElevationComponent so that using 'markExcludes' does not modify the result set or ranking of 'excluded' documents relative to not using elevation at all. (Alexey Serba via hossman) * SOLR-3569: Fixed debug output on distributed requests when there are no results found. (David Bowen via hossman) * SOLR-3811: Query Form using wrong values for dismax, edismax (steffkes) * SOLR-3779: DataImportHandler's LineEntityProcessor when used in conjunction with FileListEntityProcessor would only process the first file. (Ahmet Arslan via James Dyer) * SOLR-3791: CachedSqlEntityProcessor would throw a NullPointerException when a query returns a row with a NULL key. (Steffen Moelter via James Dyer) * SOLR-3833: When a election is started because a leader went down, the new leader candidate should decline if the last state they published was not active. (yonik, Mark Miller) * SOLR-3836: When doing peer sync, we should only count sync attempts that cannot reach the given host as success when the candidate leader is syncing with the replicas - not when replicas are syncing to the leader. (Mark Miller) * SOLR-3835: In our leader election algorithm, if on connection loss we found we did not create our election node, we should retry, not throw an exception. (Mark Miller) * SOLR-3834: A new leader on cluster startup should also run the leader sync process in case there was a bad cluster shutdown. (Mark Miller) * SOLR-3772: On cluster startup, we should wait until we see all registered replicas before running the leader process - or if they all do not come up, N amount of time. (Mark Miller) * SOLR-3756: If we are elected the leader of a shard, but we fail to publish this for any reason, we should clean up and re trigger a leader election. (Mark Miller) * SOLR-3812: ConnectionLoss during recovery can cause lost updates, leading to shard inconsistency. (Mark Miller) * SOLR-3813: When a new leader syncs, we need to ask all shards to sync back, not just those that are active. (Mark Miller) * SOLR-3641: CoreContainer is not persisting roles core attribute. (hossman, Mark Miller) * SOLR-3527: SolrCmdDistributor drops some of the important commit attributes (maxOptimizeSegments, softCommit, expungeDeletes) when sending a commit to replicas. (Andy Laird, Tomás Fernández Löbbe, Mark Miller) * SOLR-3844: SolrCore reload can fail because it tries to remove the index write lock while already holding it. (Mark Miller) * SOLR-3831: Atomic updates do not distribute correctly to other nodes. (Jim Musil, Mark Miller) * SOLR-3465: Replication causes two searcher warmups. (Michael Garski, Mark Miller) * SOLR-3645: /terms should default to distrib=false. (Nick Cotton, Mark Miller) * SOLR-3759: Various fixes to the example-DIH configs (Ahmet Arslan, hossman) * SOLR-3777: Dataimport-UI does not send unchecked checkboxes (Glenn MacStravic via steffkes) * SOLR-3850: DataImportHandler "cacheKey" parameter was incorrectly renamed "cachePk" (James Dyer) * SOLR-3087: Fixed DOMUtil so that code doing attribute validation will automatically ignore nodes in the reserved "xml" prefix - in particular this fixes some bugs related to xinclude and fieldTypes. (Amit Nithian, hossman) * SOLR-3783: Fixed Pivot Faceting to work with facet.missing=true (hossman) * SOLR-3869: A PeerSync attempt to it's replicas by a candidate leader should not fail on o.a.http.conn.ConnectTimeoutException. (Mark Miller) * SOLR-3875: Fixed index boosts on multi-valued fields when docBoost is used (hossman) * SOLR-3878: Exception when using open-ended range query with CurrencyField (janhoy) * SOLR-3891: CacheValue in CachingDirectoryFactory cannot be used outside of solr.core package. (phunt via Mark Miller) * SOLR-3892: Inconsistent locking when accessing cache in CachingDirectoryFactory from RAMDirectoryFactory and MockDirectoryFactory. (phunt via Mark Miller) * SOLR-3883: Distributed indexing forwards non-applicable request params. (Dan Sutton, Per Steffensen, yonik, Mark Miller) * SOLR-3903: Fixed MissingFormatArgumentException in ConcurrentUpdateSolrServer (hossman) * SOLR-3916: Fixed whitespace bug in parsing the fl param (hossman) Other Changes ---------------------- * SOLR-3690: Fixed binary release packages to include dependencies needed for the solr-test-framework (hossman) * SOLR-2857: The /update/json and /update/csv URLs were restored to aid in the migration of existing clients. (yonik) * SOLR-3691: SimplePostTool: Mode for crawling/posting web pages See http://wiki.apache.org/solr/ExtractingRequestHandler for examples (janhoy) * SOLR-3707: Upgrade Solr to Tika 1.2 (janhoy) * SOLR-2747: Updated changes2html.pl to handle Solr's CHANGES.txt; added target 'changes-to-html' to solr/build.xml. (Steve Rowe, Robert Muir) * SOLR-3752: When a leader goes down, have the Overseer clear the leader state in cluster.json (Mark Miller) * SOLR-3751: Add defensive checks for SolrCloud updates and requests that ensure the local state matches what we can tell the request expected. (Mark Miller) * SOLR-3773: Hash based on the external String id rather than the indexed representation for distributed updates. (Michael Garski, yonik, Mark Miller) * SOLR-3780: Maven build: Make solrj tests run separately from solr-core. (Steve Rowe) * SOLR-3772: Optionally, on cluster startup, we can wait until we see all registered replicas before running the leader process - or if they all do not come up, N amount of time. (Jan Høydahl, Per Steffensen, Mark Miller) * SOLR-3750: Optionally, on session expiration, we can explicitly wait some time before running the leader sync process so that we are sure every node participates. (Per Steffensen, Mark Miller) * SOLR-3824: Velocity: Error messages from search not displayed (janhoy) * SOLR-3826: Test framework improvements for specifying coreName on initCore (Amit Nithian, hossman) * SOLR-3749: Allow default UpdateLog syncLevel to be configured by solrconfig.xml (Raintung Li, Mark Miller) * SOLR-3845: Rename numReplicas to replicationFactor in Collections API. (yonik, Mark Miller) * SOLR-3815: SolrCloud - Add properties such as "range" to shards, which changes the clusterstate.json and puts the shard replicas under "replicas". (yonik) * SOLR-3871: SyncStrategy should use an executor for the threads it creates to request recoveries. (Mark Miller) * SOLR-3870: SyncStrategy should have a close so it can abort earlier on shutdown. (Mark Miller) ================== 4.0.0-BETA =================== Versions of Major Components --------------------- Apache Tika 1.1 Carrot2 3.5.0 Velocity 1.6.4 and Velocity Tools 2.0 Apache UIMA 2.3.1 Apache ZooKeeper 3.3.6 Upgrading from Solr 4.0.0-ALPHA ---------------------- Solr is now much more strict about requiring that the uniqueKeyField feature (if used) must refer to a field which is not multiValued. If you upgrade from an earlier version of Solr and see an error that your uniqueKeyField "can not be configured to be multivalued" please add 'multiValued="false"' to the declaration for your uniqueKeyField. See SOLR-3682 for more details. Detailed Change List ---------------------- New Features ---------------------- * LUCENE-4201: Added JapaneseIterationMarkCharFilterFactory to normalize Japanese iteration marks. (Robert Muir, Christian Moen) * SOLR-1856: In Solr Cell, literals should override Tika-parsed values. Patch adds a param "literalsOverride" which defaults to true, but can be set to "false" to let Tika-parsed values be appended to literal values (Chris Harris, janhoy) * SOLR-3488: Added a Collection management API for SolrCloud. (Tommaso Teofili, Sami Siren, yonik, Mark Miller) * SOLR-3559: Full deleteByQuery support with SolrCloud distributed indexing. All replicas of a shard will be consistent, even if updates arrive in a different order on different replicas. (yonik) * SOLR-1929: Index encrypted documents with ExtractingUpdateRequestHandler. By supplying resource.password= or specifying an external file with regular expressions matching file names, Solr will decrypt and index PDFs and DOCX formats. (janhoy, Yiannis Pericleous) * SOLR-3562: Add options to remove instance dir or data dir on core unload. (Mark Miller, Per Steffensen) * SOLR-2702: The default directory factory was changed to NRTCachingDirectoryFactory which wraps the StandardDirectoryFactory and caches small files for improved Near Real-time (NRT) performance. (Mark Miller, yonik) * SOLR-2616: Include a sample java util logging configuration file. (David Smiley, Mark Miller) * SOLR-3460: Add cloud-scripts directory and a zkcli.sh|bat tool for easy scripting and interaction with ZooKeeper. (Mark Miller) * SOLR-1725: StatelessScriptUpdateProcessorFactory allows users to implement the full ScriptUpdateProcessor API using any scripting language with a javax.script.ScriptEngineFactory (Uri Boness, ehatcher, Simon Rosenthal, hossman) * SOLR-139: Change to updateable documents to create the document if it doesn't already exist. To assert that the document must exist, use the optimistic concurrency feature by specifying a _version_ of 1. (yonik) * LUCENE-2510, LUCENE-4044: Migrated Solr's Tokenizer-, TokenFilter-, and CharFilterFactories to the lucene-analysis module. To add new analysis modules to Solr (like ICU, SmartChinese, Morfologik,...), just drop in the JAR files from Lucene's binary distribution into your Solr instance's lib folder. The factories are automatically made available with SPI. (Chris Male, Robert Muir, Uwe Schindler) * SOLR-3634, SOLR-3635: CoreContainer and CoreAdminHandler will now remember and report back information about failures to initialize SolrCores. These failures will be accessible from the web UI and CoreAdminHandler STATUS command until they are "reset" by creating/renaming a SolrCore with the same name. (hossman, steffkes) * SOLR-1280: Added commented-out example of the new script update processor to the example configuration. See http://wiki.apache.org/solr/ScriptUpdateProcessor (ehatcher) * SOLR-3672: SimplePostTool: Improvements for posting files Support for auto mode, recursive and wildcards (janhoy) Optimizations ---------------------- * SOLR-3708: Add hashCode to ClusterState so that structures built based on the ClusterState can be easily cached. (Mark Miller) * SOLR-3709: Cache the url list created from the ClusterState in CloudSolrServer on each request. (Mark Miller, yonik) * SOLR-3710: Change CloudSolrServer so that update requests are only sent to leaders by default. (Mark Miller) Bug Fixes ---------------------- * SOLR-3582: Our ZooKeeper watchers respond to session events as if they are change events, creating undesirable side effects. (Trym R. Møller, Mark Miller) * SOLR-3467: ExtendedDismax escaping is missing several reserved characters (Michael Dodsworth via janhoy) * SOLR-3587: After reloading a SolrCore, the original Analyzer is still used rather than a new one. (Alexey Serba, yonik, rmuir, Mark Miller) * LUCENE-4185: Fix a bug where CharFilters were wrongly being applied twice. (Michael Froh, rmuir) * SOLR-3610: After reloading a core, indexing would fail on any newly added fields to the schema. (Brent Mills, rmuir) * SOLR-3377: edismax fails to correctly parse a fielded query wrapped by parens. This regression was introduced in 3.6. (Bernd Fehling, Jan Høydahl, yonik) * SOLR-3621: Fix rare concurrency issue when opening a new IndexWriter for replication or rollback. (Mark Miller) * SOLR-1781: Replication index directories not always cleaned up. (Markus Jelsma, Terje Sten Bjerkseth, Mark Miller) * SOLR-3639: Update ZooKeeper to 3.3.6 for a variety of bug fixes. (Mark Miller) * SOLR-3629: Typo in solr.xml persistence when overriding the solrconfig.xml file name using the "config" attribute prevented the override file from being used. (Ryan Zezeski, hossman) * SOLR-3642: Correct broken check for multivalued fields in stats.facet (Yandong Yao, hossman) * SOLR-3660: Velocity: Link to admin page broken (janhoy) * SOLR-3658: Adding thousands of docs with one UpdateProcessorChain instance can briefly create spikes of threads in the thousands. (yonik, Mark Miller) * SOLR-3656: A core reload now always uses the same dataDir. (Mark Miller, yonik) * SOLR-3662: Core reload bugs: a reload always obtained a non-NRT searcher, which could go back in time with respect to the previous core's NRT searcher. Versioning did not work correctly across a core reload, and update handler synchronization was changed to synchronize on core state since more than on update handler can coexist for a single index during a reload. (yonik) * SOLR-3663: There are a couple of bugs in the sync process when a leader goes down and a new leader is elected. (Mark Miller) * SOLR-3623: Fixed inconsistent treatment of third-party dependencies for solr contribs analysis-extras & uima (hossman) * SOLR-3652: Fixed range faceting to error instead of looping infinitely when 'gap' is zero -- or effectively zero due to floating point arithmetic underflow. (hossman) * SOLR-3648: Fixed VelocityResponseWriter template loading in SolrCloud mode. For the example configuration, this means /browse now works with SolrCloud. (janhoy, ehatcher) * SOLR-3677: Fixed misleading error message in web ui to distinguish between no SolrCores loaded vs. no /admin/ handler available. (hossman, steffkes) * SOLR-3428: SolrCmdDistributor flushAdds/flushDeletes can cause repeated adds/deletes to be sent (Mark Miller, Per Steffensen) * SOLR-3647: DistributedQueue should use our Solr zk client rather than the std zk client. ZooKeeper expiration can be permanent otherwise. (Mark Miller) Other Changes ---------------------- * SOLR-3524: Make discarding punctuation configurable in JapaneseTokenizerFactory. The default is to discard punctuation, but this is overridable as an expert option. (Kazuaki Hiraga, Jun Ohtani via Christian Moen) * SOLR-1770: Move the default core instance directory into a collection1 folder. (Mark Miller) * SOLR-3355: Add shard and collection to SolrCore statistics. (Michael Garski, Mark Miller) * SOLR-3575: solr.xml should default to persist=true (Mark Miller) * SOLR-3563: Unloading all cores in a SolrCloud collection will now cause the removal of that collection's meta data from ZooKeeper. (Mark Miller, Per Steffensen) * SOLR-3599: Add zkClientTimeout to solr.xml so that it's obvious how to change it and so that you can change it with a system property. (Mark Miller) * SOLR-3609: Change Solr's expanded webapp directory to be at a consistent path called solr-webapp rather than a temporary directory. (Mark Miller) * SOLR-3600: Raise the default zkClientTimeout from 10 seconds to 15 seconds. (Mark Miller) * SOLR-3215: Clone SolrInputDocument when distrib indexing so that update processors after the distrib update process do not process the document twice. (Mark Miller) * SOLR-3683: Improved error handling if an contains both an explicit class attribute, as well as nested factories. (hossman) * SOLR-3682: Fail to parse schema.xml if uniqueKeyField is multivalued (hossman) * SOLR-2115: DIH no longer requires the "config" parameter to be specified in solrconfig.xml. Instead, the configuration is loaded and parsed with every import. This allows the use of a different configuration with each import, and makes correcting configuration errors simpler. Also, the configuration itself can be passed using the "dataConfig" parameter rather than using a file (this previously worked in debug mode only). When configuration errors are encountered, the error message is returned in XML format. (James Dyer) * SOLR-3439: Make SolrCell easier to use out of the box. Also improves "/browse" to display rich-text documents correctly, along with facets for author and content_type. With the new "content" field, highlighting of body is supported. See also SOLR-3672 for easier posting of a whole directory structure. (Jack Krupansky, janhoy) * SOLR-3579: SolrCloud view should default to the graph view rather than tree view. (steffkes, Mark Miller) ================== 4.0.0-ALPHA ================== More information about this release, including any errata related to the release notes, upgrade instructions, or other changes may be found online at: https://wiki.apache.org/solr/Solr4.0 Versions of Major Components --------------------- Apache Tika 1.1 Carrot2 3.5.0 Velocity 1.6.4 and Velocity Tools 2.0 Apache UIMA 2.3.1 Apache ZooKeeper 3.3.4 Upgrading from Solr 3.6-dev ---------------------- * The Lucene index format has changed and as a result, once you upgrade, previous versions of Solr will no longer be able to read your indices. In a master/slave configuration, all searchers/slaves should be upgraded before the master. If the master were to be updated first, the older searchers would not be able to read the new index format. * Setting abortOnConfigurationError=false is no longer supported (since it has never worked properly). Solr will now warn you if you attempt to set this configuration option at all. (see SOLR-1846) * The default logic for the 'mm' param of the 'dismax' QParser has been changed. If no 'mm' param is specified (either in the query, or as a default in solrconfig.xml) then the effective value of the 'q.op' param (either in the query or as a default in solrconfig.xml or from the 'defaultOperator' option in schema.xml) is used to influence the behavior. If q.op is effectively "AND" then mm=100%. If q.op is effectively "OR" then mm=0%. Users who wish to force the legacy behavior should set a default value for the 'mm' param in their solrconfig.xml file. * The VelocityResponseWriter is no longer built into the core. Its JAR and dependencies now need to be added (via or solr/home lib inclusion), and it needs to be registered in solrconfig.xml like this: * The update request parameter to choose Update Request Processor Chain is renamed from "update.processor" to "update.chain". The old parameter was deprecated but still working since Solr3.2, but is now removed entirely. * The and sections of solrconfig.xml are discontinued and replaced with the section. There are also better defaults. When migrating, if you don't know what your old settings mean, simply delete both and sections. If you have customizations, put them in section - with same syntax as before. * Two of the SolrServer subclasses in SolrJ were renamed/replaced. CommonsHttpSolrServer is now HttpSolrServer, and StreamingUpdateSolrServer is now ConcurrentUpdateSolrServer. * The PingRequestHandler no longer looks for a option in the (legacy) section of solrconfig.xml. Users who wish to take advantage of this feature should configure a "healthcheckFile" init param directly on the PingRequestHandler. As part of this change, relative file paths have been fixed to be resolved against the data dir. See the example solrconfig.xml and SOLR-1258 for more details. * Due to low level changes to support SolrCloud, the uniqueKey field can no longer be populated via or in the schema.xml. Users wishing to have Solr automatically generate a uniqueKey value when adding documents should instead use an instance of solr.UUIDUpdateProcessorFactory in their update processor chain. See SOLR-2796 for more details. Detailed Change List ---------------------- New Features ---------------------- * SOLR-3272: Solr filter factory for MorfologikFilter (Polish lemmatisation). (Rafał Kuć via Dawid Weiss, Steven Rowe, Uwe Schindler). * SOLR-571: The autowarmCount for LRUCaches (LRUCache and FastLRUCache) now supports "percentages" which get evaluated relative the current size of the cache when warming happens. (Tomás Fernández Löbbe and hossman) * SOLR-1932: New relevancy function queries: termfreq, tf, docfreq, idf norm, maxdoc, numdocs. (yonik) * SOLR-1665: Add debug component options for timings, results and query info only (gsingers, hossman, yonik) * SOLR-2112: Solrj API now supports streaming results. (ryan) * SOLR-792: Adding PivotFacetComponent for Hierarchical faceting (ehatcher, Jeremy Hinegardner, Thibaut Lassalle, ryan) * LUCENE-2507, SOLR-2571, SOLR-2576: Added DirectSolrSpellChecker, which uses Lucene's DirectSpellChecker to retrieve correction candidates directly from the term dictionary using levenshtein automata. (James Dyer, rmuir) * SOLR-1873, SOLR-2358: SolrCloud - added shared/central config and core/shard management via zookeeper, built-in load balancing, and distributed indexing. (Jamie Johnson, Sami Siren, Ted Dunning, yonik, Mark Miller) Additional Work: - SOLR-2324: SolrCloud solr.xml parameters are not persisted by CoreContainer. (Massimo Schiavon, Mark Miller) - SOLR-2287: Allow users to query by multiple, compatible collections with SolrCloud. (Soheb Mahmood, Alex Cowell, Mark Miller) - SOLR-2622: ShowFileRequestHandler does not work in SolrCloud mode. (Stefan Matheis, Mark Miller) - SOLR-3108: Error in SolrCloud's replica lookup code when replica's are hosted in same Solr instance. (Bruno Dumon, Sami Siren, Mark Miller) - SOLR-3080: Remove shard info from zookeeper when SolrCore is explicitly unloaded. (yonik, Mark Miller, siren) - SOLR-3437: Recovery issues a spurious commit to the cluster. (Trym R. Møller via Mark Miller) - SOLR-2822: Skip update processors already run on other nodes (hossman) * SOLR-1566: Transforming documents in the ResponseWriters. This will allow for more complex results in responses and open the door for function queries as results. (ryan with patches from grant, noble, cmale, yonik, Jan Høydahl, Arul Kalaipandian, Luca Cavanna, hossman) - SOLR-2037: Thanks to SOLR-1566, documents boosted by the QueryElevationComponent can be marked as boosted. (gsingers, ryan, yonik) * SOLR-2396: Add CollationField, which is much more efficient than the Solr 3.x CollationKeyFilterFactory, and also supports Locale-sensitive range queries. (rmuir) * SOLR-2338: Add support for using in a schema's fieldType, for customizing scoring on a per-field basis. (hossman, yonik, rmuir) * SOLR-2335: New 'field("...")' function syntax for referring to complex field names (containing whitespace or special characters) in functions. * SOLR-2383: /browse improvements: generalize range and date facet display (Jan Høydahl via yonik) * SOLR-2272: Pseudo-join queries / filters. Examples: - To restrict to the set of parents with at least one blue-eyed child: fq={!join from=parent to=name}eyes:blue - To restrict to the set of children with at least one blue-eyed parent: fq={!join from=name to=parent}eyes:blue (yonik) * SOLR-1942: Added the ability to select postings format per fieldType in schema.xml as well as support custom Codecs in solrconfig.xml. (simonw via rmuir) * SOLR-2136: Boolean type added to function queries, along with new functions exists(), if(), and(), or(), xor(), not(), def(), and true and false constants. (yonik) * SOLR-2491: Add support for using spellcheck collation in conjunction with grouping. Note that the number of hits returned for collations is the number of ungrouped hits. (James Dyer via rmuir) * SOLR-1298: Return FunctionQuery as pseudo field. The solr 'fl' param now supports functions. For example: fl=id,sum(x,y) -- NOTE: only functions with fast random access are recommended. (yonik, ryan) * SOLR-705: Optionally return shard info with each document in distributed search. Use fl=id,[shard] to return the shard url. (ryan) * SOLR-2417: Add explain info directly to return documents using ?fl=id,[explain] (ryan) * SOLR-2533: Converted ValueSource.ValueSourceSortField over to new rewriteable Lucene SortFields. ValueSourceSortField instances must be rewritten before they can be used. This is done by SolrIndexSearcher when necessary. (Chris Male). * SOLR-2193, SOLR-2565: You may now specify a 'soft' commit when committing. This will use Lucene's NRT feature to avoid guaranteeing documents are on stable storage in exchange for faster reopen times. There is also a new 'soft' autocommit tracker that can be configured. (Mark Miller, Robert Muir) * SOLR-2399: Updated Solr Admin interface. New look and feel with per core administration and many new options. (Stefan Matheis via ryan) * SOLR-1032: CSV handler now supports "literal.field_name=value" parameters. (Simon Rosenthal, ehatcher) * SOLR-2656: realtime-get, efficiently retrieves the latest stored fields for specified documents, even if they are not yet searchable (i.e. without reopening a searcher) (yonik) * SOLR-2703: Added support for Lucene's "surround" query parser. (Simon Rosenthal, ehatcher) * SOLR-2754: Added factories for several ranking algorithms: - BM25SimilarityFactory: Okapi BM25 - DFRSimilarityFactory: Divergence from Randomness models - IBSimilarityFactory: Information-based models - LMDirichletSimilarity: LM with Dirichlet smoothing - LMJelinekMercerSimilarity: LM with Jelinek-Mercer smoothing (David Mark Nemeskey, Robert Muir) * SOLR-2134 Trie* fields should support sortMissingLast=true, and deprecate Sortable* Field Types (Ryan McKinley, Mike McCandless, Uwe Schindler, Erick Erickson) * SOLR-2438 added MultiTermAwareComponent to the various classes to allow automatic lowercasing for multiterm queries (wildcards, regex, prefix, range, etc). You can now optionally specify a "multiterm" analyzer in our schema.xml, but Solr should "do the right thing" if you don't specify (Pete Sturge Erick Erickson, Mentoring from Seeley and Muir) * SOLR-2481: Add support for commitWithin in DataImportHandler (Sami Siren via yonik) * SOLR-2992: Add support for IndexWriter.prepareCommit() via prepareCommit=true on update URLs. (yonik) * SOLR-2906: Added LFU cache options to Solr. (Shawn Heisey via Erick Erickson) * SOLR-3069: Ability to add openSearcher=false to not open a searcher when doing a hard commit. commitWithin now only invokes a softCommit. (yonik) * SOLR-2802: New FieldMutatingUpdateProcessor and Factory to simplify the development of UpdateProcessors that modify field values of documents as they are indexed. Also includes several useful new implementations: - RemoveBlankFieldUpdateProcessorFactory - TrimFieldUpdateProcessorFactory - HTMLStripFieldUpdateProcessorFactory - RegexReplaceProcessorFactory - FieldLengthUpdateProcessorFactory - ConcatFieldUpdateProcessorFactory - FirstFieldValueUpdateProcessorFactory - LastFieldValueUpdateProcessorFactory - MinFieldValueUpdateProcessorFactory - MaxFieldValueUpdateProcessorFactory - TruncateFieldUpdateProcessorFactory - IgnoreFieldUpdateProcessorFactory (hossman, janhoy) * SOLR-3120: Optional post filtering for spatial queries bbox and geofilt for LatLonType. (yonik) * SOLR-2459: Expose LogLevel selection with a RequestHandler rather then servlet (Stefan Matheis, Upayavira, ryan) * SOLR-3134: Include shard info in distributed response when shards.info=true (Russell Black, ryan) * SOLR-2898: Support grouped faceting. (Martijn van Groningen) Additional Work: - SOLR-3406: Extended grouped faceting support to facet.query and facet.range parameters. (David Boychuck, Martijn van Groningen) * SOLR-2949: QueryElevationComponent is now supported with distributed search. (Mark Miller, yonik) * SOLR-3221: Added the ability to directly configure aspects of the concurrency and thread-pooling used within distributed search in solr. This allows for finer grained controlled and can be tuned by end users to target their own specific requirements. This builds on the work of the HttpCommComponent and uses the same configuration block to configure the thread pool. The default configuration has the same behaviour as solr 3.5, favouring throughput over latency. More information can be found on the wiki (http://wiki.apache.org/solr/SolrConfigXml) (Greg Bowyer) * SOLR-3278: Negative boost support to the Extended Dismax Query Parser Boost Query (bq). (James Dyer) * SOLR-3255: OpenExchangeRates.Org Exchange Rate Provider for CurrencyField (janhoy) * SOLR-3358: Logging events are captured and available from the /admin/logging request handler. (ryan) * SOLR-1535: PreAnalyzedField type provides a functionality to index (and optionally store) field content that was already processed and split into tokens using some external processing chain. Serialization format is pluggable, and defaults to JSON. (ab) * SOLR-3363: Consolidated Exceptions in Analysis Factories so they only throw InitializationExceptions (Chris Male) * SOLR-2690: New support for a "TZ" request param which overrides the TimeZone used when rounding Dates in DateMath expressions for the entire request (all date range queries and date faceting is affected). The default TZ is still UTC. (David Schlotfeldt, hossman) * SOLR-3402: Analysis Factories are now configured with their Lucene Version throw setLuceneMatchVersion, rather than through the Map passed to init. Parsing and simple error checking for the Version is now done inside the code that creates the Analysis Factories. (Chris Male) * SOLR-3178: Optimistic locking. If a _version_ is provided with an update that does not match the version in the index, an HTTP 409 error (Conflict) will result. (Per Steffensen, yonik) * SOLR-139: Updateable documents. JSON Example: {"id":"mydoc", "f1":{"set":10}, "f2":{"add":20}} will result in field "f1" being set to 10, "f2" having an additional value of 20 added, and all other existing fields unchanged. All source fields must be stored for this feature to work correctly. (Ryan McKinley, Erik Hatcher, yonik) * SOLR-2857: Support XML,CSV,JSON, and javabin in a single RequestHandler and choose the correct ContentStreamLoader based on Content-Type header. This also deprecates the existing [Xml,JSON,CSV,Binary,Xslt]UpdateRequestHandler. (ryan) * SOLR-2585: Context-Sensitive Spelling Suggestions & Collations. This adds support for the "spellcheck.alternativeTermCount" & "spellcheck.maxResultsForSuggest" parameters, letting users receive suggestions even when all the queried terms exist in the dictionary. This differs from "spellcheck.onlyMorePopular" in that the suggestions need not consist entirely of terms with a greater document frequency than the queried terms. (James Dyer) * SOLR-2058: Edismax query parser to allow "phrase slop" to be specified per-field on the pf/pf2/pf3 parameters using optional "FieldName~slop^boost" syntax. The prior "FieldName^boost" syntax is still accepted. In such cases the value on the "ps" parameter serves as the default slop. (Ron Mayer via James Dyer) * SOLR-3495: New UpdateProcessors have been added to create default values for configured fields. These works similarly to the option in schema.xml, but are applied in the UpdateProcessorChain, so they may be used prior to other UpdateProcessors, or to generate a uniqueKey field value when using the DistributedUpdateProcessor (ie: SolrCloud) TimestampUpdateProcessorFactory UUIDUpdateProcessorFactory DefaultValueUpdateProcessorFactory (hossman) * SOLR-2993: Add WordBreakSolrSpellChecker to offer suggestions by combining adjacent query terms and/or breaking terms into multiple words. This spellchecker can be configured with a traditional checker (ie: DirectSolrSpellChecker). The results are combined and collations can contain a mix of corrections from both spellcheckers. (James Dyer) * SOLR-3508: Simplify JSON update format for deletes as well as allow version specification for optimistic locking. Examples: - {"delete":"myid"} - {"delete":["id1","id2","id3"]} - {"delete":{"id":"myid", "_version_":123456789}} (yonik) * SOLR-3211: Allow parameter overrides in conjunction with "spellcheck.maxCollationTries". To do so, use parameters starting with "spellcheck.collateParam." For instance, to override the "mm" parameter, specify "spellcheck.collateParam.mm". This is helpful in cases where testing spellcheck collations for result counts should use different parameters from the main query (James Dyer) * SOLR-2599: CloneFieldUpdateProcessorFactory provides similar functionality to schema.xml's declaration but as an update processor that can be combined with other processors in any order. (Jan Høydahl & hossman) * SOLR-3351: eDismax: ps2 and ps3 params (janhoy) * SOLR-3542: Add WeightedFragListBuilder for FVH and set it to default fragListBuilder in example solrconfig.xml. (Sebastian Lutze, koji) * SOLR-2396: Add ICUCollationField to contrib/analysis-extras, which is much more efficient than the Solr 3.x ICUCollationKeyFilterFactory, and also supports Locale-sensitive range queries. (rmuir) Optimizations ---------------------- * SOLR-1875: Per-segment field faceting for single valued string fields. Enable with facet.method=fcs, control the number of threads used with the "threads" local param on the facet.field param. This algorithm will only be faster in the presence of rapid index changes. (yonik) * SOLR-1904: When facet.enum.cache.minDf > 0 and the base doc set is a SortedIntSet, convert to HashDocSet for better performance. (yonik) * SOLR-2092: Speed up single-valued and multi-valued "fc" faceting. Typical improvement is 5%, but can be much greater (up to 10x faster) when facet.offset is very large (deep paging). (yonik) * SOLR-2193, SOLR-2565: The default Solr update handler has been improved so that it uses fewer locks, keeps the IndexWriter open rather than closing it on each commit (ie commits no longer wait for background merges to complete), works with SolrCore to provide faster 'soft' commits, and has an improved API that requires less instanceof special casing. (Mark Miller, Robert Muir) Additional Work: - SOLR-2697: commit and autocommit operations don't reset DirectUpdateHandler2.numDocsPending stats attribute. (Alexey Serba, Mark Miller) * SOLR-2950: The QueryElevationComponent now avoids using the FieldCache and looking up every document id (gsingers, yonik) Bug Fixes ---------------------- * SOLR-3139: Make ConcurrentUpdateSolrServer send UpdateRequest.getParams() as HTTP request params (siren) * SOLR-3165: Cannot use DIH in Solrcloud + Zookeeper (Alexey Serba, Mark Miller, siren) * SOLR-3068: Occasional NPE in ThreadDumpHandler (siren) * SOLR-2762: FSTLookup could return duplicate results or one results less than requested. (David Smiley, Dawid Weiss) * SOLR-2741: Bugs in facet range display in trunk (janhoy) * SOLR-1908: Fixed SignatureUpdateProcessor to fail to initialize on invalid config. Specifically: a signatureField that does not exist, or overwriteDupes=true with a signatureField that is not indexed. (hossman) * SOLR-1824: IndexSchema will now fail to initialize if there is a problem initializing one of the fields or field types. (hossman) * SOLR-1928: TermsComponent didn't correctly break ties for non-text fields sorted by count. (yonik) * SOLR-2107: MoreLikeThisHandler doesn't work with alternate qparsers. (yonik) * SOLR-2108: Fixed false positives when using wildcard queries on fields with reversed wildcard support. For example, a query of *zemog* would match documents that contain 'gomez'. (Landon Kuhn via Robert Muir) * SOLR-1962: SolrCore#initIndex should not use a mix of indexPath and newIndexPath (Mark Miller) * SOLR-2275: fix DisMax 'mm' parsing to be tolerant of whitespace (Erick Erickson via hossman) * SOLR-2193, SOLR-2565, SOLR-2651: SolrCores now properly share IndexWriters across SolrCore reloads. (Mark Miller, Robert Muir) Additional Work: - SOLR-2705: On reload, IndexWriterProvider holds onto the initial SolrCore it was created with. (Yury Kats, Mark Miller) * SOLR-2682: Remove addException() in SimpleFacet. FacetComponent no longer catches and embeds exceptions occurred during facet processing, it throws HTTP 400 or 500 exceptions instead. (koji) * SOLR-2654: Directorys used by a SolrCore are now closed when they are no longer used. (Mark Miller) * SOLR-2854: Now load URL content stream data (via stream.url) when called for during request handling, rather than loading URL content streams automatically regardless of use. (David Smiley and Ryan McKinley via ehatcher) * SOLR-2829: Fix problem with false-positives due to incorrect equals methods. (Yonik Seeley, Hossman, Erick Erickson. Marc Tinnemeyer caught the bug) * SOLR-2848: Removed 'instanceof AbstractLuceneSpellChecker' hacks from distributed spellchecking code, and added a merge() method to SolrSpellChecker instead. Previously if you extended SolrSpellChecker your spellchecker would not work in distributed fashion. (James Dyer via rmuir) * SOLR-2509: StringIndexOutOfBoundsException in the spellchecker collate when the term contains a hyphen. (Thomas Gambier caught the bug, Steffen Godskesen did the patch, via Erick Erickson) * SOLR-1730: Made it clearer when a core failed to load as well as better logging when the QueryElevationComponent fails to properly initialize (gsingers) * SOLR-1520: QueryElevationComponent now supports non-string ids (gsingers) * SOLR-3037: When using binary format in solrj the codec screws up parameters (Sami Siren, Jörg Maier via yonik) * SOLR-3062: A join in the main query was not respecting any filters pushed down to it via acceptDocs since LUCENE-1536. (Mike Hugo, yonik) * SOLR-3214: If you use multiple fl entries rather than a comma separated list, all but the first entry can be ignored if you are using distributed search. (Tomás Fernández Löbbe via Mark Miller) * SOLR-3352: eDismax: pf2 should kick in for a query with 2 terms (janhoy) * SOLR-3361: ReplicationHandler "maxNumberOfBackups" doesn't work if backups are triggered on commit (James Dyer, Tomás Fernández Löbbe) * SOLR-2605: fixed tracking of the 'defaultCoreName' in CoreContainer so that CoreAdminHandler could return consistent information regardless of whether there is a a default core name or not. (steffkes, hossman) * SOLR-3370: fixed CSVResponseWriter to respect globs in the 'fl' param (Keith Fligg via hossman) * SOLR-3436: Group count incorrect when not all shards are queried in the second pass. (Francois Perron, Martijn van Groningen) * SOLR-3454: Exception when using result grouping with main=true and using wt=javabin. (Ludovic Boutros, Martijn van Groningen) * SOLR-3446: Better errors when PatternTokenizerFactory is configured with an invalid pattern, and include the 'name' whenever possible in plugin init error messages. (hossman) * LUCENE-4075: Cleaner path usage in TestXPathEntityProcessor (Greg Bowyer via hossman) * SOLR-2923: IllegalArgumentException when using useFilterForSortedQuery on an empty index. (Adrien Grand via Mark Miller) * SOLR-2352: Fixed TermVectorComponent so that it will not fail if the fl param contains globs or psuedo-fields (hossman) * SOLR-3541: add missing solrj dependencies to binary packages. (Thijs Vonk via siren) * SOLR-3522: fixed parsing of the 'literal()' function (hossman) * SOLR-3548: Fixed a bug in the cachability of queries using the {!join} parser or the strdist() function, as well as some minor improvements to the hashCode implementation of {!bbox} and {!geofilt} queries. (hossman) * SOLR-3470: contrib/clustering: custom Carrot2 tokenizer and stemmer factories are respected now (Stanislaw Osinski, Dawid Weiss) * SOLR-3430: Added a new DIH test against a real SQL database. Fixed problems revealed by this new test related to the expanded cache support added to 3.6/SOLR-2382 (James Dyer) * SOLR-1958: When using the MailEntityProcessor, import would fail if fetchMailsSince was not specified. (Max Lynch via James Dyer) * SOLR-4289: Admin UI - JVM memory bar - dark grey "used" width is too small (steffkes, elyograg) Other Changes ---------------------- * SOLR-1846: Eliminate support for the abortOnConfigurationError option. It has never worked very well, and in recent versions of Solr hasn't worked at all. (hossman) * SOLR-1889: The default logic for the 'mm' param of DismaxQParser and ExtendedDismaxQParser has been changed to be determined based on the effective value of the 'q.op' param (hossman) * SOLR-1946: Misc improvements to the SystemInfoHandler: /admin/system (hossman) * SOLR-2289: Tweak spatial coords for example docs so they are a bit more spread out (Erick Erickson via hossman) * SOLR-2288: Small tweaks to eliminate compiler warnings. primarily using Generics where applicable in method/object declarations, and adding @SuppressWarnings("unchecked") when appropriate (hossman) * SOLR-2375: Suggester Lookup implementations now store trie data and load it back on init. This means that large tries don't have to be rebuilt on every commit or core reload. (ab) * SOLR-2413: Support for returning multi-valued fields w/o tag in the XMLResponseWriter was removed. XMLResponseWriter only no longer work with values less then 2.2 (ryan) * SOLR-2423: FieldType argument changed from String to Object Conversion from SolrInputDocument > Object > Fieldable is now managed by FieldType rather then DocumentBuilder. (ryan) * SOLR-2461: QuerySenderListener and AbstractSolrEventListener are now public (hossman) * LUCENE-2995: Moved some spellchecker and suggest APIs to modules/suggest: HighFrequencyDictionary, SortedIterator, TermFreqIterator, and the suggester APIs and implementations. (rmuir) * SOLR-2576: Remove deprecated SpellingResult.add(Token, int). (James Dyer via rmuir) * LUCENE-3232: Moved MutableValue classes to new 'common' module. (Chris Male) * LUCENE-2883: FunctionQuery, DocValues (and its impls), ValueSource (and its impls) and BoostedQuery have been consolidated into the queries module. They can now be found at o.a.l.queries.function. * SOLR-2027: FacetField.getValues() now returns an empty list if there are no values, instead of null (Chris Male) * SOLR-1825: SolrQuery.addFacetQuery now enables facets automatically, like addFacetField (Chris Male) * SOLR-2663: FieldTypePluginLoader has been refactored out of IndexSchema and made public. (hossman) * SOLR-2331,SOLR-2691: Refactor CoreContainer's SolrXML serialization code and improve testing (Yury Kats, hossman, Mark Miller) * SOLR-2698: Enhance CoreAdmin STATUS command to return index size. (Yury Kats, hossman, Mark Miller) * SOLR-2654: The same Directory instance is now always used across a SolrCore so that it's easier to add other DirectoryFactory's without static caching hacks. (Mark Miller) * LUCENE-3286: 'luke' ant target has been disabled due to incompatibilities with XML queryparser location (Chris Male) * SOLR-1897: The data dir from the core descriptor should override the data dir from the solrconfig.xml rather than the other way round. (Mark Miller) * SOLR-2756: Maven configuration: Excluded transitive stax:stax-api dependency from org.codehaus.woodstox:wstx-asl dependency. (David Smiley via Steve Rowe) * SOLR-2588: Moved VelocityResponseWriter back to contrib module in order to remove it as a mandatory core dependency. (ehatcher) * SOLR-2862: More explicit lexical resources location logged if Carrot2 clustering extension is used. Fixed solr. impl. of IResource and IResourceLookup. (Dawid Weiss) * SOLR-1123: Changed JSONResponseWriter to now use application/json as its Content-Type by default. However the Content-Type can be overwritten and is set to text/plain in the example configuration. (Uri Boness, Chris Male) * SOLR-2607: Removed deprecated client/ruby directory, which included solr-ruby and flare. (ehatcher) * SOLR-3032: logOnce from SolrException logOnce and all the supporting structure is gone. abortOnConfigurationError is also gone as it is no longer referenced. Errors should be caught and logged at the top-most level or logged and NOT propagated up the chain. (Erick Erickson) * SOLR-2105: Remove support for deprecated "update.processor" (since 3.2), in favor of "update.chain" (janhoy) * SOLR-3005: Default QueryResponseWriters are now initialized via init() with an empty NamedList. (Gasol Wu, Chris Male) * SOLR-2607: Removed obsolete client/ folder (ehatcher, Eric Pugh, janhoy) * SOLR-3202, SOLR-3244: Dropping Support for JSP. New Admin UI is all client side (ryan, Aliaksandr Zhuhrou, Uwe Schindler) * SOLR-3159: Upgrade example and tests to run with Jetty 8 (ryan) * SOLR-3254: Upgrade Solr to Tika 1.1 (janhoy) * SOLR-3329: Dropped getSourceID() from SolrInfoMBean and using getClass().getPackage().getSpecificationVersion() for Version. (ryan) * SOLR-3302: Upgraded SLF4j to version 1.6.4 (hossman) * SOLR-3322: Add more context to IndexReaderFactory.newReader (ab) * SOLR-3343: Moved FastWriter, FileUtils, RegexFileFilter, RTimer and SystemIdResolver from org.apache.solr.common to org.apache.solr.util (Chris Male) * SOLR-3357: ResourceLoader.newInstance now accepts a Class representation of the expected instance type (Chris Male) * SOLR-3388: HTTP caching is now disabled by default for RequestUpdateHandlers. (ryan) * SOLR-3309: web.xml now specifies metadata-complete=true (which requires Servlet 2.5) to prevent servlet containers from scanning class annotations on startup. This allows for faster startup times on some servlet containers. (Bill Bell, hossman) * SOLR-1893: Refactored some common code from LRUCache and FastLRUCache into SolrCacheBase (Tomás Fernández Löbbe via hossman) * SOLR-3403: Deprecated Analysis Factories now log their own deprecation messages. No logging support is provided by Factory parent classes. (Chris Male) * SOLR-1258: PingRequestHandler is now directly configured with a "healthcheckFile" instead of looking for the legacy syntax. Filenames specified as relative paths have been fixed so that they are resolved against the data dir instead of the CWD of the java process. (hossman) * SOLR-3083: JMX beans now report Numbers as numeric values rather then String (Tagged Siteops, Greg Bowyer via ryan) * SOLR-2796: Due to low level changes to support SolrCloud, the uniqueKey field can no longer be populated via or in the schema.xml. * SOLR-3534: The Dismax and eDismax query parsers will fall back on the 'df' parameter when 'qf' is absent. And if neither is present nor the schema default search field then an exception will be thrown now. (dsmiley) * SOLR-3262: The "threads" feature of DIH is removed (deprecated in Solr 3.6) (James Dyer) * SOLR-3422: Refactored DIH internal data classes. All entities in data-config.xml must have a name (James Dyer) Documentation ---------------------- * SOLR-2232: Improved README info on solr.solr.home in examples (Eric Pugh and hossman) ================== 3.6.2 ================== Bug Fixes ---------------------- * SOLR-3790: ConcurrentModificationException could be thrown when using hl.fl=*. (yonik, koji) * SOLR-3589: Edismax parser does not honor mm parameter if analyzer splits a token. (Tom Burton-West, Robert Muir) ================== 3.6.1 ================== More information about this release, including any errata related to the release notes, upgrade instructions, or other changes may be found online at: https://wiki.apache.org/solr/Solr3.6.1 Bug Fixes * LUCENE-3969: Throw IAE on bad arguments that could cause confusing errors in PatternTokenizer. CommonGrams populates PositionLengthAttribute correctly. (Uwe Schindler, Mike McCandless, Robert Muir) * SOLR-3361: ReplicationHandler "maxNumberOfBackups" doesn't work if backups are triggered on commit (James Dyer, Tomás Fernández Löbbe) * SOLR-3375: Fix charset problems with HttpSolrServer (Roger Håkansson, yonik, siren) * SOLR-3436: Group count incorrect when not all shards are queried in the second pass. (Francois Perron, Martijn van Groningen) * SOLR-3454: Exception when using result grouping with main=true and using wt=javabin. (Ludovic Boutros, Martijn van Groningen) * SOLR-3489: Config file replication less error prone (Jochen Just via janhoy) * SOLR-3477: SOLR does not start up when no cores are defined (Tomás Fernández Löbbe via tommaso) * SOLR-3470: contrib/clustering: custom Carrot2 tokenizer and stemmer factories are respected now (Stanislaw Osinski, Dawid Weiss) * SOLR-3360: More DIH bug fixes for the deprecated "threads" parameter. (Mikhail Khludnev, Claudio R, via James Dyer) * SOLR-3430: Added a new DIH test against a real SQL database. Fixed problems revealed by this new test related to the expanded cache support added to 3.6/SOLR-2382 (James Dyer) * SOLR-3336: SolrEntityProcessor substitutes most variables at query time. (Michael Kroh, Lance Norskog, via Martijn van Groningen) ================== 3.6.0 ================== More information about this release, including any errata related to the release notes, upgrade instructions, or other changes may be found online at: https://wiki.apache.org/solr/Solr3.6 Upgrading from Solr 3.5 ---------------------- * SOLR-2983: As a consequence of moving the code which sets a MergePolicy from SolrIndexWriter to SolrIndexConfig, (custom) MergePolicies should now have an empty constructor; thus an IndexWriter should not be passed as constructor parameter but instead set using the setIndexWriter() method. * As doGet() methods in SimplePostTool was changed to static, the client applications of this class need to be recompiled. * In Solr version 3.5 and earlier, HTMLStripCharFilter had known bugs in the character offsets it provided, triggering e.g. exceptions in highlighting. HTMLStripCharFilter has been re-implemented, addressing this and other issues. See the entry for LUCENE-3690 in the Bug Fixes section below for a detailed list of changes. For people who depend on the behavior of HTMLStripCharFilter in Solr version 3.5 and earlier: the old implementation (bugs and all) is preserved as LegacyHTMLStripCharFilter. * As of Solr 3.6, the and sections of solrconfig.xml are deprecated and replaced with a new section. Read more in SOLR-1052 below. * SOLR-3040: The DIH's admin UI (dataimport.jsp) now requires DIH request handlers to start with a '/'. (dsmiley) * SOLR-3161: is now the default. An existing config will probably work as-is because handleSelect was explicitly enabled in default configs. HandleSelect makes /select work as well as enables the 'qt' parameter. Instead, consider explicitly configuring /select as is done in the example solrconfig.xml, and register your other search handlers with a leading '/' which is a recommended practice. (David Smiley, Erik Hatcher) * SOLR-3161: Don't use the 'qt' parameter with a leading '/'. It probably won't work in 4.0 and it's now limited in 3.6 to SearchHandler subclasses that aren't lazy-loaded. * SOLR-2724: Specifying and in schema.xml is now considered deprecated. Instead you are encouraged to specify these via the "df" and "q.op" parameters in your request handler definition. (David Smiley) * Bugs found and fixed in the SignatureUpdateProcessor that previously caused some documents to produce the same signature even when the configured fields contained distinct (non-String) values. Users of SignatureUpdateProcessor are strongly advised that they should re-index as document signatures may have now changed. (see SOLR-3200 & SOLR-3226 for details) New Features ---------------------- * SOLR-2020: Add Java client that uses Apache Http Components http client (4.x). (Chantal Ackermann, Ryan McKinley, Yonik Seeley, siren) * SOLR-2854: Now load URL content stream data (via stream.url) when called for during request handling, rather than loading URL content streams automatically regardless of use. (David Smiley and Ryan McKinley via ehatcher) * SOLR-2904: BinaryUpdateRequestHandler should be able to accept multiple update requests from a stream (shalin) * SOLR-1565: StreamingUpdateSolrServer supports RequestWriter API and therefore, javabin update format (shalin) * SOLR-2438 added MultiTermAwareComponent to the various classes to allow automatic lowercasing for multiterm queries (wildcards, regex, prefix, range, etc). You can now optionally specify a "multiterm" analyzer in our schema.xml, but Solr should "do the right thing" if you don't specify (Pete Sturge Erick Erickson, Mentoring from Seeley and Muir) * SOLR-2919: Added support for localized range queries when the analysis chain uses CollationKeyFilter or ICUCollationKeyFilter. (Michael Sokolov, rmuir) * SOLR-2982: Added BeiderMorseFilterFactory for Beider-Morse (BMPM) phonetic encoder. Upgrades commons-codec to version 1.6 (Brooke Schreier Ganz, rmuir) * SOLR-1843: A new "rootName" attribute is now available when configuring in solrconfig.xml. If this attribute is set, Solr will use it as the root name for all MBeans Solr exposes via JMX. The default root name is "solr" followed by the core name. (Constantijn Visinescu, hossman) * SOLR-2906: Added LFU cache options to Solr. (Shawn Heisey via Erick Erickson) * SOLR-3036: Ability to specify overwrite=false on the URL for XML updates. (Sami Siren via yonik) * SOLR-2603: Add the encoding function for alternate fields in highlighting. (Massimo Schiavon, koji) * SOLR-1729: Evaluation of NOW for date math is done only once per request for consistency, and is also propagated to shards in distributed search. Adding a parameter NOW= to the request will override the current time. (Peter Sturge, yonik, Simon Willnauer) * SOLR-1709: Distributed support for Date and Numeric Range Faceting (Peter Sturge, David Smiley, hossman, Simon Willnauer) * SOLR-3054, LUCENE-3671: Add TypeTokenFilterFactory that creates TypeTokenFilter that filters tokens based on their TypeAttribute. (Tommaso Teofili via Uwe Schindler) * LUCENE-3305, SOLR-3056: Added Kuromoji morphological analyzer for Japanese. See the 'text_ja' fieldtype in the example to get started. (Christian Moen, Masaru Hasegawa via Robert Muir) * SOLR-1860: StopFilterFactory, CommonGramsFilterFactory, and CommonGramsQueryFilterFactory can optionally read stopwords in Snowball format (specify format="snowball"). (Robert Muir) * SOLR-3105: ElisionFilterFactory optionally allows the parameter ignoreCase (default=false). (Robert Muir) * LUCENE-3714: Add WFSTLookupFactory, a suggester that uses a weighted FST for more fine-grained suggestions. (Mike McCandless, Dawid Weiss, Robert Muir) * SOLR-3143: Add SuggestQueryConverter, a QueryConverter intended for auto-suggesters. (Robert Muir) * SOLR-3033: ReplicationHandler's backup command now supports a 'maxNumberOfBackups' init param that can be used to delete all but the most recent N backups. (Torsten Krah, James Dyer) * SOLR-2202: Currency FieldType, whith support for currencies and exchange rates (Greg Fodor & Andrew Morrison via janhoy, rmuir, Uwe Schindler) * SOLR-3026: eDismax: Locking down which fields can be explicitly queried (user fields aka uf) (janhoy, hossmann, Tomás Fernández Löbbe) * SOLR-2826: URLClassify Update Processor (janhoy) * SOLR-2764: Create a NorwegianLightStemmer and NorwegianMinimalStemmer (janhoy) * SOLR-3221: Added the ability to directly configure aspects of the concurrency and thread-pooling used within distributed search in solr. This allows for finer grained controlled and can be tuned by end users to target their own specific requirements. This builds on the work of the HttpCommComponent and uses the same configuration block to configure the thread pool. The default configuration has the same behaviour as solr 3.5, favouring throughput over latency. More information can be found on the wiki (http://wiki.apache.org/solr/SolrConfigXml) (Greg Bowyer) * SOLR-2001: The query component will substitute an empty query that matches no documents if the query parser returns null. This also prevents an exception from being thrown by the default parser if "q" is missing. (yonik) - SOLR-435: if q is "" then it's also acceptable. (dsmiley, hoss) * SOLR-2919: Added parametric tailoring options to ICUCollationKeyFilterFactory. These can be used to customize range query/sort behavior, for example to support numeric collation, ignore punctuation/whitespace, ignore accents but not case, control whether upper/lowercase values are sorted first, etc. (rmuir) * SOLR-2346: Add a chance to set content encoding explicitly via content type of stream for extracting request handler. This is convenient when Tika's auto detector cannot detect encoding, especially the text file is too short to detect encoding. (koji) * SOLR-1499: Added SolrEntityProcessor that imports data from another Solr core or instance based on a specified query. (Lance Norskog, Erik Hatcher, Pulkit Singhal, Ahmet Arslan, Luca Cavanna, Martijn van Groningen) * SOLR-3190: Minor improvements to SolrEntityProcessor. Add more consistency between solr parameters and parameters used in SolrEntityProcessor and ability to specify a custom HttpClient instance. (Luca Cavanna via Martijn van Groningen) * SOLR-2382: Added pluggable cache support to DIH so that any Entity can be made cache-able by adding the "cacheImpl" parameter. Include "SortedMapBackedCache" to provide in-memory caching (as previously this was the only option when using CachedSqlEntityProcessor). Users can provide their own implementations of DIHCache for other caching strategies. Deprecate CachedSqlEntityProcessor in favor of specifing "cacheImpl" with SqlEntityProcessor. Make SolrWriter implement DIHWriter and allow the possibility of pluggable Writers (DIH writing to something other than Solr). (James Dyer, Noble Paul) Optimizations ---------------------- * SOLR-1931: Speedup for LukeRequestHandler and admin/schema browser. New parameter reportDocCount defaults to 'false'. Old behavior still possible by specifying this as 'true' (Erick Erickson) * SOLR-3012: Move System.getProperty("type") in postData() to main() and add type argument so that the client applications of SimplePostTool can set content type via method argument. (koji) * SOLR-2888: FSTSuggester refactoring: internal storage is now UTF-8, external sorting (on disk) prevents OOMs even with large data sets (the bottleneck is now FST construction), code cleanups and API cleanups. (Dawid Weiss, Robert Muir) Bug Fixes ---------------------- * SOLR-3187 SystemInfoHandler leaks filehandles (siren) * LUCENE-3820: Fixed invalid position indexes by reimplementing PatternReplaceCharFilter. This change also drops real support for boundary characters -- all input is prebuffered for pattern matching. (Dawid Weiss) * SOLR-3068: Fixed NPE in ThreadDumpHandler (siren) * SOLR-2912: Fixed File descriptor leak in ShowFileRequestHandler (Michael Ryan, shalin) * SOLR-2819: Improved speed of parsing hex entities in HTMLStripCharFilter (Bernhard Berger, hossman) * SOLR-2509: StringIndexOutOfBoundsException in the spellchecker collate when the term contains a hyphen. (Thomas Gambier caught the bug, Steffen Godskesen did the patch, via Erick Erickson) * SOLR-2955: Fixed IllegalStateException when querying with group.sort=score desc in sharded environment. (Steffen Elberg Godskesen, Martijn van Groningen) * SOLR-2956: Fixed inconsistencies in the flags (and flag key) reported by the LukeRequestHandler (hossman) * SOLR-1730: Made it clearer when a core failed to load as well as better logging when the QueryElevationComponent fails to properly initialize (gsingers) * SOLR-1520: QueryElevationComponent now supports non-string ids (gsingers) * SOLR-3024: Fixed JSONTestUtil.matchObj, in previous releases it was not respecting the 'delta' arg (David Smiley via hossman) * SOLR-2542: Fixed DIH Context variables which were broken for all scopes other then SCOPE_ENTITY (Linbin Chen & Frank Wesemann via hossman) * SOLR-3042: Fixed Maven Jetty plugin configuration. (David Smiley via Steve Rowe) * SOLR-2970: CSV ResponseWriter returns fields defined as stored=false in schema (janhoy) * LUCENE-3690, LUCENE-2208, SOLR-882, SOLR-42: Re-implemented HTMLStripCharFilter as a JFlex-generated scanner and moved it to lucene/contrib/analyzers/common/. See below for a list of bug fixes and other changes. To get the same behavior as HTMLStripCharFilter in Solr version 3.5 and earlier (including the bugs), use LegacyHTMLStripCharFilter, which is the previous implementation. Behavior changes from the previous version: - Known offset bugs are fixed. - The "Mark invalid" exceptions reported in SOLR-1283 are no longer triggered (the bug is still present in LegacyHTMLStripCharFilter). - The character entity "'" is now always properly decoded. - More cases of