lucene/solr
Shalin Shekhar Mangar 179c8f9b48 SOLR-11990: Make it possible to co-locate replicas of multiple collections together in a node.
A collection may be co-located with another collection during collection creation time by specifying a
'withCollection' parameter. It can also be co-located afterwards by using the modify collection API.
The co-location guarantee is enforced regardless of future cluster operations whether they are invoked
manually via the Collection API or automatically by the Autoscaling framework.

Squashed commit of the following:

commit 3827703b38c598f1247c90ab57d3d640ab3a9e21
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date:   Sat Jul 28 11:54:10 2018 +0530

    SOLR-11990: Added change log entry

commit 7977222e07ba47274062cb8d8a69e7956d644000
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date:   Sat Jul 28 11:52:17 2018 +0530

    SOLR-11990: Added change log entry

commit 1857075fdb9d535b6149ad4369fed8b64b0c01f6
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date:   Sat Jul 28 11:49:51 2018 +0530

    SOLR-11990: Added note about co-location guarantees being one way only

commit 8557cbc8a511f21d1fcad99e11ea9d2104d0bef4
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date:   Sat Jul 28 10:43:37 2018 +0530

    SOLR-11990: Remove unused import

commit 864b013fd744edca9b6b84a8a7573fab3c5310d5
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date:   Sat Jul 28 10:21:59 2018 +0530

    SOLR-11990: Fixing compilation issues after merging master

commit dd840a2f7e765ee96c899d4d9ea89b6b67c5ae62
Merge: bb4ffb3 828d281
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date:   Sat Jul 28 10:03:50 2018 +0530

    Merge branch 'master' into jira/solr-11990

    # Conflicts:
    #	solr/solr-ref-guide/src/collections-api.adoc
    #	solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Clause.java
    #	solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Suggestion.java

commit bb4ffb32c4960a2809ac8927e214e1e012204a73
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date:   Fri Jul 27 14:09:44 2018 +0530

    SOLR-11990: Ensure that the suggestion are validated by the policy engine otherwise move to the next candidate replica or the next candidate node

commit a97d45b22f9c232e939f979502c761001be9ae24
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date:   Fri Jul 27 13:22:10 2018 +0530

    SOLR-11990: Autoscaling suggestions for withCollection violations should prefer moving replicas before adding replicas

commit 7b5a84338dfe7335599a5e96aff2d26cb4eeaac6
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date:   Fri Jul 27 12:22:45 2018 +0530

    SOLR-11990: Fix statement about the behavior of the modify collection API when modifying the withCollection parameter

commit 63aec4fe0de7025c16b6ebc47dad1004531ecee1
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date:   Thu Jul 26 07:29:07 2018 +0530

    SOLR-11990: Added new page to the reference guide describing how to colocate collections together including guarantees and limitations

commit 6bfcd0786bb30353de9c26a01ec97ce3191b58f8
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date:   Wed Jul 25 21:42:25 2018 +0530

    SOLR-11990: Added another test which creates two collections which are colocated with two different collections and ensures that create collection and add replica operations work correctly

commit 4cead778f0044b6fb4012b085abf7b60350f495b
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date:   Wed Jul 25 21:07:47 2018 +0530

    SOLR-11990: Stop or start jettys in test setup to ensure that we always have exactly 2 replicas running before a test starts

commit 70dbfd042c2164fcd76d406eeab1518e4d3147fb
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date:   Wed Jul 25 19:19:07 2018 +0530

    SOLR-11990: Added description of the new withCollection parameter in the reference guide

commit 9d8260852b9d667d4d8e026432fd7727b7789393
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date:   Wed Jul 25 19:16:46 2018 +0530

    SOLR-11990: Reset count down latch during test setup

commit ae508165571b1afde54337859b8d5fdbb1d67312
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date:   Wed Jul 25 15:43:54 2018 +0530

    SOLR-11990: Add support for withCollection in simulated create collection API

commit 84f026b8c4cc25edb548430b8f5ad09d2486b3b5
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date:   Tue Jul 24 17:21:33 2018 +0530

    SOLR-11990: Ported the refactoring made in CreateCollectionCmd to the simulated version so that simulation tests are able to create collections correctly

commit defe111c9d31c8e4f0f00b4f2f3c875f5b2fa602
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date:   Tue Jul 24 16:17:52 2018 +0530

    SOLR-11990: Add missing javadoc for return statement

commit 8e47d5bc4545548c5441909c3fcc1a7901b38185
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date:   Tue Jul 24 16:11:45 2018 +0530

    SOLR-11990: Replace usage of forbidden Charsets with StandardCharsets class

commit 2d1b9eb25ea96a3a42c000ae654400ed44c17554
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date:   Tue Jul 24 16:07:36 2018 +0530

    SOLR-11990: Extract ConditionType to an interface VarType along with a WithCollectionVarType implementation

commit 1de2a4f52a59afca28de75bfa5156a3d6567a4f5
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date:   Tue Jul 24 12:53:26 2018 +0530

    SOLR-11990: Pass strict-ness parameter to the ConditionType so that WITH_COLLECTION can choose not to project add replica in strict mode.

    This ensures that add replica or move replica suggesters always choose nodes that already have withCollection replicas first unless there are violations in doing so. Only if the first pass fails to find a suitable replica, do we go to the other nodes in the cluster. This also removes the need for the majority of changes in AddReplicaSuggester and so they've been reverted.

commit 0d616ed9e9bad791548c87086cba7760d724350d
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date:   Tue Jul 24 11:36:34 2018 +0530

    SOLR-11990: Minor changes to formatting and code comments

commit 1228538f934f35f15797d89c2c66f2deb9cddd8c
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date:   Mon Jul 23 14:26:19 2018 +0530

    SOLR-11990: Added a test which simulates a lost node and asserts that move replica suggester moves the replica on the lost node to a node already having the withCollection present

commit 582f1fd98de93ab73c74a1f623749dd031beb381
Author: Noble Paul <noble@apache.org>
Date:   Mon Jul 23 18:35:22 2018 +1000

    SOLR-11990: NPE removing unnecessary System.out.println

commit 501bc6c1d066321b344bbb8b1de3c2ead52f8c49
Author: Noble Paul <noble@apache.org>
Date:   Mon Jul 23 18:31:07 2018 +1000

    SOLR-11990: NPE during class init

commit acbf4a69321e16cff11cc7cf0a1f076fd9ac0037
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date:   Mon Jul 23 13:55:30 2018 +0530

    SOLR-11990: Added asserts on the nodes that should be selected by the add replica suggester

commit 4824933fd6eb7d1773acbff1a1a0c5e670226e0b
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date:   Fri Jul 20 14:30:52 2018 +0530

    SOLR-11990: Added WITH_COLLECTION to global tags. Fixed implementation of addViolatingReplicas and getSuggestions in the clause impl. Added more asserts in testWithCollectionSuggestions.

commit dbadb33211c190026e08d8e3ea587b6f8df8720b
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date:   Fri Jul 20 13:44:36 2018 +0530

    SOLR-11990: Added support for comparing violations, generating suggestions and adding violating replicas

commit ada1f17d5c93a4186260473e4822d2bee1da0e16
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date:   Wed Jul 18 19:14:56 2018 +0530

    SOLR-11990: Fix mock node state provider in TestPolicy to use the right cluster state. Added nocommits to ensure that we return the right suggestions for this feature.

commit ef2d61812e0d96eb2275b3411906d9de57ab835e
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date:   Wed Jul 18 18:39:51 2018 +0530

    SOLR-11990: Add missing node in nodeValues configuration

commit 34841fc01fea4a9f1e6a9f64050e576f2247a72b
Author: Shalin Shekhar Mangar <shalin@apache.org>
Date:   Wed Jul 18 16:32:57 2018 +0530

    SOLR-11990: Make it possible to co-locate replicas of multiple collections together in a node
2018-07-29 07:26:13 +05:30
..
bin SOLR-12008: Settle a location for the log4j2.xml file 2018-07-11 20:57:38 -07:00
bin-test SOLR-12279: Reject invalid 'blockUnknown' values for 'bin/solr auth' 2018-05-07 07:41:13 -04:00
cloud-dev LUCENE-6732: More filetypes to check 2015-08-12 17:43:08 +00:00
contrib SOLR-10243: Apply @AwaitsFix on TestExtractionDateUtil.testParseDate to be fixed by SOLR-12593 2018-07-25 11:10:32 -04:00
core SOLR-11990: Make it possible to co-locate replicas of multiple collections together in a node. 2018-07-29 07:26:13 +05:30
example SOLR-12008: Settle a location for the log4j2.xml file 2018-07-11 20:57:38 -07:00
licenses LUCENE-8420: Upgrade OpenNLP to 1.9.0 2018-07-23 13:55:22 +09:00
server SOLR-12008: Settle a location for the log4j2.xml file 2018-07-11 20:57:38 -07:00
site SOLR-10842: Convert all remaining {{quickstart.html}} links to {{guide/solr-tutorial.html}}; remove all references to quickstart from the build; and version the link to the ref guide's tutorial in Solr's versioned top-level documentation page. 2017-10-04 15:25:20 -04:00
solr-ref-guide SOLR-11990: Make it possible to co-locate replicas of multiple collections together in a node. 2018-07-29 07:26:13 +05:30
solrj SOLR-11990: Make it possible to co-locate replicas of multiple collections together in a node. 2018-07-29 07:26:13 +05:30
test-framework SOLR-12412: Fix test failure 2018-07-17 18:42:29 +07:00
webapp LUCENE-7976: Make TieredMergePolicy respect maxSegmentSizeMB and allow singleton merges of very large segments 2018-06-15 12:33:02 -07:00
.gitignore SOLR-10713: Add common output files to .gitignore 2017-06-05 12:49:50 -05:00
CHANGES.txt SOLR-11990: Make it possible to co-locate replicas of multiple collections together in a node. 2018-07-29 07:26:13 +05:30
LICENSE.txt
NOTICE.txt Copy the latest lucene's NOTICE.txt contents into Solr's NOTICE.txt 2018-06-18 14:42:08 +02:00
README.txt SOLR-11312: Add docs on how to stop Solr to README.txt 2017-12-04 15:28:20 -06:00
build.xml SOLR-12134: hook ref-guide 'bare-bones-html' validation into top level documentation target using relative javadoc URL prefixess that are now validated to point to real files 2018-04-03 16:15:09 -07:00
common-build.xml SOLR-11694: Remove outdated UIMA module 2018-07-07 09:58:57 -04:00

README.txt

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.


Welcome to the Apache Solr project!
-----------------------------------

Solr is the popular, blazing fast open source enterprise search platform
from the Apache Lucene project.

For a complete description of the Solr project, team composition, source
code repositories, and other details, please see the Solr web site at
http://lucene.apache.org/solr


Getting Started
---------------

To start Solr for the first time after installation, simply do:

  bin/solr start

This will launch a standalone Solr server in the background of your shell,
listening on port 8983. Alternatively, you can launch Solr in "cloud" mode,
which allows you to scale out using sharding and replication. To launch Solr
in cloud mode, do:

  bin/solr start -cloud

To see all available options for starting Solr, please do:

  bin/solr start -help

After starting Solr, create either a core or collection depending on whether
Solr is running in standalone (core) or SolrCloud mode (collection) by doing:

  bin/solr create -c <name>

This will create a collection that uses a data-driven schema which tries to guess
the correct field type when you add documents to the index. To see all available
options for creating a new collection, execute:

  bin/solr create -help

After starting Solr, direct your Web browser to the Solr Admin Console at:

  http://localhost:8983/solr/

When finished with your Solr installation, shut it down by executing:

  bin/solr stop -all

The `-p PORT` option can also be used to identify the Solr instance to shutdown,
where more than one Solr is running on the machine.


Solr Examples
---------------

Solr includes a few examples to help you get started. To run a specific example, do:

  bin/solr -e <EXAMPLE> where <EXAMPLE> is one of:

    cloud        : SolrCloud example
    dih          : Data Import Handler (rdbms, mail, atom, tika)
    schemaless   : Schema-less example (schema is inferred from data during indexing)
    techproducts : Kitchen sink example providing comprehensive examples of Solr features

For instance, if you want to run the Solr Data Import Handler example, do:

  bin/solr -e dih


Indexing Documents
---------------

To add documents to the index, use bin/post.  For example:

     bin/post -c <collection_name> example/exampledocs/*.xml

For more information about Solr examples please read...

 * example/README.txt
   For more information about the "Solr Home" and Solr specific configuration
 * https://lucene.apache.org/solr/guide/solr-tutorial.html
   For a Solr tutorial
 * http://lucene.apache.org/solr/resources.html
   For a list of other tutorials and introductory articles.

or linked from "docs/index.html" in a binary distribution.

Also, there are Solr clients for many programming languages, see
   http://wiki.apache.org/solr/IntegratingSolr


Files included in an Apache Solr binary distribution
----------------------------------------------------

server/
  A self-contained Solr instance, complete with a sample
  configuration and documents to index. Please see: bin/solr start -help
  for more information about starting a Solr server.

example/
  Contains example documents and an alternative Solr home
  directory containing examples of how to use the Data Import Handler,
  see example/example-DIH/README.txt for more information.

dist/solr-<component>-XX.jar
  The Apache Solr libraries.  To compile Apache Solr Plugins,
  one or more of these will be required.  The core library is
  required at a minimum. (see http://wiki.apache.org/solr/SolrPlugins
  for more information).

docs/index.html
  A link to the online version of Apache Solr Javadoc API documentation and Tutorial


Instructions for Building Apache Solr from Source
-------------------------------------------------

1. Download the Java SE 8 JDK (Java Development Kit) or later from http://www.oracle.com/java/
   You will need the JDK installed, and the $JAVA_HOME/bin (Windows: %JAVA_HOME%\bin)
   folder included on your command path. To test this, issue a "java -version" command
   from your shell (command prompt) and verify that the Java version is 1.8 or later.

2. Download the Apache Ant binary distribution (1.8.2+) from
   http://ant.apache.org/  You will need Ant installed and the $ANT_HOME/bin (Windows:
   %ANT_HOME%\bin) folder included on your command path. To test this, issue a
   "ant -version" command from your shell (command prompt) and verify that Ant is
   available.

   You will also need to install Apache Ivy binary distribution (2.2.0) from
   http://ant.apache.org/ivy/ and place ivy-2.2.0.jar file in ~/.ant/lib -- if you skip
   this step, the Solr build system will offer to do it for you.

3. Download the Apache Solr distribution, linked from the above web site.
   Unzip the distribution to a folder of your choice, e.g. C:\solr or ~/solr
   Alternately, you can obtain a copy of the latest Apache Solr source code
   directly from the GIT repository:

     http://lucene.apache.org/solr/versioncontrol.html

4. Navigate to the "solr" folder and issue an "ant" command to see the available options
   for building, testing, and packaging Solr.

   NOTE:
   To see Solr in action, you may want to use the "ant server" command to build
   and package Solr into the server directory. See also server/README.txt.


Export control
-------------------------------------------------
This distribution includes cryptographic software.  The country in
which you currently reside may have restrictions on the import,
possession, use, and/or re-export to another country, of
encryption software.  BEFORE using any encryption software, please
check your country's laws, regulations and policies concerning the
import, possession, or use, and re-export of encryption software, to
see if this is permitted.  See <http://www.wassenaar.org/> for more
information.

The U.S. Government Department of Commerce, Bureau of Industry and
Security (BIS), has classified this software as Export Commodity
Control Number (ECCN) 5D002.C.1, which includes information security
software using or performing cryptographic functions with asymmetric
algorithms.  The form and manner of this Apache Software Foundation
distribution makes it eligible for export under the License Exception
ENC Technology Software Unrestricted (TSU) exception (see the BIS
Export Administration Regulations, Section 740.13) for both object
code and source code.

The following provides more details on the included cryptographic
software:
    Apache Solr uses the Apache Tika which uses the Bouncy Castle generic encryption libraries for
    extracting text content and metadata from encrypted PDF files.
    See http://www.bouncycastle.org/ for more details on Bouncy Castle.