From aff647ecfaf5af3bbeb2363b82821c53c5df7f3d Mon Sep 17 00:00:00 2001 From: Cassandra Targett Date: Tue, 5 Sep 2017 09:35:45 -0500 Subject: [PATCH] SOLR-9526: Update Ref Guide for schemaless changes --- solr/solr-ref-guide/src/schemaless-mode.adoc | 62 ++++++++++++++------ 1 file changed, 43 insertions(+), 19 deletions(-) diff --git a/solr/solr-ref-guide/src/schemaless-mode.adoc b/solr/solr-ref-guide/src/schemaless-mode.adoc index 825c294ac6b..f351bd63191 100644 --- a/solr/solr-ref-guide/src/schemaless-mode.adoc +++ b/solr/solr-ref-guide/src/schemaless-mode.adoc @@ -22,7 +22,7 @@ Schemaless Mode is a set of Solr features that, when used together, allow users These Solr features, all controlled via `solrconfig.xml`, are: -. Managed schema: Schema modifications are made at runtime through Solr APIs, which requires the use of `schemaFactory` that supports these changes - see <> for more details. +. Managed schema: Schema modifications are made at runtime through Solr APIs, which requires the use of a `schemaFactory` that supports these changes. See the section <> for more details. . Field value class guessing: Previously unseen fields are run through a cascading set of value-based parsers, which guess the Java class of field values - parsers for Boolean, Integer, Long, Float, Double, and Date are currently available. . Automatic schema field addition, based on field value class(es): Previously unseen fields are added to the schema, based on field value Java classes, which are mapped to schema field types - see <>. @@ -35,7 +35,7 @@ The three features of schemaless mode are pre-configured in the `_default` <> to confirm this: `curl \http://localhost:8983/solr/gettingstarted/schema/fields` will output: @@ -84,19 +84,23 @@ You can configure the `ManagedIndexSchemaFactory` (and control the resource file ---- -=== Define an UpdateRequestProcessorChain +=== Enable Field Class Guessing -The UpdateRequestProcessorChain allows Solr to guess field types, and you can define the default field type classes to use. To start, you should define it as follows (see the javadoc links below for update processor factory documentation): +In Solr, an <> defines a chain of plugins that are applied to documents before or while they are indexed. + +The field guessing aspect of Solr's schemaless mode uses a specially-defined UpdateRequestProcessorChain that allows Solr to guess field types. You can also define the default field type classes to use. + +To start, you should define it as follows (see the javadoc links below for update processor factory documentation): [source,xml] ---- - + [^\w-\.] _ - + @@ -120,11 +124,11 @@ The UpdateRequestProcessorChain allows Solr to guess field types, and you can de yyyy-MM-dd - + - java.lang.String + java.lang.String text_general - + *_str 256 @@ -140,7 +144,7 @@ The UpdateRequestProcessorChain allows Solr to guess field types, and you can de pdates - java.lang.Long + java.lang.Long java.lang.Integer plongs @@ -152,14 +156,26 @@ The UpdateRequestProcessorChain allows Solr to guess field types, and you can de + processor="uuid,remove-blank,field-name-mutating,parse-boolean,parse-long,parse-double,parse-date,add-schema-fields"> ---- -Javadocs for update processor factories mentioned above: +There are many things defined in this chain. Let's step through a few of them. + +<1> First, we're using the FieldNameMutatingUpdateProcessorFactory to lower-case all field names. Note that this and every following `` element include a `name`. These names will be used in the final chain definition at the end of this example. +<2> Next we add several update request processors to parse different field types. Note the ParseDateFieldUpdateProcessorFactory includes a long list of possible date formations that would be parsed into valid Solr dates. If you have a custom date, you could add it to this list (see the link to the Javadocs below to get information on how). +<3> Once the fields have been parsed, we define the field types that will be assigned to those fields. You can modify any of these that you would like to change. +<4> In this definition, if the parsing step decides the incoming data in a field is a string, we will put this into a field in Solr with the field type `text_general`. This field type by default allows Solr to query on this field. +<5> After we've added the `text_general` field, we have also defined a copy field rule that will copy all data from the new `text_general` field to a field with the same name suffixed with `_str`. This is done by Solr's dynamic fields feature. By defining the target of the copy field rule as a dynamic field in this way, you can control the field type used in your schema. The default selection allows Solr to facet, highlight, and sort on these fields. +<6> This is another example of a mapping rule. In this case we define that when either of the `Long` or `Integer` field parsers identify a field, they should both map their fields to the `plongs` field type. +<7> Finally, we add a chain definition that calls the list of plugins. These plugins are each called by the names we gave to them when we defined them. We can also add other processors to the chain, as shown here. Note we have also given the entire chain a `name` ("add-unknown-fields-to-the-schema"). We'll use this name in the next section to specify that our update request handler should use this chain definition. + +CAUTION: This chain definition will make a number of copy field rules for string fields to be created from corresponding text fields. If your data causes you to end up with a lot of copy field rules, indexing may be slowed down noticeably, and your index size will be larger. To control for these issues, it's recommended that you review the copy field rules that are created, and remove any which you do not need for faceting, sorting, highlighting, etc. + +If you're interested in more information about the classes used in this chain, here are links to the Javadocs for update processor factories mentioned above: * {solr-javadocs}/solr-core/org/apache/solr/update/processor/UUIDUpdateProcessorFactory.html[UUIDUpdateProcessorFactory] * {solr-javadocs}/solr-core/org/apache/solr/update/processor/RemoveBlankFieldUpdateProcessorFactory.html[RemoveBlankFieldUpdateProcessorFactory] @@ -170,9 +186,13 @@ Javadocs for update processor factories mentioned above: * {solr-javadocs}/solr-core/org/apache/solr/update/processor/ParseDateFieldUpdateProcessorFactory.html[ParseDateFieldUpdateProcessorFactory] * {solr-javadocs}/solr-core/org/apache/solr/update/processor/AddSchemaFieldsUpdateProcessorFactory.html[AddSchemaFieldsUpdateProcessorFactory] -=== Make the UpdateRequestProcessorChain the Default for the UpdateRequestHandler +=== Set the Default UpdateRequestProcessorChain -Once the UpdateRequestProcessorChain has been defined, you must instruct your UpdateRequestHandlers to use it when working with index updates (i.e., adding, removing, replacing documents). There are two ways to do this. The update chain shown above has a `default=true` attribute which will use it for any update handler. An alternative, more explicit way is to use <> to set the defaults on all `/update` request handlers: +Once the UpdateRequestProcessorChain has been defined, you must instruct your UpdateRequestHandlers to use it when working with index updates (i.e., adding, removing, replacing documents). + +There are two ways to do this. The update chain shown above has a `default=true` attribute which will use it for any update handler. + +An alternative, more explicit way is to use <> to set the defaults on all `/update` request handlers: [source,xml] ---- @@ -183,14 +203,18 @@ Once the UpdateRequestProcessorChain has been defined, you must instruct your Up ---- -[IMPORTANT] -==== -After each of these changes have been made, Solr should be restarted (or, you can reload the cores to load the new `solrconfig.xml` definitions). -==== +IMPORTANT: After all of these changes have been made, Solr should be restarted or the cores reloaded. + +=== Disabling Automatic Field Guessing + +Automatic field creation can be disabled with the `update.autoCreateFields` property. To do this, you can use the <> with a command such as: + +[source,bash] +curl http://host:8983/solr/mycollection/config -d '{"set-user-property": {"update.autoCreateFields":"false"}}' == Examples of Indexed Documents -Once the schemaless mode has been enabled (whether you configured it manually or are using `_default`), documents that include fields that are not defined in your schema will be indexed, using the guessed field types which are automatically added to the schema. +Once the schemaless mode has been enabled (whether you configured it manually or are using the `_default` configset), documents that include fields that are not defined in your schema will be indexed, using the guessed field types which are automatically added to the schema. For example, adding a CSV document will cause unknown fields to be added, with fieldTypes based on values: