Apache
/
lucene
mirror of https://github.com/apache/lucene.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Ref Guide: Add license header, typos in Package Manager docs

This commit is contained in:
Cassandra Targett 2020-03-02 12:05:42 -06:00
parent 153d7bcfee
commit a7ebe0c906
2 changed files with 94 additions and 63 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

156
solr/solr-ref-guide/src/package-manager-internals.adoc
View File

@ -1,11 +1,29 @@
= Package Manager Internals = Package Manager Internals
:page-tocclass: right
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership. The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License. You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.
The package manager (CLI) internally uses various Solr APIs to install, deploy and update packages. This document contains an overview of those APIs. The package manager (CLI) internally uses various Solr APIs to install, deploy and update packages. This document contains an overview of those APIs.
== Salient Features == Salient Features
* Zero disruption deployment (hot deployment): Should be possible to install and update packages without node restarts or core reloads, and hence deployments should be quick and without failed requests or stale caches. * Zero disruption deployment (hot deployment): Should be possible to install and update packages without node restarts or core reloads, and hence deployments should be quick and without failed requests or stale caches.
* Easy packaging: * Easy packaging:
** Standard plugin concepts, e.g. query parser, search component, request handler, URP etc., should be supported without any special code/packaging changes. ** Standard plugin concepts, e.g., query parser, search component, request handler, URP etc., should be supported without any special code/packaging changes.
** Artifacts (jars containing custom plugins) that users have already deployed (and are using in production) should be compatible, without needing to recompile or re-package, for greater adoption. ** Artifacts (jars containing custom plugins) that users have already deployed (and are using in production) should be compatible, without needing to recompile or re-package, for greater adoption.
** Should support single jar packages as well as multiple jar packages. ** Should support single jar packages as well as multiple jar packages.
** Use familiar / standard naming ** Use familiar / standard naming
@ -44,22 +62,22 @@ $ bin/solr package add-key my_key.der
Package store is a distributed file store which can store arbitrary files in the file system. Package store is a distributed file store which can store arbitrary files in the file system.
* This is a fully replicated file system based repository. * This is a fully replicated file system based repository.
* It lives at <solr.home>/filestore on each Solr node * It lives at <solr.home>/filestore on each Solr node.
* Every entry is a file + metadata. The metadata is named .<filename>.json * Every entry is a file + metadata. The metadata is named .<filename>.json.
* The metadata file contains the sha256, signatures of the file * The metadata file contains the sha256, signatures of the file.
* Users can’t create files starting with period (.) * Users can’t create files starting with period (.).
* It is agnostic of content type of files. You may store jars as well as other files. * It is agnostic of content type of files. You may store jars as well as other files..
=== How Does the Package Store Work? === How Does the Package Store Work?
When a file is uploaded to the PackageStore , When a file is uploaded to the PackageStore, the following is true:
* It’s saved to the local file system * It’s saved to the local file system.
* It’s saved along with the metadata. The metadata file stores the sha512 and signatures of the jar files too. * It’s saved along with the metadata. The metadata file stores the sha512 and signatures of the jar files too.
* Every live node in the cluster is asked to download it as well * Every live node in the cluster is asked to download it as well.
=== Package Store APIs === Package Store APIs
The end points are The end points are:
* `PUT /api/cluster/files/{full/path/to/file}` in each node * `PUT /api/cluster/files/{full/path/to/file}` in each node
* `GET /api/node/files/{full/path/to/file}` to download the file * `GET /api/node/files/{full/path/to/file}` to download the file
@ -70,30 +88,34 @@ The end points are
Use the following steps to upload a jar signed with your public key: Use the following steps to upload a jar signed with your public key:
1) If you don't have a jar file with plugins, download a sample from github . If you don't have a jar file with plugins, download a sample from github:
+
[source, bash] [source, bash]
---- ----
$ curl -o runtimelibs.jar -LO https://github.com/apache/lucene-solr/blob/master/solr/core/src/test-files/runtimecode/runtimelibs.jar.bin?raw=true $ curl -o runtimelibs.jar -LO https://github.com/apache/lucene-solr/blob/master/solr/core/src/test-files/runtimecode/runtimelibs.jar.bin?raw=true
---- ----
2) Sign the jar with your private key
. Sign the jar with your private key:
+
[source, bash] [source, bash]
---- ----
$ openssl dgst -sha1 -sign my_key.pem runtimelibs.jar | openssl enc -base64 | sed 's/+/%2B/g' | tr -d \\n | sed $ openssl dgst -sha1 -sign my_key.pem runtimelibs.jar | openssl enc -base64 | sed 's/+/%2B/g' | tr -d \\n | sed
---- ----
3) Upload your jar with signature. (replace the `sig` param with the output you got from the previous command) . Upload your jar with signature, replacing the `sig` parameter with the output from the previous command:
+
[source, bash] [source, bash]
---- ----
$ curl --data-binary @runtimelibs.jar -X PUT http://localhost:7574/api/cluster/files/mypkg/1.0/myplugins.jar?sig=<signature-of-jar> $ curl --data-binary @runtimelibs.jar -X PUT http://localhost:7574/api/cluster/files/mypkg/1.0/myplugins.jar?sig=<signature-of-jar>
---- ----
4) Verify your jar upload . Verify your jar upload:
+
[source, bash] [source, bash]
---- ----
$ curl http://localhost:7574/api/node/files/mypkg/1.0?omitHeader=true $ curl http://localhost:7574/api/node/files/mypkg/1.0?omitHeader=true
---- ----
+
[source, json] [source, json]
---- ----
{ {
@ -104,22 +126,21 @@ $ curl http://localhost:7574/api/node/files/mypkg/1.0?omitHeader=true
"sig":["elNjhmWIOgTgbAzeZ+OcwR42N7vqL6Ig9eAqn4YoP2thT7FJuhiaZuCPivjMkD682EBo9gveSCTyXIsZKjOCbQ=="]}]}} "sig":["elNjhmWIOgTgbAzeZ+OcwR42N7vqL6Ig9eAqn4YoP2thT7FJuhiaZuCPivjMkD682EBo9gveSCTyXIsZKjOCbQ=="]}]}}
---- ----
== Packages == Packages
A Package have the following attributes: A Package have the following attributes:
* A unique name * A unique name
* One or more versions with the following attributes: * One or more versions with the following attributes:
** `version` : The version string ** `version`: The version string
** `files` : An array of files from the package store ** `files`: An array of files from the package store
For every package / version in the packages definition, there is a unique `SolrResourceLoader` instance. This is a child of the `CoreContainer` resource loader. For every package/version in the packages definition, there is a unique `SolrResourceLoader` instance. This is a child of the `CoreContainer` resource loader.
=== packages.json === packages.json
The package configurations live in a file called /packages.json in ZooKeeper. At any given moment we can have multiple versions of a given package in the package configuration. The system will always use the latest version . Versions are sorted by their numeric value and the biggest is the latest. The package configurations live in a file called `packages.json` in ZooKeeper. At any given moment we can have multiple versions of a given package in the package configuration. The system will always use the latest version. Versions are sorted by their numeric value and the highest is the latest.
*example:* For example:
[source,json] [source,json]
---- ----
@ -142,41 +163,44 @@ The package configurations live in a file called /packages.json in ZooKeeper. At
* `GET /api/cluster/package` Get the list of packages * `GET /api/cluster/package` Get the list of packages
* `POST /api/cluster/package` edit packages * `POST /api/cluster/package` edit packages
** `add` command: add a version of a package ** `add` command: add a version of a package
** `delete` command : delete a version of a package ** `delete` command: delete a version of a package
=== How to Upgrade? === How to Upgrade?
Use the add command to add a version that is bigger than the current version
=== How to Downgrade ? Use the `add` command to add a version that is higher than the current version.
Use the delete command to delete a version that is the highest and the next highest version will be chosen
=== How to Downgrade?
Use the `delete` command to delete the highest version and choose the next highest version.
=== Using Multiple Versions in Parallel === Using Multiple Versions in Parallel
We use the `params.json` in the collection config to store a version of a package it uses. by default it is the $LATEST.
We use `params.json` in the collection config to store a version of a package it uses. By default it is the `$LATEST`.
[source, json] [source, json]
---- ----
{"params":{ {"params":{
"PKG_VERSIONS": { "PKG_VERSIONS": {
"mypkg": "0.1", "mypkg": "0.1", <1>
"pkg2" : "$LATEST", "pkg2" : "$LATEST", <2>
}}} }}}
---- ----
* for `mypkg` use the version `0.1` irrespective of whether there is a newer version is available or not <1> For `mypkg`, use the version `0.1` irrespective of whether there is a newer version available or not.
* for `pkg2` use the latest. This is optional. The default is ‘$LATEST’ <2> For `pkg2`, use the latest. This is optional. The default is `$LATEST`.
=== The Workflow === Workflow
* A new version of a package is added
* The package loader loads the classes and notifies every plugin holder of the availability of the new version
* It checks if it is supposed to use a specific version, Ignore the update
* If not, reload the plugin
* A new version of a package is added.
* The package loader loads the classes and notifies every plugin holder of the availability of the new version.
* It checks if it is supposed to use a specific version, Ignore the update.
* If not, reload the plugin.
=== Using Them in Plugins === Using Packages in Plugins
Any class name can be prefixed with the packagename e.g : `mypkg:fully.qualified.ClassName` and Solr would use the latest version of the package to load the classes from. The plugins loaded from packages cannot depend on core level classes.
*Plugin declaration in `solrconfig.xml`* Any class name can be prefixed with the package name, e.g., `mypkg:fully.qualified.ClassName` and Solr would use the latest version of the package to load the classes from. The plugins loaded from packages cannot depend on core level classes.
.Plugin declaration in `solrconfig.xml`
[source, xml] [source, xml]
---- ----
<requestHandler name="/myhandler" class="mypkg:full.path.to.MyClass"> <requestHandler name="/myhandler" class="mypkg:full.path.to.MyClass">
@ -185,8 +209,8 @@ Any class name can be prefixed with the packagename e.g : `mypkg:fully.qualified
=== Full Working Example === Full Working Example
1) create a package . Create a package:
+
[source,bash] [source,bash]
---- ----
curl http://localhost:8983/api/cluster/package -H 'Content-type:application/json' -d ' curl http://localhost:8983/api/cluster/package -H 'Content-type:application/json' -d '
@ -195,13 +219,14 @@ curl http://localhost:8983/api/cluster/package -H 'Content-type:application/jso
"version":"1.0", "version":"1.0",
"files" :["/mypkg/1.0/myplugins.jar"]}}' "files" :["/mypkg/1.0/myplugins.jar"]}}'
---- ----
2) Verify the created package
. Verify the created package:
+
[source,bash] [source,bash]
---- ----
curl http://localhost:8983/api/cluster/package?omitHeader=true curl http://localhost:8983/api/cluster/package?omitHeader=true
---- ----
+
[source,json] [source,json]
---- ----
{"result":{ {"result":{
@ -211,20 +236,22 @@ curl http://localhost:8983/api/cluster/package?omitHeader=true
"files":["/mypkg/1.0/myplugins.jar"]}]}}} "files":["/mypkg/1.0/myplugins.jar"]}]}}}
---- ----
3) The Package is ready to use now. Now, register a plugin in your collection from the package. Note the `"mypkg"` prefix applied to the `class` attribute. The same result can be achieved by editing your `solrconfig.xml` as well . The package should be ready to use at this point. Next, register a plugin in your collection from the package. Note the `mypkg:` prefix applied to the `class` attribute. The same result can be achieved by editing your `solrconfig.xml` as well:
+
[source,bash] [source,bash]
---- ----
curl http://localhost:8983/solr/gettingstarted/config -H 'Content-type:application/json' -d '{ curl http://localhost:8983/solr/gettingstarted/config -H 'Content-type:application/json' -d '{
"create-requesthandler" : { "name" : "/test", "create-requesthandler": { "name": "/test",
"class": "mypkg:org.apache.solr.core.RuntimeLibReqHandler" }}' "class": "mypkg:org.apache.solr.core.RuntimeLibReqHandler" }}'
---- ----
4) Verify that the component is created and it is using the correct version of the package to load classes from . Verify that the component is created and it is using the correct version of the package to load classes from:
+
[source,bash] [source,bash]
---- ----
curl http://localhost:8983/solr/gettingstarted/config/requestHandler?componentName=/test&meta=true&omitHeader=true curl http://localhost:8983/solr/gettingstarted/config/requestHandler?componentName=/test&meta=true&omitHeader=true
---- ----
+
[source,json] [source,json]
---- ----
{ {
@ -237,11 +264,13 @@ curl http://localhost:8983/solr/gettingstarted/config/requestHandler?componentNa
"files":["/mypkg/1.0/myplugins.jar"]}}}}} "files":["/mypkg/1.0/myplugins.jar"]}}}}}
---- ----
5) Test the request handler . Test the request handler:
+
[source,bash] [source,bash]
---- ----
$ curl http://localhost:8983/solr/gettingstarted/test?omitHeader=true $ curl http://localhost:8983/solr/gettingstarted/test?omitHeader=true
---- ----
+
[source,json] [source,json]
---- ----
{ {
@ -255,23 +284,24 @@ $ curl http://localhost:8983/solr/gettingstarted/test?omitHeader=true
"loader":"java.net.FactoryURLClassLoader"} "loader":"java.net.FactoryURLClassLoader"}
---- ----
6) Update the version of our component . Update the version of our component. Get a new version of the jar, sign and upload it:
+
Get a new version of the jar, sign and upload it
[source, bash] [source, bash]
---- ----
$ curl -o runtimelibs3.jar -LO https://github.com/apache/lucene-solr/blob/master/solr/core/src/test-files/runtimecode/runtimelibs_v3.jar.bin?raw=true $ curl -o runtimelibs3.jar -LO https://github.com/apache/lucene-solr/blob/master/solr/core/src/test-files/runtimecode/runtimelibs_v3.jar.bin?raw=true
$ openssl dgst -sha1 -sign my_key.pem runtimelibs.jar | openssl enc -base64 | sed 's/+/%2B/g' | tr -d \\n | sed $ openssl dgst -sha1 -sign my_key.pem runtimelibs.jar | openssl enc -base64 | sed 's/+/%2B/g' | tr -d \\n | sed
$ curl --data-binary @runtimelibs3.jar -X PUT http://localhost:8983/api/cluster/files/mypkg/2.0/myplugins.jar?sig=<signature> $ curl --data-binary @runtimelibs3.jar -X PUT http://localhost:8983/api/cluster/files/mypkg/2.0/myplugins.jar?sig=<signature>
---- ----
7) Verify it . Verify it:
+
[source, bash] [source, bash]
---- ----
$ curl http://localhost:8983/api/node/files/mypkg/2.0?omitHeader=true $ curl http://localhost:8983/api/node/files/mypkg/2.0?omitHeader=true
---- ----
+
[source, json] [source, json]
---- ----
{ {
@ -282,8 +312,8 @@ $ curl http://localhost:8983/api/node/files/mypkg/2.0?omitHeader=true
"sig":["ICkC+nGE+AqiANM0ajhVPNCQsbPbHLSWlIe5ETV5835e5HqndWrFHiV2R6nLVjDCxov/wLPo1uK0VzvAPIioUQ=="]}]}} "sig":["ICkC+nGE+AqiANM0ajhVPNCQsbPbHLSWlIe5ETV5835e5HqndWrFHiV2R6nLVjDCxov/wLPo1uK0VzvAPIioUQ=="]}]}}
---- ----
8) Add a new version of the package . Add a new version of the package:
+
[source,bash] [source,bash]
---- ----
$ curl http://localhost:8983/api/cluster/package -H 'Content-type:application/json' -d ' $ curl http://localhost:8983/api/cluster/package -H 'Content-type:application/json' -d '
@ -293,12 +323,13 @@ $ curl http://localhost:8983/api/cluster/package -H 'Content-type:application/j
"files" :["/mypkg/2.0/myplugins.jar"]}}' "files" :["/mypkg/2.0/myplugins.jar"]}}'
---- ----
9) Verify the plugin to see if the correct version of the package is being used . Verify the plugin to see if the correct version of the package is being used:
+
[source,bash] [source,bash]
---- ----
$ curl http://localhost:8983/solr/gettingstarted/config/requestHandler?componentName=/test&meta=true&omitHeader=true $ curl http://localhost:8983/solr/gettingstarted/config/requestHandler?componentName=/test&meta=true&omitHeader=true
---- ----
+
[source,json] [source,json]
---- ----
{ {
@ -316,12 +347,13 @@ $ curl http://localhost:8983/solr/gettingstarted/config/requestHandler?component
}}}}} }}}}}
---- ----
10) Test the plugin . Test the plugin:
+
[source,bash] [source,bash]
---- ----
$ curl http://localhost:8983/solr/gettingstarted/test?omitHeader=true $ curl http://localhost:8983/solr/gettingstarted/test?omitHeader=true
---- ----
+
[source,json] [source,json]
---- ----
{ {
@ -338,11 +370,11 @@ $ curl http://localhost:8983/solr/gettingstarted/test?omitHeader=true
"Version": "2" "Version": "2"
} }
---- ----
Note that the `Version` value is `"2"` . So the plugin is updated Note that the `Version` value is `"2"`, which means the plugin is updated.
=== How to Avoid Automatic Upgrade === How to Avoid Automatic Upgrade
The default version used in any collection is always the latest. However, setting a per-collection property in the `params.json` ensures that the versions are always fixed irrespective of the new versions added The default version used in any collection is always the latest. However, setting a per-collection property in `params.json` ensures that the versions are always fixed irrespective of the new versions added.
[source,bash] [source,bash]
---- ----

1
solr/solr-ref-guide/src/package-manager.adoc
View File

@ -1,7 +1,6 @@
= Package Management = Package Management
:page-children: package-manager-internals :page-children: package-manager-internals
:page-tocclass: right :page-tocclass: right
// Licensed to the Apache Software Foundation (ASF) under one // Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file // or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information // distributed with this work for additional information