From a7ebe0c906f44cfb542f21c1e3a5faee4c3c1fe0 Mon Sep 17 00:00:00 2001 From: Cassandra Targett Date: Mon, 2 Mar 2020 12:05:42 -0600 Subject: [PATCH] Ref Guide: Add license header, typos in Package Manager docs --- .../src/package-manager-internals.adoc | 156 +++++++++++------- solr/solr-ref-guide/src/package-manager.adoc | 1 - 2 files changed, 94 insertions(+), 63 deletions(-) diff --git a/solr/solr-ref-guide/src/package-manager-internals.adoc b/solr/solr-ref-guide/src/package-manager-internals.adoc index 467691bfdeb..1c73f26048a 100644 --- a/solr/solr-ref-guide/src/package-manager-internals.adoc +++ b/solr/solr-ref-guide/src/package-manager-internals.adoc @@ -1,11 +1,29 @@ = Package Manager Internals +:page-tocclass: right +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + The package manager (CLI) internally uses various Solr APIs to install, deploy and update packages. This document contains an overview of those APIs. == Salient Features * Zero disruption deployment (hot deployment): Should be possible to install and update packages without node restarts or core reloads, and hence deployments should be quick and without failed requests or stale caches. * Easy packaging: -** Standard plugin concepts, e.g. query parser, search component, request handler, URP etc., should be supported without any special code/packaging changes. +** Standard plugin concepts, e.g., query parser, search component, request handler, URP etc., should be supported without any special code/packaging changes. ** Artifacts (jars containing custom plugins) that users have already deployed (and are using in production) should be compatible, without needing to recompile or re-package, for greater adoption. ** Should support single jar packages as well as multiple jar packages. ** Use familiar / standard naming @@ -44,22 +62,22 @@ $ bin/solr package add-key my_key.der Package store is a distributed file store which can store arbitrary files in the file system. * This is a fully replicated file system based repository. -* It lives at /filestore on each Solr node -* Every entry is a file + metadata. The metadata is named ..json -* The metadata file contains the sha256, signatures of the file -* Users can’t create files starting with period (.) -* It is agnostic of content type of files. You may store jars as well as other files. +* It lives at /filestore on each Solr node. +* Every entry is a file + metadata. The metadata is named ..json. +* The metadata file contains the sha256, signatures of the file. +* Users can’t create files starting with period (.). +* It is agnostic of content type of files. You may store jars as well as other files.. === How Does the Package Store Work? -When a file is uploaded to the PackageStore , +When a file is uploaded to the PackageStore, the following is true: -* It’s saved to the local file system +* It’s saved to the local file system. * It’s saved along with the metadata. The metadata file stores the sha512 and signatures of the jar files too. -* Every live node in the cluster is asked to download it as well +* Every live node in the cluster is asked to download it as well. === Package Store APIs -The end points are +The end points are: * `PUT /api/cluster/files/{full/path/to/file}` in each node * `GET /api/node/files/{full/path/to/file}` to download the file @@ -70,30 +88,34 @@ The end points are Use the following steps to upload a jar signed with your public key: -1) If you don't have a jar file with plugins, download a sample from github - +. If you don't have a jar file with plugins, download a sample from github: ++ [source, bash] ---- $ curl -o runtimelibs.jar -LO https://github.com/apache/lucene-solr/blob/master/solr/core/src/test-files/runtimecode/runtimelibs.jar.bin?raw=true - ---- -2) Sign the jar with your private key + +. Sign the jar with your private key: ++ [source, bash] ---- $ openssl dgst -sha1 -sign my_key.pem runtimelibs.jar | openssl enc -base64 | sed 's/+/%2B/g' | tr -d \\n | sed ---- -3) Upload your jar with signature. (replace the `sig` param with the output you got from the previous command) +. Upload your jar with signature, replacing the `sig` parameter with the output from the previous command: ++ [source, bash] ---- $ curl --data-binary @runtimelibs.jar -X PUT http://localhost:7574/api/cluster/files/mypkg/1.0/myplugins.jar?sig= ---- -4) Verify your jar upload +. Verify your jar upload: ++ [source, bash] ---- $ curl http://localhost:7574/api/node/files/mypkg/1.0?omitHeader=true ---- ++ [source, json] ---- { @@ -104,22 +126,21 @@ $ curl http://localhost:7574/api/node/files/mypkg/1.0?omitHeader=true "sig":["elNjhmWIOgTgbAzeZ+OcwR42N7vqL6Ig9eAqn4YoP2thT7FJuhiaZuCPivjMkD682EBo9gveSCTyXIsZKjOCbQ=="]}]}} ---- - == Packages A Package have the following attributes: * A unique name * One or more versions with the following attributes: -** `version` : The version string -** `files` : An array of files from the package store +** `version`: The version string +** `files`: An array of files from the package store -For every package / version in the packages definition, there is a unique `SolrResourceLoader` instance. This is a child of the `CoreContainer` resource loader. +For every package/version in the packages definition, there is a unique `SolrResourceLoader` instance. This is a child of the `CoreContainer` resource loader. === packages.json -The package configurations live in a file called /packages.json in ZooKeeper. At any given moment we can have multiple versions of a given package in the package configuration. The system will always use the latest version . Versions are sorted by their numeric value and the biggest is the latest. +The package configurations live in a file called `packages.json` in ZooKeeper. At any given moment we can have multiple versions of a given package in the package configuration. The system will always use the latest version. Versions are sorted by their numeric value and the highest is the latest. -*example:* +For example: [source,json] ---- @@ -142,41 +163,44 @@ The package configurations live in a file called /packages.json in ZooKeeper. At * `GET /api/cluster/package` Get the list of packages * `POST /api/cluster/package` edit packages ** `add` command: add a version of a package -** `delete` command : delete a version of a package +** `delete` command: delete a version of a package === How to Upgrade? -Use the add command to add a version that is bigger than the current version -=== How to Downgrade ? -Use the delete command to delete a version that is the highest and the next highest version will be chosen +Use the `add` command to add a version that is higher than the current version. + +=== How to Downgrade? + +Use the `delete` command to delete the highest version and choose the next highest version. === Using Multiple Versions in Parallel -We use the `params.json` in the collection config to store a version of a package it uses. by default it is the $LATEST. + +We use `params.json` in the collection config to store a version of a package it uses. By default it is the `$LATEST`. [source, json] ---- {"params":{ "PKG_VERSIONS": { - "mypkg": "0.1", - "pkg2" : "$LATEST", + "mypkg": "0.1", <1> + "pkg2" : "$LATEST", <2> }}} ---- -* for `mypkg` use the version `0.1` irrespective of whether there is a newer version is available or not -* for `pkg2` use the latest. This is optional. The default is ‘$LATEST’ +<1> For `mypkg`, use the version `0.1` irrespective of whether there is a newer version available or not. +<2> For `pkg2`, use the latest. This is optional. The default is `$LATEST`. -=== The Workflow -* A new version of a package is added -* The package loader loads the classes and notifies every plugin holder of the availability of the new version -* It checks if it is supposed to use a specific version, Ignore the update -* If not, reload the plugin +=== Workflow +* A new version of a package is added. +* The package loader loads the classes and notifies every plugin holder of the availability of the new version. +* It checks if it is supposed to use a specific version, Ignore the update. +* If not, reload the plugin. -=== Using Them in Plugins -Any class name can be prefixed with the packagename e.g : `mypkg:fully.qualified.ClassName` and Solr would use the latest version of the package to load the classes from. The plugins loaded from packages cannot depend on core level classes. +=== Using Packages in Plugins -*Plugin declaration in `solrconfig.xml`* +Any class name can be prefixed with the package name, e.g., `mypkg:fully.qualified.ClassName` and Solr would use the latest version of the package to load the classes from. The plugins loaded from packages cannot depend on core level classes. +.Plugin declaration in `solrconfig.xml` [source, xml] ---- @@ -185,8 +209,8 @@ Any class name can be prefixed with the packagename e.g : `mypkg:fully.qualified === Full Working Example -1) create a package - +. Create a package: ++ [source,bash] ---- curl http://localhost:8983/api/cluster/package -H 'Content-type:application/json' -d ' @@ -195,13 +219,14 @@ curl http://localhost:8983/api/cluster/package -H 'Content-type:application/jso "version":"1.0", "files" :["/mypkg/1.0/myplugins.jar"]}}' ---- -2) Verify the created package +. Verify the created package: ++ [source,bash] ---- curl http://localhost:8983/api/cluster/package?omitHeader=true ---- - ++ [source,json] ---- {"result":{ @@ -211,20 +236,22 @@ curl http://localhost:8983/api/cluster/package?omitHeader=true "files":["/mypkg/1.0/myplugins.jar"]}]}}} ---- -3) The Package is ready to use now. Now, register a plugin in your collection from the package. Note the `"mypkg"` prefix applied to the `class` attribute. The same result can be achieved by editing your `solrconfig.xml` as well +. The package should be ready to use at this point. Next, register a plugin in your collection from the package. Note the `mypkg:` prefix applied to the `class` attribute. The same result can be achieved by editing your `solrconfig.xml` as well: ++ [source,bash] ---- curl http://localhost:8983/solr/gettingstarted/config -H 'Content-type:application/json' -d '{ - "create-requesthandler" : { "name" : "/test", + "create-requesthandler": { "name": "/test", "class": "mypkg:org.apache.solr.core.RuntimeLibReqHandler" }}' ---- -4) Verify that the component is created and it is using the correct version of the package to load classes from - +. Verify that the component is created and it is using the correct version of the package to load classes from: ++ [source,bash] ---- curl http://localhost:8983/solr/gettingstarted/config/requestHandler?componentName=/test&meta=true&omitHeader=true ---- ++ [source,json] ---- { @@ -237,11 +264,13 @@ curl http://localhost:8983/solr/gettingstarted/config/requestHandler?componentNa "files":["/mypkg/1.0/myplugins.jar"]}}}}} ---- -5) Test the request handler +. Test the request handler: ++ [source,bash] ---- $ curl http://localhost:8983/solr/gettingstarted/test?omitHeader=true ---- ++ [source,json] ---- { @@ -255,23 +284,24 @@ $ curl http://localhost:8983/solr/gettingstarted/test?omitHeader=true "loader":"java.net.FactoryURLClassLoader"} ---- -6) Update the version of our component - -Get a new version of the jar, sign and upload it - +. Update the version of our component. Get a new version of the jar, sign and upload it: ++ [source, bash] ---- $ curl -o runtimelibs3.jar -LO https://github.com/apache/lucene-solr/blob/master/solr/core/src/test-files/runtimecode/runtimelibs_v3.jar.bin?raw=true + $ openssl dgst -sha1 -sign my_key.pem runtimelibs.jar | openssl enc -base64 | sed 's/+/%2B/g' | tr -d \\n | sed + $ curl --data-binary @runtimelibs3.jar -X PUT http://localhost:8983/api/cluster/files/mypkg/2.0/myplugins.jar?sig= ---- -7) Verify it +. Verify it: ++ [source, bash] ---- $ curl http://localhost:8983/api/node/files/mypkg/2.0?omitHeader=true ---- - ++ [source, json] ---- { @@ -282,8 +312,8 @@ $ curl http://localhost:8983/api/node/files/mypkg/2.0?omitHeader=true "sig":["ICkC+nGE+AqiANM0ajhVPNCQsbPbHLSWlIe5ETV5835e5HqndWrFHiV2R6nLVjDCxov/wLPo1uK0VzvAPIioUQ=="]}]}} ---- -8) Add a new version of the package - +. Add a new version of the package: ++ [source,bash] ---- $ curl http://localhost:8983/api/cluster/package -H 'Content-type:application/json' -d ' @@ -293,12 +323,13 @@ $ curl http://localhost:8983/api/cluster/package -H 'Content-type:application/j "files" :["/mypkg/2.0/myplugins.jar"]}}' ---- -9) Verify the plugin to see if the correct version of the package is being used +. Verify the plugin to see if the correct version of the package is being used: ++ [source,bash] ---- $ curl http://localhost:8983/solr/gettingstarted/config/requestHandler?componentName=/test&meta=true&omitHeader=true ---- - ++ [source,json] ---- { @@ -316,12 +347,13 @@ $ curl http://localhost:8983/solr/gettingstarted/config/requestHandler?component }}}}} ---- -10) Test the plugin - +. Test the plugin: ++ [source,bash] ---- $ curl http://localhost:8983/solr/gettingstarted/test?omitHeader=true ---- ++ [source,json] ---- { @@ -338,11 +370,11 @@ $ curl http://localhost:8983/solr/gettingstarted/test?omitHeader=true "Version": "2" } ---- -Note that the `Version` value is `"2"` . So the plugin is updated +Note that the `Version` value is `"2"`, which means the plugin is updated. === How to Avoid Automatic Upgrade -The default version used in any collection is always the latest. However, setting a per-collection property in the `params.json` ensures that the versions are always fixed irrespective of the new versions added +The default version used in any collection is always the latest. However, setting a per-collection property in `params.json` ensures that the versions are always fixed irrespective of the new versions added. [source,bash] ---- diff --git a/solr/solr-ref-guide/src/package-manager.adoc b/solr/solr-ref-guide/src/package-manager.adoc index 94e948ea650..57a021200c1 100644 --- a/solr/solr-ref-guide/src/package-manager.adoc +++ b/solr/solr-ref-guide/src/package-manager.adoc @@ -1,7 +1,6 @@ = Package Management :page-children: package-manager-internals :page-tocclass: right - // Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information