Apache
/
lucene
mirror of https://github.com/apache/lucene.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

SOLR-14789: Absorb the docker-solr repo. (#1769)

This commit is contained in:
Houston Putman 2020-09-11 12:29:29 -04:00 committed by GitHub
parent ed930f4b29
commit 485d5fb41a
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
52 changed files with 2862 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
build.gradle
View File

@ -25,6 +25,7 @@ plugins {
id 'de.thetaphi.forbiddenapis' version '3.0.1' apply false
id "de.undercouch.download" version "4.0.2" apply false
id "net.ltgt.errorprone" version "1.2.1" apply false
id "com.palantir.docker" version "0.25.0" apply false
}
apply from: file('gradle/defaults.gradle')

2
settings.gradle
View File

@ -68,3 +68,5 @@ include "solr:solr-ref-guide"
include "solr:example"
include "solr:packaging"
include "solr:docker"
include "solr:docker:package"

2
solr/CHANGES.txt
View File

@ -20,6 +20,8 @@ should be used instead, such as autoscaling policy or rules-based placement.
New Features
---------------------
* SOLR-14789: Migrate docker image creation from docker-solr repo to solr/docker. (Houston Putman, Tim Potter, David Smiley, janhoy, Mike Drob)
* SOLR-14440: Introduce new Certificate Authentication Plugin to load Principal from certificate subject. (Mike Drob)
* SOLR-14588: Introduce Circuit Breaker Infrastructure and a JVM heap usage memory tracking circuit breaker implementation (Atri Sharma)

320
solr/docker/Docker-FAQ.md Normal file
View File

@ -0,0 +1,320 @@
Docker Solr FAQ
===============
How do I persist Solr data and config?
--------------------------------------
Your data is persisted already, in your container's filesystem.
If you `docker run`, add data to Solr, then `docker stop` and later
`docker start`, then your data is still there. The same is true for
changes to configuration files.
Equally, if you `docker commit` your container, you can later create a new
container from that image, and that will have your data in it.
For some use-cases it is convenient to provide a modified `solr.in.sh` file to Solr.
For example to point Solr to a ZooKeeper host:
```
docker create --name my_solr -P solr
docker cp my_solr:/opt/solr/bin/solr.in.sh .
sed -i -e 's/#ZK_HOST=.*/ZK_HOST=cylon.lan:2181/' solr.in.sh
docker cp solr.in.sh my_solr:/opt/solr/bin/solr.in.sh
docker start my_solr
# With a browser go to http://cylon.lan:32873/solr/#/ and confirm "-DzkHost=cylon.lan:2181" in the JVM Args section.
```
But usually when people ask this question, what they are after is a way
to store Solr data and config in a separate [Docker Volume](https://docs.docker.com/userguide/dockervolumes/).
That is explained in the next two questions.
How can I mount a host directory as a data volume?
--------------------------------------------------
This is useful if you want to inspect or modify the data in the Docker host
when the container is not running, and later easily run new containers against that data.
This is indeed possible, but there are a few gotchas.
Solr stores its core data in the `server/solr` directory, in sub-directories
for each core. The `server/solr` directory also contains configuration files
that are part of the Solr distribution.
Now, if we mounted volumes for each core individually, then that would
interfere with Solr trying to create those directories. If instead we make
the whole directory a volume, then we need to provide those configuration files
in our volume, which we can do by copying them from a temporary container.
For example:
```
# create a directory to store the server/solr directory
$ mkdir /home/docker-volumes/mysolr1
# make sure its host owner matches the container's solr user
$ sudo chown 8983:8983 /home/docker-volumes/mysolr1
# copy the solr directory from a temporary container to the volume
$ docker run -it --rm -v /home/docker-volumes/mysolr1:/target apache/solr cp -r server/solr /target/
# pass the solr directory to a new container running solr
$ SOLR_CONTAINER=$(docker run -d -P -v /home/docker-volumes/mysolr1/solr:/opt/solr/server/solr apache/solr)
# create a new core
$ docker exec -it --user=solr $SOLR_CONTAINER solr create_core -c gettingstarted
# check the volume on the host:
$ ls /home/docker-volumes/mysolr1/solr/
configsets gettingstarted README.txt solr.xml zoo.cfg
```
Note that if you add or modify files in that directory from the host, you must `chown 8983:8983` them.
How can I use a Data Volume Container?
--------------------------------------
You can avoid the concerns about UID mismatches above, by using data volumes only from containers.
You can create a container with a volume, then point future containers at that same volume.
This can be handy if you want to modify the solr image, for example if you want to add a program.
By separating the data and the code, you can change the code and re-use the data.
But there are pitfalls:
- if you remove the container that owns the volume, then you lose your data.
Docker does not even warn you that a running container is dependent on it.
- if you point multiple solr containers at the same volume, you will have multiple instances
write to the same files, which will undoubtedly lead to corruption
- if you do want to remove that volume, you must do `docker rm -v containername`;
if you forget the `-v` there will be a dangling volume which you can not easily clean up.
Here is an example:
```
# create a container with a volume on the path that solr uses to store data.
docker create -v /opt/solr/server/solr --name mysolr1data solr /bin/true
# pass the volume to a new container running solr
SOLR_CONTAINER=$(docker run -d -P --volumes-from=mysolr1data apache/solr)
# create a new core
$ docker exec -it --user=solr $SOLR_CONTAINER solr create_core -c gettingstarted
# make a change to the config, using the config API
docker exec -it --user=solr $SOLR_CONTAINER curl http://localhost:8983/solr/gettingstarted/config -H 'Content-type:application/json' -d'{
"set-property" : {"query.filterCache.autowarmCount":1000},
"unset-property" :"query.filterCache.size"}'
# verify the change took effect
docker exec -it --user=solr $SOLR_CONTAINER curl http://localhost:8983/solr/gettingstarted/config/overlay?omitHeader=true
# stop the solr container
docker exec -it --user=solr $SOLR_CONTAINER bash -c 'cd server; java -DSTOP.PORT=7983 -DSTOP.KEY=solrrocks -jar start.jar --stop'
# create a new container
SOLR_CONTAINER=$(docker run -d -P --volumes-from=mysolr1data apache/solr)
# check our core is still there:
docker exec -it --user=solr $SOLR_CONTAINER ls server/solr/gettingstarted
# check the config modification is still there:
docker exec -it --user=solr $SOLR_CONTAINER curl http://localhost:8983/solr/gettingstarted/config/overlay?omitHeader=true
```
Can I use volumes with SOLR_HOME?
---------------------------------
Solr supports a SOLR_HOME environment variable to point to a non-standard location of the Solr home directory.
You can use this in Solr docker, in combination with volumes:
```
docker run -it -v $PWD/mysolrhome:/mysolrhome -e SOLR_HOME=/mysolrhome apache/solr
```
This does need a pre-configured directory at that location.
To make this easier, Solr docker supports a INIT_SOLR_HOME setting, which copies the contents
from the default directory in the image to the SOLR_HOME (if it is empty).
```
mkdir mysolrhome
sudo chown 8983:8983 mysolrhome
docker run -it -v $PWD/mysolrhome:/mysolrhome -e SOLR_HOME=/mysolrhome -e INIT_SOLR_HOME=yes apache/solr
```
Note: If SOLR_HOME is set, the "solr-precreate" command will put the created core in the SOLR_HOME directory
rather than the "mycores" directory.
Can I run ZooKeeper and Solr clusters under Docker?
---------------------------------------------------
At the network level the ZooKeeper nodes need to be able to talk to eachother,
and the Solr nodes need to be able to talk to the ZooKeeper nodes and to each other.
At the application level, different nodes need to be able to identify and locate each other.
In ZooKeeper that is done with a configuration file that lists hostnames or IP addresses for each node.
In Solr that is done with a parameter that specifies a host or IP address, which is then stored in ZooKeeper.
In typical clusters, those hostnames/IP addresses are pre-defined and remain static through the lifetime of the cluster.
In Docker, inter-container communication and multi-host networking can be facilitated by [Docker Networks](https://docs.docker.com/engine/userguide/networking/).
But, crucially, Docker does not normally guarantee that IP addresses of containers remain static during the lifetime of a container.
In non-networked Docker, the IP address seems to change everytime you stop/start.
In a networked Docker, containers can lose their IP address in certain sequences of starting/stopping, unless you take steps to prevent that.
IP changes causes problems:
- If you use hardcoded IP addresses in configuration, and the addresses of your containers change after a stops/start, then your cluster will stop working and may corrupt itself.
- If you use hostnames in configuration, and the addresses of your containers change, then you might run into problems with cached hostname lookups.
- And if you use hostnames there is another problem: the names are not defined until the respective container is running,
So when for example the first ZooKeeper node starts up, it will attempt a hostname lookup for the other nodes, and that will fail.
This is especially a problem for ZooKeeper 3.4.6; future versions are better at recovering.
Docker 1.10 has a new `--ip` configuration option that allows you to specify an IP address for a container.
It also has a `--ip-range` option that allows you to specify the range that other containers get addresses from.
Used together, you can implement static addresses. See [this example](docs/docker-networking.md).
Can I run ZooKeeper and Solr with Docker Links?
-----------------------------------------------
Docker's [Legacy container links](https://docs.docker.com/engine/userguide/networking/default_network/dockerlinks/) provide a way to
pass connection configuration between containers. It only works on a single machine, on the default bridge.
It provides no facilities for static IPs.
Note: this feature is expected to be deprecated and removed in a future release.
So really, see the "Can I run ZooKeeper and Solr clusters under Docker?" option above instead.
But for some use-cases, such as quick demos or one-shot automated testing, it can be convenient.
Run ZooKeeper, and define a name so we can link to it:
```console
$ docker run --name zookeeper -d -p 2181:2181 -p 2888:2888 -p 3888:3888 jplock/zookeeper
```
Run two Solr nodes, linked to the zookeeper container:
```console
$ docker run --name solr1 --link zookeeper:ZK -d -p 8983:8983 \
apache/solr \
bash -c 'solr start -f -z $ZK_PORT_2181_TCP_ADDR:$ZK_PORT_2181_TCP_PORT'
$ docker run --name solr2 --link zookeeper:ZK -d -p 8984:8983 \
apache/solr \
bash -c 'solr start -f -z $ZK_PORT_2181_TCP_ADDR:$ZK_PORT_2181_TCP_PORT'
```
Create a collection:
```console
$ docker exec -i -t solr1 solr create_collection \
-c gettingstarted -shards 2 -p 8983
```
Then go to `http://localhost:8983/solr/#/~cloud` (adjust the hostname for your docker host) to see the two shards and Solr nodes.
How can I run ZooKeeper and Solr with Docker Compose?
-----------------------------------------------------
See the [docker compose example](docs/docker-compose.yml).
I'm confused about the different invocations of solr -- help?
-------------------------------------------------------------
The different invocations of the Solr docker image can look confusing, because the name of the
image is "apache/solr" and the Solr command is also "solr", and the image interprets various arguments in
special ways. I'll illustrate the various invocations:
To run an arbitrary command in the image:
```
docker run -it apache/solr date
```
here "apache/solr" is the name of the image, and "date" is the command.
This does not invoke any solr functionality.
To run the Solr server:
```
docker run -it apache/solr
```
Here "apache/solr" is the name of the image, and there is no specific command,
so the image defaults to run the "solr" command with "-f" to run it in the foreground.
To run the Solr server with extra arguments:
```
docker run -it apache/solr -h myhostname
```
This is the same as the previous one, but an additional argument is passed.
The image will run the "solr" command with "-f -h myhostname"
To run solr as an arbitrary command:
```
docker run -it apache/solr solr zk --help
```
here the first "apache/solr" is the image name, and the second "solr"
is the "solr" command. The image runs the command exactly as specified;
no "-f" is implicitly added. The container will print help text, and exit.
If you find this visually confusing, it might be helpful to use more specific image tags,
and specific command paths. For example:
```
docker run -it apache/solr:6 bin/solr -f -h myhostname
```
Finally, the Solr docker image offers several commands that do some work before
then invoking the Solr server, like "solr-precreate" and "solr-demo".
See the README.md for usage.
These are implemented by the `docker-entrypoint.sh` script, and must be passed
as the first argument to the image. For example:
```
docker run -it apache/solr:6 solr-demo
```
It's important to understand an implementation detail here. The Dockerfile uses
`solr-foreground` as the `CMD`, and the `docker-entrypoint.sh` implements
that by by running "solr -f". So these two are equivalent:
```
docker run -it apache/solr:6
docker run -it apache/solr:6 solr-foreground
```
whereas:
```
docker run -it apache/solr:6 solr -f
```
is slightly different: the "solr" there is a generic command, not treated in any
special way by `docker-entrypoint.sh`. In particular, this means that the
`docker-entrypoint-initdb.d` mechanism is not applied.
So, if you want to use `docker-entrypoint-initdb.d`, then you must use one
of the other two invocations.
You also need to keep that in mind when you want to invoke solr from the bash
command. For example, this does NOT run `docker-entrypoint-initdb.d` scripts:
```
docker run -it -v $PWD/set-heap.sh:/docker-entrypoint-initdb.d/set-heap.sh \
apache/solr:6 bash -c "echo hello; solr -f"
```
but this does:
```
docker run -it $PWD/set-heap.sh:/docker-entrypoint-initdb.d/set-heap.sh \
apache/solr:6 bash -c "echo hello; /opt/docker-solr/scripts/docker-entrypoint.sh solr-foreground"
```

75
solr/docker/Dockerfile Normal file
View File

@ -0,0 +1,75 @@
ARG SOLR_PACKAGE_IMAGE
ARG BASE_IMAGE=openjdk:11-jre-slim
FROM $SOLR_PACKAGE_IMAGE as solr_package
FROM $BASE_IMAGE as runtime
LABEL maintainer="The Apache Lucene/Solr Project"
LABEL repository="https://github.com/apache/lucene-solr"
# Override the default github URL to provide a mirror for github releases.
ARG GITHUB_URL=github.com
RUN set -ex; \
apt-get update; \
apt-get -y install acl dirmngr gpg lsof procps wget netcat gosu tini; \
rm -rf /var/lib/apt/lists/*; \
cd /usr/local/bin; wget -nv https://${GITHUB_URL}/apangin/jattach/releases/download/v1.5/jattach; chmod 755 jattach; \
echo >jattach.sha512 "d8eedbb3e192a8596c08efedff99b9acf1075331e1747107c07cdb1718db2abe259ef168109e46bd4cf80d47d43028ff469f95e6ddcbdda4d7ffa73a20e852f9 jattach"; \
sha512sum -c jattach.sha512; rm jattach.sha512
ENV SOLR_USER="solr" \
SOLR_UID="8983" \
SOLR_GROUP="solr" \
SOLR_GID="8983" \
PATH="/opt/solr/bin:/opt/docker-solr/scripts:$PATH" \
SOLR_INCLUDE=/etc/default/solr.in.sh \
SOLR_HOME=/var/solr/data \
SOLR_PID_DIR=/var/solr \
SOLR_LOGS_DIR=/var/solr/logs \
LOG4J_PROPS=/var/solr/log4j2.xml \
SOLR_JETTY_HOST="0.0.0.0"
RUN set -ex; \
groupadd -r --gid "$SOLR_GID" "$SOLR_GROUP"; \
useradd -r --uid "$SOLR_UID" --gid "$SOLR_GID" "$SOLR_USER"
COPY --chown=0:0 scripts /opt/docker-solr/scripts
ARG SOLR_VERSION
COPY --from=solr_package "/opt/solr-$SOLR_VERSION.tgz" "/opt/solr-$SOLR_VERSION.tgz"
RUN set -ex; \
tar -C /opt --extract --file "/opt/solr-$SOLR_VERSION.tgz" && \
rm "/opt/solr-$SOLR_VERSION.tgz"; \
(cd /opt; ln -s "solr-$SOLR_VERSION" solr); \
rm -Rf /opt/solr/docs/ /opt/solr/dist/{solr-solrj-$SOLR_VERSION.jar,solrj-lib,solr-test-framework-$SOLR_VERSION.jar,test-framework}; \
mkdir -p /opt/solr/server/solr/lib /docker-entrypoint-initdb.d /opt/docker-solr; \
chown -R 0:0 "/opt/solr-$SOLR_VERSION"; \
find "/opt/solr-$SOLR_VERSION" -type d -print0 | xargs -0 chmod 0755; \
find "/opt/solr-$SOLR_VERSION" -type f -print0 | xargs -0 chmod 0644; \
chmod -R 0755 "/opt/solr-$SOLR_VERSION/bin" "/opt/solr-$SOLR_VERSION/contrib/prometheus-exporter/bin/solr-exporter" /opt/solr-$SOLR_VERSION/server/scripts/cloud-scripts; \
cp /opt/solr/bin/solr.in.sh /etc/default/solr.in.sh; \
mv /opt/solr/bin/solr.in.sh /opt/solr/bin/solr.in.sh.orig; \
mv /opt/solr/bin/solr.in.cmd /opt/solr/bin/solr.in.cmd.orig; \
chown root:0 /etc/default/solr.in.sh; \
chmod 0664 /etc/default/solr.in.sh; \
mkdir -p /var/solr/data /var/solr/logs; \
(cd /opt/solr/server/solr; cp solr.xml zoo.cfg /var/solr/data/); \
cp /opt/solr/server/resources/log4j2.xml /var/solr/log4j2.xml; \
find /var/solr -type d -print0 | xargs -0 chmod 0770; \
find /var/solr -type f -print0 | xargs -0 chmod 0660; \
sed -i -e "s/\"\$(whoami)\" == \"root\"/\$(id -u) == 0/" /opt/solr/bin/solr; \
sed -i -e 's/lsof -PniTCP:/lsof -t -PniTCP:/' /opt/solr/bin/solr; \
chown -R "0:0" /opt/solr-$SOLR_VERSION /docker-entrypoint-initdb.d /opt/docker-solr; \
chown -R "$SOLR_USER:0" /var/solr;
VOLUME /var/solr
EXPOSE 8983
WORKDIR /opt/solr
USER $SOLR_USER
ENTRYPOINT ["docker-entrypoint.sh"]
CMD ["solr-foreground"]

275
solr/docker/README.md Normal file
View File

@ -0,0 +1,275 @@
# Supported tags and respective `Dockerfile` links
See [Docker Hub](https://hub.docker.com/apache/solr?tab=tags) for a list of image tags available to pull.
# Getting started with the Docker image
Instructions below apply to `apache/solr:8.0.0` and above.
## Running Solr with host-mounted directories
Typically users first want to run a single standalone Solr server in a container, with a single core for data, while storing data in a local directory.
This is a convenient mechanism for developers, and could be used for single-server production hosts too.
```console
$ mkdir solrdata
$ docker run -d -v "$PWD/solrdata:/var/solr" -p 8983:8983 --name my_solr apache/solr solr-precreate gettingstarted
```
Then with a web browser go to `http://localhost:8983/` to see the Admin Console (adjust the hostname for your docker host).
In the web UI if you click on "Core Admin" you should now see the "gettingstarted" core.
Next load some of the example data that is included in the container:
```console
$ docker exec -it my_solr post -c gettingstarted example/exampledocs/manufacturers.xml
```
In the UI, find the "Core selector" popup menu and select the "gettingstarted" core, then select the "Query" menu item. This gives you a default search for `*:*` which returns all docs. Hit the "Execute Query" button, and you should see a few docs with data. Congratulations!
# Single server with Docker-compose
You can use Docker Compose to run a single standalone server.
And you could use Docker Volumes instead of host-mounted directories.
For example, with a `docker-compose.yml` containing the following:
```yaml
version: '3'
services:
solr:
image: apache/solr
ports:
- "8983:8983"
volumes:
- data:/var/solr
command:
- solr-precreate
- gettingstarted
volumes:
data:
```
you can simply run:
```console
docker-compose up -d
```
## Single-command demo
For quick demos of Solr docker, there is a single command that starts Solr, creates a collection called "demo", and loads sample data into it:
```console
$ docker run --name solr_demo -d -p 8983:8983 apache/solr solr-demo
```
## Distributed Solr
See [this example](docs/docker-compose.yml)
for an example Docker Compose file that starts up Solr in a simple cluster configuration.
# How the image works
The container contains an installation of Solr, as installed by the [service installation script](https://lucene.apache.org/solr/guide/7_7/taking-solr-to-production.html#service-installation-script).
This stores the Solr distribution in `/opt/solr`, and configures Solr to use `/var/solr` to store data and logs, using the `/etc/default/solr` file for configuration.
If you want to persist the data, mount a volume or directory on `/var/solr`.
Solr expects some files and directories in `/var/solr`; if you use your own directory or volume you can either pre-populate them, or let Solr docker copy them for you. See [init-var-solr](scripts/init-var-solr).
If you want to use custom configuration, mount it in the appropriate place. See below for examples.
The Solr docker distribution adds [scripts](include/scripts) in `/opt/docker-solr/scripts` to make it easier to use under Docker, for example to create cores on container startup.
## Creating cores
When Solr runs in standalone mode, you create "cores" to store data. On a non-Docker Solr, you would run the server in the background, then use the [Solr control script](https://lucene.apache.org/solr/guide/7_7/solr-control-script-reference.html) to create cores and load data. With Solr docker you have various options.
The first is exactly the same: start Solr running in a container, then execute the control script manually in the same container:
```console
$ docker run -d -p 8983:8983 --name my_solr apache/solr
$ docker exec -it my_solr solr create_core -c gettingstarted
```
This is not very convenient for users, and makes it harder to turn it into configuration for Docker Compose and orchestration tools like Kubernetes.
So, typically you will use the `solr-precreate` command which prepares the specified core and then runs Solr:
```console
$ docker run -d -p 8983:8983 --name my_solr apache/solr solr-precreate gettingstarted
```
The `solr-precreate` command takes an optional extra argument to specify a configset directory below `/opt/solr/server/solr/configsets/`.
This allows you to specify your own config. See [this example](https://github.com/docker-solr/docker-solr-examples/tree/master/custom-configset).
The third option is to use the `solr-create` command. This runs a Solr in the background in the container, then uses the Solr control script to create the core, then stops the Solr server and restarts it in the foreground. This method is less popular because the double Solr run can be confusing.
```console
$ docker run -d -p 8983:8983 --name my_solr apache/solr solr-create -c gettingstarted
```
Finally, you can run your own command-line and specify what to do, and even invoke mounted scripts. For example:
```console
$ docker run -p 8983:8983 -v $PWD/mysetup.sh:/mysetup.sh --name my_solr apache/solr bash -c "precreate-core gettingstarted && source /mysetup.sh && solr-foreground"
```
## Creating collections
In a "SolrCloud" cluster you create "collections" to store data; and again you have several options for creating a core.
These examples assume you're running [this example cluster](docs/docker-compose.yml)
The first way to create a collection is to go to the [Solr Admin UI](http://localhost:8983/), select "Collections" from the left-hand side navigation menu, then press the "Add Collection" button, give it a name, select the `_default` config set, then press the "Add Collection" button.
The second way is through the Solr control script on one of the containers:
```console
$ docker exec solr1 solr create -c gettingstarted2
```
The third way is to use a separate container:
```console
$ docker run -e SOLR_HOST=solr1 --network docs_solr apache/solr solr create_collection -c gettingstarted3 -p 8983
```
The fourth way is to use the remote API, from the host or from one of the containers, or some new container on the same network (adjust the hostname accordingly):
```console
curl 'http://localhost:8983/solr/admin/collections?action=CREATE&name=gettingstarted3&numShards=1&collection.configName=_default'
```
If you want to use a custom config for your collection, you first need to upload it, and then refer to it by name when you create the collection.
See the Ref guide on how to use the [ZooKeeper upload](https://lucene.apache.org/solr/guide/7_7/solr-control-script-reference.html) or the [configset API](https://lucene.apache.org/solr/guide/7_0/configsets-api.html).
## Loading your own data
There are several ways to load data; let's look at the most common ones.
The most common first deployment is to run Solr standalone (not in a cluster), on a workstation or server, where you have local data you wish to load.
One way of doing that is using a separate container, with a mounted volume containing the data, using the host network so you can connect to the mapped port:
```console
# start Solr. Listens on localhost:8983
$ docker run --name my_solr -p 8983:8983 apache/solr solr-precreate books
# get data
$ mkdir mydata
$ wget -O mydata/books.csv https://raw.githubusercontent.com/apache/lucene-solr/master/solr/example/exampledocs/books.csv
$ docker run --rm -v "$PWD/mydata:/mydata" --network=host apache/solr post -c books /mydata/books.csv
```
If you use the [this example cluster](docs/docker-compose.yml) the same works, or you can just start your loading container in the same network:
```console
$ docker run -e SOLR_HOST=solr1 --network=mycluster_solr apache/solr solr create_collection -c books -p 8983
$ docker run --rm -v "$PWD/mydata:/mydata" --network=mycluster_solr apache/solr post -c books /mydata/books.csv -host solr1
```
Alternatively, you can make the data available on a volume at Solr start time, and then load it from `docker exec` or a custom start script.
## solr.in.sh configuration
In Solr it is common to configure settings in [solr.in.sh](https://github.com/apache/lucene-solr/blob/master/solr/bin/solr.in.sh),
as documented in the [Solr Reference Guide](https://cwiki.apache.org/confluence/display/solr/Taking+Solr+to+Production#TakingSolrtoProduction-Environmentoverridesincludefile).
The `solr.in.sh` file can be found in `/etc/default`:
```console
$ docker run apache/solr cat /etc/default/solr.in.sh
```
It has various commented-out values, which you can override when running the container, like:
```
$ docker run -d -p 8983:8983 -e SOLR_HEAP=800m apache/solr
```
You can also mount your own config file. Do no modify the values that are set at the end of the file.
## Extending the image
The Solr docker image has an extension mechanism. At run time, before starting Solr, the container will execute scripts
in the `/docker-entrypoint-initdb.d/` directory. You can add your own scripts there either by using mounted volumes
or by using a custom Dockerfile. These scripts can for example copy a core directory with pre-loaded data for continuous
integration testing, or modify the Solr configuration.
Here is a simple example. With a `custom.sh` script like:
```console
#!/bin/bash
set -e
echo "this is running inside the container before Solr starts"
```
you can run:
```console
$ docker run --name solr_custom1 -d -v $PWD/custom.sh:/docker-entrypoint-initdb.d/custom.sh apache/solr
$ sleep 5
$ docker logs solr_custom1 | head
/opt/docker-solr/scripts/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/set-heap.sh
this is running inside the container before Solr starts
Starting Solr on port 8983 from /opt/solr/server
```
With this extension mechanism it can be useful to see the shell commands that are being executed by the `docker-entrypoint.sh`
script in the docker log. To do that, set an environment variable using Docker's `-e VERBOSE=yes`.
Instead of using this mechanism, you can of course create your own script that does setup and then call `solr-foreground`, mount that script into the container, and execute it as a command when running the container.
Other ways of extending the image are to create custom Docker images that inherit from this one.
## Debugging with jattach
The `jcmd`, `jmap` `jstack` tools can be useful for debugging Solr inside the container. These tools are not included with the JRE, but this image includes the [jattach](https://github.com/apangin/jattach) utility which lets you do much of the same.
Usage: jattach <pid> <cmd> [args ...]
Commands:
load : load agent library
properties : print system properties
agentProperties : print agent properties
datadump : show heap and thread summary
threaddump : dump all stack traces (like jstack)
dumpheap : dump heap (like jmap)
inspectheap : heap histogram (like jmap -histo)
setflag : modify manageable VM flag
printflag : print VM flag
jcmd : execute jcmd command
Example comands to do a thread dump and get heap info for PID 10:
jattach 10 threaddump
jattach 10 jcmd GC.heap_info
# Updating from Docker-solr5-7 to 8
For Solr 8, the docker-solr distribution switched from just extracting the Solr tar, to using the [service installation script](https://lucene.apache.org/solr/guide/7_7/taking-solr-to-production.html#service-installation-script). This was done for various reasons: to bring it in line with the recommendations by the Solr Ref Guide, to make it easier to mount volumes, and because we were [asked to](https://github.com/docker-solr/docker-solr/issues/173).
This is a backwards incompatible change, and means that if you're upgrading from an older version, you will most likely need to make some changes. If you don't want to upgrade at this time, specify `apache/solr:7` as your container image. If you use `solr:8` you will use the new style. If you use just `solr` then you risk being tripped up by backwards incompatible changes; always specify at least a major version.
Changes:
- The Solr data is now stored in `/var/solr/data` rather than `/opt/solr/server/solr`. The `/opt/solr/server/solr/mycores` no longer exists
- The custom `SOLR_HOME` can no longer be used, because various scripts depend on the new locations. Consequently, `INIT_SOLR_HOME` is also no longer supported.
# Running under tini
The Solr docker image runs Solr under [tini](https://github.com/krallin/tini), to make signal handling work better; in particular, this allows you to `kill -9` the JVM. If you run `docker run --init`, or use `init: true` in `docker-compose.yml`, or have added `--init` to `dockerd`, docker will start its `tini` and docker-solr will notice it is not PID 1, and just `exec` Solr. If you do not run with `--init`, then the docker entrypoint script detects that it is running as PID 1, and will start the `tini` present in the docker-solr image, and run Solr under that. If you really do not want to run `tini`, and just run Solr as PID 1 instead, then you can set the `TINI=no` environment variable.
# Out of memory handling
You can use the `OOM` environment variable to control the behaviour of the Solr JVM when an out-of-memory error occurs.
If you specify `OOM=exit`, Solr docker will add `-XX:+ExitOnOutOfMemoryError` to the JVM arguments, so that the JVM will exit.
If you specify `OOM=crash`, Solr docker will add `-XX:+CrashOnOutOfMemoryError` to the JVM arguments, so the JVM will crash and produces text and binary crash files (if core files are enabled).
If you specify `OOM=script`, Solr docker will add `-XX:OnOutOfMemoryError=/opt/docker-solr/scripts/oom_solr.sh`, so the JVM will run that script (and if you want to you can mount your own in its place).
# About this repository
This repository is available on , and the official build is on the [Docker Hub](https://hub.docker.com/apache/solr/).
# History
This project was started in 2015 by [Martijn Koster](https://github.com/makuk66) in the [docker-solr](https://github.com/docker-solr/docker-solr) repository. In 2019 maintainership and copyright was transferred to the Apache Lucene/Solr project, and in 2020 the project was migrated to live within the Solr project. Many thanks to Martijn for all your contributions over the years!

92
solr/docker/build.gradle Normal file
View File

@ -0,0 +1,92 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
apply plugin: 'base'
apply plugin: 'com.palantir.docker'
subprojects {
apply plugin: 'base'
apply plugin: 'com.palantir.docker'
}
description = 'Solr Docker image'
def dockerPackage = project(':solr:docker:package')
dependencies {
docker dockerPackage
}
docker {
name = "apache/solr:${version}"
files file('include')
buildArgs(['BASE_IMAGE' : 'openjdk:11-jre-slim', 'SOLR_PACKAGE_IMAGE' : 'apache/solr-build:local-package', 'SOLR_VERSION': "${version}"])
}
// In order to create the solr docker image, the solr package image must be created first.
tasks.docker.dependsOn(dockerPackage.tasks.docker)
abstract class DockerTestSuite extends DefaultTask {
private List<String> tests = new ArrayList<>();
private List<String> ignore = new ArrayList<>();
@OutputDirectory
abstract DirectoryProperty getOutputDir()
@Option(option = "tests", description = "Only run these specified tests, comma separated.")
public void setTests(List<String> tests) {
this.tests = tests;
}
@Input
public List<String> getTests() {
return tests;
}
@Option(option = "ignore", description = "Ignore these tests, comma separated.")
public void setIgnore(List<String> ignore) {
this.ignore = ignore;
}
@Input
public List<String> getIgnore() {
return ignore;
}
@TaskAction
void execute() {
def sourceDir = project.file("tests/cases")
sourceDir.eachFile { file ->
def testName = file.getName()
def testCaseBuildDir = outputDir.dir(testName).get().toString()
// If specific tests are specified, only run those. Otherwise run all that are not ignored.
def runTest = !this.tests.isEmpty() ? tests.contains(testName) : !ignore.contains(testName)
if (runTest) {
project.exec {
environment "TEST_DIR", "$file"
environment "BUILD_DIR", "$testCaseBuildDir"
commandLine "bash", "$file/test.sh", "apache/solr:${project.version}"
}
}
}
}
}
task test(type: DockerTestSuite) {
outputDir = project.file("$buildDir/tmp/tests")
}

13
solr/docker/docs/docker-compose.yml Normal file
View File

@ -0,0 +1,13 @@
version: '3'
services:
solr:
image: apache/solr
ports:
- "8983:8983"
volumes:
- data:/var/solr
command:
- solr-precreate
- gettingstarted
volumes:
data:

249
solr/docker/docs/docker-networking.md Normal file
View File

@ -0,0 +1,249 @@
Example of Zookeeper and Solr cluster with Docker networking
------------------------------------------------------------
_Note: this article dates from Jan 2016. While this approach would still work, in Jan 2019 this would typically done with Docker cluster and orchestration tools like Kubernetes. See for example [this blog post](https://lucidworks.com/2019/02/07/running-solr-on-kubernetes-part-1/)._
In this example I'll create a cluster with 3 ZooKeeper nodes and 3 Solr nodes, distributed over 3 machines (trinity10, trinity20, trinity30).
I'll use an overlay network, specify fixed IP addresses when creating containers, and I'll pass in explicit `/etc/hosts` entries to make sure they are available even when nodes are down.
I won't show the configuration of the key-value store to configuration to enable networking, see [the docs](https://docs.docker.com/engine/userguide/networking/get-started-overlay/) for that.
I'll not use Docker Swarm in this example, but specifically place and configure containers where I want them by ssh'ing into the appropriate Docker host.
To make this example easier to understand I'll just use shell commands.
For actual use you may want to use a fancier deployment tool like [Fabric](http://www.fabfile.org).
Note: this example requires Docker 1.10.
I'll run these commands from the first machine, trinity10.
Create a network named "netzksolr" for this cluster. The `--ip-range` specifies the range of
addresses to use for containers, whereas the `--subnet` specifies all possible addresses in this
network. So effectively, addresses in the subnet but outside the range are reserved for containers
that specifically use the `--ip` option.
```
docker network create --driver=overlay --subnet 192.168.22.0/24 --ip-range=192.168.22.128/25 netzksolr
```
As a simple test, check the automatic assignment and specific assignment work:
```
$ docker run -i --rm --net=netzksolr busybox ip -4 addr show eth0 | grep inet
inet 192.168.23.129/24 scope global eth0
$ docker run -i --rm --net=netzksolr --ip=192.168.22.5 busybox ip -4 addr show eth0 | grep inet
inet 192.168.22.5/24 scope global eth0
```
So next create containers for ZooKeeper nodes.
First define some environment variables for convenience:
```
# the machine to run the container on
ZK1_HOST=trinity10.lan
ZK2_HOST=trinity20.lan
ZK3_HOST=trinity30.lan
# the IP address for the container
ZK1_IP=192.168.22.10
ZK2_IP=192.168.22.11
ZK3_IP=192.168.22.12
# the Docker image
ZK_IMAGE=jplock/zookeeper
```
Then create the containers:
```
ssh -n $ZK1_HOST "docker pull jplock/zookeeper && docker create --ip=$ZK1_IP --net netzksolr --name zk1 --hostname=zk1 --add-host zk2:$ZK2_IP --add-host zk3:$ZK3_IP -it $ZK_IMAGE"
ssh -n $ZK2_HOST "docker pull jplock/zookeeper && docker create --ip=$ZK2_IP --net netzksolr --name zk2 --hostname=zk2 --add-host zk1:$ZK1_IP --add-host zk3:$ZK3_IP -it $ZK_IMAGE"
ssh -n $ZK3_HOST "docker pull jplock/zookeeper && docker create --ip=$ZK3_IP --net netzksolr --name zk3 --hostname=zk3 --add-host zk1:$ZK1_IP --add-host zk2:$ZK2_IP -it $ZK_IMAGE"
```
Next configure those containers by creating ZooKeeper's `zoo.cfg` and `myid` files:
```
# Add ZooKeeper nodes to the ZooKeeper config.
# If you use hostnames here, ZK will complain with UnknownHostException about the other nodes.
# In ZooKeeper 3.4.6 that stays broken forever; in 3.4.7 that does recover.
# If you use IP addresses you avoid the UnknownHostException and get a quorum more quickly,
# but IP address changes can impact you.
docker cp zk1:/opt/zookeeper/conf/zoo.cfg .
cat >>zoo.cfg <<EOM
server.1=zk1:2888:3888
server.2=zk2:2888:3888
server.3=zk3:2888:3888
EOM
cat zoo.cfg | ssh $ZK1_HOST 'dd of=zoo.cfg.tmp && docker cp zoo.cfg.tmp zk1:/opt/zookeeper/conf/zoo.cfg && rm zoo.cfg.tmp'
cat zoo.cfg | ssh $ZK2_HOST 'dd of=zoo.cfg.tmp && docker cp zoo.cfg.tmp zk2:/opt/zookeeper/conf/zoo.cfg && rm zoo.cfg.tmp'
cat zoo.cfg | ssh $ZK3_HOST 'dd of=zoo.cfg.tmp && docker cp zoo.cfg.tmp zk3:/opt/zookeeper/conf/zoo.cfg && rm zoo.cfg.tmp'
rm zoo.cfg
echo 1 | ssh $ZK1_HOST 'dd of=myid && docker cp myid zk1:/tmp/zookeeper/myid && rm myid'
echo 2 | ssh $ZK2_HOST 'dd of=myid && docker cp myid zk2:/tmp/zookeeper/myid && rm myid'
echo 3 | ssh $ZK3_HOST 'dd of=myid && docker cp myid zk3:/tmp/zookeeper/myid && rm myid'
```
Now start the containers:
```
ssh -n $ZK1_HOST 'docker start zk1'
ssh -n $ZK2_HOST 'docker start zk2'
ssh -n $ZK3_HOST 'docker start zk3'
# Optional: verify containers are running
ssh -n $ZK1_HOST 'docker ps'
ssh -n $ZK2_HOST 'docker ps'
ssh -n $ZK3_HOST 'docker ps'
# Optional: inspect IP addresses of the containers
ssh -n $ZK1_HOST "docker inspect --format '{{ .NetworkSettings.Networks.netzksolr.IPAddress }}' zk1"
ssh -n $ZK2_HOST "docker inspect --format '{{ .NetworkSettings.Networks.netzksolr.IPAddress }}' zk2"
ssh -n $ZK3_HOST "docker inspect --format '{{ .NetworkSettings.Networks.netzksolr.IPAddress }}' zk3"
# Optional: verify connectivity and hostnames
ssh -n $ZK1_HOST 'docker run --rm --net netzksolr -i ubuntu bash -c "echo -n zk1,zk2,zk3 | xargs -n 1 --delimiter=, /bin/ping -c 1"'
ssh -n $ZK2_HOST 'docker run --rm --net netzksolr -i ubuntu bash -c "echo -n zk1,zk2,zk3 | xargs -n 1 --delimiter=, /bin/ping -c 1"'
ssh -n $ZK3_HOST 'docker run --rm --net netzksolr -i ubuntu bash -c "echo -n zk1,zk2,zk3 | xargs -n 1 --delimiter=, /bin/ping -c 1"'
# Optional: verify cluster got a leader
ssh -n $ZK1_HOST "docker exec -i zk1 bash -c 'echo stat | nc localhost 2181'"
ssh -n $ZK2_HOST "docker exec -i zk2 bash -c 'echo stat | nc localhost 2181'"
ssh -n $ZK3_HOST "docker exec -i zk3 bash -c 'echo stat | nc localhost 2181'"
# Optional: verify we can connect a zookeeper client. This should show the `[zookeeper]` znode.
printf "ls /\nquit\n" | ssh $ZK1_HOST docker exec -i zk1 /opt/zookeeper/bin/zkCli.sh
```
That's the ZooKeeper cluster running.
Next, we create Solr containers in much the same way:
```
ZKSOLR1_HOST=trinity10.lan
ZKSOLR2_HOST=trinity20.lan
ZKSOLR3_HOST=trinity30.lan
ZKSOLR1_IP=192.168.22.20
ZKSOLR2_IP=192.168.22.21
ZKSOLR3_IP=192.168.22.22
# the Docker image
SOLR_IMAGE=solr
HOST_OPTIONS="--add-host zk1:$ZK1_IP --add-host zk2:$ZK2_IP --add-host zk3:$ZK3_IP"
ssh -n $ZKSOLR1_HOST "docker pull $SOLR_IMAGE && docker create --ip=$ZKSOLR1_IP --net netzksolr --name zksolr1 --hostname=zksolr1 -it $HOST_OPTIONS $SOLR_IMAGE"
ssh -n $ZKSOLR2_HOST "docker pull $SOLR_IMAGE && docker create --ip=$ZKSOLR2_IP --net netzksolr --name zksolr2 --hostname=zksolr2 -it $HOST_OPTIONS $SOLR_IMAGE"
ssh -n $ZKSOLR3_HOST "docker pull $SOLR_IMAGE && docker create --ip=$ZKSOLR3_IP --net netzksolr --name zksolr3 --hostname=zksolr3 -it $HOST_OPTIONS $SOLR_IMAGE"
```
Now configure Solr to know where its ZooKeeper cluster is, and start the containers:
```
for h in zksolr1 zksolr2 zksolr3; do
docker cp zksolr1:/opt/solr/bin/solr.in.sh .
sed -i -e 's/#ZK_HOST=""/ZK_HOST="zk1:2181,zk2:2181,zk3:2181"/' solr.in.sh
sed -i -e 's/#*SOLR_HOST=.*/SOLR_HOST="'$h'"/' solr.in.sh
mv solr.in.sh solr.in.sh-$h
done
cat solr.in.sh-zksolr1 | ssh $ZKSOLR1_HOST "dd of=solr.in.sh && docker cp solr.in.sh zksolr1:/opt/solr/bin/solr.in.sh && rm solr.in.sh"
cat solr.in.sh-zksolr2 | ssh $ZKSOLR2_HOST "dd of=solr.in.sh && docker cp solr.in.sh zksolr2:/opt/solr/bin/solr.in.sh && rm solr.in.sh"
cat solr.in.sh-zksolr3 | ssh $ZKSOLR3_HOST "dd of=solr.in.sh && docker cp solr.in.sh zksolr3:/opt/solr/bin/solr.in.sh && rm solr.in.sh"
rm solr.in.sh*
ssh -n $ZKSOLR1_HOST docker start zksolr1
ssh -n $ZKSOLR2_HOST docker start zksolr2
ssh -n $ZKSOLR3_HOST docker start zksolr3
# Optional: print IP addresses to verify
ssh -n $ZKSOLR1_HOST 'docker inspect --format "{{ .NetworkSettings.Networks.netzksolr.IPAddress }}" zksolr1'
ssh -n $ZKSOLR2_HOST 'docker inspect --format "{{ .NetworkSettings.Networks.netzksolr.IPAddress }}" zksolr2'
ssh -n $ZKSOLR3_HOST 'docker inspect --format "{{ .NetworkSettings.Networks.netzksolr.IPAddress }}" zksolr3'
# Optional: check logs
ssh -n $ZKSOLR1_HOST docker logs zksolr1
ssh -n $ZKSOLR2_HOST docker logs zksolr2
ssh -n $ZKSOLR3_HOST docker logs zksolr3
# Optional: check the webserver
ssh -n $ZKSOLR1_HOST "docker exec -i zksolr1 /bin/bash -c 'wget -O - http://zksolr1:8983/'"
ssh -n $ZKSOLR2_HOST "docker exec -i zksolr2 /bin/bash -c 'wget -O - http://zksolr2:8983/'"
ssh -n $ZKSOLR3_HOST "docker exec -i zksolr3 /bin/bash -c 'wget -O - http://zksolr3:8983/'"
```
Next let's create a collection:
```
ssh -n $ZKSOLR1_HOST docker exec -i zksolr1 /opt/solr/bin/solr create_collection -c my_collection1 -shards 2 -p 8983
```
To load data, and see it was split over shards:
```
mak@trinity10:~$ docker exec -it --user=solr zksolr1 bin/post -c my_collection1 example/exampledocs/manufacturers.xml
/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -classpath /opt/solr/dist/solr-core-5.4.0.jar -Dauto=yes -Dc=my_collection1 -Ddata=files org.apache.solr.util.SimplePostTool example/exampledocs/manufacturers.xml
SimplePostTool version 5.0.0
Posting files to [base] url http://localhost:8983/solr/my_collection1/update...
Entering auto mode. File endings considered are xml,json,csv,pdf,doc,docx,ppt,pptx,xls,xlsx,odt,odp,ods,ott,otp,ots,rtf,htm,html,txt,log
POSTing file manufacturers.xml (application/xml) to [base]
1 files indexed.
COMMITting Solr index changes to http://localhost:8983/solr/my_collection1/update...
Time spent: 0:00:01.093
mak@trinity10:~$ docker exec -it --user=solr zksolr1 bash -c "wget -q -O - 'http://zksolr1:8983/solr/my_collection1/select?q=*:*&indent=true&rows=100&fl=id' | egrep '<str name=.id.>' | wc -l"
11
mak@trinity10:~$ docker exec -it --user=solr zksolr1 bash -c "wget -q -O - 'http://zksolr1:8983/solr/my_collection1/select?q=*:*&shards=shard1&rows=100&indent=true&fl=id' | grep '<str name=.id.>' | wc -l"
4
mak@trinity10:~$ docker exec -it --user=solr zksolr1 bash -c "wget -q -O - 'http://zksolr1:8983/solr/my_collection1/select?q=*:*&shards=shard2&rows=100&indent=true&fl=id' | grep '<str name=.id.>' | wc -l"
7
```
Now to get external access to this overlay network from outside we can use a container to proxy the connections.
For a simple TCP proxy container with an exposed port on the Docker host, proxying to a single Solr node, you can use [brandnetworks/tcpproxy](https://github.com/brandnetworks/tcpproxy):
```
ssh -n trinity10.lan "docker pull brandnetworks/tcpproxy && docker run -p 8001 -p 8002 --net netzksolr --name zksolrproxy --hostname=zksolrproxy.netzksolr -tid brandnetworks/tcpproxy --connections 8002:zksolr1:8983"
docker port zksolrproxy 8002
```
Or use a suitably configured HAProxy to round-robin between all Solr nodes. Or, instead of the overlay network, use [Project Calico](http://www.projectcalico.org) and configure L3 routing so you do not need to mess with proxies.
Now I can get to Solr on http://trinity10:32774/solr/#/. In the Cloud -> Tree -> /live_nodes view I see the Solr nodes.
From the Solr UI select the collection1 core, and click on Cloud -> Graph to see how it has created
two shards across our Solr nodes.
Now, by way of test, we'll stop the Solr containers, and start them out-of-order, and verify the IP addresses are unchanged, and check the same results come back:
```
ssh -n $ZKSOLR1_HOST docker kill zksolr1
ssh -n $ZKSOLR2_HOST docker kill zksolr2
ssh -n $ZKSOLR3_HOST docker kill zksolr3
ssh -n $ZKSOLR1_HOST docker start zksolr1
sleep 3
ssh -n $ZKSOLR3_HOST docker start zksolr3
sleep 3
ssh -n $ZKSOLR2_HOST docker start zksolr2
ssh -n $ZKSOLR1_HOST 'docker inspect --format "{{ .NetworkSettings.Networks.netzksolr.IPAddress }}" zksolr1'
ssh -n $ZKSOLR2_HOST 'docker inspect --format "{{ .NetworkSettings.Networks.netzksolr.IPAddress }}" zksolr2'
ssh -n $ZKSOLR3_HOST 'docker inspect --format "{{ .NetworkSettings.Networks.netzksolr.IPAddress }}" zksolr3'
docker exec -it --user=solr zksolr1 bash -c "wget -q -O - 'http://zksolr1:8983/solr/my_collection1/select?q=*:*&indent=true&rows=100&fl=id' | egrep '<str name=.id.>' | wc -l"
docker exec -it --user=solr zksolr1 bash -c "wget -q -O - 'http://zksolr1:8983/solr/my_collection1/select?q=*:*&shards=shard1&rows=100&indent=true&fl=id' | grep '<str name=.id.>' | wc -l"
docker exec -it --user=solr zksolr1 bash -c "wget -q -O - 'http://zksolr1:8983/solr/my_collection1/select?q=*:*&shards=shard2&rows=100&indent=true&fl=id' | grep '<str name=.id.>' | wc -l"
```
Good, that works.
Finally To clean up this example:
```
ssh -n $ZK1_HOST "docker kill zk1; docker rm zk1"
ssh -n $ZK2_HOST "docker kill zk2; docker rm zk2"
ssh -n $ZK3_HOST "docker kill zk3; docker rm zk3"
ssh -n $ZKSOLR1_HOST "docker kill zksolr1; docker rm zksolr1"
ssh -n $ZKSOLR2_HOST "docker kill zksolr2; docker rm zksolr2"
ssh -n $ZKSOLR3_HOST "docker kill zksolr3; docker rm zksolr3"
ssh -n trinity10.lan "docker kill zksolrproxy; docker rm zksolrproxy"
docker network rm netzksolr
```

20
solr/docker/docs/precreate-collection.sh Normal file
View File

@ -0,0 +1,20 @@
#!/bin/bash
#
# A script that creates a core by copying config before starting solr.
#
# To use this, map this file into your container's docker-entrypoint-initdb.d directory:
#
# docker run -d -P -v $PWD/precreate-collection.sh:/docker-entrypoint-initdb.d/precreate-collection.sh solr
CORE=${CORE:-gettingstarted}
if [[ -d "/opt/solr/server/solr/$CORE" ]]; then
echo "$CORE is already present on disk"
exit 0
fi
mkdir -p "/opt/solr/server/solr/$CORE/"
cd "/opt/solr/server/solr/$CORE" || exit
touch core.properties
# TODO: we may want a more minimal example here
cp -r /opt/solr/example/files/* .
echo created "/opt/solr/server/solr/$CORE"

16
solr/docker/docs/set-heap.sh Executable file
View File

@ -0,0 +1,16 @@
#!/bin/bash
#
# This script is mainly an illustration for the docker-entrypoint-initdb.d extension mechanism.
# Run it with e.g.:
#
# docker run -d -P -v $PWD/docs/set-heap.sh:/docker-entrypoint-initdb.d/set-heap.sh solr
#
# The SOLR_HEAP configuration technique here is usable for older versions of Solr.
# From Solr 6.3 setting the SOLR_HEAP can be done more easily with:
#
# docker run -d -P -e SOLR_HEAP=800m apache/solr
#
set -e
cp /opt/solr/bin/solr.in.sh /opt/solr/bin/solr.in.sh.orig
sed -e 's/SOLR_HEAP=".*"/SOLR_HEAP="1024m"/' </opt/solr/bin/solr.in.sh.orig >/opt/solr/bin/solr.in.sh
grep '^SOLR_HEAP=' /opt/solr/bin/solr.in.sh

31
solr/docker/include/scripts/docker-entrypoint.sh Executable file
View File

@ -0,0 +1,31 @@
#!/bin/bash
#
# docker-entrypoint for Solr docker
set -e
# Clear some variables that we don't want runtime
unset SOLR_USER SOLR_UID SOLR_GROUP SOLR_GID \
SOLR_CLOSER_URL SOLR_DIST_URL SOLR_ARCHIVE_URL SOLR_DOWNLOAD_URL SOLR_DOWNLOAD_SERVER SOLR_KEYS SOLR_SHA512
if [[ "$VERBOSE" == "yes" ]]; then
set -x
fi
if ! [[ ${SOLR_PORT:-} =~ ^[0-9]+$ ]]; then
SOLR_PORT=8983
export SOLR_PORT
fi
# when invoked with e.g.: docker run solr -help
if [ "${1:0:1}" == '-' ]; then
set -- solr-foreground "$@"
fi
# execute command passed in as arguments.
# The Dockerfile has specified the PATH to include
# /opt/solr/bin (for Solr) and /opt/docker-solr/scripts (for our scripts
# like solr-foreground, solr-create, solr-precreate, solr-demo).
# Note: if you specify "solr", you'll typically want to add -f to run it in
# the foreground.
exec "$@"

61
solr/docker/include/scripts/init-var-solr Executable file
View File

@ -0,0 +1,61 @@
#!/bin/bash
#
# A helper script to initialise an empty $DIR
# If you use volumes then Docker will copy the $DIR content from the container to the volume.
# If you use bind mounts, that does not happen, so we do it here.
set -e
if [[ "$VERBOSE" == "yes" ]]; then
set -x
fi
if [[ -n "${NO_INIT_VAR_SOLR:-}" ]]; then
exit 0
fi
DIR=${1:-/var/solr}
if [ ! -d "$DIR" ]; then
echo "Missing $DIR"
exit 1
fi
function check_dir_writability {
local dir="$1"
if [ ! -w "$dir" ]; then
echo "Cannot write to $dir as $(id -u):$(id -g)"
ls -ld "$dir"
exit 1
fi
}
if [ ! -d "$DIR/data" ]; then
echo "Creating $DIR/data"
check_dir_writability "$DIR"
mkdir "$DIR/data"
chmod 0770 "$DIR/data"
fi
if [ ! -d "$DIR/logs" ]; then
echo "Creating $DIR/logs"
check_dir_writability "$DIR"
mkdir "$DIR/logs"
chmod 0770 "$DIR/logs"
fi
if [ ! -f "$DIR/data/solr.xml" ]; then
echo "Copying solr.xml"
cp -a /opt/solr/server/solr/solr.xml "$DIR/data/solr.xml"
fi
if [ ! -f "$DIR/data/zoo.cfg" ]; then
echo "Copying zoo.cfg"
cp -a /opt/solr/server/solr/zoo.cfg "$DIR/data/zoo.cfg"
fi
if [ ! -f "$DIR/log4j2.xml" ]; then
echo "Copying log4j2.xml"
cp -a /opt/solr/server/resources/log4j2.xml "$DIR/log4j2.xml"
fi

38
solr/docker/include/scripts/oom_solr.sh Executable file
View File

@ -0,0 +1,38 @@
#!/bin/bash
# Custom oom handler loosely based on
# https://github.com/apache/lucene-solr/blob/master/solr/bin/oom_solr.sh
# See solr-forgeground for how to configure OOM behaviour
if [[ -z "${SOLR_LOGS_DIR:-}" ]]; then
if [ -d /var/solr/logs ]; then
SOLR_LOGS_DIR=/var/solr/logs
elif [ -d /opt/solr/server/logs ]; then
SOLR_LOGS_DIR=/opt/solr/server/logs
else
echo "Cannot determine SOLR_LOGS_DIR!"
exit 1
fi
fi
SOLR_PID=$(pgrep -f start.jar)
if [[ -z "$SOLR_PID" ]]; then
echo "Couldn't find Solr process running!"
exit
fi
NOW=$(date +"%F_%H_%M_%S")
(
echo "Running OOM killer script for Solr process $SOLR_PID"
if [[ "$SOLR_PID" == 1 ]]; then
# under Docker, when running as pid 1, a SIGKILL is ignored,
# so use the default SIGTERM
kill "$SOLR_PID"
sleep 2
# if that hasn't worked, send SIGKILL
kill -SIGILL "$SOLR_PID"
else
# if we're running with `--init` or under tini or similar,
# follow the upstream behaviour
kill -9 "$SOLR_PID"
fi
) | tee "$SOLR_LOGS_DIR/solr_oom_killer-$SOLR_PORT-$NOW.log"

40
solr/docker/include/scripts/precreate-core Executable file
View File

@ -0,0 +1,40 @@
#!/bin/bash
#
# Create a core on disk
# arguments are: corename configdir
set -e
echo "Executing $0" "$@"
if [[ "${VERBOSE:-}" == "yes" ]]; then
set -x
fi
CORE=${1:-gettingstarted}
CONFIG_SOURCE="${2:-}"
if [[ -z "$CONFIG_SOURCE" ]]; then
DEFAULT_CONFIGS=(_default data_driven_schema_configs)
for config_dir in "${DEFAULT_CONFIGS[@]}"; do
config_dir="/opt/solr/server/solr/configsets/$config_dir"
if [ -d "$config_dir" ]; then
CONFIG_SOURCE="$config_dir"
break
fi
done
if [[ -z $CONFIG_SOURCE ]]; then
echo "Cannot find default config"
exit 1
fi
fi
coresdir=/var/solr/data
coredir="$coresdir/$CORE"
if [[ ! -d $coredir ]]; then
cp -r "$CONFIG_SOURCE/" "$coredir"
touch "$coredir/core.properties"
echo "Created $CORE"
else
echo "Core $CORE already exists"
fi

28
solr/docker/include/scripts/run-initdb Executable file
View File

@ -0,0 +1,28 @@
#!/bin/bash
#
# Run the init-solr-home script and source any '.sh' scripts in
# /docker-entrypoint-initdb.d.
# This script is sourced by some of the solr-* commands, so that
# you can run eg:
#
# mkdir initdb; echo "echo hi" > initdb/hi.sh
# docker run -v $PWD/initdb:/docker-entrypoint-initdb.d solr
#
# and have your script execute before Solr starts.
#
# Note: scripts can modify the environment, which will affect
# subsequent scripts and ultimately Solr. That allows you to set
# environment variables from your scripts (though you usually just
# use "docker run -e"). If this is undesirable in your use-case,
# have your scripts execute a sub-shell.
set -e
# execute files in /docker-entrypoint-initdb.d before starting solr
while read -r f; do
case "$f" in
*.sh) echo "$0: running $f"; . "$f" ;;
*) echo "$0: ignoring $f" ;;
esac
echo
done < <(find /docker-entrypoint-initdb.d/ -mindepth 1 -type f | sort -n)

66
solr/docker/include/scripts/solr-create Executable file
View File

@ -0,0 +1,66 @@
#!/bin/bash
#
# This script starts Solr on localhost, creates a core with "solr create",
# stops Solr, and then starts Solr as normal.
# Any arguments are passed to the "solr create".
# To simply create a core:
# docker run -P -d solr solr-create -c mycore
# To create a core from mounted config:
# docker run -P -d -v $PWD/myconfig:/myconfig solr solr-create -c mycore -d /myconfig
set -euo pipefail
echo "Executing $0" "$@"
if [[ "${VERBOSE:-}" == "yes" ]]; then
set -x
fi
# init script for handling an empty /var/solr
/opt/docker-solr/scripts/init-var-solr
. /opt/docker-solr/scripts/run-initdb
# solr uses "-c corename". Parse the arguments to determine the core name.
CORE_NAME="$(
while (( $# > 0 )); do
if [[ "$1" == '-c' ]]; then
shift
echo "$1"
fi
shift
done
)"
if [[ -z "${CORE_NAME:-}" ]]; then
echo "Could not determine core name"
exit 1
fi
coresdir=/var/solr/data
CORE_DIR="$coresdir/$CORE_NAME"
if [[ -d $CORE_DIR ]]; then
echo "Directory $CORE_DIR exists; skipping core creation"
else
start-local-solr
echo "Creating core with:" "${@:1}"
/opt/solr/bin/solr create "${@:1}"
# See https://github.com/docker-solr/docker-solr/issues/27
echo "Checking core"
if ! wget -O - "http://localhost:${SOLR_PORT:-8983}/solr/admin/cores?action=STATUS" | grep instanceDir >/dev/null; then
echo "Could not find any cores"
exit 1
fi
echo "Created core with:" "${@:1}"
stop-local-solr
# check the core_dir exists; otherwise the detecting above will fail after stop/start
if [ ! -d "$CORE_DIR" ]; then
echo "Missing $CORE_DIR"
exit 1
fi
fi
exec solr-fg

42
solr/docker/include/scripts/solr-demo Executable file
View File

@ -0,0 +1,42 @@
#!/bin/bash
#
# Configure a Solr demo and then run solr in the foreground
set -euo pipefail
if [[ "${VERBOSE:-}" == "yes" ]]; then
set -x
fi
. /opt/docker-solr/scripts/run-initdb
CORE=demo
coresdir=/var/solr/data
CORE_DIR="$coresdir/demo"
if [ -d "$CORE_DIR" ]; then
echo "$CORE_DIR exists; skipping demo creation"
else
start-local-solr
echo "Creating $CORE"
/opt/solr/bin/solr create -c "$CORE"
echo "Created $CORE"
echo "Loading example data"
post_args=()
if [[ -n "${SOLR_PORT:-}" ]]; then
post_args+=(-p "$SOLR_PORT")
fi
/opt/solr/bin/post "${post_args[@]}" -c $CORE -commit no example/exampledocs/*.xml
/opt/solr/bin/post "${post_args[@]}" -c $CORE -commit no example/exampledocs/books.json
/opt/solr/bin/post "${post_args[@]}" -c $CORE -commit yes example/exampledocs/books.csv
echo "Loaded example data"
stop-local-solr
# check the core_dir exists; otherwise the detecting above will fail after stop/start
if [ ! -d "$CORE_DIR" ]; then
echo "Missing $CORE_DIR"
exit 1
fi
fi
exec solr-fg

56
solr/docker/include/scripts/solr-fg Executable file
View File

@ -0,0 +1,56 @@
#!/bin/bash
#
# start solr in the foreground
set -e
if [[ "$VERBOSE" == "yes" ]]; then
set -x
fi
EXTRA_ARGS=()
# Allow easy setting of the OOM behaviour
# Test with: docker run -p 8983:8983 -it -e OOM=script -e SOLR_JAVA_MEM="-Xms25m -Xmx25m" solr
if [[ -z "${OOM:-}" ]]; then
OOM='none'
fi
case "$OOM" in
'script')
EXTRA_ARGS+=(-a '-XX:OnOutOfMemoryError=/opt/docker-solr/scripts/oom_solr.sh')
;;
'exit')
# recommended
EXTRA_ARGS+=(-a '-XX:+ExitOnOutOfMemoryError')
;;
'crash')
EXTRA_ARGS+=(-a '-XX:+CrashOnOutOfMemoryError')
;;
'none'|'')
;;
*)
echo "Unsupported value in OOM=$OOM"
exit 1
esac
echo "Starting Solr $SOLR_VERSION"
# determine TINI default. If it is already set, assume the user knows what they want
if [[ -z "${TINI:-}" ]]; then
if [[ "$$" == 1 ]]; then
# Default to running tini, so we can run with an OOM script and have 'kill -9' work
TINI=yes
else
# Presumably we're already running under tini through 'docker --init', in which case we
# don't need to run it twice.
# It's also possible that we're run from a wrapper script without exec,
# in which case running tini would not be ideal either.
TINI=no
fi
fi
if [[ "$TINI" == yes ]]; then
exec tini -- solr -f "$@" "${EXTRA_ARGS[@]}"
elif [[ "$TINI" == no ]]; then
exec solr -f "$@" "${EXTRA_ARGS[@]}"
else
echo "invalid value TINI=$TINI"
exit 1
fi

15
solr/docker/include/scripts/solr-foreground Executable file
View File

@ -0,0 +1,15 @@
#!/bin/bash
#
# Run the initdb, then start solr in the foreground
set -e
if [[ "$VERBOSE" == "yes" ]]; then
set -x
fi
# init script for handling an empty /var/solr
/opt/docker-solr/scripts/init-var-solr
. /opt/docker-solr/scripts/run-initdb
exec solr-fg "$@"

27
solr/docker/include/scripts/solr-precreate Executable file
View File

@ -0,0 +1,27 @@
#!/bin/bash
#
# Create a core on disk and then run solr in the foreground
# arguments are: corename configdir
# To simply create a core:
# docker run -P -d solr solr-precreate mycore
# To create a core from mounted config:
# docker run -P -d -v $PWD/myconfig:/myconfig solr solr-precreate mycore /myconfig
# To create a core in a mounted directory:
# mkdir myvarsolr; chown 8983:8983 myvarsolr
# docker run -it --rm -P -v $PWD/myvarsolr://var/solr solr solr-precreate mycore
set -e
echo "Executing $0" "$@"
if [[ "${VERBOSE:-}" == "yes" ]]; then
set -x
fi
# init script for handling an empty /var/solr
/opt/docker-solr/scripts/init-var-solr
. /opt/docker-solr/scripts/run-initdb
/opt/docker-solr/scripts/precreate-core "$@"
exec solr-fg

21
solr/docker/include/scripts/start-local-solr Executable file
View File

@ -0,0 +1,21 @@
#!/bin/bash
# configure Solr to run on the local interface, and start it running in the background
set -euo pipefail
if [[ "${VERBOSE:-}" == "yes" ]]; then
set -x
fi
echo "Running solr in the background. Logs are in /var/solr/logs"
SOLR_OPTS="-Djetty.host=${SOLR_LOCAL_HOST:-localhost}" solr start
max_try=${MAX_TRY:-12}
wait_seconds=${WAIT_SECONDS:-5}
if ! /opt/docker-solr/scripts/wait-for-solr.sh --max-attempts "$max_try" --wait-seconds "$wait_seconds"; then
echo "Could not start Solr."
if [ -f "/var/solr/logs/solr.log" ]; then
echo "Here is the log:"
cat "/var/solr/logs/solr.log"
fi
exit 1
fi

11
solr/docker/include/scripts/stop-local-solr Executable file
View File

@ -0,0 +1,11 @@
#!/bin/bash
# stop the background Solr, and restore the normal configuration
set -e
if [[ "$VERBOSE" == "yes" ]]; then
set -x
fi
echo "Shutting down the background Solr"
solr stop

98
solr/docker/include/scripts/wait-for-solr.sh Executable file
View File

@ -0,0 +1,98 @@
#!/bin/bash
#
# A helper script to wait for solr
#
# Usage: wait-for-solr.sh [--max-attempts count] [--wait-seconds seconds] [--solr-url url]
# Deprecated usage: wait-for-solr.sh [ max_attempts [ wait_seconds ] ]
set -euo pipefail
SCRIPT="$0"
if [[ "${VERBOSE:-}" == "yes" ]]; then
set -x
fi
function usage {
echo "$1"
echo "Usage: $SCRIPT [--max-attempts count] [--wait-seconds seconds ] [--solr-url url]"
exit 1
}
max_attempts=12
wait_seconds=5
if ! [[ ${SOLR_PORT:-8983} =~ ^[0-9]+$ ]]; then
echo "Invalid SOLR_PORT=${SOLR_PORT:-} environment variable specified"
exit 1
fi
solr_url="http://localhost:${SOLR_PORT:-8983}"
while (( $# > 0 )); do
case "$1" in
--help)
cat <<EOM
Usage: $SCRIPT [options]
Options:
--max-attempts count: number of attempts to check Solr is up. Default: $max_attempts
--wait-seconds seconds: number of seconds to wait between attempts. Default: $wait_seconds
--solr-url url: URL for Solr server to check. Default: $solr_url
EOM
exit 0
;;
--solr-url)
solr_url="$2";
shift 2
;;
--max-attempts)
max_attempts="$2";
shift 2;
;;
--wait-seconds)
wait_seconds="$2";
shift 2;
;;
* )
# deprecated invocation, kept for backwards compatibility
max_attempts=$1;
wait_seconds=$2;
echo "WARNING: deprecated invocation. Use $SCRIPT [--max-attempts count] [--wait-seconds seconds]"
shift 2;
break;
;;
esac
done
grep -q -E '^[0-9]+$' <<<"$max_attempts" || usage "--max-attempts $max_attempts: not a number"
if (( max_attempts == 0 )); then
echo "The --max-attempts argument should be >0"
exit 1
fi
grep -q -E '^[0-9]+$' <<<"$wait_seconds" || usage "--wait-seconds $wait_seconds: not a number"
grep -q -E '^https?://' <<<"$solr_url" || usage "--solr-url $solr_url: not a URL"
((attempts_left=max_attempts))
while (( attempts_left > 0 )); do
if wget -q -O - "$solr_url" | grep -i solr >/dev/null; then
break
fi
(( attempts_left-- ))
if (( attempts_left == 0 )); then
echo "Solr is still not running; giving up"
exit 1
fi
if (( attempts_left == 1 )); then
attempts=attempt
else
attempts=attempts
fi
echo "Solr is not running yet on $solr_url. $attempts_left $attempts left"
sleep "$wait_seconds"
done
echo "Solr is running on $solr_url"

165
solr/docker/include/scripts/wait-for-zookeeper.sh Executable file
View File

@ -0,0 +1,165 @@
#!/bin/bash
#
# A helper script to wait for ZooKeeper
#
# This script waits for a ZooKeeper master to appear.
# It repeatedly looks up the name passed as argument
# in the DNS using getent, and then connects to the
# ZooKeeper admin port and uses the 'srvr' command to
# obtain the server's status.
# You can use this in a Kubernetes init container to
# delay Solr pods starting until the ZooKeeper service
# has settled down. Or you could explicitly run this in
# the Solr container before exec'ing Solr.
#
# Inspired by https://github.com/helm/charts/blob/9eba7b1c80990233a68dce48f4a8fe0baf9b7fa5/incubator/solr/templates/statefulset.yaml#L60
#
# Usage: wait-for-zookeeper.sh [--max-attempts count] [--wait-seconds seconds] zookeeper-service-name
#
# If no argument is provided, but a Solr-style ZK_HOST is set,
# that will be used. If neither is provided, the default
# name is 'solr-zookeeper-headless', to match the helm chart.
set -euo pipefail
SCRIPT="$0"
if [[ "${VERBOSE:-}" == "yes" ]]; then
set -x
fi
function usage {
echo "$1"
echo "Usage: $SCRIPT [--max-attempts count] [--wait-seconds seconds ] zookeeper-service-name"
exit 1
}
TMP_HOSTS="/tmp/hosts.$$"
TMP_STATUS="/tmp/status.$$"
function cleanup {
rm -f $TMP_HOSTS $TMP_STATUS
}
trap cleanup EXIT
function check_zookeeper {
local host=$1
local port="${2:-2181}"
if ! echo srvr | nc "$host" "$port" > $TMP_STATUS; then
echo "Failed to get status from $host"
return
fi
if [ ! -s $TMP_STATUS ]; then
echo "No data from $ip"
return
fi
if grep -q 'not currently serving requests' $TMP_STATUS; then
echo "Node $ip is not currently serving requests"
return
fi
mode=$(grep "Mode: " $TMP_STATUS | sed 's/Mode: //');
if [ -z "$mode" ]; then
echo "Cannot determine mode from:"
cat $TMP_STATUS
return
fi
echo "Node $ip is a $mode"
if [ "$mode" = "leader" ] || [ "$mode" = "standalone" ]; then
echo "Done"
exit 0
fi
}
max_attempts=120
wait_seconds=2
while (( $# > 0 )); do
case "$1" in
--help)
cat <<EOM
Usage: $SCRIPT [options] zookeeper-service-name
Options:
--max-attempts count: number of attempts to check Solr is up. Default: $max_attempts
--wait-seconds seconds: number of seconds to wait between attempts. Default: $wait_seconds
EOM
exit 0
;;
--max-attempts)
max_attempts="$2";
shift 2;
;;
--wait-seconds)
wait_seconds="$2";
shift 2;
;;
*)
if [ -n "${lookup_arg:-}" ]; then
usage "Cannot specify multiple zookeeper service names"
fi
lookup_arg=$1;
shift;
break;
;;
esac
done
grep -q -E '^[0-9]+$' <<<"$max_attempts" || usage "--max-attempts $max_attempts: not a number"
if (( max_attempts == 0 )); then
echo "The --max-attempts argument should be >0"
exit 1
fi
grep -q -E '^[0-9]+$' <<<"$wait_seconds" || usage "--wait-seconds $wait_seconds: not a number"
if [ -z "${lookup_arg:-}" ]; then
if [ -n "$ZK_HOST" ]; then
lookup_arg="$ZK_HOST"
else
lookup_arg=solr-zookeeper-headless
fi
fi
echo "Looking up '$lookup_arg'"
# split on commas, for when a ZK_HOST string like zoo1:2181,zoo2:2181 is used
IFS=',' read -ra lookups <<< "$lookup_arg"
((attempts_left=max_attempts))
while (( attempts_left > 0 )); do
for lookup in "${lookups[@]}"; do
if grep -q -E "^\[[0-9].*\]" <<<"$lookup"; then
# looks like an IPv6 address, eg [2001:DB8::1] or [2001:DB8::1]:2181
# getent does not support the bracket notation, but does support IPv6 addresses
host=$(sed -E 's/\[(.*)\].*/\1/' <<<"$lookup")
port=$(sed -E 's/^\[(.*)\]:?//' <<<"$lookup")
else
# IPv4, just split on :
IFS=: read -ra split <<<"$lookup"
host="${split[0]}"
port="${split[1]:-}"
fi
if [[ "${VERBOSE:-}" == "yes" ]]; then
echo "Parsed host=$host port=${port:-}"
fi
if getent hosts "$host" > $TMP_HOSTS; then
while read -r ip hostname ; do
echo "${hostname:-}">/dev/null # consume for shellcheck
check_zookeeper "$ip" "$port"
done <$TMP_HOSTS
else
echo "Cannot find $lookup yet"
fi
done
(( attempts_left-- ))
if (( attempts_left == 0 )); then echo "Still no master found; giving up"
exit 1
fi
sleep "$wait_seconds"
done
# To test the parsing:
# bash scripts/wait-for-zookeeper.sh foo
# bash scripts/wait-for-zookeeper.sh 'ZK_HOST=[2001:DB8::1]:2181,[2001:DB8::1],127.0.0.1:2181,127.0.0.2'
# ZK_HOST=[2001:DB8::1]:2181,[2001:DB8::1],127.0.0.1:2181,127.0.0.2 bash scripts/wait-for-zookeeper.sh

3
solr/docker/package/Dockerfile.local-package Normal file
View File

@ -0,0 +1,3 @@
FROM scratch
COPY releases/ /opt/

74
solr/docker/package/Dockerfile.release-package Normal file
View File

@ -0,0 +1,74 @@
ARG BASE_IMAGE=openjdk:11-jre
FROM $BASE_IMAGE as downloader
ARG SOLR_VERSION
ARG SOLR_SHA512
ARG SOLR_KEYS
# If specified, this will override SOLR_DOWNLOAD_SERVER and all ASF mirrors. Typically used downstream for custom builds
ARG SOLR_DOWNLOAD_URL
# Override the solr download location with e.g.:
# docker build -t mine --build-arg SOLR_DOWNLOAD_SERVER=http://www-eu.apache.org/dist/lucene/solr .
ARG SOLR_DOWNLOAD_SERVER
# This is only applicable when SOLR_DOWNLOAD_URL is not provided. Skips the GPG check for Solr downloads.
ARG SKIP_GPG_CHECK="true"
ENV SOLR_CLOSER_URL="http://www.apache.org/dyn/closer.lua?filename=lucene/solr/$SOLR_VERSION/solr-$SOLR_VERSION.tgz&action=download" \
SOLR_DIST_URL="https://www.apache.org/dist/lucene/solr/$SOLR_VERSION/solr-$SOLR_VERSION.tgz" \
SOLR_ARCHIVE_URL="https://archive.apache.org/dist/lucene/solr/$SOLR_VERSION/solr-$SOLR_VERSION.tgz"
RUN set -ex; \
apt-get update; \
apt-get -y install dirmngr gpg wget; \
rm -rf /var/lib/apt/lists/*;
RUN set -ex; \
export GNUPGHOME="/tmp/gnupg_home"; \
mkdir -p "$GNUPGHOME"; \
chmod 700 "$GNUPGHOME"; \
echo "disable-ipv6" >> "$GNUPGHOME/dirmngr.conf"; \
for key in $SOLR_KEYS; do \
found=''; \
for server in \
ha.pool.sks-keyservers.net \
hkp://keyserver.ubuntu.com:80 \
hkp://p80.pool.sks-keyservers.net:80 \
pgp.mit.edu \
; do \
echo " trying $server for $key"; \
gpg --batch --keyserver "$server" --keyserver-options timeout=10 --recv-keys "$key" && found=yes && break; \
gpg --batch --keyserver "$server" --keyserver-options timeout=10 --recv-keys "$key" && found=yes && break; \
done; \
test -z "$found" && echo >&2 "error: failed to fetch $key from several disparate servers -- network issues?" && exit 1; \
done; \
exit 0
RUN set -ex; \
export GNUPGHOME="/tmp/gnupg_home"; \
MAX_REDIRECTS=1; \
if [ -n "$SOLR_DOWNLOAD_URL" ]; then \
# If a custom URL is defined, we download from non-ASF mirror URL and allow more redirects and skip GPG step
# This takes effect only if the SOLR_DOWNLOAD_URL build-arg is specified, typically in downstream Dockerfiles
MAX_REDIRECTS=4; \
SKIP_GPG_CHECK="true"; \
elif [ -n "$SOLR_DOWNLOAD_SERVER" ]; then \
SOLR_DOWNLOAD_URL="$SOLR_DOWNLOAD_SERVER/$SOLR_VERSION/solr-$SOLR_VERSION.tgz"; \
fi; \
for url in $SOLR_DOWNLOAD_URL $SOLR_CLOSER_URL $SOLR_DIST_URL $SOLR_ARCHIVE_URL; do \
if [ -f "/opt/solr-$SOLR_VERSION.tgz" ]; then break; fi; \
echo "downloading $url"; \
if wget -t 10 --max-redirect $MAX_REDIRECTS --retry-connrefused -nv "$url" -O "/opt/solr-$SOLR_VERSION.tgz"; then break; else rm -f "/opt/solr-$SOLR_VERSION.tgz"; fi; \
done; \
if [ ! -f "/opt/solr-$SOLR_VERSION.tgz" ]; then echo "failed all download attempts for solr-$SOLR_VERSION.tgz"; exit 1; fi; \
if [ "$SKIP_GPG_CHECK" != "true" ]; then \
echo "downloading $SOLR_ARCHIVE_URL.asc"; \
wget -nv "$SOLR_ARCHIVE_URL.asc" -O "/opt/solr-$SOLR_VERSION.tgz.asc"; \
echo "$SOLR_SHA512 */opt/solr-$SOLR_VERSION.tgz" | sha512sum -c -; \
(>&2 ls -l "/opt/solr-$SOLR_VERSION.tgz" "/opt/solr-$SOLR_VERSION.tgz.asc"); \
gpg --batch --verify "/opt/solr-$SOLR_VERSION.tgz.asc" "/opt/solr-$SOLR_VERSION.tgz"; \
else \
echo "Skipping GPG validation due to non-Apache build"; \
fi; \
{ command -v gpgconf; gpgconf --kill all || :; }; \
rm -r "$GNUPGHOME";

36
solr/docker/package/build.gradle Normal file
View File

@ -0,0 +1,36 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
description = 'Solr Docker Package image'
// The solr package docker image relies on the output of the solr:packaging project.
Project solrPackaging = project(':solr:packaging')
dependencies {
docker solrPackaging
}
docker {
name = 'apache/solr-build:local-package'
dockerfile file('Dockerfile.local-package')
files(solrPackaging.tasks.distTar.outputs)
getCopySpec().into('releases')
}
// Only allow the following docker tasks
def availableDockerTasks = ["docker", "dockerClean", "dockerPrepare", "dockerfileZip"]
project.tasks.configureEach { t -> t.enabled = t.getGroup() != "Docker" || availableDockerTasks.contains(t.getName()) }

6
solr/docker/tests/README.md Normal file
View File

@ -0,0 +1,6 @@
To run tests, specify a fully-versioned image tag:
```
./test.sh apache/solr:6.6.0
```

16
solr/docker/tests/cases/create_bad_core/test.sh Executable file
View File

@ -0,0 +1,16 @@
#!/bin/bash
#
set -euo pipefail
TEST_DIR="${TEST_DIR:-$(dirname -- "${BASH_SOURCE[0]}")}"
source "${TEST_DIR}/../../shared.sh"
echo "Running $container_name"
if docker run --name "$container_name" "$tag" solr-create -c 'bad/core?name:here'; then
echo "Bad core creation did not return a failure"
exit 1
fi
container_cleanup "$container_name"
echo "Test $TEST_NAME $tag succeeded"

24
solr/docker/tests/cases/create_core/test.sh Executable file
View File

@ -0,0 +1,24 @@
#!/bin/bash
#
set -euo pipefail
TEST_DIR="${TEST_DIR:-$(dirname -- "${BASH_SOURCE[0]}")}"
source "${TEST_DIR}/../../shared.sh"
echo "Running $container_name"
docker run --name "$container_name" -d "$tag" solr-create -c gettingstarted
wait_for_container_and_solr "$container_name"
echo "Loading data"
docker exec --user=solr "$container_name" bin/post -c gettingstarted example/exampledocs/manufacturers.xml
sleep 1
echo "Checking data"
data=$(docker exec --user=solr "$container_name" wget -q -O - 'http://localhost:8983/solr/gettingstarted/select?q=id%3Adell')
if ! grep -E -q 'One Dell Way Round Rock, Texas 78682' <<<"$data"; then
echo "Test $TEST_NAME $tag failed; data did not load"
exit 1
fi
container_cleanup "$container_name"
echo "Test $TEST_NAME $tag succeeded"

26
solr/docker/tests/cases/create_core_exec/test.sh Executable file
View File

@ -0,0 +1,26 @@
#!/bin/bash
#
set -euo pipefail
TEST_DIR="${TEST_DIR:-$(dirname -- "${BASH_SOURCE[0]}")}"
source "${TEST_DIR}/../../shared.sh"
echo "Running $container_name"
docker run --name "$container_name" -d "$tag"
wait_for_container_and_solr "$container_name"
echo "Creating core"
docker exec --user=solr "$container_name" bin/solr create_core -c gettingstarted
echo "Loading data"
docker exec --user=solr "$container_name" bin/post -c gettingstarted example/exampledocs/manufacturers.xml
sleep 1
echo "Checking data"
data=$(docker exec --user=solr "$container_name" wget -q -O - 'http://localhost:8983/solr/gettingstarted/select?q=id%3Adell')
if ! grep -E -q 'One Dell Way Round Rock, Texas 78682' <<<"$data"; then
echo "Test $TEST_NAME $tag failed; data did not load"
exit 1
fi
container_cleanup "$container_name"
echo "Test $TEST_NAME $tag succeeded"

30
solr/docker/tests/cases/create_core_randomuser_rootgroup/test.sh Executable file
View File

@ -0,0 +1,30 @@
#!/bin/bash
#
# Simulate openshift by running with a random uid
#
set -euo pipefail
TEST_DIR="${TEST_DIR:-$(dirname -- "${BASH_SOURCE[0]}")}"
source "${TEST_DIR}/../../shared.sh"
echo "Running $container_name"
docker run --user 7777:0 --name "$container_name" -d "$tag" solr-create -c gettingstarted
wait_for_container_and_solr "$container_name"
echo "Loading data"
docker exec --user=solr "$container_name" bin/post -c gettingstarted example/exampledocs/manufacturers.xml
sleep 1
echo "Checking data"
data=$(docker exec --user=solr "$container_name" wget -q -O - 'http://localhost:8983/solr/gettingstarted/select?q=id%3Adell')
if ! grep -E -q 'One Dell Way Round Rock, Texas 78682' <<<"$data"; then
echo "Test $TEST_NAME $tag failed; data did not load"
exit 1
fi
docker exec --user=root "$container_name" ls -lR /var/solr
container_cleanup "$container_name"
echo "Test $TEST_NAME $tag succeeded"

25
solr/docker/tests/cases/demo-tini/test.sh Executable file
View File

@ -0,0 +1,25 @@
#!/bin/bash
#
set -euo pipefail
TEST_DIR="${TEST_DIR:-$(dirname -- "${BASH_SOURCE[0]}")}"
source "${TEST_DIR}/../../shared.sh"
echo "Running $container_name"
docker run --name "$container_name" -d -e TINI=yes "$tag" "solr-demo"
wait_for_container_and_solr "$container_name"
echo "Checking data"
data=$(docker exec --user=solr "$container_name" wget -q -O - 'http://localhost:8983/solr/demo/select?q=id%3Adell')
if ! grep -E -q 'One Dell Way Round Rock, Texas 78682' <<<"$data"; then
echo "Test $TEST_NAME $tag failed; data did not load"
exit 1
fi
data=$(docker exec --user=solr "$container_name" pgrep tini)
echo "$data"
container_cleanup "$container_name"
echo "Test $TEST_NAME $tag succeeded"

21
solr/docker/tests/cases/demo/test.sh Executable file
View File

@ -0,0 +1,21 @@
#!/bin/bash
#
set -euo pipefail
TEST_DIR="${TEST_DIR:-$(dirname -- "${BASH_SOURCE[0]}")}"
source "${TEST_DIR}/../../shared.sh"
echo "Running $container_name"
docker run --name "$container_name" -d "$tag" "solr-demo"
wait_for_container_and_solr "$container_name"
echo "Checking data"
data=$(docker exec --user=solr "$container_name" wget -q -O - 'http://localhost:8983/solr/demo/select?q=id%3Adell')
if ! grep -E -q 'One Dell Way Round Rock, Texas 78682' <<<"$data"; then
echo "Test $TEST_NAME $tag failed; data did not load"
exit 1
fi
container_cleanup "$container_name"
echo "Test $TEST_NAME $tag succeeded"

44
solr/docker/tests/cases/empty-varsolr-dir-ramdomuser-rootgroup/test.sh Executable file
View File

@ -0,0 +1,44 @@
#!/bin/bash
#
set -euo pipefail
TEST_DIR="${TEST_DIR:-$(dirname -- "${BASH_SOURCE[0]}")}"
source "${TEST_DIR}/../../shared.sh"
container_cleanup "$container_name-copier"
myvarsolr="${BUILD_DIR}/myvarsolr-${container_name}"
prepare_dir_to_mount 7777 "$myvarsolr"
echo "Running $container_name"
docker run \
--user 7777:0 \
-v "$myvarsolr:/var/solr" \
--name "$container_name" \
-d "$tag" solr-precreate getting-started
wait_for_container_and_solr "$container_name"
echo "Loading data"
docker exec --user=solr "$container_name" bin/post -c getting-started example/exampledocs/manufacturers.xml
sleep 1
echo "Checking data"
data=$(docker exec --user=solr "$container_name" wget -q -O - 'http://localhost:8983/solr/getting-started/select?q=id%3Adell')
if ! grep -E -q 'One Dell Way Round Rock, Texas 78682' <<<"$data"; then
echo "Test $TEST_NAME $tag failed; data did not load"
exit 1
fi
docker exec --user=7777 "$container_name" ls -l /var/solr/data
container_cleanup "$container_name"
# remove the solr-owned files from inside a container
docker run --rm -e VERBOSE=yes \
--user root \
-v "$myvarsolr:/myvarsolr" "$tag" \
bash -c "rm -fr /myvarsolr/*"
rm -fr "$myvarsolr"
echo "Test $TEST_NAME $tag succeeded"

45
solr/docker/tests/cases/empty-varsolr-dir-solr/test.sh Executable file
View File

@ -0,0 +1,45 @@
#!/bin/bash
#
set -euo pipefail
TEST_DIR="${TEST_DIR:-$(dirname -- "${BASH_SOURCE[0]}")}"
source "${TEST_DIR}/../../shared.sh"
container_cleanup "$container_name-copier"
myvarsolr="${BUILD_DIR}/myvarsolr-${container_name}"
prepare_dir_to_mount 8983 "$myvarsolr"
echo "Running $container_name"
docker run \
-v "$myvarsolr:/var/solr" \
--name "$container_name" \
-d "$tag" solr-precreate getting-started
wait_for_container_and_solr "$container_name"
echo "Loading data"
docker exec --user=solr "$container_name" bin/post -c getting-started example/exampledocs/manufacturers.xml
sleep 1
echo "Checking data"
data=$(docker exec --user=solr "$container_name" wget -q -O - 'http://localhost:8983/solr/getting-started/select?q=id%3Adell')
if ! grep -E -q 'One Dell Way Round Rock, Texas 78682' <<<"$data"; then
echo "Test $TEST_NAME $tag failed; data did not load"
exit 1
fi
docker exec --user=solr "$container_name" ls -l /var/solr/data
container_cleanup "$container_name"
ls -l "$myvarsolr"/
# remove the solr-owned files from inside a container
docker run --rm -e VERBOSE=yes \
-v "$myvarsolr:/myvarsolr" "$tag" \
bash -c "rm -fr /myvarsolr/*"
ls -l "$myvarsolr/"
rm -fr "$myvarsolr"
echo "Test $TEST_NAME $tag succeeded"

42
solr/docker/tests/cases/empty-varsolr-dir-user/test.sh Executable file
View File

@ -0,0 +1,42 @@
#!/bin/bash
#
set -euo pipefail
TEST_DIR="${TEST_DIR:-$(dirname -- "${BASH_SOURCE[0]}")}"
source "${TEST_DIR}/../../shared.sh"
container_cleanup "$container_name-copier"
myvarsolr="${BUILD_DIR}/myvarsolr-${container_name}"
prepare_dir_to_mount 8983 "$myvarsolr"
echo "Running $container_name"
docker run \
-v "$myvarsolr:/var/solr" \
--name "$container_name" \
-u "$(id -u):$(id -g)" \
-d "$tag" solr-precreate getting-started
wait_for_container_and_solr "$container_name"
echo "Loading data"
docker exec --user=solr "$container_name" bin/post -c getting-started example/exampledocs/manufacturers.xml
sleep 1
echo "Checking data"
data=$(docker exec --user=solr "$container_name" wget -q -O - 'http://localhost:8983/solr/getting-started/select?q=id%3Adell')
if ! grep -E -q 'One Dell Way Round Rock, Texas 78682' <<<"$data"; then
echo "Test $TEST_NAME $tag failed; data did not load"
exit 1
fi
docker exec --user="$(id -u)" "$container_name" ls -lR /var/solr/data
container_cleanup "$container_name"
docker run --rm --user 0:0 -d -e VERBOSE=yes \
-v "$myvarsolr:/myvarsolr" "$tag" \
bash -c "chown -R $(id -u):$(id -g) /myvarsolr; ls -ld /myvarsolr"
rm -fr "$myvarsolr"
echo "Test $TEST_NAME $tag succeeded"

42
solr/docker/tests/cases/empty-varsolr-vol-ramdomuser-rootgroup/test.sh Executable file
View File

@ -0,0 +1,42 @@
#!/bin/bash
#
set -euo pipefail
TEST_DIR="${TEST_DIR:-$(dirname -- "${BASH_SOURCE[0]}")}"
source "${TEST_DIR}/../../shared.sh"
container_cleanup "$container_name-copier"
myvarsolr="myvarsolr-${container_name}"
docker volume rm "$myvarsolr" >/dev/null 2>&1 || true
docker volume create "$myvarsolr"
# when we mount onto /var/solr, that will be owned by "solr"
echo "Running $container_name"
docker run \
--user 777:0 \
-v "$myvarsolr:/var/solr" \
--name "$container_name" \
-d "$tag" solr-precreate getting-started
wait_for_container_and_solr "$container_name"
echo "Loading data"
docker exec --user=solr "$container_name" bin/post -c getting-started example/exampledocs/manufacturers.xml
sleep 1
echo "Checking data"
data=$(docker exec --user=solr "$container_name" wget -q -O - 'http://localhost:8983/solr/getting-started/select?q=id%3Adell')
if ! grep -E -q 'One Dell Way Round Rock, Texas 78682' <<<"$data"; then
echo "Test $TEST_NAME $tag failed; data did not load"
exit 1
fi
docker exec --user=7777 "$container_name" ls -l /var/solr/data
container_cleanup "$container_name"
docker volume rm "$myvarsolr"
echo "Test $TEST_NAME $tag succeeded"

47
solr/docker/tests/cases/empty-varsolr-vol-solr-nocopy/test.sh Executable file
View File

@ -0,0 +1,47 @@
#!/bin/bash
#
set -euo pipefail
TEST_DIR="${TEST_DIR:-$(dirname -- "${BASH_SOURCE[0]}")}"
source "${TEST_DIR}/../../shared.sh"
container_cleanup "$container_name-copier"
myvarsolr="myvarsolr-${container_name}"
docker volume rm "$myvarsolr" >/dev/null 2>&1 || true
docker volume create "$myvarsolr"
# with nocopy, the /var/solr ends up owned by root, so we need to chown it first
docker run \
-v "$myvarsolr:/var/solr:nocopy" \
--rm \
-u "0:0" \
"$tag" bash -c "chown 8983:8983 /var/solr"
echo "Running $container_name"
docker run \
-v "$myvarsolr:/var/solr:nocopy" \
--name "$container_name" \
-d "$tag" solr-precreate getting-started
wait_for_container_and_solr "$container_name"
echo "Loading data"
docker exec --user=solr "$container_name" bin/post -c getting-started example/exampledocs/manufacturers.xml
sleep 1
echo "Checking data"
data=$(docker exec --user=solr "$container_name" wget -q -O - 'http://localhost:8983/solr/getting-started/select?q=id%3Adell')
if ! grep -E -q 'One Dell Way Round Rock, Texas 78682' <<<"$data"; then
echo "Test $TEST_NAME $tag failed; data did not load"
exit 1
fi
docker exec --user=solr "$container_name" ls -l /var/solr/data
container_cleanup "$container_name"
docker volume rm "$myvarsolr"
echo "Test $TEST_NAME $tag succeeded"

41
solr/docker/tests/cases/empty-varsolr-vol-solr/test.sh Executable file
View File

@ -0,0 +1,41 @@
#!/bin/bash
#
set -euo pipefail
TEST_DIR="${TEST_DIR:-$(dirname -- "${BASH_SOURCE[0]}")}"
source "${TEST_DIR}/../../shared.sh"
container_cleanup "$container_name-copier"
myvarsolr="myvarsolr-${container_name}"
docker volume rm "$myvarsolr" >/dev/null 2>&1 || true
docker volume create "$myvarsolr"
# when we mount onto /var/solr, that will be owned by "solr"
echo "Running $container_name"
docker run \
-v "$myvarsolr:/var/solr" \
--name "$container_name" \
-d "$tag" solr-precreate getting-started
wait_for_container_and_solr "$container_name"
echo "Loading data"
docker exec --user=solr "$container_name" bin/post -c getting-started example/exampledocs/manufacturers.xml
sleep 1
echo "Checking data"
data=$(docker exec --user=solr "$container_name" wget -q -O - 'http://localhost:8983/solr/getting-started/select?q=id%3Adell')
if ! grep -E -q 'One Dell Way Round Rock, Texas 78682' <<<"$data"; then
echo "Test $TEST_NAME $tag failed; data did not load"
exit 1
fi
docker exec --user=solr "$container_name" ls -l /var/solr/data
container_cleanup "$container_name"
docker volume rm "$myvarsolr"
echo "Test $TEST_NAME $tag succeeded"

48
solr/docker/tests/cases/empty-varsolr-vol-user/test.sh Executable file
View File

@ -0,0 +1,48 @@
#!/bin/bash
#
set -euo pipefail
TEST_DIR="${TEST_DIR:-$(dirname -- "${BASH_SOURCE[0]}")}"
source "${TEST_DIR}/../../shared.sh"
container_cleanup "$container_name-copier"
myvarsolr="myvarsolr-${container_name}"
docker volume rm "$myvarsolr" >/dev/null 2>&1 || true
docker volume create "$myvarsolr"
# when we mount onto /var/solr, it will be owned by "solr", and it will copy
# the solr-owned directories and files from the container filesystem onto the
# the container. So from a container running as solr, modify permissions with
# setfacl to allow our user to write.
# If you don't have setfacl then run as root and do: chown -R $(id -u):$(id -g) /var/solr
docker run \
-v "$myvarsolr:/var/solr" \
--rm \
"$tag" bash -c "setfacl -R -m u:$(id -u):rwx /var/solr"
echo "Running $container_name as $(id -u):$(id -g)"
docker run \
-v "$myvarsolr:/var/solr" \
--name "$container_name" \
-u "$(id -u):$(id -g)" \
-d "$tag" solr-precreate getting-started
wait_for_container_and_solr "$container_name"
echo "Loading data"
docker exec --user=solr "$container_name" bin/post -c getting-started example/exampledocs/manufacturers.xml
sleep 1
echo "Checking data"
data=$(docker exec --user=solr "$container_name" wget -q -O - 'http://localhost:8983/solr/getting-started/select?q=id%3Adell')
if ! grep -E -q 'One Dell Way Round Rock, Texas 78682' <<<"$data"; then
echo "Test $TEST_NAME $tag failed; data did not load"
exit 1
fi
container_cleanup "$container_name"
docker volume rm "$myvarsolr"
echo "Test $TEST_NAME $tag succeeded"

68
solr/docker/tests/cases/gosu/test.sh Executable file
View File

@ -0,0 +1,68 @@
#!/bin/bash
#
# A simple test of gosu. We create a myvarsolr, and chown it
#
if [[ "$OSTYPE" == "darwin"* ]]; then
# TODO: Fix this test on Mac
echo "WARNING: Ignoring test 'gosu' on macOS"
exit 0
fi
set -euo pipefail
TEST_DIR="${TEST_DIR:-$(dirname -- "${BASH_SOURCE[0]}")}"
source "${TEST_DIR}/../../shared.sh"
myvarsolr="${BUILD_DIR}/myvarsolr-${container_name}"
prepare_dir_to_mount 8983 "$myvarsolr"
echo "Running $container_name"
docker run --user 0:0 --name "$container_name" -d -e VERBOSE=yes \
-v "$myvarsolr:/var/solr" "$tag" \
bash -c "chown -R solr:solr /var/solr; touch /var/solr/root_was_here; exec gosu solr:solr solr-precreate gettingstarted"
wait_for_container_and_solr "$container_name"
echo "Loading data"
docker exec --user=solr "$container_name" bin/post -c gettingstarted example/exampledocs/manufacturers.xml
sleep 1
echo "Checking data"
data=$(docker exec --user=solr "$container_name" wget -q -O - 'http://localhost:8983/solr/gettingstarted/select?q=id%3Adell')
if ! grep -E -q 'One Dell Way Round Rock, Texas 78682' <<<"$data"; then
echo "Test $TEST_NAME $tag failed; data did not load"
exit 1
fi
# check test file was created by root
data=$(docker exec --user=root "$container_name" stat -c %U /var/solr/root_was_here )
if [[ "$data" == *'No such file or directory' ]]; then
echo "Missing /var/solr/root_was_here"
exit 1
fi
if [[ "$data" != root ]]; then
echo "/var/solr/root_was_here is owned by $data"
exit 1
fi
# check core is created by solr
data=$(docker exec --user=root "$container_name" stat -c %U /var/solr/data/gettingstarted/core.properties )
if [[ "$data" == *'No such file or directory' ]]; then
echo "Missing /var/solr/data/gettingstarted/core.properties"
exit 1
fi
if [[ "$data" != solr ]]; then
echo "/var/solr/data/gettingstarted/core.properties is owned by $data"
exit 1
fi
container_cleanup "$container_name"
# chown it back
docker run --rm --user 0:0 -d -e VERBOSE=yes \
-v "$myvarsolr:/myvarsolr" "$tag" \
bash -c "chown -R $(id -u):$(id -g) /myvarsolr; ls -ld /myvarsolr"
rm -fr "$myvarsolr"
echo "Test $TEST_NAME $tag succeeded"

50
solr/docker/tests/cases/initdb/test.sh Executable file
View File

@ -0,0 +1,50 @@
#!/bin/bash
#
set -euo pipefail
TEST_DIR="${TEST_DIR:-$(dirname -- "${BASH_SOURCE[0]}")}"
source "${TEST_DIR}/../../shared.sh"
initdb="$BUILD_DIR/initdb-$container_name"
prepare_dir_to_mount 8983 "$initdb"
cat > "$initdb/create-was-here.sh" <<EOM
touch /var/solr/initdb-was-here
EOM
cat > "$initdb/ignore-me" <<EOM
touch /var/solr/should-not-be
EOM
echo "Running $container_name"
docker run --name "$container_name" -d -e VERBOSE=yes -v "$initdb:/docker-entrypoint-initdb.d" "$tag"
wait_for_server_started "$container_name"
echo "Checking initdb"
data=$(docker exec --user=solr "$container_name" ls /var/solr/initdb-was-here)
if [[ "$data" != /var/solr/initdb-was-here ]]; then
echo "Test $TEST_DIR $tag failed; script did not run"
exit 1
fi
data=$(docker exec --user=solr "$container_name" ls /var/solr/should-not-be; true)
if [[ -n "$data" ]]; then
echo "Test $TEST_DIR $tag failed; should-not-be was"
exit 1
fi
echo "Checking docker logs"
log="${BUILD_DIR}/docker.log-$container_name"
if ! docker logs "$container_name" >"$log" 2>&1; then
echo "Could not get logs for $container_name"
exit
fi
if ! grep -q 'ignoring /docker-entrypoint-initdb.d/ignore-me' "$log"; then
echo "missing ignoring message"
cat "$log"
exit 1
fi
rm "$log"
rm -fr "$initdb"
container_cleanup "$container_name"
echo "Test $TEST_NAME $tag succeeded"

24
solr/docker/tests/cases/precreate_core/test.sh Executable file
View File

@ -0,0 +1,24 @@
#!/bin/bash
#
set -euo pipefail
TEST_DIR="${TEST_DIR:-$(dirname -- "${BASH_SOURCE[0]}")}"
source "${TEST_DIR}/../../shared.sh"
echo "Running $container_name"
docker run --name "$container_name" -d -e VERBOSE=yes "$tag" solr-precreate gettingstarted
wait_for_container_and_solr "$container_name"
echo "Loading data"
docker exec --user=solr "$container_name" bin/post -c gettingstarted example/exampledocs/manufacturers.xml
sleep 1
echo "Checking data"
data=$(docker exec --user=solr "$container_name" wget -q -O - 'http://localhost:8983/solr/gettingstarted/select?q=id%3Adell')
if ! grep -E -q 'One Dell Way Round Rock, Texas 78682' <<<"$data"; then
echo "Test $TEST_NAME $tag failed; data did not load"
exit 1
fi
container_cleanup "$container_name"
echo "Test $TEST_NAME $tag succeeded"

29
solr/docker/tests/cases/precreate_core_randomuser_rootgroup/test.sh Executable file
View File

@ -0,0 +1,29 @@
#!/bin/bash
#
set -euo pipefail
TEST_DIR="${TEST_DIR:-$(dirname -- "${BASH_SOURCE[0]}")}"
source "${TEST_DIR}/../../shared.sh"
echo "Running $container_name"
docker run \
--user 7777:0 \
--name "$container_name" \
-d \
-e VERBOSE=yes \
"$tag" solr-precreate gettingstarted
wait_for_container_and_solr "$container_name"
echo "Loading data"
docker exec --user=solr "$container_name" bin/post -c gettingstarted example/exampledocs/manufacturers.xml
sleep 1
echo "Checking data"
data=$(docker exec --user=solr "$container_name" wget -q -O - 'http://localhost:8983/solr/gettingstarted/select?q=id%3Adell')
if ! grep -E -q 'One Dell Way Round Rock, Texas 78682' <<<"$data"; then
echo "Test $TEST_NAME $tag failed; data did not load"
exit 1
fi
container_cleanup "$container_name"
echo "Test $TEST_NAME $tag succeeded"

1
solr/docker/tests/cases/test_log4j/bogus-log4j2.xml Normal file
View File

@ -0,0 +1 @@
this is not xml < !

76
solr/docker/tests/cases/test_log4j/log4j2.xml Normal file
View File

@ -0,0 +1,76 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<Configuration>
<Appenders>
<Console name="STDOUT" target="SYSTEM_OUT">
<PatternLayout>
<Pattern>
%d{yyyy-MM-dd HH:mm:ss.SSS} %-5p (%t) [%X{collection} %X{shard} %X{replica} %X{core}] %c{1.} %m%n
</Pattern>
</PatternLayout>
</Console>
<RollingFile
name="RollingFile"
fileName="${sys:solr.log.dir}/solr.log"
filePattern="${sys:solr.log.dir}/solr.log.%i" >
<PatternLayout>
<Pattern>
%d{yyyy-MM-dd HH:mm:ss.SSS} %-5p (%t) [%X{collection} %X{shard} %X{replica} %X{core}] %c{1.} %m%n
</Pattern>
</PatternLayout>
<Policies>
<OnStartupTriggeringPolicy />
<SizeBasedTriggeringPolicy size="32 MB"/>
</Policies>
<DefaultRolloverStrategy max="10"/>
</RollingFile>
<RollingFile
name="SlowFile"
fileName="${sys:solr.log.dir}/solr_slow_requests.log"
filePattern="${sys:solr.log.dir}/solr_slow_requests.log.%i" >
<PatternLayout>
<Pattern>
%d{yyyy-MM-dd HH:mm:ss.SSS} %-5p (%t) [%X{collection} %X{shard} %X{replica} %X{core}] %c{1.} %m%n
</Pattern>
</PatternLayout>
<Policies>
<OnStartupTriggeringPolicy />
<SizeBasedTriggeringPolicy size="32 MB"/>
</Policies>
<DefaultRolloverStrategy max="10"/>
</RollingFile>
</Appenders>
<Loggers>
<Logger name="org.apache.hadoop" level="warn"/>
<Logger name="org.apache.solr.update.LoggingInfoStream" level="off"/>
<Logger name="org.apache.zookeeper" level="warn"/>
<Logger name="org.apache.solr.core.SolrCore.SlowRequest" level="info" additivity="false">
<AppenderRef ref="SlowFile"/>
</Logger>
<Root level="debug">
<AppenderRef ref="RollingFile"/>
<AppenderRef ref="STDOUT"/>
</Root>
</Loggers>
</Configuration>

34
solr/docker/tests/cases/test_log4j/test.sh Executable file
View File

@ -0,0 +1,34 @@
#!/bin/bash
#
set -euo pipefail
TEST_DIR="${TEST_DIR:-$(dirname -- "${BASH_SOURCE[0]}")}"
source "${TEST_DIR}/../../shared.sh"
echo "Running $container_name"
docker run --name "$container_name" -d -e VERBOSE=yes \
-e LOG4J_PROPS=/opt/solr/server/resources/log4j2.xml \
-v "$TEST_DIR/log4j2.xml:/opt/solr/server/resources/log4j2.xml" \
-v "$TEST_DIR/bogus-log4j2.xml:/var/solr/log4j2.xml" \
"$tag" solr-precreate gettingstarted
wait_for_container_and_solr "$container_name"
echo "Loading data"
docker exec --user=solr "$container_name" bin/post -c gettingstarted example/exampledocs/manufacturers.xml
sleep 1
echo "Checking data"
data=$(docker exec --user=solr "$container_name" wget -q -O - 'http://localhost:8983/solr/gettingstarted/select?q=id%3Adell')
if ! grep -E -q 'One Dell Way Round Rock, Texas 78682' <<<"$data"; then
echo "Test $TEST_NAME $tag failed; data did not load"
exit 1
fi
data=$(docker exec --user=solr "$container_name" grep 'DEBUG (main)' /var/solr/logs/solr.log | wc -l)
if (( data == 0 )); then
echo "missing DEBUG lines in the log"
exit 1
fi
container_cleanup "$container_name"
echo "Test $TEST_NAME $tag succeeded"

68
solr/docker/tests/cases/user_volume/test.sh Executable file
View File

@ -0,0 +1,68 @@
#!/bin/bash
#
set -euo pipefail
TEST_DIR="${TEST_DIR:-$(dirname -- "${BASH_SOURCE[0]}")}"
source "${TEST_DIR}/../../shared.sh"
container_cleanup "$container_name-copier"
myvarsolr="${BUILD_DIR}/myvarsolr-${container_name}"
prepare_dir_to_mount 8983 "$myvarsolr"
mylogs="${BUILD_DIR}/mylogs-${container_name}"
prepare_dir_to_mount 8983 "$mylogs"
myconf="${BUILD_DIR}/myconf-${container_name}"
configsets="${BUILD_DIR}/configsets-${container_name}"
# create a core by hand:
rm -fr "$myconf" "$configsets" 2>/dev/null
docker create --name "$container_name-copier" "$tag"
docker cp "$container_name-copier:/opt/solr/server/solr/configsets" "$configsets"
docker rm "$container_name-copier"
for d in data_driven_schema_configs _default; do
if [ -d "$configsets/$d" ]; then
cp -r "$configsets/$d/conf" "$myconf"
break
fi
done
rm -fr "$configsets"
if [ ! -d "$myconf" ]; then
echo "Could not get config"
exit 1
fi
if [ ! -f "$myconf/solrconfig.xml" ]; then
find "$myconf"
echo "ERROR: no solrconfig.xml"
exit 1
fi
# create a directory for the core
mkdir -p "$myvarsolr/data/mycore"
mkdir -p "$myvarsolr/logs"
touch "$myvarsolr/data/mycore/core.properties"
echo "Running $container_name"
docker run \
-v "$myvarsolr:/var/solr" \
-v "$myconf:/var/solr/data/mycore/conf:ro" \
-v "$mylogs:/var/solr/logs" \
--user "$(id -u):$(id -g)" \
--name "$container_name" \
-d "$tag"
wait_for_container_and_solr "$container_name"
echo "Loading data"
docker exec --user=solr "$container_name" bin/post -c mycore example/exampledocs/manufacturers.xml
sleep 1
echo "Checking data"
data=$(docker exec --user=solr "$container_name" wget -q -O - 'http://localhost:8983/solr/mycore/select?q=id%3Adell')
if ! grep -E -q 'One Dell Way Round Rock, Texas 78682' <<<"$data"; then
echo "Test $TEST_NAME $tag failed; data did not load"
exit 1
fi
container_cleanup "$container_name"
rm -fr "$myconf" "$myvarsolr" "$mylogs" "$configsets"
echo "Test $TEST_NAME $tag succeeded"

45
solr/docker/tests/cases/version/test.sh Executable file
View File

@ -0,0 +1,45 @@
#!/bin/bash
#
set -euo pipefail
TEST_DIR="${TEST_DIR:-$(dirname -- "${BASH_SOURCE[0]}")}"
source "${TEST_DIR}/../../shared.sh"
echo "Running $container_name"
docker run --name "$container_name" -d "$tag"
wait_for_server_started "$container_name"
echo "Checking that the OS matches the tag '$tag'"
if echo "$tag" | grep -q -- -alpine; then
alpine_version=$(docker exec --user=solr "$container_name" cat /etc/alpine-release || true)
if [[ -z $alpine_version ]]; then
echo "Could not get alpine version from container $container_name"
container_cleanup "$container_name"
exit 1
fi
echo "Alpine $alpine_version"
else
debian_version=$(docker exec --user=solr "$container_name" cat /etc/debian_version || true)
if [[ -z $debian_version ]]; then
echo "Could not get debian version from container $container_name"
container_cleanup "$container_name"
exit 1
fi
echo "Debian $debian_version"
fi
# check that the version of Solr matches the tag
changelog_version=$(docker exec --user=solr "$container_name" bash -c "grep -E '^==========* ' /opt/solr/CHANGES.txt | head -n 1 | tr -d '= '")
echo "Solr version $changelog_version"
solr_version_from_tag=$(echo "$tag" | sed -e 's/^.*://' -e 's/-.*//')
if [[ $changelog_version != "$solr_version_from_tag" ]]; then
echo "Solr version mismatch"
container_cleanup "$container_name"
exit 1
fi
container_cleanup "$container_name"
echo "Test $TEST_NAME $tag succeeded"

133
solr/docker/tests/shared.sh Executable file
View File

@ -0,0 +1,133 @@
#!/bin/bash
#
# Shared functions for testing
function container_cleanup {
local container_name
container_name=$1
previous=$(docker inspect "$container_name" --format '{{.ID}}' 2>/dev/null || true)
if [[ -n $previous ]]; then
container_status=$(docker inspect --format='{{.State.Status}}' "$previous" 2>/dev/null)
if [[ $container_status == 'running' ]]; then
echo "killing $previous"
docker kill "$previous" 2>/dev/null || true
sleep 2
fi
echo "removing $previous"
docker rm "$previous" 2>/dev/null || true
fi
}
function wait_for_container_and_solr {
local container_name
container_name=$1
wait_for_server_started "$container_name" 0
printf '\nWaiting for Solr...\n'
local status
status=$(docker exec "$container_name" /opt/docker-solr/scripts/wait-for-solr.sh --max-attempts 60 --wait-seconds 1)
# echo "Got status from Solr: $status"
if ! grep -E -i -q 'Solr is running' <<<"$status"; then
echo "Solr did not start"
container_cleanup "$container_name"
exit 1
else
echo "Solr is running"
fi
sleep 4
}
function wait_for_server_started {
local container_name
container_name=$1
local sleep_time
sleep_time=5
if [ -n "${2:-}" ]; then
sleep_time=$2
fi
echo "Waiting for container start: $container_name"
local TIMEOUT_SECONDS
TIMEOUT_SECONDS=$(( 5 * 60 ))
local started
started=$(date +%s)
local log
log="${BUILD_DIR}/${container_name}.log"
while true; do
docker logs "$container_name" > "${log}" 2>&1
if grep -E -q '(o\.e\.j\.s\.Server Started|Started SocketConnector)' "${log}" ; then
break
fi
local container_status
container_status=$(docker inspect --format='{{.State.Status}}' "$container_name")
if [[ $container_status == 'exited' ]]; then
echo "container exited"
exit 1
fi
if (( $(date +%s) > started + TIMEOUT_SECONDS )); then
echo "giving up after $TIMEOUT_SECONDS seconds"
exit 1
fi
printf '.'
sleep 2
done
echo "Server started"
rm "${log}"
sleep "$sleep_time"
}
function prepare_dir_to_mount {
local userid
userid=8983
local folder
folder="${BUILD_DIR}/myvarsolr"
if [ -n "$1" ]; then
userid=$1
fi
if [ -n "$2" ]; then
folder=$2
fi
rm -fr "$folder" >/dev/null 2>&1
mkdir "$folder"
#echo "***** Created varsolr folder $BUILD_DIR / $folder"
# The /var/solr mountpoint is owned by solr, so when we bind mount there our directory,
# owned by the current user in the host, will show as owned by solr, and our attempts
# to write to it as the user will fail. To deal with that, set the ACL to allow that.
# If you can't use setfacl (eg on macOS), you'll have to chown the directory to 8983, or apply world
# write permissions.
if command -v setfacl &> /dev/null; then
setfacl -m "u:$userid:rwx" "$folder"
fi
}
# Shared setup
if (( $# == 0 )); then
echo "Usage: ${TEST_DIR}/test.sh <tag>"
exit
fi
tag=$1
if [[ -n "${DEBUG:-}" ]]; then
set -x
fi
TEST_NAME="$(basename -- "${TEST_DIR}")"
# Create build directory if it hasn't been provided already
if [[ -z ${BUILD_DIR:-} ]]; then
BASE_DIR="$(dirname -- "${BASH_SOURCE-$0}")"
BASE_DIR="$(cd "${BASE_DIR}/.." && pwd)"
BUILD_DIR="${BASE_DIR}/build/tmp/tests/${TEST_NAME}"
fi
mkdir -p "${BUILD_DIR}"
echo "Test $TEST_DIR $tag"
echo "Test logs and build files can be found at: ${BUILD_DIR}"
container_name="test-$(echo "${TEST_NAME}" | tr ':/-' '_')-$(echo "${tag}" | tr ':/-' '_')"
echo "Cleaning up left-over containers from previous runs"
container_cleanup "${container_name}"