mirror of https://github.com/apache/nifi.git
NIFI-259: - Adding additional documentation for embedded ZK.
Signed-off-by: Aldrin Piri <aldrin@apache.org>
This commit is contained in:
parent
8a668fd344
commit
7711106d62
|
@ -415,7 +415,7 @@ file and will actually be ignored if they are populated. However, the `local-pro
|
||||||
Additionally, if NiFi is run in a cluster, each node must also have the `cluster-provider` element present and properly configured.
|
Additionally, if NiFi is run in a cluster, each node must also have the `cluster-provider` element present and properly configured.
|
||||||
Otherwise, NiFi will fail to startup.
|
Otherwise, NiFi will fail to startup.
|
||||||
|
|
||||||
While there are not many properties that need to be configured for these providers, they were externalized into a separate _state-providers.xml_
|
While there are not many properties that need to be configured for these providers, they were externalized into a separate _state-management.xml_
|
||||||
file, rather than being configured via the _nifi.properties_ file, simply because different implementations may require different properties,
|
file, rather than being configured via the _nifi.properties_ file, simply because different implementations may require different properties,
|
||||||
and it is easier to maintain and understand the configuration in an XML-based file such as this, than to mix the properties of the Provider
|
and it is easier to maintain and understand the configuration in an XML-based file such as this, than to mix the properties of the Provider
|
||||||
in with all of the other NiFi framework-specific properties.
|
in with all of the other NiFi framework-specific properties.
|
||||||
|
@ -442,16 +442,17 @@ However, it is up to the administrator to determine the number of nodes most app
|
||||||
|
|
||||||
If the `nifi.state.management.embedded.zookeeper.start` property is set to `true`, the `nifi.state.management.embedded.zookeeper.properties` property
|
If the `nifi.state.management.embedded.zookeeper.start` property is set to `true`, the `nifi.state.management.embedded.zookeeper.properties` property
|
||||||
in _nifi.properties_ also becomes relevant. This specifies the ZooKeeper properties file to use. At a minimum, this properties file needs to be populated
|
in _nifi.properties_ also becomes relevant. This specifies the ZooKeeper properties file to use. At a minimum, this properties file needs to be populated
|
||||||
with the list of ZooKeeper servers. Each of these servers is configured as <hostname>:<quorum port>[:<leader election port>]. For example, `myhost:2888:3888`.
|
with the list of ZooKeeper servers. The servers are specified as properties in the form of `server.1`, `server.2`, to `server.n`. Each of these servers is
|
||||||
This list of nodes should be the same nodes in the NiFi cluster that have the `nifi.state.management.embedded.zookeeper.start`
|
configured as <hostname>:<quorum port>[:<leader election port>]. For example, `myhost:2888:3888`. This list of nodes should be the same nodes in the NiFi
|
||||||
property set to `true`. Also note that because ZooKeeper will be listening on these ports, the firewall may need to be configured to open these ports
|
cluster that have the `nifi.state.management.embedded.zookeeper.start` property set to `true`. Also note that because ZooKeeper will be listening on these
|
||||||
for incoming traffic, at least between nodes in the cluster. Additionally, the port to listen on for client connections must be opened in the firewall.
|
ports, the firewall may need to be configured to open these ports for incoming traffic, at least between nodes in the cluster. Additionally, the port to
|
||||||
The default value for this is _2181_ but can be configured via the _clientPort_ property in the _zookeeper.properties_ file.
|
listen on for client connections must be opened in the firewall. The default value for this is _2181_ but can be configured via the _clientPort_ property
|
||||||
|
in the _zookeeper.properties_ file.
|
||||||
|
|
||||||
When using an embedded ZooKeeper, the _conf/zookeeper.properties_ file has a property named `dataDir`. By default, this value is set to `./state/zookeeper`.
|
When using an embedded ZooKeeper, the _conf/zookeeper.properties_ file has a property named `dataDir`. By default, this value is set to `./state/zookeeper`.
|
||||||
If more than one NiFi node is running an embedded ZooKeeper, it is important to tell the server which one it is. This is accomplished by creating a file named
|
If more than one NiFi node is running an embedded ZooKeeper, it is important to tell the server which one it is. This is accomplished by creating a file named
|
||||||
_myid_ and placing it in ZooKeeper's data directory. The contents of this file should be index of the server. So for one of the ZooKeeper servers, we will
|
_myid_ and placing it in ZooKeeper's data directory. The contents of this file should be the index of the server as specific by the `server.<number>`. So for
|
||||||
accomplish this by performing the following commands:
|
one of the ZooKeeper servers, we will accomplish this by performing the following commands:
|
||||||
|
|
||||||
[source]
|
[source]
|
||||||
cd $NIFI_HOME
|
cd $NIFI_HOME
|
||||||
|
@ -508,7 +509,8 @@ NiFi ZooKeeper client and embedded ZooKeeper server to use Kerberos are provided
|
||||||
[[zk_kerberos_client]]
|
[[zk_kerberos_client]]
|
||||||
==== Kerberizing NiFi's ZooKeeper Client
|
==== Kerberizing NiFi's ZooKeeper Client
|
||||||
The preferred mechanism for authenticating users with ZooKeeper is to use Kerberos. In order to use Kerberos to authenticate, we must configure a few
|
The preferred mechanism for authenticating users with ZooKeeper is to use Kerberos. In order to use Kerberos to authenticate, we must configure a few
|
||||||
system properties, so that the ZooKeeper client knows who the user is and where the KeyTab file is.
|
system properties, so that the ZooKeeper client knows who the user is and where the KeyTab file is. All nodes configured to store cluster-wide state
|
||||||
|
using `ZooKeeperStateProvider` and using Kerberos should follow these steps.
|
||||||
|
|
||||||
First, we must create the Principal that we will use when communicating with ZooKeeper. This is generally done via the `kadmin` tool:
|
First, we must create the Principal that we will use when communicating with ZooKeeper. This is generally done via the `kadmin` tool:
|
||||||
|
|
||||||
|
@ -525,7 +527,7 @@ After we have created our Principal, we will need to create a KeyTab for the Pri
|
||||||
kadmin: xst -k nifi.keytab nifi@EXAMPLE.COM
|
kadmin: xst -k nifi.keytab nifi@EXAMPLE.COM
|
||||||
|
|
||||||
This will create a file in the current directory named `nifi.keytab`. We can now copy that file into the _$NIFI_HOME/conf/_ directory. We should ensure
|
This will create a file in the current directory named `nifi.keytab`. We can now copy that file into the _$NIFI_HOME/conf/_ directory. We should ensure
|
||||||
that only the user that will be running NiF is allowed to read this file.
|
that only the user that will be running NiFi is allowed to read this file.
|
||||||
|
|
||||||
Next, we need to configure NiFi to use this KeyTab for authentication. Since ZooKeeper uses the Java Authentication and Authorization Service (JAAS), we need to
|
Next, we need to configure NiFi to use this KeyTab for authentication. Since ZooKeeper uses the Java Authentication and Authorization Service (JAAS), we need to
|
||||||
create a JAAS-compatible file. In the `$NIFI_HOME/conf/` directory, create a file named `zookeeper-jaas.conf` and add to it the following snippet:
|
create a JAAS-compatible file. In the `$NIFI_HOME/conf/` directory, create a file named `zookeeper-jaas.conf` and add to it the following snippet:
|
||||||
|
@ -565,8 +567,10 @@ Now, when we start NiFi, it will use Kerberos to authentication as the `nifi` us
|
||||||
|
|
||||||
[[zk_kerberos_server]]
|
[[zk_kerberos_server]]
|
||||||
==== Kerberizing Embedded ZooKeeper Server
|
==== Kerberizing Embedded ZooKeeper Server
|
||||||
When using the embedded ZooKeeper server, we may choose to secure the server by using Kerberos. If Kerberos is not already setup in your environment, you can find
|
When using the embedded ZooKeeper server, we may choose to secure the server by using Kerberos. All nodes configured to launch an embedded ZooKeeper
|
||||||
information on installing and setting up a Kerberos Server at
|
and using Kerberos should follow these steps.
|
||||||
|
|
||||||
|
If Kerberos is not already setup in your environment, you can find information on installing and setting up a Kerberos Server at
|
||||||
link:https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Managing_Smart_Cards/Configuring_a_Kerberos_5_Server.html[https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Managing_Smart_Cards/Configuring_a_Kerberos_5_Server.html]
|
link:https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Managing_Smart_Cards/Configuring_a_Kerberos_5_Server.html[https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Managing_Smart_Cards/Configuring_a_Kerberos_5_Server.html]
|
||||||
. This guide assumes that Kerberos already has been installed in the environment in which NiFi is running.
|
. This guide assumes that Kerberos already has been installed in the environment in which NiFi is running.
|
||||||
|
|
||||||
|
@ -942,7 +946,7 @@ for components to persist state. See the <<state_management>> section for more i
|
||||||
|nifi.state.management.provider.cluster|The ID of the Cluster State Provider to use. This value must match the value of the `id` element of one of the `cluster-provider` elements in the _state-management.xml_ file. This value is ignored if not clustered but is required for nodes in a cluster.
|
|nifi.state.management.provider.cluster|The ID of the Cluster State Provider to use. This value must match the value of the `id` element of one of the `cluster-provider` elements in the _state-management.xml_ file. This value is ignored if not clustered but is required for nodes in a cluster.
|
||||||
|nifi.state.management.embedded.zookeeper.start|Specifies whether or not this instance of NiFi should start an embedded ZooKeeper Server. This is used in conjunction with the ZooKeeperStateProvider.
|
|nifi.state.management.embedded.zookeeper.start|Specifies whether or not this instance of NiFi should start an embedded ZooKeeper Server. This is used in conjunction with the ZooKeeperStateProvider.
|
||||||
|nifi.state.management.embedded.zookeeper.properties|Specifies a properties file that contains the configuration for the embedded ZooKeeper Server that is started (if the `|nifi.state.management.embedded.zookeeper.start` property is set to `true`)
|
|nifi.state.management.embedded.zookeeper.properties|Specifies a properties file that contains the configuration for the embedded ZooKeeper Server that is started (if the `|nifi.state.management.embedded.zookeeper.start` property is set to `true`)
|
||||||
||====
|
|====
|
||||||
|
|
||||||
|
|
||||||
*H2 Settings* +
|
*H2 Settings* +
|
||||||
|
|
|
@ -27,4 +27,19 @@ syncLimit=5
|
||||||
tickTime=2000
|
tickTime=2000
|
||||||
dataDir=./state/zookeeper
|
dataDir=./state/zookeeper
|
||||||
autopurge.snapRetainCount=30
|
autopurge.snapRetainCount=30
|
||||||
|
|
||||||
|
#
|
||||||
|
# Specifies the servers that are part of this zookeeper ensemble. For
|
||||||
|
# every NiFi instance running an embedded zookeeper, there needs to be
|
||||||
|
# a server entry below. For instance:
|
||||||
|
#
|
||||||
|
# server.1=nifi-node1-hostname:2888:3888
|
||||||
|
# server.2=nifi-node2-hostname:2888:3888
|
||||||
|
# server.3=nifi-node3-hostname:2888:3888
|
||||||
|
#
|
||||||
|
# The index of the server corresponds to the myid file that gets created
|
||||||
|
# in the dataDir of each node running an embedded zookeeper. See the
|
||||||
|
# administration guide for more details.
|
||||||
|
#
|
||||||
|
|
||||||
server.1=
|
server.1=
|
Loading…
Reference in New Issue