Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

HADOOP-11633. Convert remaining branch-2 .apt.vm files to markdown. Contributed by Masatake Iwasaki.

This commit is contained in:
Haohui Mai 2015-03-11 14:23:44 -07:00
parent 5e99b7cd2f
commit e75e6c66ea
46 changed files with 10303 additions and 12862 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

70
hadoop-common-project/hadoop-auth/src/site/apt/BuildingIt.apt.vm
View File

@ -1,70 +0,0 @@
~~ Licensed under the Apache License, Version 2.0 (the "License");
~~ you may not use this file except in compliance with the License.
~~ You may obtain a copy of the License at
~~
~~ http://www.apache.org/licenses/LICENSE-2.0
~~
~~ Unless required by applicable law or agreed to in writing, software
~~ distributed under the License is distributed on an "AS IS" BASIS,
~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~~ See the License for the specific language governing permissions and
~~ limitations under the License. See accompanying LICENSE file.
---
Hadoop Auth, Java HTTP SPNEGO ${project.version} - Building It
---
---
${maven.build.timestamp}
Hadoop Auth, Java HTTP SPNEGO ${project.version} - Building It
* Requirements
* Java 6+
* Maven 3+
* Kerberos KDC (for running Kerberos test cases)
* Building
Use Maven goals: clean, test, compile, package, install
Available profiles: docs, testKerberos
* Testing
By default Kerberos testcases are not run.
The requirements to run Kerberos testcases are a running KDC, a keytab
file with a client principal and a kerberos principal.
To run Kerberos tescases use the <<<testKerberos>>> Maven profile:
+---+
$ mvn test -PtestKerberos
+---+
The following Maven <<<-D>>> options can be used to change the default
values:
* <<<hadoop-auth.test.kerberos.realm>>>: default value <<LOCALHOST>>
* <<<hadoop-auth.test.kerberos.client.principal>>>: default value <<client>>
* <<<hadoop-auth.test.kerberos.server.principal>>>: default value
<<HTTP/localhost>> (it must start 'HTTP/')
* <<<hadoop-auth.test.kerberos.keytab.file>>>: default value
<<${HOME}/${USER}.keytab>>
** Generating Documentation
To create the documentation use the <<<docs>>> Maven profile:
+---+
$ mvn package -Pdocs
+---+
The generated documentation is available at
<<<hadoop-auth/target/site/>>>.

377
hadoop-common-project/hadoop-auth/src/site/apt/Configuration.apt.vm
View File

@ -1,377 +0,0 @@
~~ Licensed under the Apache License, Version 2.0 (the "License");
~~ you may not use this file except in compliance with the License.
~~ You may obtain a copy of the License at
~~
~~ http://www.apache.org/licenses/LICENSE-2.0
~~
~~ Unless required by applicable law or agreed to in writing, software
~~ distributed under the License is distributed on an "AS IS" BASIS,
~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~~ See the License for the specific language governing permissions and
~~ limitations under the License. See accompanying LICENSE file.
---
Hadoop Auth, Java HTTP SPNEGO ${project.version} - Server Side
Configuration
---
---
${maven.build.timestamp}
Hadoop Auth, Java HTTP SPNEGO ${project.version} - Server Side
Configuration
* Server Side Configuration Setup
The AuthenticationFilter filter is Hadoop Auth's server side component.
This filter must be configured in front of all the web application resources
that required authenticated requests. For example:
The Hadoop Auth and dependent JAR files must be in the web application
classpath (commonly the <<<WEB-INF/lib>>> directory).
Hadoop Auth uses SLF4J-API for logging. Auth Maven POM dependencies define
the SLF4J API dependency but it does not define the dependency on a concrete
logging implementation, this must be addded explicitly to the web
application. For example, if the web applicationan uses Log4j, the
SLF4J-LOG4J12 and LOG4J jar files must be part part of the web application
classpath as well as the Log4j configuration file.
** Common Configuration parameters
* <<<config.prefix>>>: If specified, all other configuration parameter names
must start with the prefix. The default value is no prefix.
* <<<[PREFIX.]type>>>: the authentication type keyword (<<<simple>>> or
<<<kerberos>>>) or a Authentication handler implementation.
* <<<[PREFIX.]signature.secret>>>: When <<<signer.secret.provider>>> is set to
<<<string>>> or not specified, this is the value for the secret used to sign
the HTTP cookie.
* <<<[PREFIX.]token.validity>>>: The validity -in seconds- of the generated
authentication token. The default value is <<<3600>>> seconds. This is also
used for the rollover interval when <<<signer.secret.provider>>> is set to
<<<random>>> or <<<zookeeper>>>.
* <<<[PREFIX.]cookie.domain>>>: domain to use for the HTTP cookie that stores
the authentication token.
* <<<[PREFIX.]cookie.path>>>: path to use for the HTTP cookie that stores the
authentication token.
* <<<signer.secret.provider>>>: indicates the name of the SignerSecretProvider
class to use. Possible values are: <<<string>>>, <<<random>>>,
<<<zookeeper>>>, or a classname. If not specified, the <<<string>>>
implementation will be used; and failing that, the <<<random>>>
implementation will be used.
** Kerberos Configuration
<<IMPORTANT>>: A KDC must be configured and running.
To use Kerberos SPNEGO as the authentication mechanism, the authentication
filter must be configured with the following init parameters:
* <<<[PREFIX.]type>>>: the keyword <<<kerberos>>>.
* <<<[PREFIX.]kerberos.principal>>>: The web-application Kerberos principal
name. The Kerberos principal name must start with <<<HTTP/...>>>. For
example: <<<HTTP/localhost@LOCALHOST>>>. There is no default value.
* <<<[PREFIX.]kerberos.keytab>>>: The path to the keytab file containing
the credentials for the kerberos principal. For example:
<<</Users/tucu/tucu.keytab>>>. There is no default value.
<<Example>>:
+---+
<web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee">
...
<filter>
<filter-name>kerberosFilter</filter-name>
<filter-class>org.apache.hadoop.security.auth.server.AuthenticationFilter</filter-class>
<init-param>
<param-name>type</param-name>
<param-value>kerberos</param-value>
</init-param>
<init-param>
<param-name>token.validity</param-name>
<param-value>30</param-value>
</init-param>
<init-param>
<param-name>cookie.domain</param-name>
<param-value>.foo.com</param-value>
</init-param>
<init-param>
<param-name>cookie.path</param-name>
<param-value>/</param-value>
</init-param>
<init-param>
<param-name>kerberos.principal</param-name>
<param-value>HTTP/localhost@LOCALHOST</param-value>
</init-param>
<init-param>
<param-name>kerberos.keytab</param-name>
<param-value>/tmp/auth.keytab</param-value>
</init-param>
</filter>
<filter-mapping>
<filter-name>kerberosFilter</filter-name>
<url-pattern>/kerberos/*</url-pattern>
</filter-mapping>
...
</web-app>
+---+
** Pseudo/Simple Configuration
To use Pseudo/Simple as the authentication mechanism (trusting the value of
the query string parameter 'user.name'), the authentication filter must be
configured with the following init parameters:
* <<<[PREFIX.]type>>>: the keyword <<<simple>>>.
* <<<[PREFIX.]simple.anonymous.allowed>>>: is a boolean parameter that
indicates if anonymous requests are allowed or not. The default value is
<<<false>>>.
<<Example>>:
+---+
<web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee">
...
<filter>
<filter-name>simpleFilter</filter-name>
<filter-class>org.apache.hadoop.security.auth.server.AuthenticationFilter</filter-class>
<init-param>
<param-name>type</param-name>
<param-value>simple</param-value>
</init-param>
<init-param>
<param-name>token.validity</param-name>
<param-value>30</param-value>
</init-param>
<init-param>
<param-name>cookie.domain</param-name>
<param-value>.foo.com</param-value>
</init-param>
<init-param>
<param-name>cookie.path</param-name>
<param-value>/</param-value>
</init-param>
<init-param>
<param-name>simple.anonymous.allowed</param-name>
<param-value>false</param-value>
</init-param>
</filter>
<filter-mapping>
<filter-name>simpleFilter</filter-name>
<url-pattern>/simple/*</url-pattern>
</filter-mapping>
...
</web-app>
+---+
** AltKerberos Configuration
<<IMPORTANT>>: A KDC must be configured and running.
The AltKerberos authentication mechanism is a partially implemented derivative
of the Kerberos SPNEGO authentication mechanism which allows a "mixed" form of
authentication where Kerberos SPNEGO is used by non-browsers while an
alternate form of authentication (to be implemented by the user) is used for
browsers. To use AltKerberos as the authentication mechanism (besides
providing an implementation), the authentication filter must be configured
with the following init parameters, in addition to the previously mentioned
Kerberos SPNEGO ones:
* <<<[PREFIX.]type>>>: the full class name of the implementation of
AltKerberosAuthenticationHandler to use.
* <<<[PREFIX.]alt-kerberos.non-browser.user-agents>>>: a comma-separated
list of which user-agents should be considered non-browsers.
<<Example>>:
+---+
<web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee">
...
<filter>
<filter-name>kerberosFilter</filter-name>
<filter-class>org.apache.hadoop.security.auth.server.AuthenticationFilter</filter-class>
<init-param>
<param-name>type</param-name>
<param-value>org.my.subclass.of.AltKerberosAuthenticationHandler</param-value>
</init-param>
<init-param>
<param-name>alt-kerberos.non-browser.user-agents</param-name>
<param-value>java,curl,wget,perl</param-value>
</init-param>
<init-param>
<param-name>token.validity</param-name>
<param-value>30</param-value>
</init-param>
<init-param>
<param-name>cookie.domain</param-name>
<param-value>.foo.com</param-value>
</init-param>
<init-param>
<param-name>cookie.path</param-name>
<param-value>/</param-value>
</init-param>
<init-param>
<param-name>kerberos.principal</param-name>
<param-value>HTTP/localhost@LOCALHOST</param-value>
</init-param>
<init-param>
<param-name>kerberos.keytab</param-name>
<param-value>/tmp/auth.keytab</param-value>
</init-param>
</filter>
<filter-mapping>
<filter-name>kerberosFilter</filter-name>
<url-pattern>/kerberos/*</url-pattern>
</filter-mapping>
...
</web-app>
+---+
** SignerSecretProvider Configuration
The SignerSecretProvider is used to provide more advanced behaviors for the
secret used for signing the HTTP Cookies.
These are the relevant configuration properties:
* <<<signer.secret.provider>>>: indicates the name of the
SignerSecretProvider class to use. Possible values are: "string",
"random", "zookeeper", or a classname. If not specified, the "string"
implementation will be used; and failing that, the "random" implementation
will be used.
* <<<[PREFIX.]signature.secret>>>: When <<<signer.secret.provider>>> is set
to <<<string>>> or not specified, this is the value for the secret used to
sign the HTTP cookie.
* <<<[PREFIX.]token.validity>>>: The validity -in seconds- of the generated
authentication token. The default value is <<<3600>>> seconds. This is
also used for the rollover interval when <<<signer.secret.provider>>> is
set to <<<random>>> or <<<zookeeper>>>.
The following configuration properties are specific to the <<<zookeeper>>>
implementation:
* <<<signer.secret.provider.zookeeper.connection.string>>>: Indicates the
ZooKeeper connection string to connect with.
* <<<signer.secret.provider.zookeeper.path>>>: Indicates the ZooKeeper path
to use for storing and retrieving the secrets. All servers
that need to coordinate their secret should point to the same path
* <<<signer.secret.provider.zookeeper.auth.type>>>: Indicates the auth type
to use. Supported values are <<<none>>> and <<<sasl>>>. The default
value is <<<none>>>.
* <<<signer.secret.provider.zookeeper.kerberos.keytab>>>: Set this to the
path with the Kerberos keytab file. This is only required if using
Kerberos.
* <<<signer.secret.provider.zookeeper.kerberos.principal>>>: Set this to the
Kerberos principal to use. This only required if using Kerberos.
<<Example>>:
+---+
<web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee">
...
<filter>
<!-- AuthenticationHandler configs not shown -->
<init-param>
<param-name>signer.secret.provider</param-name>
<param-value>string</param-value>
</init-param>
<init-param>
<param-name>signature.secret</param-name>
<param-value>my_secret</param-value>
</init-param>
</filter>
...
</web-app>
+---+
<<Example>>:
+---+
<web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee">
...
<filter>
<!-- AuthenticationHandler configs not shown -->
<init-param>
<param-name>signer.secret.provider</param-name>
<param-value>random</param-value>
</init-param>
<init-param>
<param-name>token.validity</param-name>
<param-value>30</param-value>
</init-param>
</filter>
...
</web-app>
+---+
<<Example>>:
+---+
<web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee">
...
<filter>
<!-- AuthenticationHandler configs not shown -->
<init-param>
<param-name>signer.secret.provider</param-name>
<param-value>zookeeper</param-value>
</init-param>
<init-param>
<param-name>token.validity</param-name>
<param-value>30</param-value>
</init-param>
<init-param>
<param-name>signer.secret.provider.zookeeper.connection.string</param-name>
<param-value>zoo1:2181,zoo2:2181,zoo3:2181</param-value>
</init-param>
<init-param>
<param-name>signer.secret.provider.zookeeper.path</param-name>
<param-value>/myapp/secrets</param-value>
</init-param>
<init-param>
<param-name>signer.secret.provider.zookeeper.use.kerberos.acls</param-name>
<param-value>true</param-value>
</init-param>
<init-param>
<param-name>signer.secret.provider.zookeeper.kerberos.keytab</param-name>
<param-value>/tmp/auth.keytab</param-value>
</init-param>
<init-param>
<param-name>signer.secret.provider.zookeeper.kerberos.principal</param-name>
<param-value>HTTP/localhost@LOCALHOST</param-value>
</init-param>
</filter>
...
</web-app>
+---+

133
hadoop-common-project/hadoop-auth/src/site/apt/Examples.apt.vm
View File

@ -1,133 +0,0 @@
~~ Licensed under the Apache License, Version 2.0 (the "License");
~~ you may not use this file except in compliance with the License.
~~ You may obtain a copy of the License at
~~
~~ http://www.apache.org/licenses/LICENSE-2.0
~~
~~ Unless required by applicable law or agreed to in writing, software
~~ distributed under the License is distributed on an "AS IS" BASIS,
~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~~ See the License for the specific language governing permissions and
~~ limitations under the License. See accompanying LICENSE file.
---
Hadoop Auth, Java HTTP SPNEGO ${project.version} - Examples
---
---
${maven.build.timestamp}
Hadoop Auth, Java HTTP SPNEGO ${project.version} - Examples
* Accessing a Hadoop Auth protected URL Using a browser
<<IMPORTANT:>> The browser must support HTTP Kerberos SPNEGO. For example,
Firefox or Internet Explorer.
For Firefox access the low level configuration page by loading the
<<<about:config>>> page. Then go to the
<<<network.negotiate-auth.trusted-uris>>> preference and add the hostname or
the domain of the web server that is HTTP Kerberos SPNEGO protected (if using
multiple domains and hostname use comma to separate them).
* Accessing a Hadoop Auth protected URL Using <<<curl>>>
<<IMPORTANT:>> The <<<curl>>> version must support GSS, run <<<curl -V>>>.
+---+
$ curl -V
curl 7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7 OpenSSL/0.9.8l zlib/1.2.3
Protocols: tftp ftp telnet dict ldap http file https ftps
Features: GSS-Negotiate IPv6 Largefile NTLM SSL libz
+---+
Login to the KDC using <<kinit>> and then use <<<curl>>> to fetch protected
URL:
+---+
$ kinit
Please enter the password for tucu@LOCALHOST:
$ curl --negotiate -u foo -b ~/cookiejar.txt -c ~/cookiejar.txt http://localhost:8080/hadoop-auth-examples/kerberos/who
Enter host password for user 'tucu':
Hello Hadoop Auth Examples!
+---+
* The <<<--negotiate>>> option enables SPNEGO in <<<curl>>>.
* The <<<-u foo>>> option is required but the user ignored (the principal
that has been kinit-ed is used).
* The <<<-b>>> and <<<-c>>> are use to store and send HTTP Cookies.
* Using the Java Client
Use the <<<AuthenticatedURL>>> class to obtain an authenticated HTTP
connection:
+---+
...
URL url = new URL("http://localhost:8080/hadoop-auth/kerberos/who");
AuthenticatedURL.Token token = new AuthenticatedURL.Token();
...
HttpURLConnection conn = new AuthenticatedURL(url, token).openConnection();
...
conn = new AuthenticatedURL(url, token).openConnection();
...
+---+
* Building and Running the Examples
Download Hadoop-Auth's source code, the examples are in the
<<<src/main/examples>>> directory.
** Server Example:
Edit the <<<hadoop-auth-examples/src/main/webapp/WEB-INF/web.xml>>> and set the
right configuration init parameters for the <<<AuthenticationFilter>>>
definition configured for Kerberos (the right Kerberos principal and keytab
file must be specified). Refer to the {{{./Configuration.html}Configuration
document}} for details.
Create the web application WAR file by running the <<<mvn package>>> command.
Deploy the WAR file in a servlet container. For example, if using Tomcat,
copy the WAR file to Tomcat's <<<webapps/>>> directory.
Start the servlet container.
** Accessing the server using <<<curl>>>
Try accessing protected resources using <<<curl>>>. The protected resources
are:
+---+
$ kinit
Please enter the password for tucu@LOCALHOST:
$ curl http://localhost:8080/hadoop-auth-examples/anonymous/who
$ curl http://localhost:8080/hadoop-auth-examples/simple/who?user.name=foo
$ curl --negotiate -u foo -b ~/cookiejar.txt -c ~/cookiejar.txt http://localhost:8080/hadoop-auth-examples/kerberos/who
+---+
** Accessing the server using the Java client example
+---+
$ kinit
Please enter the password for tucu@LOCALHOST:
$ cd examples
$ mvn exec:java -Durl=http://localhost:8080/hadoop-auth-examples/kerberos/who
....
Token value: "u=tucu,p=tucu@LOCALHOST,t=kerberos,e=1295305313146,s=sVZ1mpSnC5TKhZQE3QLN5p2DWBo="
Status code: 200 OK
You are: user[tucu] principal[tucu@LOCALHOST]
....
+---+

59
hadoop-common-project/hadoop-auth/src/site/apt/index.apt.vm
View File

@ -1,59 +0,0 @@
~~ Licensed under the Apache License, Version 2.0 (the "License");
~~ you may not use this file except in compliance with the License.
~~ You may obtain a copy of the License at
~~
~~ http://www.apache.org/licenses/LICENSE-2.0
~~
~~ Unless required by applicable law or agreed to in writing, software
~~ distributed under the License is distributed on an "AS IS" BASIS,
~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~~ See the License for the specific language governing permissions and
~~ limitations under the License. See accompanying LICENSE file.
---
Hadoop Auth, Java HTTP SPNEGO ${project.version}
---
---
${maven.build.timestamp}
Hadoop Auth, Java HTTP SPNEGO ${project.version}
Hadoop Auth is a Java library consisting of a client and a server
components to enable Kerberos SPNEGO authentication for HTTP.
Hadoop Auth also supports additional authentication mechanisms on the client
and the server side via 2 simple interfaces.
Additionally, it provides a partially implemented derivative of the Kerberos
SPNEGO authentication to allow a "mixed" form of authentication where Kerberos
SPNEGO is used by non-browsers while an alternate form of authentication
(to be implemented by the user) is used for browsers.
* License
Hadoop Auth is distributed under {{{http://www.apache.org/licenses/}Apache
License 2.0}}.
* How Does Auth Works?
Hadoop Auth enforces authentication on protected resources, once authentiation
has been established it sets a signed HTTP Cookie that contains an
authentication token with the user name, user principal, authentication type
and expiration time.
Subsequent HTTP client requests presenting the signed HTTP Cookie have access
to the protected resources until the HTTP Cookie expires.
The secret used to sign the HTTP Cookie has multiple implementations that
provide different behaviors, including a hardcoded secret string, a rolling
randomly generated secret, and a rolling randomly generated secret
synchronized between multiple servers using ZooKeeper.
* User Documentation
* {{{./Examples.html}Examples}}
* {{{./Configuration.html}Configuration}}
* {{{./BuildingIt.html}Building It}}

56
hadoop-common-project/hadoop-auth/src/site/markdown/BuildingIt.md Normal file
View File

@ -0,0 +1,56 @@
<!---
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
Hadoop Auth, Java HTTP SPNEGO - Building It
===========================================
Requirements
------------
* Java 6+
* Maven 3+
* Kerberos KDC (for running Kerberos test cases)
Building
--------
Use Maven goals: clean, test, compile, package, install
Available profiles: docs, testKerberos
Testing
-------
By default Kerberos testcases are not run.
The requirements to run Kerberos testcases are a running KDC, a keytab file with a client principal and a kerberos principal.
To run Kerberos tescases use the `testKerberos` Maven profile:
$ mvn test -PtestKerberos
The following Maven `-D` options can be used to change the default values:
* `hadoop-auth.test.kerberos.realm`: default value **LOCALHOST**
* `hadoop-auth.test.kerberos.client.principal`: default value **client**
* `hadoop-auth.test.kerberos.server.principal`: default value **HTTP/localhost** (it must start 'HTTP/')
* `hadoop-auth.test.kerberos.keytab.file`: default value **$HOME/$USER.keytab**
### Generating Documentation
To create the documentation use the `docs` Maven profile:
$ mvn package -Pdocs
The generated documentation is available at `hadoop-auth/target/site/`.

341
hadoop-common-project/hadoop-auth/src/site/markdown/Configuration.md Normal file
View File

@ -0,0 +1,341 @@
<!---
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
Hadoop Auth, Java HTTP SPNEGO - Server Side Configuration
=========================================================
Server Side Configuration Setup
-------------------------------
The AuthenticationFilter filter is Hadoop Auth's server side component.
This filter must be configured in front of all the web application resources that required authenticated requests. For example:
The Hadoop Auth and dependent JAR files must be in the web application classpath (commonly the `WEB-INF/lib` directory).
Hadoop Auth uses SLF4J-API for logging. Auth Maven POM dependencies define the SLF4J API dependency but it does not define the dependency on a concrete logging implementation, this must be addded explicitly to the web application. For example, if the web applicationan uses Log4j, the SLF4J-LOG4J12 and LOG4J jar files must be part part of the web application classpath as well as the Log4j configuration file.
### Common Configuration parameters
* `config.prefix`: If specified, all other configuration parameter names
must start with the prefix. The default value is no prefix.
* `[PREFIX.]type`: the authentication type keyword (`simple` or \
`kerberos`) or a Authentication handler implementation.
* `[PREFIX.]signature.secret`: When `signer.secret.provider` is set to
`string` or not specified, this is the value for the secret used to sign
the HTTP cookie.
* `[PREFIX.]token.validity`: The validity -in seconds- of the generated
authentication token. The default value is `3600` seconds. This is also
used for the rollover interval when `signer.secret.provider` is set to
`random` or `zookeeper`.
* `[PREFIX.]cookie.domain`: domain to use for the HTTP cookie that stores
the authentication token.
* `[PREFIX.]cookie.path`: path to use for the HTTP cookie that stores the
authentication token.
* `signer.secret.provider`: indicates the name of the SignerSecretProvider
class to use. Possible values are: `string`, `random`,
`zookeeper`, or a classname. If not specified, the `string`
implementation will be used; and failing that, the `random`
implementation will be used.
### Kerberos Configuration
**IMPORTANT**: A KDC must be configured and running.
To use Kerberos SPNEGO as the authentication mechanism, the authentication filter must be configured with the following init parameters:
* `[PREFIX.]type`: the keyword `kerberos`.
* `[PREFIX.]kerberos.principal`: The web-application Kerberos principal
name. The Kerberos principal name must start with `HTTP/...`. For
example: `HTTP/localhost@LOCALHOST`. There is no default value.
* `[PREFIX.]kerberos.keytab`: The path to the keytab file containing
the credentials for the kerberos principal. For example:
`/Users/tucu/tucu.keytab`. There is no default value.
**Example**:
<web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee">
...
<filter>
<filter-name>kerberosFilter</filter-name>
<filter-class>org.apache.hadoop.security.auth.server.AuthenticationFilter</filter-class>
<init-param>
<param-name>type</param-name>
<param-value>kerberos</param-value>
</init-param>
<init-param>
<param-name>token.validity</param-name>
<param-value>30</param-value>
</init-param>
<init-param>
<param-name>cookie.domain</param-name>
<param-value>.foo.com</param-value>
</init-param>
<init-param>
<param-name>cookie.path</param-name>
<param-value>/</param-value>
</init-param>
<init-param>
<param-name>kerberos.principal</param-name>
<param-value>HTTP/localhost@LOCALHOST</param-value>
</init-param>
<init-param>
<param-name>kerberos.keytab</param-name>
<param-value>/tmp/auth.keytab</param-value>
</init-param>
</filter>
<filter-mapping>
<filter-name>kerberosFilter</filter-name>
<url-pattern>/kerberos/*</url-pattern>
</filter-mapping>
...
</web-app>
### Pseudo/Simple Configuration
To use Pseudo/Simple as the authentication mechanism (trusting the value of the query string parameter 'user.name'), the authentication filter must be configured with the following init parameters:
* `[PREFIX.]type`: the keyword `simple`.
* `[PREFIX.]simple.anonymous.allowed`: is a boolean parameter that
indicates if anonymous requests are allowed or not. The default value is
`false`.
**Example**:
<web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee">
...
<filter>
<filter-name>simpleFilter</filter-name>
<filter-class>org.apache.hadoop.security.auth.server.AuthenticationFilter</filter-class>
<init-param>
<param-name>type</param-name>
<param-value>simple</param-value>
</init-param>
<init-param>
<param-name>token.validity</param-name>
<param-value>30</param-value>
</init-param>
<init-param>
<param-name>cookie.domain</param-name>
<param-value>.foo.com</param-value>
</init-param>
<init-param>
<param-name>cookie.path</param-name>
<param-value>/</param-value>
</init-param>
<init-param>
<param-name>simple.anonymous.allowed</param-name>
<param-value>false</param-value>
</init-param>
</filter>
<filter-mapping>
<filter-name>simpleFilter</filter-name>
<url-pattern>/simple/*</url-pattern>
</filter-mapping>
...
</web-app>
### AltKerberos Configuration
**IMPORTANT**: A KDC must be configured and running.
The AltKerberos authentication mechanism is a partially implemented derivative of the Kerberos SPNEGO authentication mechanism which allows a "mixed" form of authentication where Kerberos SPNEGO is used by non-browsers while an alternate form of authentication (to be implemented by the user) is used for browsers. To use AltKerberos as the authentication mechanism (besides providing an implementation), the authentication filter must be configured with the following init parameters, in addition to the previously mentioned Kerberos SPNEGO ones:
* `[PREFIX.]type`: the full class name of the implementation of
AltKerberosAuthenticationHandler to use.
* `[PREFIX.]alt-kerberos.non-browser.user-agents`: a comma-separated
list of which user-agents should be considered non-browsers.
**Example**:
<web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee">
...
<filter>
<filter-name>kerberosFilter</filter-name>
<filter-class>org.apache.hadoop.security.auth.server.AuthenticationFilter</filter-class>
<init-param>
<param-name>type</param-name>
<param-value>org.my.subclass.of.AltKerberosAuthenticationHandler</param-value>
</init-param>
<init-param>
<param-name>alt-kerberos.non-browser.user-agents</param-name>
<param-value>java,curl,wget,perl</param-value>
</init-param>
<init-param>
<param-name>token.validity</param-name>
<param-value>30</param-value>
</init-param>
<init-param>
<param-name>cookie.domain</param-name>
<param-value>.foo.com</param-value>
</init-param>
<init-param>
<param-name>cookie.path</param-name>
<param-value>/</param-value>
</init-param>
<init-param>
<param-name>kerberos.principal</param-name>
<param-value>HTTP/localhost@LOCALHOST</param-value>
</init-param>
<init-param>
<param-name>kerberos.keytab</param-name>
<param-value>/tmp/auth.keytab</param-value>
</init-param>
</filter>
<filter-mapping>
<filter-name>kerberosFilter</filter-name>
<url-pattern>/kerberos/*</url-pattern>
</filter-mapping>
...
</web-app>
### SignerSecretProvider Configuration
The SignerSecretProvider is used to provide more advanced behaviors for the secret used for signing the HTTP Cookies.
These are the relevant configuration properties:
* `signer.secret.provider`: indicates the name of the
SignerSecretProvider class to use. Possible values are: "string",
"random", "zookeeper", or a classname. If not specified, the "string"
implementation will be used; and failing that, the "random" implementation
will be used.
* `[PREFIX.]signature.secret`: When `signer.secret.provider` is set
to `string` or not specified, this is the value for the secret used to
sign the HTTP cookie.
* `[PREFIX.]token.validity`: The validity -in seconds- of the generated
authentication token. The default value is `3600` seconds. This is
also used for the rollover interval when `signer.secret.provider` is
set to `random` or `zookeeper`.
The following configuration properties are specific to the `zookeeper` implementation:
* `signer.secret.provider.zookeeper.connection.string`: Indicates the
ZooKeeper connection string to connect with.
* `signer.secret.provider.zookeeper.path`: Indicates the ZooKeeper path
to use for storing and retrieving the secrets. All servers
that need to coordinate their secret should point to the same path
* `signer.secret.provider.zookeeper.auth.type`: Indicates the auth type
to use. Supported values are `none` and `sasl`. The default
value is `none`.
* `signer.secret.provider.zookeeper.kerberos.keytab`: Set this to the
path with the Kerberos keytab file. This is only required if using
Kerberos.
* `signer.secret.provider.zookeeper.kerberos.principal`: Set this to the
Kerberos principal to use. This only required if using Kerberos.
**Example**:
<web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee">
...
<filter>
<!-- AuthenticationHandler configs not shown -->
<init-param>
<param-name>signer.secret.provider</param-name>
<param-value>string</param-value>
</init-param>
<init-param>
<param-name>signature.secret</param-name>
<param-value>my_secret</param-value>
</init-param>
</filter>
...
</web-app>
**Example**:
<web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee">
...
<filter>
<!-- AuthenticationHandler configs not shown -->
<init-param>
<param-name>signer.secret.provider</param-name>
<param-value>random</param-value>
</init-param>
<init-param>
<param-name>token.validity</param-name>
<param-value>30</param-value>
</init-param>
</filter>
...
</web-app>
**Example**:
<web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee">
...
<filter>
<!-- AuthenticationHandler configs not shown -->
<init-param>
<param-name>signer.secret.provider</param-name>
<param-value>zookeeper</param-value>
</init-param>
<init-param>
<param-name>token.validity</param-name>
<param-value>30</param-value>
</init-param>
<init-param>
<param-name>signer.secret.provider.zookeeper.connection.string</param-name>
<param-value>zoo1:2181,zoo2:2181,zoo3:2181</param-value>
</init-param>
<init-param>
<param-name>signer.secret.provider.zookeeper.path</param-name>
<param-value>/myapp/secrets</param-value>
</init-param>
<init-param>
<param-name>signer.secret.provider.zookeeper.use.kerberos.acls</param-name>
<param-value>true</param-value>
</init-param>
<init-param>
<param-name>signer.secret.provider.zookeeper.kerberos.keytab</param-name>
<param-value>/tmp/auth.keytab</param-value>
</init-param>
<init-param>
<param-name>signer.secret.provider.zookeeper.kerberos.principal</param-name>
<param-value>HTTP/localhost@LOCALHOST</param-value>
</init-param>
</filter>
...
</web-app>

109
hadoop-common-project/hadoop-auth/src/site/markdown/Examples.md Normal file
View File

@ -0,0 +1,109 @@
<!---
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
Hadoop Auth, Java HTTP SPNEGO - Examples
========================================
Accessing a Hadoop Auth protected URL Using a browser
-----------------------------------------------------
**IMPORTANT:** The browser must support HTTP Kerberos SPNEGO. For example, Firefox or Internet Explorer.
For Firefox access the low level configuration page by loading the `about:config` page. Then go to the `network.negotiate-auth.trusted-uris` preference and add the hostname or the domain of the web server that is HTTP Kerberos SPNEGO protected (if using multiple domains and hostname use comma to separate them).
Accessing a Hadoop Auth protected URL Using `curl`
--------------------------------------------------
**IMPORTANT:** The `curl` version must support GSS, run `curl -V`.
$ curl -V
curl 7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7 OpenSSL/0.9.8l zlib/1.2.3
Protocols: tftp ftp telnet dict ldap http file https ftps
Features: GSS-Negotiate IPv6 Largefile NTLM SSL libz
Login to the KDC using **kinit** and then use `curl` to fetch protected URL:
$ kinit
Please enter the password for tucu@LOCALHOST:
$ curl --negotiate -u foo -b ~/cookiejar.txt -c ~/cookiejar.txt http://localhost:8080/hadoop-auth-examples/kerberos/who
Enter host password for user 'tucu':
Hello Hadoop Auth Examples!
* The `--negotiate` option enables SPNEGO in `curl`.
* The `-u foo` option is required but the user ignored (the principal
that has been kinit-ed is used).
* The `-b` and `-c` are use to store and send HTTP Cookies.
Using the Java Client
---------------------
Use the `AuthenticatedURL` class to obtain an authenticated HTTP connection:
...
URL url = new URL("http://localhost:8080/hadoop-auth/kerberos/who");
AuthenticatedURL.Token token = new AuthenticatedURL.Token();
...
HttpURLConnection conn = new AuthenticatedURL(url, token).openConnection();
...
conn = new AuthenticatedURL(url, token).openConnection();
...
Building and Running the Examples
---------------------------------
Download Hadoop-Auth's source code, the examples are in the `src/main/examples` directory.
### Server Example:
Edit the `hadoop-auth-examples/src/main/webapp/WEB-INF/web.xml` and set the right configuration init parameters for the `AuthenticationFilter` definition configured for Kerberos (the right Kerberos principal and keytab file must be specified). Refer to the [Configuration document](./Configuration.html) for details.
Create the web application WAR file by running the `mvn package` command.
Deploy the WAR file in a servlet container. For example, if using Tomcat, copy the WAR file to Tomcat's `webapps/` directory.
Start the servlet container.
### Accessing the server using `curl`
Try accessing protected resources using `curl`. The protected resources are:
$ kinit
Please enter the password for tucu@LOCALHOST:
$ curl http://localhost:8080/hadoop-auth-examples/anonymous/who
$ curl http://localhost:8080/hadoop-auth-examples/simple/who?user.name=foo
$ curl --negotiate -u foo -b ~/cookiejar.txt -c ~/cookiejar.txt http://localhost:8080/hadoop-auth-examples/kerberos/who
### Accessing the server using the Java client example
$ kinit
Please enter the password for tucu@LOCALHOST:
$ cd examples
$ mvn exec:java -Durl=http://localhost:8080/hadoop-auth-examples/kerberos/who
....
Token value: "u=tucu,p=tucu@LOCALHOST,t=kerberos,e=1295305313146,s=sVZ1mpSnC5TKhZQE3QLN5p2DWBo="
Status code: 200 OK
You are: user[tucu] principal[tucu@LOCALHOST]
....

43
hadoop-common-project/hadoop-auth/src/site/markdown/index.md Normal file
View File

@ -0,0 +1,43 @@
<!---
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
Hadoop Auth, Java HTTP SPNEGO
=============================
Hadoop Auth is a Java library consisting of a client and a server components to enable Kerberos SPNEGO authentication for HTTP.
Hadoop Auth also supports additional authentication mechanisms on the client and the server side via 2 simple interfaces.
Additionally, it provides a partially implemented derivative of the Kerberos SPNEGO authentication to allow a "mixed" form of authentication where Kerberos SPNEGO is used by non-browsers while an alternate form of authentication (to be implemented by the user) is used for browsers.
License
-------
Hadoop Auth is distributed under [Apache License 2.0](http://www.apache.org/licenses/).
How Does Auth Works?
--------------------
Hadoop Auth enforces authentication on protected resources, once authentiation has been established it sets a signed HTTP Cookie that contains an authentication token with the user name, user principal, authentication type and expiration time.
Subsequent HTTP client requests presenting the signed HTTP Cookie have access to the protected resources until the HTTP Cookie expires.
The secret used to sign the HTTP Cookie has multiple implementations that provide different behaviors, including a hardcoded secret string, a rolling randomly generated secret, and a rolling randomly generated secret synchronized between multiple servers using ZooKeeper.
User Documentation
------------------
* [Examples](./Examples.html)
* [Configuration](./Configuration.html)
* [Building It](./BuildingIt.html)

3
hadoop-common-project/hadoop-common/CHANGES.txt
View File

@ -275,6 +275,9 @@ Release 2.7.0 - UNRELEASED
HADOOP-11183. Memory-based S3AOutputstream. (Thomas Demoor via stevel)
HADOOP-11633. Convert remaining branch-2 .apt.vm files to markdown.
(Masatake Iwasaki via wheat9)
BUG FIXES
HADOOP-11512. Use getTrimmedStrings when reading serialization keys

703
hadoop-common-project/hadoop-common/src/site/apt/ClusterSetup.apt.vm
View File

@ -1,703 +0,0 @@
~~ Licensed under the Apache License, Version 2.0 (the "License");
~~ you may not use this file except in compliance with the License.
~~ You may obtain a copy of the License at
~~
~~ http://www.apache.org/licenses/LICENSE-2.0
~~
~~ Unless required by applicable law or agreed to in writing, software
~~ distributed under the License is distributed on an "AS IS" BASIS,
~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~~ See the License for the specific language governing permissions and
~~ limitations under the License. See accompanying LICENSE file.
---
Hadoop Map Reduce Next Generation-${project.version} - Cluster Setup
---
---
${maven.build.timestamp}
%{toc|section=1|fromDepth=0}
Hadoop MapReduce Next Generation - Cluster Setup
* {Purpose}
This document describes how to install, configure and manage non-trivial
Hadoop clusters ranging from a few nodes to extremely large clusters
with thousands of nodes.
To play with Hadoop, you may first want to install it on a single
machine (see {{{./SingleCluster.html}Single Node Setup}}).
* {Prerequisites}
Download a stable version of Hadoop from Apache mirrors.
* {Installation}
Installing a Hadoop cluster typically involves unpacking the software on all
the machines in the cluster or installing RPMs.
Typically one machine in the cluster is designated as the NameNode and
another machine the as ResourceManager, exclusively. These are the masters.
The rest of the machines in the cluster act as both DataNode and NodeManager.
These are the slaves.
* {Running Hadoop in Non-Secure Mode}
The following sections describe how to configure a Hadoop cluster.
{Configuration Files}
Hadoop configuration is driven by two types of important configuration files:
* Read-only default configuration - <<<core-default.xml>>>,
<<<hdfs-default.xml>>>, <<<yarn-default.xml>>> and
<<<mapred-default.xml>>>.
* Site-specific configuration - <<conf/core-site.xml>>,
<<conf/hdfs-site.xml>>, <<conf/yarn-site.xml>> and
<<conf/mapred-site.xml>>.
Additionally, you can control the Hadoop scripts found in the bin/
directory of the distribution, by setting site-specific values via the
<<conf/hadoop-env.sh>> and <<yarn-env.sh>>.
{Site Configuration}
To configure the Hadoop cluster you will need to configure the
<<<environment>>> in which the Hadoop daemons execute as well as the
<<<configuration parameters>>> for the Hadoop daemons.
The Hadoop daemons are NameNode/DataNode and ResourceManager/NodeManager.
** {Configuring Environment of Hadoop Daemons}
Administrators should use the <<conf/hadoop-env.sh>> and
<<conf/yarn-env.sh>> script to do site-specific customization of the
Hadoop daemons' process environment.
At the very least you should specify the <<<JAVA_HOME>>> so that it is
correctly defined on each remote node.
In most cases you should also specify <<<HADOOP_PID_DIR>>> and
<<<HADOOP_SECURE_DN_PID_DIR>>> to point to directories that can only be
written to by the users that are going to run the hadoop daemons.
Otherwise there is the potential for a symlink attack.
Administrators can configure individual daemons using the configuration
options shown below in the table:
*--------------------------------------+--------------------------------------+
|| Daemon || Environment Variable |
*--------------------------------------+--------------------------------------+
| NameNode | HADOOP_NAMENODE_OPTS |
*--------------------------------------+--------------------------------------+
| DataNode | HADOOP_DATANODE_OPTS |
*--------------------------------------+--------------------------------------+
| Secondary NameNode | HADOOP_SECONDARYNAMENODE_OPTS |
*--------------------------------------+--------------------------------------+
| ResourceManager | YARN_RESOURCEMANAGER_OPTS |
*--------------------------------------+--------------------------------------+
| NodeManager | YARN_NODEMANAGER_OPTS |
*--------------------------------------+--------------------------------------+
| WebAppProxy | YARN_PROXYSERVER_OPTS |
*--------------------------------------+--------------------------------------+
| Map Reduce Job History Server | HADOOP_JOB_HISTORYSERVER_OPTS |
*--------------------------------------+--------------------------------------+
For example, To configure Namenode to use parallelGC, the following
statement should be added in hadoop-env.sh :
----
export HADOOP_NAMENODE_OPTS="-XX:+UseParallelGC ${HADOOP_NAMENODE_OPTS}"
----
Other useful configuration parameters that you can customize include:
* <<<HADOOP_LOG_DIR>>> / <<<YARN_LOG_DIR>>> - The directory where the
daemons' log files are stored. They are automatically created if they
don't exist.
* <<<HADOOP_HEAPSIZE>>> / <<<YARN_HEAPSIZE>>> - The maximum amount of
heapsize to use, in MB e.g. if the varibale is set to 1000 the heap
will be set to 1000MB. This is used to configure the heap
size for the daemon. By default, the value is 1000. If you want to
configure the values separately for each deamon you can use.
*--------------------------------------+--------------------------------------+
|| Daemon || Environment Variable |
*--------------------------------------+--------------------------------------+
| ResourceManager | YARN_RESOURCEMANAGER_HEAPSIZE |
*--------------------------------------+--------------------------------------+
| NodeManager | YARN_NODEMANAGER_HEAPSIZE |
*--------------------------------------+--------------------------------------+
| WebAppProxy | YARN_PROXYSERVER_HEAPSIZE |
*--------------------------------------+--------------------------------------+
| Map Reduce Job History Server | HADOOP_JOB_HISTORYSERVER_HEAPSIZE |
*--------------------------------------+--------------------------------------+
** {Configuring the Hadoop Daemons in Non-Secure Mode}
This section deals with important parameters to be specified in
the given configuration files:
* <<<conf/core-site.xml>>>
*-------------------------+-------------------------+------------------------+
|| Parameter || Value || Notes |
*-------------------------+-------------------------+------------------------+
| <<<fs.defaultFS>>> | NameNode URI | <hdfs://host:port/> |
*-------------------------+-------------------------+------------------------+
| <<<io.file.buffer.size>>> | 131072 | |
| | | Size of read/write buffer used in SequenceFiles. |
*-------------------------+-------------------------+------------------------+
* <<<conf/hdfs-site.xml>>>
* Configurations for NameNode:
*-------------------------+-------------------------+------------------------+
|| Parameter || Value || Notes |
*-------------------------+-------------------------+------------------------+
| <<<dfs.namenode.name.dir>>> | | |
| | Path on the local filesystem where the NameNode stores the namespace | |
| | and transactions logs persistently. | |
| | | If this is a comma-delimited list of directories then the name table is |
| | | replicated in all of the directories, for redundancy. |
*-------------------------+-------------------------+------------------------+
| <<<dfs.namenode.hosts>>> / <<<dfs.namenode.hosts.exclude>>> | | |
| | List of permitted/excluded DataNodes. | |
| | | If necessary, use these files to control the list of allowable |
| | | datanodes. |
*-------------------------+-------------------------+------------------------+
| <<<dfs.blocksize>>> | 268435456 | |
| | | HDFS blocksize of 256MB for large file-systems. |
*-------------------------+-------------------------+------------------------+
| <<<dfs.namenode.handler.count>>> | 100 | |
| | | More NameNode server threads to handle RPCs from large number of |
| | | DataNodes. |
*-------------------------+-------------------------+------------------------+
* Configurations for DataNode:
*-------------------------+-------------------------+------------------------+
|| Parameter || Value || Notes |
*-------------------------+-------------------------+------------------------+
| <<<dfs.datanode.data.dir>>> | | |
| | Comma separated list of paths on the local filesystem of a | |
| | <<<DataNode>>> where it should store its blocks. | |
| | | If this is a comma-delimited list of directories, then data will be |
| | | stored in all named directories, typically on different devices. |
*-------------------------+-------------------------+------------------------+
* <<<conf/yarn-site.xml>>>
* Configurations for ResourceManager and NodeManager:
*-------------------------+-------------------------+------------------------+
|| Parameter || Value || Notes |
*-------------------------+-------------------------+------------------------+
| <<<yarn.acl.enable>>> | | |
| | <<<true>>> / <<<false>>> | |
| | | Enable ACLs? Defaults to <false>. |
*-------------------------+-------------------------+------------------------+
| <<<yarn.admin.acl>>> | | |
| | Admin ACL | |
| | | ACL to set admins on the cluster. |
| | | ACLs are of for <comma-separated-users><space><comma-separated-groups>. |
| | | Defaults to special value of <<*>> which means <anyone>. |
| | | Special value of just <space> means no one has access. |
*-------------------------+-------------------------+------------------------+
| <<<yarn.log-aggregation-enable>>> | | |
| | <false> | |
| | | Configuration to enable or disable log aggregation |
*-------------------------+-------------------------+------------------------+
* Configurations for ResourceManager:
*-------------------------+-------------------------+------------------------+
|| Parameter || Value || Notes |
*-------------------------+-------------------------+------------------------+
| <<<yarn.resourcemanager.address>>> | | |
| | <<<ResourceManager>>> host:port for clients to submit jobs. | |
| | | <host:port>\ |
| | | If set, overrides the hostname set in <<<yarn.resourcemanager.hostname>>>. |
*-------------------------+-------------------------+------------------------+
| <<<yarn.resourcemanager.scheduler.address>>> | | |
| | <<<ResourceManager>>> host:port for ApplicationMasters to talk to | |
| | Scheduler to obtain resources. | |
| | | <host:port>\ |
| | | If set, overrides the hostname set in <<<yarn.resourcemanager.hostname>>>. |
*-------------------------+-------------------------+------------------------+
| <<<yarn.resourcemanager.resource-tracker.address>>> | | |
| | <<<ResourceManager>>> host:port for NodeManagers. | |
| | | <host:port>\ |
| | | If set, overrides the hostname set in <<<yarn.resourcemanager.hostname>>>. |
*-------------------------+-------------------------+------------------------+
| <<<yarn.resourcemanager.admin.address>>> | | |
| | <<<ResourceManager>>> host:port for administrative commands. | |
| | | <host:port>\ |
| | | If set, overrides the hostname set in <<<yarn.resourcemanager.hostname>>>. |
*-------------------------+-------------------------+------------------------+
| <<<yarn.resourcemanager.webapp.address>>> | | |
| | <<<ResourceManager>>> web-ui host:port. | |
| | | <host:port>\ |
| | | If set, overrides the hostname set in <<<yarn.resourcemanager.hostname>>>. |
*-------------------------+-------------------------+------------------------+
| <<<yarn.resourcemanager.hostname>>> | | |
| | <<<ResourceManager>>> host. | |
| | | <host>\ |
| | | Single hostname that can be set in place of setting all <<<yarn.resourcemanager*address>>> resources. Results in default ports for ResourceManager components. |
*-------------------------+-------------------------+------------------------+
| <<<yarn.resourcemanager.scheduler.class>>> | | |
| | <<<ResourceManager>>> Scheduler class. | |
| | | <<<CapacityScheduler>>> (recommended), <<<FairScheduler>>> (also recommended), or <<<FifoScheduler>>> |
*-------------------------+-------------------------+------------------------+
| <<<yarn.scheduler.minimum-allocation-mb>>> | | |
| | Minimum limit of memory to allocate to each container request at the <<<Resource Manager>>>. | |
| | | In MBs |
*-------------------------+-------------------------+------------------------+
| <<<yarn.scheduler.maximum-allocation-mb>>> | | |
| | Maximum limit of memory to allocate to each container request at the <<<Resource Manager>>>. | |
| | | In MBs |
*-------------------------+-------------------------+------------------------+
| <<<yarn.resourcemanager.nodes.include-path>>> / | | |
| <<<yarn.resourcemanager.nodes.exclude-path>>> | | |
| | List of permitted/excluded NodeManagers. | |
| | | If necessary, use these files to control the list of allowable |
| | | NodeManagers. |
*-------------------------+-------------------------+------------------------+
* Configurations for NodeManager:
*-------------------------+-------------------------+------------------------+
|| Parameter || Value || Notes |
*-------------------------+-------------------------+------------------------+
| <<<yarn.nodemanager.resource.memory-mb>>> | | |
| | Resource i.e. available physical memory, in MB, for given <<<NodeManager>>> | |
| | | Defines total available resources on the <<<NodeManager>>> to be made |
| | | available to running containers |
*-------------------------+-------------------------+------------------------+
| <<<yarn.nodemanager.vmem-pmem-ratio>>> | | |
| | Maximum ratio by which virtual memory usage of tasks may exceed |
| | physical memory | |
| | | The virtual memory usage of each task may exceed its physical memory |
| | | limit by this ratio. The total amount of virtual memory used by tasks |
| | | on the NodeManager may exceed its physical memory usage by this ratio. |
*-------------------------+-------------------------+------------------------+
| <<<yarn.nodemanager.local-dirs>>> | | |
| | Comma-separated list of paths on the local filesystem where | |
| | intermediate data is written. ||
| | | Multiple paths help spread disk i/o. |
*-------------------------+-------------------------+------------------------+
| <<<yarn.nodemanager.log-dirs>>> | | |
| | Comma-separated list of paths on the local filesystem where logs | |
| | are written. | |
| | | Multiple paths help spread disk i/o. |
*-------------------------+-------------------------+------------------------+
| <<<yarn.nodemanager.log.retain-seconds>>> | | |
| | <10800> | |
| | | Default time (in seconds) to retain log files on the NodeManager |
| | | Only applicable if log-aggregation is disabled. |
*-------------------------+-------------------------+------------------------+
| <<<yarn.nodemanager.remote-app-log-dir>>> | | |
| | </logs> | |
| | | HDFS directory where the application logs are moved on application |
| | | completion. Need to set appropriate permissions. |
| | | Only applicable if log-aggregation is enabled. |
*-------------------------+-------------------------+------------------------+
| <<<yarn.nodemanager.remote-app-log-dir-suffix>>> | | |
| | <logs> | |
| | | Suffix appended to the remote log dir. Logs will be aggregated to |
| | | $\{yarn.nodemanager.remote-app-log-dir\}/$\{user\}/$\{thisParam\} |
| | | Only applicable if log-aggregation is enabled. |
*-------------------------+-------------------------+------------------------+
| <<<yarn.nodemanager.aux-services>>> | | |
| | mapreduce_shuffle | |
| | | Shuffle service that needs to be set for Map Reduce applications. |
*-------------------------+-------------------------+------------------------+
* Configurations for History Server (Needs to be moved elsewhere):
*-------------------------+-------------------------+------------------------+
|| Parameter || Value || Notes |
*-------------------------+-------------------------+------------------------+
| <<<yarn.log-aggregation.retain-seconds>>> | | |
| | <-1> | |
| | | How long to keep aggregation logs before deleting them. -1 disables. |
| | | Be careful, set this too small and you will spam the name node. |
*-------------------------+-------------------------+------------------------+
| <<<yarn.log-aggregation.retain-check-interval-seconds>>> | | |
| | <-1> | |
| | | Time between checks for aggregated log retention. If set to 0 or a |
| | | negative value then the value is computed as one-tenth of the |
| | | aggregated log retention time. |
| | | Be careful, set this too small and you will spam the name node. |
*-------------------------+-------------------------+------------------------+
* <<<conf/mapred-site.xml>>>
* Configurations for MapReduce Applications:
*-------------------------+-------------------------+------------------------+
|| Parameter || Value || Notes |
*-------------------------+-------------------------+------------------------+
| <<<mapreduce.framework.name>>> | | |
| | yarn | |
| | | Execution framework set to Hadoop YARN. |
*-------------------------+-------------------------+------------------------+
| <<<mapreduce.map.memory.mb>>> | 1536 | |
| | | Larger resource limit for maps. |
*-------------------------+-------------------------+------------------------+
| <<<mapreduce.map.java.opts>>> | -Xmx1024M | |
| | | Larger heap-size for child jvms of maps. |
*-------------------------+-------------------------+------------------------+
| <<<mapreduce.reduce.memory.mb>>> | 3072 | |
| | | Larger resource limit for reduces. |
*-------------------------+-------------------------+------------------------+
| <<<mapreduce.reduce.java.opts>>> | -Xmx2560M | |
| | | Larger heap-size for child jvms of reduces. |
*-------------------------+-------------------------+------------------------+
| <<<mapreduce.task.io.sort.mb>>> | 512 | |
| | | Higher memory-limit while sorting data for efficiency. |
*-------------------------+-------------------------+------------------------+
| <<<mapreduce.task.io.sort.factor>>> | 100 | |
| | | More streams merged at once while sorting files. |
*-------------------------+-------------------------+------------------------+
| <<<mapreduce.reduce.shuffle.parallelcopies>>> | 50 | |
| | | Higher number of parallel copies run by reduces to fetch outputs |
| | | from very large number of maps. |
*-------------------------+-------------------------+------------------------+
* Configurations for MapReduce JobHistory Server:
*-------------------------+-------------------------+------------------------+
|| Parameter || Value || Notes |
*-------------------------+-------------------------+------------------------+
| <<<mapreduce.jobhistory.address>>> | | |
| | MapReduce JobHistory Server <host:port> | Default port is 10020. |
*-------------------------+-------------------------+------------------------+
| <<<mapreduce.jobhistory.webapp.address>>> | | |
| | MapReduce JobHistory Server Web UI <host:port> | Default port is 19888. |
*-------------------------+-------------------------+------------------------+
| <<<mapreduce.jobhistory.intermediate-done-dir>>> | /mr-history/tmp | |
| | | Directory where history files are written by MapReduce jobs. |
*-------------------------+-------------------------+------------------------+
| <<<mapreduce.jobhistory.done-dir>>> | /mr-history/done| |
| | | Directory where history files are managed by the MR JobHistory Server. |
*-------------------------+-------------------------+------------------------+
* {Hadoop Rack Awareness}
The HDFS and the YARN components are rack-aware.
The NameNode and the ResourceManager obtains the rack information of the
slaves in the cluster by invoking an API <resolve> in an administrator
configured module.
The API resolves the DNS name (also IP address) to a rack id.
The site-specific module to use can be configured using the configuration
item <<<topology.node.switch.mapping.impl>>>. The default implementation
of the same runs a script/command configured using
<<<topology.script.file.name>>>. If <<<topology.script.file.name>>> is
not set, the rack id </default-rack> is returned for any passed IP address.
* {Monitoring Health of NodeManagers}
Hadoop provides a mechanism by which administrators can configure the
NodeManager to run an administrator supplied script periodically to
determine if a node is healthy or not.
Administrators can determine if the node is in a healthy state by
performing any checks of their choice in the script. If the script
detects the node to be in an unhealthy state, it must print a line to
standard output beginning with the string ERROR. The NodeManager spawns
the script periodically and checks its output. If the script's output
contains the string ERROR, as described above, the node's status is
reported as <<<unhealthy>>> and the node is black-listed by the
ResourceManager. No further tasks will be assigned to this node.
However, the NodeManager continues to run the script, so that if the
node becomes healthy again, it will be removed from the blacklisted nodes
on the ResourceManager automatically. The node's health along with the
output of the script, if it is unhealthy, is available to the
administrator in the ResourceManager web interface. The time since the
node was healthy is also displayed on the web interface.
The following parameters can be used to control the node health
monitoring script in <<<conf/yarn-site.xml>>>.
*-------------------------+-------------------------+------------------------+
|| Parameter || Value || Notes |
*-------------------------+-------------------------+------------------------+
| <<<yarn.nodemanager.health-checker.script.path>>> | | |
| | Node health script | |
| | | Script to check for node's health status. |
*-------------------------+-------------------------+------------------------+
| <<<yarn.nodemanager.health-checker.script.opts>>> | | |
| | Node health script options | |
| | | Options for script to check for node's health status. |
*-------------------------+-------------------------+------------------------+
| <<<yarn.nodemanager.health-checker.script.interval-ms>>> | | |
| | Node health script interval | |
| | | Time interval for running health script. |
*-------------------------+-------------------------+------------------------+
| <<<yarn.nodemanager.health-checker.script.timeout-ms>>> | | |
| | Node health script timeout interval | |
| | | Timeout for health script execution. |
*-------------------------+-------------------------+------------------------+
The health checker script is not supposed to give ERROR if only some of the
local disks become bad. NodeManager has the ability to periodically check
the health of the local disks (specifically checks nodemanager-local-dirs
and nodemanager-log-dirs) and after reaching the threshold of number of
bad directories based on the value set for the config property
yarn.nodemanager.disk-health-checker.min-healthy-disks, the whole node is
marked unhealthy and this info is sent to resource manager also. The boot
disk is either raided or a failure in the boot disk is identified by the
health checker script.
* {Slaves file}
Typically you choose one machine in the cluster to act as the NameNode and
one machine as to act as the ResourceManager, exclusively. The rest of the
machines act as both a DataNode and NodeManager and are referred to as
<slaves>.
List all slave hostnames or IP addresses in your <<<conf/slaves>>> file,
one per line.
* {Logging}
Hadoop uses the Apache log4j via the Apache Commons Logging framework for
logging. Edit the <<<conf/log4j.properties>>> file to customize the
Hadoop daemons' logging configuration (log-formats and so on).
* {Operating the Hadoop Cluster}
Once all the necessary configuration is complete, distribute the files to the
<<<HADOOP_CONF_DIR>>> directory on all the machines.
** Hadoop Startup
To start a Hadoop cluster you will need to start both the HDFS and YARN
cluster.
Format a new distributed filesystem:
----
$ $HADOOP_PREFIX/bin/hdfs namenode -format <cluster_name>
----
Start the HDFS with the following command, run on the designated NameNode:
----
$ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start namenode
----
Run a script to start DataNodes on all slaves:
----
$ $HADOOP_PREFIX/sbin/hadoop-daemons.sh --config $HADOOP_CONF_DIR --script hdfs start datanode
----
Start the YARN with the following command, run on the designated
ResourceManager:
----
$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start resourcemanager
----
Run a script to start NodeManagers on all slaves:
----
$ $HADOOP_YARN_HOME/sbin/yarn-daemons.sh --config $HADOOP_CONF_DIR start nodemanager
----
Start a standalone WebAppProxy server. If multiple servers
are used with load balancing it should be run on each of them:
----
$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start proxyserver --config $HADOOP_CONF_DIR
----
Start the MapReduce JobHistory Server with the following command, run on the
designated server:
----
$ $HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh start historyserver --config $HADOOP_CONF_DIR
----
** Hadoop Shutdown
Stop the NameNode with the following command, run on the designated
NameNode:
----
$ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs stop namenode
----
Run a script to stop DataNodes on all slaves:
----
$ $HADOOP_PREFIX/sbin/hadoop-daemons.sh --config $HADOOP_CONF_DIR --script hdfs stop datanode
----
Stop the ResourceManager with the following command, run on the designated
ResourceManager:
----
$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stop resourcemanager
----
Run a script to stop NodeManagers on all slaves:
----
$ $HADOOP_YARN_HOME/sbin/yarn-daemons.sh --config $HADOOP_CONF_DIR stop nodemanager
----
Stop the WebAppProxy server. If multiple servers are used with load
balancing it should be run on each of them:
----
$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh stop proxyserver --config $HADOOP_CONF_DIR
----
Stop the MapReduce JobHistory Server with the following command, run on the
designated server:
----
$ $HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh stop historyserver --config $HADOOP_CONF_DIR
----
* {Operating the Hadoop Cluster}
Once all the necessary configuration is complete, distribute the files to the
<<<HADOOP_CONF_DIR>>> directory on all the machines.
This section also describes the various Unix users who should be starting the
various components and uses the same Unix accounts and groups used previously:
** Hadoop Startup
To start a Hadoop cluster you will need to start both the HDFS and YARN
cluster.
Format a new distributed filesystem as <hdfs>:
----
[hdfs]$ $HADOOP_PREFIX/bin/hdfs namenode -format <cluster_name>
----
Start the HDFS with the following command, run on the designated NameNode
as <hdfs>:
----
[hdfs]$ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start namenode
----
Run a script to start DataNodes on all slaves as <root> with a special
environment variable <<<HADOOP_SECURE_DN_USER>>> set to <hdfs>:
----
[root]$ HADOOP_SECURE_DN_USER=hdfs $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start datanode
----
Start the YARN with the following command, run on the designated
ResourceManager as <yarn>:
----
[yarn]$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start resourcemanager
----
Run a script to start NodeManagers on all slaves as <yarn>:
----
[yarn]$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start nodemanager
----
Start a standalone WebAppProxy server. Run on the WebAppProxy
server as <yarn>. If multiple servers are used with load balancing
it should be run on each of them:
----
[yarn]$ $HADOOP_YARN_HOME/bin/yarn start proxyserver --config $HADOOP_CONF_DIR
----
Start the MapReduce JobHistory Server with the following command, run on the
designated server as <mapred>:
----
[mapred]$ $HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh start historyserver --config $HADOOP_CONF_DIR
----
** Hadoop Shutdown
Stop the NameNode with the following command, run on the designated NameNode
as <hdfs>:
----
[hdfs]$ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs stop namenode
----
Run a script to stop DataNodes on all slaves as <root>:
----
[root]$ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs stop datanode
----
Stop the ResourceManager with the following command, run on the designated
ResourceManager as <yarn>:
----
[yarn]$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stop resourcemanager
----
Run a script to stop NodeManagers on all slaves as <yarn>:
----
[yarn]$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stop nodemanager
----
Stop the WebAppProxy server. Run on the WebAppProxy server as
<yarn>. If multiple servers are used with load balancing it
should be run on each of them:
----
[yarn]$ $HADOOP_YARN_HOME/bin/yarn stop proxyserver --config $HADOOP_CONF_DIR
----
Stop the MapReduce JobHistory Server with the following command, run on the
designated server as <mapred>:
----
[mapred]$ $HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh stop historyserver --config $HADOOP_CONF_DIR
----
* {Web Interfaces}
Once the Hadoop cluster is up and running check the web-ui of the
components as described below:
*-------------------------+-------------------------+------------------------+
|| Daemon || Web Interface || Notes |
*-------------------------+-------------------------+------------------------+
| NameNode | http://<nn_host:port>/ | Default HTTP port is 50070. |
*-------------------------+-------------------------+------------------------+
| ResourceManager | http://<rm_host:port>/ | Default HTTP port is 8088. |
*-------------------------+-------------------------+------------------------+
| MapReduce JobHistory Server | http://<jhs_host:port>/ | |
| | | Default HTTP port is 19888. |
*-------------------------+-------------------------+------------------------+

283
hadoop-common-project/hadoop-common/src/site/apt/CommandsManual.apt.vm
View File

@ -1,283 +0,0 @@
~~ Licensed to the Apache Software Foundation (ASF) under one or more
~~ contributor license agreements. See the NOTICE file distributed with
~~ this work for additional information regarding copyright ownership.
~~ The ASF licenses this file to You under the Apache License, Version 2.0
~~ (the "License"); you may not use this file except in compliance with
~~ the License. You may obtain a copy of the License at
~~
~~ http://www.apache.org/licenses/LICENSE-2.0
~~
~~ Unless required by applicable law or agreed to in writing, software
~~ distributed under the License is distributed on an "AS IS" BASIS,
~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~~ See the License for the specific language governing permissions and
~~ limitations under the License.
---
Hadoop Commands Guide
---
---
${maven.build.timestamp}
%{toc}
Overview
All hadoop commands are invoked by the <<<bin/hadoop>>> script. Running the
hadoop script without any arguments prints the description for all
commands.
Usage: <<<hadoop [--config confdir] [--loglevel loglevel] [COMMAND]
[GENERIC_OPTIONS] [COMMAND_OPTIONS]>>>
Hadoop has an option parsing framework that employs parsing generic
options as well as running classes.
*-----------------------+---------------+
|| COMMAND_OPTION || Description
*-----------------------+---------------+
| <<<--config confdir>>>| Overwrites the default Configuration directory. Default is <<<${HADOOP_HOME}/conf>>>.
*-----------------------+---------------+
| <<<--loglevel loglevel>>>| Overwrites the log level. Valid log levels are
| | FATAL, ERROR, WARN, INFO, DEBUG, and TRACE.
| | Default is INFO.
*-----------------------+---------------+
| GENERIC_OPTIONS | The common set of options supported by multiple commands.
| COMMAND_OPTIONS | Various commands with their options are described in the following sections. The commands have been grouped into User Commands and Administration Commands.
*-----------------------+---------------+
Generic Options
The following options are supported by {{dfsadmin}}, {{fs}}, {{fsck}},
{{job}} and {{fetchdt}}. Applications should implement
{{{../../api/org/apache/hadoop/util/Tool.html}Tool}} to support
GenericOptions.
*------------------------------------------------+-----------------------------+
|| GENERIC_OPTION || Description
*------------------------------------------------+-----------------------------+
|<<<-conf \<configuration file\> >>> | Specify an application
| configuration file.
*------------------------------------------------+-----------------------------+
|<<<-D \<property\>=\<value\> >>> | Use value for given property.
*------------------------------------------------+-----------------------------+
|<<<-jt \<local\> or \<resourcemanager:port\>>>> | Specify a ResourceManager.
| Applies only to job.
*------------------------------------------------+-----------------------------+
|<<<-files \<comma separated list of files\> >>> | Specify comma separated files
| to be copied to the map
| reduce cluster. Applies only
| to job.
*------------------------------------------------+-----------------------------+
|<<<-libjars \<comma seperated list of jars\> >>>| Specify comma separated jar
| files to include in the
| classpath. Applies only to
| job.
*------------------------------------------------+-----------------------------+
|<<<-archives \<comma separated list of archives\> >>> | Specify comma separated
| archives to be unarchived on
| the compute machines. Applies
| only to job.
*------------------------------------------------+-----------------------------+
User Commands
Commands useful for users of a hadoop cluster.
* <<<archive>>>
Creates a hadoop archive. More information can be found at
{{{../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/HadoopArchives.html}
Hadoop Archives Guide}}.
* <<<credential>>>
Command to manage credentials, passwords and secrets within credential providers.
The CredentialProvider API in Hadoop allows for the separation of applications
and how they store their required passwords/secrets. In order to indicate
a particular provider type and location, the user must provide the
<hadoop.security.credential.provider.path> configuration element in core-site.xml
or use the command line option <<<-provider>>> on each of the following commands.
This provider path is a comma-separated list of URLs that indicates the type and
location of a list of providers that should be consulted.
For example, the following path:
<<<user:///,jceks://file/tmp/test.jceks,jceks://hdfs@nn1.example.com/my/path/test.jceks>>>
indicates that the current user's credentials file should be consulted through
the User Provider, that the local file located at <<</tmp/test.jceks>>> is a Java Keystore
Provider and that the file located within HDFS at <<<nn1.example.com/my/path/test.jceks>>>
is also a store for a Java Keystore Provider.
When utilizing the credential command it will often be for provisioning a password
or secret to a particular credential store provider. In order to explicitly
indicate which provider store to use the <<<-provider>>> option should be used. Otherwise,
given a path of multiple providers, the first non-transient provider will be used.
This may or may not be the one that you intended.
Example: <<<-provider jceks://file/tmp/test.jceks>>>
Usage: <<<hadoop credential <subcommand> [options]>>>
*-------------------+-------------------------------------------------------+
||COMMAND_OPTION || Description
*-------------------+-------------------------------------------------------+
| create <alias> [-v <value>][-provider <provider-path>]| Prompts the user for
| a credential to be stored as the given alias when a value
| is not provided via <<<-v>>>. The
| <hadoop.security.credential.provider.path> within the
| core-site.xml file will be used unless a <<<-provider>>> is
| indicated.
*-------------------+-------------------------------------------------------+
| delete <alias> [-i][-provider <provider-path>] | Deletes the credential with
| the provided alias and optionally warns the user when
| <<<--interactive>>> is used.
| The <hadoop.security.credential.provider.path> within the
| core-site.xml file will be used unless a <<<-provider>>> is
| indicated.
*-------------------+-------------------------------------------------------+
| list [-provider <provider-path>] | Lists all of the credential aliases
| The <hadoop.security.credential.provider.path> within the
| core-site.xml file will be used unless a <<<-provider>>> is
| indicated.
*-------------------+-------------------------------------------------------+
* <<<distcp>>>
Copy file or directories recursively. More information can be found at
{{{../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/DistCp.html}
Hadoop DistCp Guide}}.
* <<<fs>>>
Deprecated, use {{{../hadoop-hdfs/HDFSCommands.html#dfs}<<<hdfs dfs>>>}}
instead.
* <<<fsck>>>
Deprecated, use {{{../hadoop-hdfs/HDFSCommands.html#fsck}<<<hdfs fsck>>>}}
instead.
* <<<fetchdt>>>
Deprecated, use {{{../hadoop-hdfs/HDFSCommands.html#fetchdt}
<<<hdfs fetchdt>>>}} instead.
* <<<jar>>>
Runs a jar file. Users can bundle their Map Reduce code in a jar file and
execute it using this command.
Usage: <<<hadoop jar <jar> [mainClass] args...>>>
The streaming jobs are run via this command. Examples can be referred from
Streaming examples
Word count example is also run using jar command. It can be referred from
Wordcount example
Use {{{../../hadoop-yarn/hadoop-yarn-site/YarnCommands.html#jar}<<<yarn jar>>>}}
to launch YARN applications instead.
* <<<job>>>
Deprecated. Use
{{{../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredCommands.html#job}
<<<mapred job>>>}} instead.
* <<<pipes>>>
Deprecated. Use
{{{../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredCommands.html#pipes}
<<<mapred pipes>>>}} instead.
* <<<queue>>>
Deprecated. Use
{{{../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredCommands.html#queue}
<<<mapred queue>>>}} instead.
* <<<version>>>
Prints the version.
Usage: <<<hadoop version>>>
* <<<CLASSNAME>>>
hadoop script can be used to invoke any class.
Usage: <<<hadoop CLASSNAME>>>
Runs the class named <<<CLASSNAME>>>.
* <<<classpath>>>
Prints the class path needed to get the Hadoop jar and the required
libraries. If called without arguments, then prints the classpath set up by
the command scripts, which is likely to contain wildcards in the classpath
entries. Additional options print the classpath after wildcard expansion or
write the classpath into the manifest of a jar file. The latter is useful in
environments where wildcards cannot be used and the expanded classpath exceeds
the maximum supported command line length.
Usage: <<<hadoop classpath [--glob|--jar <path>|-h|--help]>>>
*-----------------+-----------------------------------------------------------+
|| COMMAND_OPTION || Description
*-----------------+-----------------------------------------------------------+
| --glob | expand wildcards
*-----------------+-----------------------------------------------------------+
| --jar <path> | write classpath as manifest in jar named <path>
*-----------------+-----------------------------------------------------------+
| -h, --help | print help
*-----------------+-----------------------------------------------------------+
Administration Commands
Commands useful for administrators of a hadoop cluster.
* <<<balancer>>>
Deprecated, use {{{../hadoop-hdfs/HDFSCommands.html#balancer}
<<<hdfs balancer>>>}} instead.
* <<<daemonlog>>>
Get/Set the log level for each daemon.
Usage: <<<hadoop daemonlog -getlevel <host:port> <name> >>>
Usage: <<<hadoop daemonlog -setlevel <host:port> <name> <level> >>>
*------------------------------+-----------------------------------------------------------+
|| COMMAND_OPTION || Description
*------------------------------+-----------------------------------------------------------+
| -getlevel <host:port> <name> | Prints the log level of the daemon running at
| <host:port>. This command internally connects
| to http://<host:port>/logLevel?log=<name>
*------------------------------+-----------------------------------------------------------+
| -setlevel <host:port> <name> <level> | Sets the log level of the daemon
| running at <host:port>. This command internally
| connects to http://<host:port>/logLevel?log=<name>
*------------------------------+-----------------------------------------------------------+
* <<<datanode>>>
Deprecated, use {{{../hadoop-hdfs/HDFSCommands.html#datanode}
<<<hdfs datanode>>>}} instead.
* <<<dfsadmin>>>
Deprecated, use {{{../hadoop-hdfs/HDFSCommands.html#dfsadmin}
<<<hdfs dfsadmin>>>}} instead.
* <<<namenode>>>
Deprecated, use {{{../hadoop-hdfs/HDFSCommands.html#namenode}
<<<hdfs namenode>>>}} instead.
* <<<secondarynamenode>>>
Deprecated, use {{{../hadoop-hdfs/HDFSCommands.html#secondarynamenode}
<<<hdfs secondarynamenode>>>}} instead.

337
hadoop-common-project/hadoop-common/src/site/markdown/ClusterSetup.md Normal file
View File

@ -0,0 +1,337 @@
<!---
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
* [Hadoop Cluster Setup](#Hadoop_Cluster_Setup)
* [Purpose](#Purpose)
* [Prerequisites](#Prerequisites)
* [Installation](#Installation)
* [Configuring Hadoop in Non-Secure Mode](#Configuring_Hadoop_in_Non-Secure_Mode)
* [Configuring Environment of Hadoop Daemons](#Configuring_Environment_of_Hadoop_Daemons)
* [Configuring the Hadoop Daemons](#Configuring_the_Hadoop_Daemons)
* [Monitoring Health of NodeManagers](#Monitoring_Health_of_NodeManagers)
* [Slaves File](#Slaves_File)
* [Hadoop Rack Awareness](#Hadoop_Rack_Awareness)
* [Logging](#Logging)
* [Operating the Hadoop Cluster](#Operating_the_Hadoop_Cluster)
* [Hadoop Startup](#Hadoop_Startup)
* [Hadoop Shutdown](#Hadoop_Shutdown)
* [Web Interfaces](#Web_Interfaces)
Hadoop Cluster Setup
====================
Purpose
-------
This document describes how to install and configure Hadoop clusters ranging from a few nodes to extremely large clusters with thousands of nodes. To play with Hadoop, you may first want to install it on a single machine (see [Single Node Setup](./SingleCluster.html)).
This document does not cover advanced topics such as [Security](./SecureMode.html) or High Availability.
Prerequisites
-------------
* Install Java. See the [Hadoop Wiki](http://wiki.apache.org/hadoop/HadoopJavaVersions) for known good versions.
* Download a stable version of Hadoop from Apache mirrors.
Installation
------------
Installing a Hadoop cluster typically involves unpacking the software on all the machines in the cluster or installing it via a packaging system as appropriate for your operating system. It is important to divide up the hardware into functions.
Typically one machine in the cluster is designated as the NameNode and another machine the as ResourceManager, exclusively. These are the masters. Other services (such as Web App Proxy Server and MapReduce Job History server) are usually run either on dedicated hardware or on shared infrastrucutre, depending upon the load.
The rest of the machines in the cluster act as both DataNode and NodeManager. These are the slaves.
Configuring Hadoop in Non-Secure Mode
-------------------------------------
Hadoop's Java configuration is driven by two types of important configuration files:
* Read-only default configuration - `core-default.xml`, `hdfs-default.xml`, `yarn-default.xml` and `mapred-default.xml`.
* Site-specific configuration - `etc/hadoop/core-site.xml`, `etc/hadoop/hdfs-site.xml`, `etc/hadoop/yarn-site.xml` and `etc/hadoop/mapred-site.xml`.
Additionally, you can control the Hadoop scripts found in the bin/ directory of the distribution, by setting site-specific values via the `etc/hadoop/hadoop-env.sh` and `etc/hadoop/yarn-env.sh`.
To configure the Hadoop cluster you will need to configure the `environment` in which the Hadoop daemons execute as well as the `configuration parameters` for the Hadoop daemons.
HDFS daemons are NameNode, SecondaryNameNode, and DataNode. YARN damones are ResourceManager, NodeManager, and WebAppProxy. If MapReduce is to be used, then the MapReduce Job History Server will also be running. For large installations, these are generally running on separate hosts.
### Configuring Environment of Hadoop Daemons
Administrators should use the `etc/hadoop/hadoop-env.sh` and optionally the `etc/hadoop/mapred-env.sh` and `etc/hadoop/yarn-env.sh` scripts to do site-specific customization of the Hadoop daemons' process environment.
At the very least, you must specify the `JAVA_HOME` so that it is correctly defined on each remote node.
Administrators can configure individual daemons using the configuration options shown below in the table:
| Daemon | Environment Variable |
|:---- |:---- |
| NameNode | HADOOP\_NAMENODE\_OPTS |
| DataNode | HADOOP\_DATANODE\_OPTS |
| Secondary NameNode | HADOOP\_SECONDARYNAMENODE\_OPTS |
| ResourceManager | YARN\_RESOURCEMANAGER\_OPTS |
| NodeManager | YARN\_NODEMANAGER\_OPTS |
| WebAppProxy | YARN\_PROXYSERVER\_OPTS |
| Map Reduce Job History Server | HADOOP\_JOB\_HISTORYSERVER\_OPTS |
For example, To configure Namenode to use parallelGC, the following statement should be added in hadoop-env.sh :
export HADOOP_NAMENODE_OPTS="-XX:+UseParallelGC"
See `etc/hadoop/hadoop-env.sh` for other examples.
Other useful configuration parameters that you can customize include:
* `HADOOP_PID_DIR` - The directory where the daemons' process id files are stored.
* `HADOOP_LOG_DIR` - The directory where the daemons' log files are stored. Log files are automatically created if they don't exist.
* `HADOOP_HEAPSIZE` / `YARN_HEAPSIZE` - The maximum amount of heapsize to use, in MB e.g. if the varibale is set to 1000 the heap will be set to 1000MB. This is used to configure the heap size for the daemon. By default, the value is 1000. If you want to configure the values separately for each deamon you can use.
In most cases, you should specify the `HADOOP_PID_DIR` and `HADOOP_LOG_DIR` directories such that they can only be written to by the users that are going to run the hadoop daemons. Otherwise there is the potential for a symlink attack.
It is also traditional to configure `HADOOP_PREFIX` in the system-wide shell environment configuration. For example, a simple script inside `/etc/profile.d`:
HADOOP_PREFIX=/path/to/hadoop
export HADOOP_PREFIX
| Daemon | Environment Variable |
|:---- |:---- |
| ResourceManager | YARN\_RESOURCEMANAGER\_HEAPSIZE |
| NodeManager | YARN\_NODEMANAGER\_HEAPSIZE |
| WebAppProxy | YARN\_PROXYSERVER\_HEAPSIZE |
| Map Reduce Job History Server | HADOOP\_JOB\_HISTORYSERVER\_HEAPSIZE |
### Configuring the Hadoop Daemons
This section deals with important parameters to be specified in the given configuration files:
* `etc/hadoop/core-site.xml`
| Parameter | Value | Notes |
|:---- |:---- |:---- |
| `fs.defaultFS` | NameNode URI | hdfs://host:port/ |
| `io.file.buffer.size` | 131072 | Size of read/write buffer used in SequenceFiles. |
* `etc/hadoop/hdfs-site.xml`
* Configurations for NameNode:
| Parameter | Value | Notes |
|:---- |:---- |:---- |
| `dfs.namenode.name.dir` | Path on the local filesystem where the NameNode stores the namespace and transactions logs persistently. | If this is a comma-delimited list of directories then the name table is replicated in all of the directories, for redundancy. |
| `dfs.hosts` / `dfs.hosts.exclude` | List of permitted/excluded DataNodes. | If necessary, use these files to control the list of allowable datanodes. |
| `dfs.blocksize` | 268435456 | HDFS blocksize of 256MB for large file-systems. |
| `dfs.namenode.handler.count` | 100 | More NameNode server threads to handle RPCs from large number of DataNodes. |
* Configurations for DataNode:
| Parameter | Value | Notes |
|:---- |:---- |:---- |
| `dfs.datanode.data.dir` | Comma separated list of paths on the local filesystem of a `DataNode` where it should store its blocks. | If this is a comma-delimited list of directories, then data will be stored in all named directories, typically on different devices. |
* `etc/hadoop/yarn-site.xml`
* Configurations for ResourceManager and NodeManager:
| Parameter | Value | Notes |
|:---- |:---- |:---- |
| `yarn.acl.enable` | `true` / `false` | Enable ACLs? Defaults to *false*. |
| `yarn.admin.acl` | Admin ACL | ACL to set admins on the cluster. ACLs are of for *comma-separated-usersspacecomma-separated-groups*. Defaults to special value of **\*** which means *anyone*. Special value of just *space* means no one has access. |
| `yarn.log-aggregation-enable` | *false* | Configuration to enable or disable log aggregation |
* Configurations for ResourceManager:
| Parameter | Value | Notes |
|:---- |:---- |:---- |
| `yarn.resourcemanager.address` | `ResourceManager` host:port for clients to submit jobs. | *host:port* If set, overrides the hostname set in `yarn.resourcemanager.hostname`. |
| `yarn.resourcemanager.scheduler.address` | `ResourceManager` host:port for ApplicationMasters to talk to Scheduler to obtain resources. | *host:port* If set, overrides the hostname set in `yarn.resourcemanager.hostname`. |
| `yarn.resourcemanager.resource-tracker.address` | `ResourceManager` host:port for NodeManagers. | *host:port* If set, overrides the hostname set in `yarn.resourcemanager.hostname`. |
| `yarn.resourcemanager.admin.address` | `ResourceManager` host:port for administrative commands. | *host:port* If set, overrides the hostname set in `yarn.resourcemanager.hostname`. |
| `yarn.resourcemanager.webapp.address` | `ResourceManager` web-ui host:port. | *host:port* If set, overrides the hostname set in `yarn.resourcemanager.hostname`. |
| `yarn.resourcemanager.hostname` | `ResourceManager` host. | *host* Single hostname that can be set in place of setting all `yarn.resourcemanager*address` resources. Results in default ports for ResourceManager components. |
| `yarn.resourcemanager.scheduler.class` | `ResourceManager` Scheduler class. | `CapacityScheduler` (recommended), `FairScheduler` (also recommended), or `FifoScheduler` |
| `yarn.scheduler.minimum-allocation-mb` | Minimum limit of memory to allocate to each container request at the `Resource Manager`. | In MBs |
| `yarn.scheduler.maximum-allocation-mb` | Maximum limit of memory to allocate to each container request at the `Resource Manager`. | In MBs |
| `yarn.resourcemanager.nodes.include-path` / `yarn.resourcemanager.nodes.exclude-path` | List of permitted/excluded NodeManagers. | If necessary, use these files to control the list of allowable NodeManagers. |
* Configurations for NodeManager:
| Parameter | Value | Notes |
|:---- |:---- |:---- |
| `yarn.nodemanager.resource.memory-mb` | Resource i.e. available physical memory, in MB, for given `NodeManager` | Defines total available resources on the `NodeManager` to be made available to running containers |
| `yarn.nodemanager.vmem-pmem-ratio` | Maximum ratio by which virtual memory usage of tasks may exceed physical memory | The virtual memory usage of each task may exceed its physical memory limit by this ratio. The total amount of virtual memory used by tasks on the NodeManager may exceed its physical memory usage by this ratio. |
| `yarn.nodemanager.local-dirs` | Comma-separated list of paths on the local filesystem where intermediate data is written. | Multiple paths help spread disk i/o. |
| `yarn.nodemanager.log-dirs` | Comma-separated list of paths on the local filesystem where logs are written. | Multiple paths help spread disk i/o. |
| `yarn.nodemanager.log.retain-seconds` | *10800* | Default time (in seconds) to retain log files on the NodeManager Only applicable if log-aggregation is disabled. |
| `yarn.nodemanager.remote-app-log-dir` | */logs* | HDFS directory where the application logs are moved on application completion. Need to set appropriate permissions. Only applicable if log-aggregation is enabled. |
| `yarn.nodemanager.remote-app-log-dir-suffix` | *logs* | Suffix appended to the remote log dir. Logs will be aggregated to ${yarn.nodemanager.remote-app-log-dir}/${user}/${thisParam} Only applicable if log-aggregation is enabled. |
| `yarn.nodemanager.aux-services` | mapreduce\_shuffle | Shuffle service that needs to be set for Map Reduce applications. |
* Configurations for History Server (Needs to be moved elsewhere):
| Parameter | Value | Notes |
|:---- |:---- |:---- |
| `yarn.log-aggregation.retain-seconds` | *-1* | How long to keep aggregation logs before deleting them. -1 disables. Be careful, set this too small and you will spam the name node. |
| `yarn.log-aggregation.retain-check-interval-seconds` | *-1* | Time between checks for aggregated log retention. If set to 0 or a negative value then the value is computed as one-tenth of the aggregated log retention time. Be careful, set this too small and you will spam the name node. |
* `etc/hadoop/mapred-site.xml`
* Configurations for MapReduce Applications:
| Parameter | Value | Notes |
|:---- |:---- |:---- |
| `mapreduce.framework.name` | yarn | Execution framework set to Hadoop YARN. |
| `mapreduce.map.memory.mb` | 1536 | Larger resource limit for maps. |
| `mapreduce.map.java.opts` | -Xmx1024M | Larger heap-size for child jvms of maps. |
| `mapreduce.reduce.memory.mb` | 3072 | Larger resource limit for reduces. |
| `mapreduce.reduce.java.opts` | -Xmx2560M | Larger heap-size for child jvms of reduces. |
| `mapreduce.task.io.sort.mb` | 512 | Higher memory-limit while sorting data for efficiency. |
| `mapreduce.task.io.sort.factor` | 100 | More streams merged at once while sorting files. |
| `mapreduce.reduce.shuffle.parallelcopies` | 50 | Higher number of parallel copies run by reduces to fetch outputs from very large number of maps. |
* Configurations for MapReduce JobHistory Server:
| Parameter | Value | Notes |
|:---- |:---- |:---- |
| `mapreduce.jobhistory.address` | MapReduce JobHistory Server *host:port* | Default port is 10020. |
| `mapreduce.jobhistory.webapp.address` | MapReduce JobHistory Server Web UI *host:port* | Default port is 19888. |
| `mapreduce.jobhistory.intermediate-done-dir` | /mr-history/tmp | Directory where history files are written by MapReduce jobs. |
| `mapreduce.jobhistory.done-dir` | /mr-history/done | Directory where history files are managed by the MR JobHistory Server. |
Monitoring Health of NodeManagers
---------------------------------
Hadoop provides a mechanism by which administrators can configure the NodeManager to run an administrator supplied script periodically to determine if a node is healthy or not.
Administrators can determine if the node is in a healthy state by performing any checks of their choice in the script. If the script detects the node to be in an unhealthy state, it must print a line to standard output beginning with the string ERROR. The NodeManager spawns the script periodically and checks its output. If the script's output contains the string ERROR, as described above, the node's status is reported as `unhealthy` and the node is black-listed by the ResourceManager. No further tasks will be assigned to this node. However, the NodeManager continues to run the script, so that if the node becomes healthy again, it will be removed from the blacklisted nodes on the ResourceManager automatically. The node's health along with the output of the script, if it is unhealthy, is available to the administrator in the ResourceManager web interface. The time since the node was healthy is also displayed on the web interface.
The following parameters can be used to control the node health monitoring script in `etc/hadoop/yarn-site.xml`.
| Parameter | Value | Notes |
|:---- |:---- |:---- |
| `yarn.nodemanager.health-checker.script.path` | Node health script | Script to check for node's health status. |
| `yarn.nodemanager.health-checker.script.opts` | Node health script options | Options for script to check for node's health status. |
| `yarn.nodemanager.health-checker.script.interval-ms` | Node health script interval | Time interval for running health script. |
| `yarn.nodemanager.health-checker.script.timeout-ms` | Node health script timeout interval | Timeout for health script execution. |
The health checker script is not supposed to give ERROR if only some of the local disks become bad. NodeManager has the ability to periodically check the health of the local disks (specifically checks nodemanager-local-dirs and nodemanager-log-dirs) and after reaching the threshold of number of bad directories based on the value set for the config property yarn.nodemanager.disk-health-checker.min-healthy-disks, the whole node is marked unhealthy and this info is sent to resource manager also. The boot disk is either raided or a failure in the boot disk is identified by the health checker script.
Slaves File
-----------
List all slave hostnames or IP addresses in your `etc/hadoop/slaves` file, one per line. Helper scripts (described below) will use the `etc/hadoop/slaves` file to run commands on many hosts at once. It is not used for any of the Java-based Hadoop configuration. In order to use this functionality, ssh trusts (via either passphraseless ssh or some other means, such as Kerberos) must be established for the accounts used to run Hadoop.
Hadoop Rack Awareness
---------------------
Many Hadoop components are rack-aware and take advantage of the network topology for performance and safety. Hadoop daemons obtain the rack information of the slaves in the cluster by invoking an administrator configured module. See the [Rack Awareness](./RackAwareness.html) documentation for more specific information.
It is highly recommended configuring rack awareness prior to starting HDFS.
Logging
-------
Hadoop uses the [Apache log4j](http://logging.apache.org/log4j/2.x/) via the Apache Commons Logging framework for logging. Edit the `etc/hadoop/log4j.properties` file to customize the Hadoop daemons' logging configuration (log-formats and so on).
Operating the Hadoop Cluster
----------------------------
Once all the necessary configuration is complete, distribute the files to the `HADOOP_CONF_DIR` directory on all the machines. This should be the same directory on all machines.
In general, it is recommended that HDFS and YARN run as separate users. In the majority of installations, HDFS processes execute as 'hdfs'. YARN is typically using the 'yarn' account.
### Hadoop Startup
To start a Hadoop cluster you will need to start both the HDFS and YARN cluster.
The first time you bring up HDFS, it must be formatted. Format a new distributed filesystem as *hdfs*:
[hdfs]$ $HADOOP_PREFIX/bin/hdfs namenode -format <cluster_name>
Start the HDFS NameNode with the following command on the designated node as *hdfs*:
[hdfs]$ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start namenode
Start a HDFS DataNode with the following command on each designated node as *hdfs*:
[hdfs]$ $HADOOP_PREFIX/sbin/hadoop-daemons.sh --config $HADOOP_CONF_DIR --script hdfs start datanode
If `etc/hadoop/slaves` and ssh trusted access is configured (see [Single Node Setup](./SingleCluster.html)), all of the HDFS processes can be started with a utility script. As *hdfs*:
[hdfs]$ $HADOOP_PREFIX/sbin/start-dfs.sh
Start the YARN with the following command, run on the designated ResourceManager as *yarn*:
[yarn]$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start resourcemanager
Run a script to start a NodeManager on each designated host as *yarn*:
[yarn]$ $HADOOP_YARN_HOME/sbin/yarn-daemons.sh --config $HADOOP_CONF_DIR start nodemanager
Start a standalone WebAppProxy server. Run on the WebAppProxy server as *yarn*. If multiple servers are used with load balancing it should be run on each of them:
[yarn]$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start proxyserver
If `etc/hadoop/slaves` and ssh trusted access is configured (see [Single Node Setup](./SingleCluster.html)), all of the YARN processes can be started with a utility script. As *yarn*:
[yarn]$ $HADOOP_PREFIX/sbin/start-yarn.sh
Start the MapReduce JobHistory Server with the following command, run on the designated server as *mapred*:
[mapred]$ $HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh --config $HADOOP_CONF_DIR start historyserver
### Hadoop Shutdown
Stop the NameNode with the following command, run on the designated NameNode as *hdfs*:
[hdfs]$ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs stop namenode
Run a script to stop a DataNode as *hdfs*:
[hdfs]$ $HADOOP_PREFIX/sbin/hadoop-daemons.sh --config $HADOOP_CONF_DIR --script hdfs stop datanode
If `etc/hadoop/slaves` and ssh trusted access is configured (see [Single Node Setup](./SingleCluster.html)), all of the HDFS processes may be stopped with a utility script. As *hdfs*:
[hdfs]$ $HADOOP_PREFIX/sbin/stop-dfs.sh
Stop the ResourceManager with the following command, run on the designated ResourceManager as *yarn*:
[yarn]$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stop resourcemanager
Run a script to stop a NodeManager on a slave as *yarn*:
[yarn]$ $HADOOP_YARN_HOME/sbin/yarn-daemons.sh --config $HADOOP_CONF_DIR stop nodemanager
If `etc/hadoop/slaves` and ssh trusted access is configured (see [Single Node Setup](./SingleCluster.html)), all of the YARN processes can be stopped with a utility script. As *yarn*:
[yarn]$ $HADOOP_PREFIX/sbin/stop-yarn.sh
Stop the WebAppProxy server. Run on the WebAppProxy server as *yarn*. If multiple servers are used with load balancing it should be run on each of them:
[yarn]$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stop proxyserver
Stop the MapReduce JobHistory Server with the following command, run on the designated server as *mapred*:
[mapred]$ $HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh --config $HADOOP_CONF_DIR stop historyserver
Web Interfaces
--------------
Once the Hadoop cluster is up and running check the web-ui of the components as described below:
| Daemon | Web Interface | Notes |
|:---- |:---- |:---- |
| NameNode | http://nn_host:port/ | Default HTTP port is 50070. |
| ResourceManager | http://rm_host:port/ | Default HTTP port is 8088. |
| MapReduce JobHistory Server | http://jhs_host:port/ | Default HTTP port is 19888. |

178
hadoop-common-project/hadoop-common/src/site/markdown/CommandsManual.md Normal file
View File

@ -0,0 +1,178 @@
<!---
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
* [Hadoop Commands Guide](#Hadoop_Commands_Guide)
* [Overview](#Overview)
* [Generic Options](#Generic_Options)
* [Hadoop Common Commands](#Hadoop_Common_Commands)
* [User Commands](#User_Commands)
* [archive](#archive)
* [checknative](#checknative)
* [classpath](#classpath)
* [credential](#credential)
* [distcp](#distcp)
* [fs](#fs)
* [jar](#jar)
* [key](#key)
* [trace](#trace)
* [version](#version)
* [CLASSNAME](#CLASSNAME)
* [Administration Commands](#Administration_Commands)
* [daemonlog](#daemonlog)
Hadoop Commands Guide
=====================
Overview
--------
All hadoop commands are invoked by the `bin/hadoop` script. Running the
hadoop script without any arguments prints the description for all
commands.
Usage: `hadoop [--config confdir] [--loglevel loglevel] [COMMAND] [GENERIC_OPTIONS] [COMMAND_OPTIONS]`
| FIELD | Description |
|:---- |:---- |
| `--config confdir` | Overwrites the default Configuration directory. Default is `${HADOOP_HOME}/conf`. |
| `--loglevel loglevel` | Overwrites the log level. Valid log levels are FATAL, ERROR, WARN, INFO, DEBUG, and TRACE. Default is INFO. |
| GENERIC\_OPTIONS | The common set of options supported by multiple commands. |
| COMMAND\_OPTIONS | Various commands with their options are described in this documention for the Hadoop common sub-project. HDFS and YARN are covered in other documents. |
### Generic Options
Many subcommands honor a common set of configuration options to alter their behavior:
| GENERIC\_OPTION | Description |
|:---- |:---- |
| `-archives <comma separated list of archives> ` | Specify comma separated archives to be unarchived on the compute machines. Applies only to job. |
| `-conf <configuration file> ` | Specify an application configuration file. |
| `-D <property>=<value> ` | Use value for given property. |
| `-files <comma separated list of files> ` | Specify comma separated files to be copied to the map reduce cluster. Applies only to job. |
| `-jt <local> or <resourcemanager:port>` | Specify a ResourceManager. Applies only to job. |
| `-libjars <comma seperated list of jars> ` | Specify comma separated jar files to include in the classpath. Applies only to job. |
Hadoop Common Commands
======================
All of these commands are executed from the `hadoop` shell command. They have been broken up into [User Commands](#User_Commands) and [Admininistration Commands](#Administration_Commands).
User Commands
-------------
Commands useful for users of a hadoop cluster.
### `archive`
Creates a hadoop archive. More information can be found at [Hadoop Archives Guide](../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/HadoopArchives.html).
### `checknative`
Usage: `hadoop checknative [-a] [-h] `
| COMMAND\_OPTION | Description |
|:---- |:---- |
| `-a` | Check all libraries are available. |
| `-h` | print help |
This command checks the availability of the Hadoop native code. See [\#NativeLibraries.html](#NativeLibraries.html) for more information. By default, this command only checks the availability of libhadoop.
### `classpath`
Usage: `hadoop classpath [--glob |--jar <path> |-h |--help]`
| COMMAND\_OPTION | Description |
|:---- |:---- |
| `--glob` | expand wildcards |
| `--jar` *path* | write classpath as manifest in jar named *path* |
| `-h`, `--help` | print help |
Prints the class path needed to get the Hadoop jar and the required libraries. If called without arguments, then prints the classpath set up by the command scripts, which is likely to contain wildcards in the classpath entries. Additional options print the classpath after wildcard expansion or write the classpath into the manifest of a jar file. The latter is useful in environments where wildcards cannot be used and the expanded classpath exceeds the maximum supported command line length.
### `credential`
Usage: `hadoop credential <subcommand> [options]`
| COMMAND\_OPTION | Description |
|:---- |:---- |
| create *alias* [-v *value*][-provider *provider-path*] | Prompts the user for a credential to be stored as the given alias when a value is not provided via `-v`. The *hadoop.security.credential.provider.path* within the core-site.xml file will be used unless a `-provider` is indicated. |
| delete *alias* [-i][-provider *provider-path*] | Deletes the credential with the provided alias and optionally warns the user when `--interactive` is used. The *hadoop.security.credential.provider.path* within the core-site.xml file will be used unless a `-provider` is indicated. |
| list [-provider *provider-path*] | Lists all of the credential aliases The *hadoop.security.credential.provider.path* within the core-site.xml file will be used unless a `-provider` is indicated. |
Command to manage credentials, passwords and secrets within credential providers.
The CredentialProvider API in Hadoop allows for the separation of applications and how they store their required passwords/secrets. In order to indicate a particular provider type and location, the user must provide the *hadoop.security.credential.provider.path* configuration element in core-site.xml or use the command line option `-provider` on each of the following commands. This provider path is a comma-separated list of URLs that indicates the type and location of a list of providers that should be consulted. For example, the following path: `user:///,jceks://file/tmp/test.jceks,jceks://hdfs@nn1.example.com/my/path/test.jceks`
indicates that the current user's credentials file should be consulted through the User Provider, that the local file located at `/tmp/test.jceks` is a Java Keystore Provider and that the file located within HDFS at `nn1.example.com/my/path/test.jceks` is also a store for a Java Keystore Provider.
When utilizing the credential command it will often be for provisioning a password or secret to a particular credential store provider. In order to explicitly indicate which provider store to use the `-provider` option should be used. Otherwise, given a path of multiple providers, the first non-transient provider will be used. This may or may not be the one that you intended.
Example: `hadoop credential list -provider jceks://file/tmp/test.jceks`
### `distcp`
Copy file or directories recursively. More information can be found at [Hadoop DistCp Guide](../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/DistCp.html).
### `fs`
This command is documented in the [File System Shell Guide](./FileSystemShell.html). It is a synonym for `hdfs dfs` when HDFS is in use.
### `jar`
Usage: `hadoop jar <jar> [mainClass] args...`
Runs a jar file.
Use [`yarn jar`](../../hadoop-yarn/hadoop-yarn-site/YarnCommands.html#jar) to launch YARN applications instead.
### `key`
Manage keys via the KeyProvider.
### `trace`
View and modify Hadoop tracing settings. See the [Tracing Guide](./Tracing.html).
### `version`
Usage: `hadoop version`
Prints the version.
### `CLASSNAME`
Usage: `hadoop CLASSNAME`
Runs the class named `CLASSNAME`.
Administration Commands
-----------------------
Commands useful for administrators of a hadoop cluster.
### `daemonlog`
Usage:
hadoop daemonlog -getlevel <host:httpport> <classname>
hadoop daemonlog -setlevel <host:httpport> <classname> <level>
| COMMAND\_OPTION | Description |
|:---- |:---- |
| `-getlevel` *host:httpport* *classname* | Prints the log level of the log identified by a qualified *classname*, in the daemon running at *host:httpport*. This command internally connects to `http://<host:httpport>/logLevel?log=<classname>` |
| `-setlevel` *host:httpport* *classname* *level* | Sets the log level of the log identified by a qualified *classname*, in the daemon running at *host:httpport*. This command internally connects to `http://<host:httpport>/logLevel?log=<classname>&level=<level>` |
Get/Set the log level for a Log identified by a qualified class name in the daemon.
Example: $ bin/hadoop daemonlog -setlevel 127.0.0.1:50070 org.apache.hadoop.hdfs.server.namenode.NameNode DEBUG

1022
hadoop-common-project/hadoop-kms/src/site/apt/index.apt.vm
View File

File diff suppressed because it is too large Load Diff

864
hadoop-common-project/hadoop-kms/src/site/markdown/index.md.vm Normal file
View File

@ -0,0 +1,864 @@
<!---
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
#set ( $H3 = '###' )
#set ( $H4 = '####' )
#set ( $H5 = '#####' )
Hadoop Key Management Server (KMS) - Documentation Sets
=======================================================
Hadoop KMS is a cryptographic key management server based on Hadoop's **KeyProvider** API.
It provides a client and a server components which communicate over HTTP using a REST API.
The client is a KeyProvider implementation interacts with the KMS using the KMS HTTP REST API.
KMS and its client have built-in security and they support HTTP SPNEGO Kerberos authentication and HTTPS secure transport.
KMS is a Java web-application and it runs using a pre-configured Tomcat bundled with the Hadoop distribution.
KMS Client Configuration
------------------------
The KMS client `KeyProvider` uses the **kms** scheme, and the embedded URL must be the URL of the KMS. For example, for a KMS running on `http://localhost:16000/kms`, the KeyProvider URI is `kms://http@localhost:16000/kms`. And, for a KMS running on `https://localhost:16000/kms`, the KeyProvider URI is `kms://https@localhost:16000/kms`
KMS
---
$H3 KMS Configuration
Configure the KMS backing KeyProvider properties in the `etc/hadoop/kms-site.xml` configuration file:
```xml
<property>
<name>hadoop.kms.key.provider.uri</name>
<value>jceks://file@/${user.home}/kms.keystore</value>
</property>
<property>
<name>hadoop.security.keystore.java-keystore-provider.password-file</name>
<value>kms.keystore.password</value>
</property>
```
The password file is looked up in the Hadoop's configuration directory via the classpath.
NOTE: You need to restart the KMS for the configuration changes to take effect.
$H3 KMS Cache
KMS caches keys for short period of time to avoid excessive hits to the underlying key provider.
The Cache is enabled by default (can be dissabled by setting the `hadoop.kms.cache.enable` boolean property to false)
The cache is used with the following 3 methods only, `getCurrentKey()` and `getKeyVersion()` and `getMetadata()`.
For the `getCurrentKey()` method, cached entries are kept for a maximum of 30000 millisecond regardless the number of times the key is being access (to avoid stale keys to be considered current).
For the `getKeyVersion()` method, cached entries are kept with a default inactivity timeout of 600000 milliseconds (10 mins). This time out is configurable via the following property in the `etc/hadoop/kms-site.xml` configuration file:
```xml
<property>
<name>hadoop.kms.cache.enable</name>
<value>true</value>
</property>
<property>
<name>hadoop.kms.cache.timeout.ms</name>
<value>600000</value>
</property>
<property>
<name>hadoop.kms.current.key.cache.timeout.ms</name>
<value>30000</value>
</property>
```
$H3 KMS Aggregated Audit logs
Audit logs are aggregated for API accesses to the GET\_KEY\_VERSION, GET\_CURRENT\_KEY, DECRYPT\_EEK, GENERATE\_EEK operations.
Entries are grouped by the (user,key,operation) combined key for a configurable aggregation interval after which the number of accesses to the specified end-point by the user for a given key is flushed to the audit log.
The Aggregation interval is configured via the property :
<property>
<name>hadoop.kms.aggregation.delay.ms</name>
<value>10000</value>
</property>
$H3 Start/Stop the KMS
To start/stop KMS use KMS's bin/kms.sh script. For example:
hadoop-${project.version} $ sbin/kms.sh start
NOTE: Invoking the script without any parameters list all possible parameters (start, stop, run, etc.). The `kms.sh` script is a wrapper for Tomcat's `catalina.sh` script that sets the environment variables and Java System properties required to run KMS.
$H3 Embedded Tomcat Configuration
To configure the embedded Tomcat go to the `share/hadoop/kms/tomcat/conf`.
KMS pre-configures the HTTP and Admin ports in Tomcat's `server.xml` to 16000 and 16001.
Tomcat logs are also preconfigured to go to Hadoop's `logs/` directory.
The following environment variables (which can be set in KMS's `etc/hadoop/kms-env.sh` script) can be used to alter those values:
* KMS_HTTP_PORT
* KMS_ADMIN_PORT
* KMS_MAX_THREADS
* KMS_LOGNOTE: You need to restart the KMS for the configuration changes to take effect.
$H3 Loading native libraries
The following environment variable (which can be set in KMS's `etc/hadoop/kms-env.sh` script) can be used to specify the location of any required native libraries. For eg. Tomact native Apache Portable Runtime (APR) libraries:
* JAVA_LIBRARY_PATH
$H3 KMS Security Configuration
$H4 Enabling Kerberos HTTP SPNEGO Authentication
Configure the Kerberos `etc/krb5.conf` file with the information of your KDC server.
Create a service principal and its keytab for the KMS, it must be an `HTTP` service principal.
Configure KMS `etc/hadoop/kms-site.xml` with the correct security values, for example:
```xml
<property>
<name>hadoop.kms.authentication.type</name>
<value>kerberos</value>
</property>
<property>
<name>hadoop.kms.authentication.kerberos.keytab</name>
<value>${user.home}/kms.keytab</value>
</property>
<property>
<name>hadoop.kms.authentication.kerberos.principal</name>
<value>HTTP/localhost</value>
</property>
<property>
<name>hadoop.kms.authentication.kerberos.name.rules</name>
<value>DEFAULT</value>
</property>
```
NOTE: You need to restart the KMS for the configuration changes to take effect.
$H4 KMS Proxyuser Configuration
Each proxyuser must be configured in `etc/hadoop/kms-site.xml` using the following properties:
```xml
<property>
<name>hadoop.kms.proxyuser.#USER#.users</name>
<value>*</value>
</property>
<property>
<name>hadoop.kms.proxyuser.#USER#.groups</name>
<value>*</value>
</property>
<property>
<name>hadoop.kms.proxyuser.#USER#.hosts</name>
<value>*</value>
</property>
```
`#USER#` is the username of the proxyuser to configure.
The `users` property indicates the users that can be impersonated.
The `groups` property indicates the groups users being impersonated must belong to.
At least one of the `users` or `groups` properties must be defined. If both are specified, then the configured proxyuser will be able to impersonate and user in the `users` list and any user belonging to one of the groups in the `groups` list.
The `hosts` property indicates from which host the proxyuser can make impersonation requests.
If `users`, `groups` or `hosts` has a `*`, it means there are no restrictions for the proxyuser regarding users, groups or hosts.
$H4 KMS over HTTPS (SSL)
To configure KMS to work over HTTPS the following 2 properties must be set in the `etc/hadoop/kms_env.sh` script (shown with default values):
* KMS_SSL_KEYSTORE_FILE=$HOME/.keystore
* KMS_SSL_KEYSTORE_PASS=password
In the KMS `tomcat/conf` directory, replace the `server.xml` file with the provided `ssl-server.xml` file.
You need to create an SSL certificate for the KMS. As the `kms` Unix user, using the Java `keytool` command to create the SSL certificate:
$ keytool -genkey -alias tomcat -keyalg RSA
You will be asked a series of questions in an interactive prompt. It will create the keystore file, which will be named **.keystore** and located in the `kms` user home directory.
The password you enter for "keystore password" must match the value of the `KMS_SSL_KEYSTORE_PASS` environment variable set in the `kms-env.sh` script in the configuration directory.
The answer to "What is your first and last name?" (i.e. "CN") must be the hostname of the machine where the KMS will be running.
NOTE: You need to restart the KMS for the configuration changes to take effect.
$H4 KMS Access Control
KMS ACLs configuration are defined in the KMS `etc/hadoop/kms-acls.xml` configuration file. This file is hot-reloaded when it changes.
KMS supports both fine grained access control as well as blacklist for kms operations via a set ACL configuration properties.
A user accessing KMS is first checked for inclusion in the Access Control List for the requested operation and then checked for exclusion in the Black list for the operation before access is granted.
```xml
<configuration>
<property>
<name>hadoop.kms.acl.CREATE</name>
<value>*</value>
<description>
ACL for create-key operations.
If the user is not in the GET ACL, the key material is not returned
as part of the response.
</description>
</property>
<property>
<name>hadoop.kms.blacklist.CREATE</name>
<value>hdfs,foo</value>
<description>
Blacklist for create-key operations.
If the user is in the Blacklist, the key material is not returned
as part of the response.
</description>
</property>
<property>
<name>hadoop.kms.acl.DELETE</name>
<value>*</value>
<description>
ACL for delete-key operations.
</description>
</property>
<property>
<name>hadoop.kms.blacklist.DELETE</name>
<value>hdfs,foo</value>
<description>
Blacklist for delete-key operations.
</description>
</property>
<property>
<name>hadoop.kms.acl.ROLLOVER</name>
<value>*</value>
<description>
ACL for rollover-key operations.
If the user is not in the GET ACL, the key material is not returned
as part of the response.
</description>
</property>
<property>
<name>hadoop.kms.blacklist.ROLLOVER</name>
<value>hdfs,foo</value>
<description>
Blacklist for rollover-key operations.
</description>
</property>
<property>
<name>hadoop.kms.acl.GET</name>
<value>*</value>
<description>
ACL for get-key-version and get-current-key operations.
</description>
</property>
<property>
<name>hadoop.kms.blacklist.GET</name>
<value>hdfs,foo</value>
<description>
ACL for get-key-version and get-current-key operations.
</description>
</property>
<property>
<name>hadoop.kms.acl.GET_KEYS</name>
<value>*</value>
<description>
ACL for get-keys operation.
</description>
</property>
<property>
<name>hadoop.kms.blacklist.GET_KEYS</name>
<value>hdfs,foo</value>
<description>
Blacklist for get-keys operation.
</description>
</property>
<property>
<name>hadoop.kms.acl.GET_METADATA</name>
<value>*</value>
<description>
ACL for get-key-metadata and get-keys-metadata operations.
</description>
</property>
<property>
<name>hadoop.kms.blacklist.GET_METADATA</name>
<value>hdfs,foo</value>
<description>
Blacklist for get-key-metadata and get-keys-metadata operations.
</description>
</property>
<property>
<name>hadoop.kms.acl.SET_KEY_MATERIAL</name>
<value>*</value>
<description>
Complimentary ACL for CREATE and ROLLOVER operation to allow the client
to provide the key material when creating or rolling a key.
</description>
</property>
<property>
<name>hadoop.kms.blacklist.SET_KEY_MATERIAL</name>
<value>hdfs,foo</value>
<description>
Complimentary Blacklist for CREATE and ROLLOVER operation to allow the client
to provide the key material when creating or rolling a key.
</description>
</property>
<property>
<name>hadoop.kms.acl.GENERATE_EEK</name>
<value>*</value>
<description>
ACL for generateEncryptedKey
CryptoExtension operations
</description>
</property>
<property>
<name>hadoop.kms.blacklist.GENERATE_EEK</name>
<value>hdfs,foo</value>
<description>
Blacklist for generateEncryptedKey
CryptoExtension operations
</description>
</property>
<property>
<name>hadoop.kms.acl.DECRYPT_EEK</name>
<value>*</value>
<description>
ACL for decrypt EncryptedKey
CryptoExtension operations
</description>
</property>
<property>
<name>hadoop.kms.blacklist.DECRYPT_EEK</name>
<value>hdfs,foo</value>
<description>
Blacklist for decrypt EncryptedKey
CryptoExtension operations
</description>
</property>
</configuration>
```
$H4 Key Access Control
KMS supports access control for all non-read operations at the Key level. All Key Access operations are classified as :
* MANAGEMENT - createKey, deleteKey, rolloverNewVersion
* GENERATE_EEK - generateEncryptedKey, warmUpEncryptedKeys
* DECRYPT_EEK - decryptEncryptedKey
* READ - getKeyVersion, getKeyVersions, getMetadata, getKeysMetadata, getCurrentKey
* ALL - all of the above
These can be defined in the KMS `etc/hadoop/kms-acls.xml` as follows
For all keys for which a key access has not been explicitly configured, It is possible to configure a default key access control for a subset of the operation types.
It is also possible to configure a "whitelist" key ACL for a subset of the operation types. The whitelist key ACL is a whitelist in addition to the explicit or default per-key ACL. That is, if no per-key ACL is explicitly set, a user will be granted access if they are present in the default per-key ACL or the whitelist key ACL. If a per-key ACL is explicitly set, a user will be granted access if they are present in the per-key ACL or the whitelist key ACL.
If no ACL is configured for a specific key AND no default ACL is configured AND no root key ACL is configured for the requested operation, then access will be DENIED.
**NOTE:** The default and whitelist key ACL does not support `ALL` operation qualifier.
```xml
<property>
<name>key.acl.testKey1.MANAGEMENT</name>
<value>*</value>
<description>
ACL for create-key, deleteKey and rolloverNewVersion operations.
</description>
</property>
<property>
<name>key.acl.testKey2.GENERATE_EEK</name>
<value>*</value>
<description>
ACL for generateEncryptedKey operations.
</description>
</property>
<property>
<name>key.acl.testKey3.DECRYPT_EEK</name>
<value>admink3</value>
<description>
ACL for decryptEncryptedKey operations.
</description>
</property>
<property>
<name>key.acl.testKey4.READ</name>
<value>*</value>
<description>
ACL for getKeyVersion, getKeyVersions, getMetadata, getKeysMetadata,
getCurrentKey operations
</description>
</property>
<property>
<name>key.acl.testKey5.ALL</name>
<value>*</value>
<description>
ACL for ALL operations.
</description>
</property>
<property>
<name>whitelist.key.acl.MANAGEMENT</name>
<value>admin1</value>
<description>
whitelist ACL for MANAGEMENT operations for all keys.
</description>
</property>
<!--
'testKey3' key ACL is defined. Since a 'whitelist'
key is also defined for DECRYPT_EEK, in addition to
admink3, admin1 can also perform DECRYPT_EEK operations
on 'testKey3'
-->
<property>
<name>whitelist.key.acl.DECRYPT_EEK</name>
<value>admin1</value>
<description>
whitelist ACL for DECRYPT_EEK operations for all keys.
</description>
</property>
<property>
<name>default.key.acl.MANAGEMENT</name>
<value>user1,user2</value>
<description>
default ACL for MANAGEMENT operations for all keys that are not
explicitly defined.
</description>
</property>
<property>
<name>default.key.acl.GENERATE_EEK</name>
<value>user1,user2</value>
<description>
default ACL for GENERATE_EEK operations for all keys that are not
explicitly defined.
</description>
</property>
<property>
<name>default.key.acl.DECRYPT_EEK</name>
<value>user1,user2</value>
<description>
default ACL for DECRYPT_EEK operations for all keys that are not
explicitly defined.
</description>
</property>
<property>
<name>default.key.acl.READ</name>
<value>user1,user2</value>
<description>
default ACL for READ operations for all keys that are not
explicitly defined.
</description>
</property>
```
$H3 KMS Delegation Token Configuration
KMS delegation token secret manager can be configured with the following properties:
```xml
<property>
<name>hadoop.kms.authentication.delegation-token.update-interval.sec</name>
<value>86400</value>
<description>
How often the master key is rotated, in seconds. Default value 1 day.
</description>
</property>
<property>
<name>hadoop.kms.authentication.delegation-token.max-lifetime.sec</name>
<value>604800</value>
<description>
Maximum lifetime of a delagation token, in seconds. Default value 7 days.
</description>
</property>
<property>
<name>hadoop.kms.authentication.delegation-token.renew-interval.sec</name>
<value>86400</value>
<description>
Renewal interval of a delagation token, in seconds. Default value 1 day.
</description>
</property>
<property>
<name>hadoop.kms.authentication.delegation-token.removal-scan-interval.sec</name>
<value>3600</value>
<description>
Scan interval to remove expired delegation tokens.
</description>
</property>
```
$H3 Using Multiple Instances of KMS Behind a Load-Balancer or VIP
KMS supports multiple KMS instances behind a load-balancer or VIP for scalability and for HA purposes.
When using multiple KMS instances behind a load-balancer or VIP, requests from the same user may be handled by different KMS instances.
KMS instances behind a load-balancer or VIP must be specially configured to work properly as a single logical service.
$H4 HTTP Kerberos Principals Configuration
When KMS instances are behind a load-balancer or VIP, clients will use the hostname of the VIP. For Kerberos SPNEGO authentication, the hostname of the URL is used to construct the Kerberos service name of the server, `HTTP/#HOSTNAME#`. This means that all KMS instances must have a Kerberos service name with the load-balancer or VIP hostname.
In order to be able to access directly a specific KMS instance, the KMS instance must also have Keberos service name with its own hostname. This is required for monitoring and admin purposes.
Both Kerberos service principal credentials (for the load-balancer/VIP hostname and for the actual KMS instance hostname) must be in the keytab file configured for authentication. And the principal name specified in the configuration must be '\*'. For example:
```xml
<property>
<name>hadoop.kms.authentication.kerberos.principal</name>
<value>*</value>
</property>
```
**NOTE:** If using HTTPS, the SSL certificate used by the KMS instance must be configured to support multiple hostnames (see Java 7 `keytool` SAN extension support for details on how to do this).
$H4 HTTP Authentication Signature
KMS uses Hadoop Authentication for HTTP authentication. Hadoop Authentication issues a signed HTTP Cookie once the client has authenticated successfully. This HTTP Cookie has an expiration time, after which it will trigger a new authentication sequence. This is done to avoid triggering the authentication on every HTTP request of a client.
A KMS instance must verify the HTTP Cookie signatures signed by other KMS instances. To do this all KMS instances must share the signing secret.
This secret sharing can be done using a Zookeeper service which is configured in KMS with the following properties in the `kms-site.xml`:
```xml
<property>
<name>hadoop.kms.authentication.signer.secret.provider</name>
<value>zookeeper</value>
<description>
Indicates how the secret to sign the authentication cookies will be
stored. Options are 'random' (default), 'string' and 'zookeeper'.
If using a setup with multiple KMS instances, 'zookeeper' should be used.
</description>
</property>
<property>
<name>hadoop.kms.authentication.signer.secret.provider.zookeeper.path</name>
<value>/hadoop-kms/hadoop-auth-signature-secret</value>
<description>
The Zookeeper ZNode path where the KMS instances will store and retrieve
the secret from.
</description>
</property>
<property>
<name>hadoop.kms.authentication.signer.secret.provider.zookeeper.connection.string</name>
<value>#HOSTNAME#:#PORT#,...</value>
<description>
The Zookeeper connection string, a list of hostnames and port comma
separated.
</description>
</property>
<property>
<name>hadoop.kms.authentication.signer.secret.provider.zookeeper.auth.type</name>
<value>kerberos</value>
<description>
The Zookeeper authentication type, 'none' or 'sasl' (Kerberos).
</description>
</property>
<property>
<name>hadoop.kms.authentication.signer.secret.provider.zookeeper.kerberos.keytab</name>
<value>/etc/hadoop/conf/kms.keytab</value>
<description>
The absolute path for the Kerberos keytab with the credentials to
connect to Zookeeper.
</description>
</property>
<property>
<name>hadoop.kms.authentication.signer.secret.provider.zookeeper.kerberos.principal</name>
<value>kms/#HOSTNAME#</value>
<description>
The Kerberos service principal used to connect to Zookeeper.
</description>
</property>
```
$H4 Delegation Tokens
TBD
$H3 KMS HTTP REST API
$H4 Create a Key
*REQUEST:*
POST http://HOST:PORT/kms/v1/keys
Content-Type: application/json
{
"name" : "<key-name>",
"cipher" : "<cipher>",
"length" : <length>, //int
"material" : "<material>", //base64
"description" : "<description>"
}
*RESPONSE:*
201 CREATED
LOCATION: http://HOST:PORT/kms/v1/key/<key-name>
Content-Type: application/json
{
"name" : "versionName",
"material" : "<material>", //base64, not present without GET ACL
}
$H4 Rollover Key
*REQUEST:*
POST http://HOST:PORT/kms/v1/key/<key-name>
Content-Type: application/json
{
"material" : "<material>",
}
*RESPONSE:*
200 OK
Content-Type: application/json
{
"name" : "versionName",
"material" : "<material>", //base64, not present without GET ACL
}
$H4 Delete Key
*REQUEST:*
DELETE http://HOST:PORT/kms/v1/key/<key-name>
*RESPONSE:*
200 OK
$H4 Get Key Metadata
*REQUEST:*
GET http://HOST:PORT/kms/v1/key/<key-name>/_metadata
*RESPONSE:*
200 OK
Content-Type: application/json
{
"name" : "<key-name>",
"cipher" : "<cipher>",
"length" : <length>, //int
"description" : "<description>",
"created" : <millis-epoc>, //long
"versions" : <versions> //int
}
$H4 Get Current Key
*REQUEST:*
GET http://HOST:PORT/kms/v1/key/<key-name>/_currentversion
*RESPONSE:*
200 OK
Content-Type: application/json
{
"name" : "versionName",
"material" : "<material>", //base64
}
$H4 Generate Encrypted Key for Current KeyVersion
*REQUEST:*
GET http://HOST:PORT/kms/v1/key/<key-name>/_eek?eek_op=generate&num_keys=<number-of-keys-to-generate>
*RESPONSE:*
200 OK
Content-Type: application/json
[
{
"versionName" : "encryptionVersionName",
"iv" : "<iv>", //base64
"encryptedKeyVersion" : {
"versionName" : "EEK",
"material" : "<material>", //base64
}
},
{
"versionName" : "encryptionVersionName",
"iv" : "<iv>", //base64
"encryptedKeyVersion" : {
"versionName" : "EEK",
"material" : "<material>", //base64
}
},
...
]
$H4 Decrypt Encrypted Key
*REQUEST:*
POST http://HOST:PORT/kms/v1/keyversion/<version-name>/_eek?ee_op=decrypt
Content-Type: application/json
{
"name" : "<key-name>",
"iv" : "<iv>", //base64
"material" : "<material>", //base64
}
*RESPONSE:*
200 OK
Content-Type: application/json
{
"name" : "EK",
"material" : "<material>", //base64
}
$H4 Get Key Version
*REQUEST:*
GET http://HOST:PORT/kms/v1/keyversion/<version-name>
*RESPONSE:*
200 OK
Content-Type: application/json
{
"name" : "versionName",
"material" : "<material>", //base64
}
$H4 Get Key Versions
*REQUEST:*
GET http://HOST:PORT/kms/v1/key/<key-name>/_versions
*RESPONSE:*
200 OK
Content-Type: application/json
[
{
"name" : "versionName",
"material" : "<material>", //base64
},
{
"name" : "versionName",
"material" : "<material>", //base64
},
...
]
$H4 Get Key Names
*REQUEST:*
GET http://HOST:PORT/kms/v1/keys/names
*RESPONSE:*
200 OK
Content-Type: application/json
[
"<key-name>",
"<key-name>",
...
]
$H4 Get Keys Metadata
GET http://HOST:PORT/kms/v1/keys/metadata?key=<key-name>&key=<key-name>,...
*RESPONSE:*
200 OK
Content-Type: application/json
[
{
"name" : "<key-name>",
"cipher" : "<cipher>",
"length" : <length>, //int
"description" : "<description>",
"created" : <millis-epoc>, //long
"versions" : <versions> //int
},
{
"name" : "<key-name>",
"cipher" : "<cipher>",
"length" : <length>, //int
"description" : "<description>",
"created" : <millis-epoc>, //long
"versions" : <versions> //int
},
...
]

159
hadoop-hdfs-project/hadoop-hdfs-httpfs/src/site/apt/ServerSetup.apt.vm
View File

@ -1,159 +0,0 @@
~~ Licensed under the Apache License, Version 2.0 (the "License");
~~ you may not use this file except in compliance with the License.
~~ You may obtain a copy of the License at
~~
~~ http://www.apache.org/licenses/LICENSE-2.0
~~
~~ Unless required by applicable law or agreed to in writing, software
~~ distributed under the License is distributed on an "AS IS" BASIS,
~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~~ See the License for the specific language governing permissions and
~~ limitations under the License.
---
Hadoop HDFS over HTTP ${project.version} - Server Setup
---
---
${maven.build.timestamp}
Hadoop HDFS over HTTP ${project.version} - Server Setup
This page explains how to quickly setup HttpFS with Pseudo authentication
against a Hadoop cluster with Pseudo authentication.
* Requirements
* Java 6+
* Maven 3+
* Install HttpFS
+---+
~ $ tar xzf httpfs-${project.version}.tar.gz
+---+
* Configure HttpFS
By default, HttpFS assumes that Hadoop configuration files
(<<<core-site.xml & hdfs-site.xml>>>) are in the HttpFS
configuration directory.
If this is not the case, add to the <<<httpfs-site.xml>>> file the
<<<httpfs.hadoop.config.dir>>> property set to the location
of the Hadoop configuration directory.
* Configure Hadoop
Edit Hadoop <<<core-site.xml>>> and defined the Unix user that will
run the HttpFS server as a proxyuser. For example:
+---+
...
<property>
<name>hadoop.proxyuser.#HTTPFSUSER#.hosts</name>
<value>httpfs-host.foo.com</value>
</property>
<property>
<name>hadoop.proxyuser.#HTTPFSUSER#.groups</name>
<value>*</value>
</property>
...
+---+
IMPORTANT: Replace <<<#HTTPFSUSER#>>> with the Unix user that will
start the HttpFS server.
* Restart Hadoop
You need to restart Hadoop for the proxyuser configuration ot become
active.
* Start/Stop HttpFS
To start/stop HttpFS use HttpFS's bin/httpfs.sh script. For example:
+---+
httpfs-${project.version} $ bin/httpfs.sh start
+---+
NOTE: Invoking the script without any parameters list all possible
parameters (start, stop, run, etc.). The <<<httpfs.sh>>> script is a wrapper
for Tomcat's <<<catalina.sh>>> script that sets the environment variables
and Java System properties required to run HttpFS server.
* Test HttpFS is working
+---+
~ $ curl -i "http://<HTTPFSHOSTNAME>:14000?user.name=babu&op=homedir"
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
{"homeDir":"http:\/\/<HTTPFS_HOST>:14000\/user\/babu"}
+---+
* Embedded Tomcat Configuration
To configure the embedded Tomcat go to the <<<tomcat/conf>>>.
HttpFS preconfigures the HTTP and Admin ports in Tomcat's <<<server.xml>>> to
14000 and 14001.
Tomcat logs are also preconfigured to go to HttpFS's <<<logs/>>> directory.
The following environment variables (which can be set in HttpFS's
<<<conf/httpfs-env.sh>>> script) can be used to alter those values:
* HTTPFS_HTTP_PORT
* HTTPFS_ADMIN_PORT
* HTTPFS_LOG
* HttpFS Configuration
HttpFS supports the following {{{./httpfs-default.html}configuration properties}}
in the HttpFS's <<<conf/httpfs-site.xml>>> configuration file.
* HttpFS over HTTPS (SSL)
To configure HttpFS to work over SSL edit the {{httpfs-env.sh}} script in the
configuration directory setting the {{HTTPFS_SSL_ENABLED}} to {{true}}.
In addition, the following 2 properties may be defined (shown with default
values):
* HTTPFS_SSL_KEYSTORE_FILE=${HOME}/.keystore
* HTTPFS_SSL_KEYSTORE_PASS=password
In the HttpFS <<<tomcat/conf>>> directory, replace the <<<server.xml>>> file
with the <<<ssl-server.xml>>> file.
You need to create an SSL certificate for the HttpFS server. As the
<<<httpfs>>> Unix user, using the Java <<<keytool>>> command to create the
SSL certificate:
+---+
$ keytool -genkey -alias tomcat -keyalg RSA
+---+
You will be asked a series of questions in an interactive prompt. It will
create the keystore file, which will be named <<.keystore>> and located in the
<<<httpfs>>> user home directory.
The password you enter for "keystore password" must match the value of the
<<<HTTPFS_SSL_KEYSTORE_PASS>>> environment variable set in the
<<<httpfs-env.sh>>> script in the configuration directory.
The answer to "What is your first and last name?" (i.e. "CN") must be the
hostname of the machine where the HttpFS Server will be running.
Start HttpFS. It should work over HTTPS.
Using the Hadoop <<<FileSystem>>> API or the Hadoop FS shell, use the
<<<swebhdfs://>>> scheme. Make sure the JVM is picking up the truststore
containing the public key of the SSL certificate if using a self-signed
certificate.

87
hadoop-hdfs-project/hadoop-hdfs-httpfs/src/site/apt/UsingHttpTools.apt.vm
View File

@ -1,87 +0,0 @@
~~ Licensed under the Apache License, Version 2.0 (the "License");
~~ you may not use this file except in compliance with the License.
~~ You may obtain a copy of the License at
~~
~~ http://www.apache.org/licenses/LICENSE-2.0
~~
~~ Unless required by applicable law or agreed to in writing, software
~~ distributed under the License is distributed on an "AS IS" BASIS,
~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~~ See the License for the specific language governing permissions and
~~ limitations under the License.
---
Hadoop HDFS over HTTP ${project.version} - Using HTTP Tools
---
---
${maven.build.timestamp}
Hadoop HDFS over HTTP ${project.version} - Using HTTP Tools
* Security
Out of the box HttpFS supports both pseudo authentication and Kerberos HTTP
SPNEGO authentication.
** Pseudo Authentication
With pseudo authentication the user name must be specified in the
<<<user.name=\<USERNAME\>>>> query string parameter of a HttpFS URL.
For example:
+---+
$ curl "http://<HTTFS_HOST>:14000/webhdfs/v1?op=homedir&user.name=babu"
+---+
** Kerberos HTTP SPNEGO Authentication
Kerberos HTTP SPNEGO authentication requires a tool or library supporting
Kerberos HTTP SPNEGO protocol.
IMPORTANT: If using <<<curl>>>, the <<<curl>>> version being used must support
GSS (<<<curl -V>>> prints out 'GSS' if it supports it).
For example:
+---+
$ kinit
Please enter the password for tucu@LOCALHOST:
$ curl --negotiate -u foo "http://<HTTPFS_HOST>:14000/webhdfs/v1?op=homedir"
Enter host password for user 'foo':
+---+
NOTE: the <<<-u USER>>> option is required by the <<<--negotiate>>> but it is
not used. Use any value as <<<USER>>> and when asked for the password press
[ENTER] as the password value is ignored.
** {Remembering Who I Am} (Establishing an Authenticated Session)
As most authentication mechanisms, Hadoop HTTP authentication authenticates
users once and issues a short-lived authentication token to be presented in
subsequent requests. This authentication token is a signed HTTP Cookie.
When using tools like <<<curl>>>, the authentication token must be stored on
the first request doing authentication, and submitted in subsequent requests.
To do this with curl the <<<-b>>> and <<<-c>>> options to save and send HTTP
Cookies must be used.
For example, the first request doing authentication should save the received
HTTP Cookies.
Using Pseudo Authentication:
+---+
$ curl -c ~/.httpfsauth "http://<HTTPFS_HOST>:14000/webhdfs/v1?op=homedir&user.name=babu"
+---+
Using Kerberos HTTP SPNEGO authentication:
+---+
$ curl --negotiate -u foo -c ~/.httpfsauth "http://<HTTPFS_HOST>:14000/webhdfs/v1?op=homedir"
+---+
Then, subsequent requests forward the previously received HTTP Cookie:
+---+
$ curl -b ~/.httpfsauth "http://<HTTPFS_HOST>:14000/webhdfs/v1?op=liststatus"
+---+

83
hadoop-hdfs-project/hadoop-hdfs-httpfs/src/site/apt/index.apt.vm
View File

@ -1,83 +0,0 @@
~~ Licensed under the Apache License, Version 2.0 (the "License");
~~ you may not use this file except in compliance with the License.
~~ You may obtain a copy of the License at
~~
~~ http://www.apache.org/licenses/LICENSE-2.0
~~
~~ Unless required by applicable law or agreed to in writing, software
~~ distributed under the License is distributed on an "AS IS" BASIS,
~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~~ See the License for the specific language governing permissions and
~~ limitations under the License.
---
Hadoop HDFS over HTTP - Documentation Sets ${project.version}
---
---
${maven.build.timestamp}
Hadoop HDFS over HTTP - Documentation Sets ${project.version}
HttpFS is a server that provides a REST HTTP gateway supporting all HDFS
File System operations (read and write). And it is inteoperable with the
<<webhdfs>> REST HTTP API.
HttpFS can be used to transfer data between clusters running different
versions of Hadoop (overcoming RPC versioning issues), for example using
Hadoop DistCP.
HttpFS can be used to access data in HDFS on a cluster behind of a firewall
(the HttpFS server acts as a gateway and is the only system that is allowed
to cross the firewall into the cluster).
HttpFS can be used to access data in HDFS using HTTP utilities (such as curl
and wget) and HTTP libraries Perl from other languages than Java.
The <<webhdfs>> client FileSytem implementation can be used to access HttpFS
using the Hadoop filesystem command (<<<hadoop fs>>>) line tool as well as
from Java aplications using the Hadoop FileSystem Java API.
HttpFS has built-in security supporting Hadoop pseudo authentication and
HTTP SPNEGO Kerberos and other pluggable authentication mechanims. It also
provides Hadoop proxy user support.
* How Does HttpFS Works?
HttpFS is a separate service from Hadoop NameNode.
HttpFS itself is Java web-application and it runs using a preconfigured Tomcat
bundled with HttpFS binary distribution.
HttpFS HTTP web-service API calls are HTTP REST calls that map to a HDFS file
system operation. For example, using the <<<curl>>> Unix command:
* <<<$ curl http://httpfs-host:14000/webhdfs/v1/user/foo/README.txt>>> returns
the contents of the HDFS <<</user/foo/README.txt>>> file.
* <<<$ curl http://httpfs-host:14000/webhdfs/v1/user/foo?op=list>>> returns the
contents of the HDFS <<</user/foo>>> directory in JSON format.
* <<<$ curl -X POST http://httpfs-host:14000/webhdfs/v1/user/foo/bar?op=mkdirs>>>
creates the HDFS <<</user/foo.bar>>> directory.
* How HttpFS and Hadoop HDFS Proxy differ?
HttpFS was inspired by Hadoop HDFS proxy.
HttpFS can be seen as a full rewrite of Hadoop HDFS proxy.
Hadoop HDFS proxy provides a subset of file system operations (read only),
HttpFS provides support for all file system operations.
HttpFS uses a clean HTTP REST API making its use with HTTP tools more
intuitive.
HttpFS supports Hadoop pseudo authentication, Kerberos SPNEGOS authentication
and Hadoop proxy users. Hadoop HDFS proxy did not.
* User and Developer Documentation
* {{{./ServerSetup.html}HttpFS Server Setup}}
* {{{./UsingHttpTools.html}Using HTTP Tools}}

121
hadoop-hdfs-project/hadoop-hdfs-httpfs/src/site/markdown/ServerSetup.md.vm Normal file
View File

@ -0,0 +1,121 @@
<!---
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
Hadoop HDFS over HTTP - Server Setup
====================================
This page explains how to quickly setup HttpFS with Pseudo authentication against a Hadoop cluster with Pseudo authentication.
Install HttpFS
--------------
~ $ tar xzf httpfs-${project.version}.tar.gz
Configure HttpFS
----------------
By default, HttpFS assumes that Hadoop configuration files (`core-site.xml & hdfs-site.xml`) are in the HttpFS configuration directory.
If this is not the case, add to the `httpfs-site.xml` file the `httpfs.hadoop.config.dir` property set to the location of the Hadoop configuration directory.
Configure Hadoop
----------------
Edit Hadoop `core-site.xml` and defined the Unix user that will run the HttpFS server as a proxyuser. For example:
```xml
<property>
<name>hadoop.proxyuser.#HTTPFSUSER#.hosts</name>
<value>httpfs-host.foo.com</value>
</property>
<property>
<name>hadoop.proxyuser.#HTTPFSUSER#.groups</name>
<value>*</value>
</property>
```
IMPORTANT: Replace `#HTTPFSUSER#` with the Unix user that will start the HttpFS server.
Restart Hadoop
--------------
You need to restart Hadoop for the proxyuser configuration ot become active.
Start/Stop HttpFS
-----------------
To start/stop HttpFS use HttpFS's sbin/httpfs.sh script. For example:
$ sbin/httpfs.sh start
NOTE: Invoking the script without any parameters list all possible parameters (start, stop, run, etc.). The `httpfs.sh` script is a wrapper for Tomcat's `catalina.sh` script that sets the environment variables and Java System properties required to run HttpFS server.
Test HttpFS is working
----------------------
~ $ curl -i "http://<HTTPFSHOSTNAME>:14000?user.name=babu&op=homedir"
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
{"homeDir":"http:\/\/<HTTPFS_HOST>:14000\/user\/babu"}
Embedded Tomcat Configuration
-----------------------------
To configure the embedded Tomcat go to the `tomcat/conf`.
HttpFS preconfigures the HTTP and Admin ports in Tomcat's `server.xml` to 14000 and 14001.
Tomcat logs are also preconfigured to go to HttpFS's `logs/` directory.
The following environment variables (which can be set in HttpFS's `etc/hadoop/httpfs-env.sh` script) can be used to alter those values:
* HTTPFS\_HTTP\_PORT
* HTTPFS\_ADMIN\_PORT
* HADOOP\_LOG\_DIR
HttpFS Configuration
--------------------
HttpFS supports the following [configuration properties](./httpfs-default.html) in the HttpFS's `etc/hadoop/httpfs-site.xml` configuration file.
HttpFS over HTTPS (SSL)
-----------------------
To configure HttpFS to work over SSL edit the [httpfs-env.sh](#httpfs-env.sh) script in the configuration directory setting the [HTTPFS\_SSL\_ENABLED](#HTTPFS_SSL_ENABLED) to [true](#true).
In addition, the following 2 properties may be defined (shown with default values):
* HTTPFS\_SSL\_KEYSTORE\_FILE=$HOME/.keystore
* HTTPFS\_SSL\_KEYSTORE\_PASS=password
In the HttpFS `tomcat/conf` directory, replace the `server.xml` file with the `ssl-server.xml` file.
You need to create an SSL certificate for the HttpFS server. As the `httpfs` Unix user, using the Java `keytool` command to create the SSL certificate:
$ keytool -genkey -alias tomcat -keyalg RSA
You will be asked a series of questions in an interactive prompt. It will create the keystore file, which will be named **.keystore** and located in the `httpfs` user home directory.
The password you enter for "keystore password" must match the value of the `HTTPFS_SSL_KEYSTORE_PASS` environment variable set in the `httpfs-env.sh` script in the configuration directory.
The answer to "What is your first and last name?" (i.e. "CN") must be the hostname of the machine where the HttpFS Server will be running.
Start HttpFS. It should work over HTTPS.
Using the Hadoop `FileSystem` API or the Hadoop FS shell, use the `swebhdfs://` scheme. Make sure the JVM is picking up the truststore containing the public key of the SSL certificate if using a self-signed certificate.

62
hadoop-hdfs-project/hadoop-hdfs-httpfs/src/site/markdown/UsingHttpTools.md Normal file
View File

@ -0,0 +1,62 @@
<!---
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
Hadoop HDFS over HTTP - Using HTTP Tools
========================================
Security
--------
Out of the box HttpFS supports both pseudo authentication and Kerberos HTTP SPNEGO authentication.
### Pseudo Authentication
With pseudo authentication the user name must be specified in the `user.name=<USERNAME>` query string parameter of a HttpFS URL. For example:
$ curl "http://<HTTFS_HOST>:14000/webhdfs/v1?op=homedir&user.name=babu"
### Kerberos HTTP SPNEGO Authentication
Kerberos HTTP SPNEGO authentication requires a tool or library supporting Kerberos HTTP SPNEGO protocol.
IMPORTANT: If using `curl`, the `curl` version being used must support GSS (`curl -V` prints out 'GSS' if it supports it).
For example:
$ kinit
Please enter the password for user@LOCALHOST:
$ curl --negotiate -u foo "http://<HTTPFS_HOST>:14000/webhdfs/v1?op=homedir"
Enter host password for user 'foo':
NOTE: the `-u USER` option is required by the `--negotiate` but it is not used. Use any value as `USER` and when asked for the password press [ENTER] as the password value is ignored.
### Remembering Who I Am (Establishing an Authenticated Session)
As most authentication mechanisms, Hadoop HTTP authentication authenticates users once and issues a short-lived authentication token to be presented in subsequent requests. This authentication token is a signed HTTP Cookie.
When using tools like `curl`, the authentication token must be stored on the first request doing authentication, and submitted in subsequent requests. To do this with curl the `-b` and `-c` options to save and send HTTP Cookies must be used.
For example, the first request doing authentication should save the received HTTP Cookies.
Using Pseudo Authentication:
$ curl -c ~/.httpfsauth "http://<HTTPFS_HOST>:14000/webhdfs/v1?op=homedir&user.name=foo"
Using Kerberos HTTP SPNEGO authentication:
$ curl --negotiate -u foo -c ~/.httpfsauth "http://<HTTPFS_HOST>:14000/webhdfs/v1?op=homedir"
Then, subsequent requests forward the previously received HTTP Cookie:
$ curl -b ~/.httpfsauth "http://<HTTPFS_HOST>:14000/webhdfs/v1?op=liststatus"

50
hadoop-hdfs-project/hadoop-hdfs-httpfs/src/site/markdown/index.md Normal file
View File

@ -0,0 +1,50 @@
<!---
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
Hadoop HDFS over HTTP - Documentation Sets
==========================================
HttpFS is a server that provides a REST HTTP gateway supporting all HDFS File System operations (read and write). And it is inteoperable with the **webhdfs** REST HTTP API.
HttpFS can be used to transfer data between clusters running different versions of Hadoop (overcoming RPC versioning issues), for example using Hadoop DistCP.
HttpFS can be used to access data in HDFS on a cluster behind of a firewall (the HttpFS server acts as a gateway and is the only system that is allowed to cross the firewall into the cluster).
HttpFS can be used to access data in HDFS using HTTP utilities (such as curl and wget) and HTTP libraries Perl from other languages than Java.
The **webhdfs** client FileSytem implementation can be used to access HttpFS using the Hadoop filesystem command (`hadoop fs`) line tool as well as from Java aplications using the Hadoop FileSystem Java API.
HttpFS has built-in security supporting Hadoop pseudo authentication and HTTP SPNEGO Kerberos and other pluggable authentication mechanims. It also provides Hadoop proxy user support.
How Does HttpFS Works?
----------------------
HttpFS is a separate service from Hadoop NameNode.
HttpFS itself is Java web-application and it runs using a preconfigured Tomcat bundled with HttpFS binary distribution.
HttpFS HTTP web-service API calls are HTTP REST calls that map to a HDFS file system operation. For example, using the `curl` Unix command:
* `$ curl http://httpfs-host:14000/webhdfs/v1/user/foo/README.txt` returns the contents of the HDFS `/user/foo/README.txt` file.
* `$ curl http://httpfs-host:14000/webhdfs/v1/user/foo?op=list` returns the contents of the HDFS `/user/foo` directory in JSON format.
* `$ curl -X POST http://httpfs-host:14000/webhdfs/v1/user/foo/bar?op=mkdirs` creates the HDFS `/user/foo.bar` directory.
User and Developer Documentation
--------------------------------
* [HttpFS Server Setup](./ServerSetup.html)
* [Using HTTP Tools](./UsingHttpTools.html)

151
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/apt/DistributedCacheDeploy.apt.vm
View File

@ -1,151 +0,0 @@
~~ Licensed under the Apache License, Version 2.0 (the "License");
~~ you may not use this file except in compliance with the License.
~~ You may obtain a copy of the License at
~~
~~ http://www.apache.org/licenses/LICENSE-2.0
~~
~~ Unless required by applicable law or agreed to in writing, software
~~ distributed under the License is distributed on an "AS IS" BASIS,
~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~~ See the License for the specific language governing permissions and
~~ limitations under the License. See accompanying LICENSE file.
---
Hadoop Map Reduce Next Generation-${project.version} - Distributed Cache Deploy
---
---
${maven.build.timestamp}
Hadoop MapReduce Next Generation - Distributed Cache Deploy
* Introduction
The MapReduce application framework has rudimentary support for deploying a
new version of the MapReduce framework via the distributed cache. By setting
the appropriate configuration properties, users can run a different version
of MapReduce than the one initially deployed to the cluster. For example,
cluster administrators can place multiple versions of MapReduce in HDFS and
configure <<<mapred-site.xml>>> to specify which version jobs will use by
default. This allows the administrators to perform a rolling upgrade of the
MapReduce framework under certain conditions.
* Preconditions and Limitations
The support for deploying the MapReduce framework via the distributed cache
currently does not address the job client code used to submit and query
jobs. It also does not address the <<<ShuffleHandler>>> code that runs as an
auxilliary service within each NodeManager. As a result the following
limitations apply to MapReduce versions that can be successfully deployed via
the distributed cache in a rolling upgrade fashion:
* The MapReduce version must be compatible with the job client code used to
submit and query jobs. If it is incompatible then the job client must be
upgraded separately on any node from which jobs using the new MapReduce
version will be submitted or queried.
* The MapReduce version must be compatible with the configuration files used
by the job client submitting the jobs. If it is incompatible with that
configuration (e.g.: a new property must be set or an existing property
value changed) then the configuration must be updated first.
* The MapReduce version must be compatible with the <<<ShuffleHandler>>>
version running on the nodes in the cluster. If it is incompatible then the
new <<<ShuffleHandler>>> code must be deployed to all the nodes in the
cluster, and the NodeManagers must be restarted to pick up the new
<<<ShuffleHandler>>> code.
* Deploying a New MapReduce Version via the Distributed Cache
Deploying a new MapReduce version consists of three steps:
[[1]] Upload the MapReduce archive to a location that can be accessed by the
job submission client. Ideally the archive should be on the cluster's default
filesystem at a publicly-readable path. See the archive location discussion
below for more details.
[[2]] Configure <<<mapreduce.application.framework.path>>> to point to the
location where the archive is located. As when specifying distributed cache
files for a job, this is a URL that also supports creating an alias for the
archive if a URL fragment is specified. For example,
<<<hdfs:/mapred/framework/hadoop-mapreduce-${project.version}.tar.gz#mrframework>>>
will be localized as <<<mrframework>>> rather than
<<<hadoop-mapreduce-${project.version}.tar.gz>>>.
[[3]] Configure <<<mapreduce.application.classpath>>> to set the proper
classpath to use with the MapReduce archive configured above. NOTE: An error
occurs if <<<mapreduce.application.framework.path>>> is configured but
<<<mapreduce.application.classpath>>> does not reference the base name of the
archive path or the alias if an alias was specified.
** Location of the MapReduce Archive and How It Affects Job Performance
Note that the location of the MapReduce archive can be critical to job
submission and job startup performance. If the archive is not located on the
cluster's default filesystem then it will be copied to the job staging
directory for each job and localized to each node where the job's tasks
run. This will slow down job submission and task startup performance.
If the archive is located on the default filesystem then the job client will
not upload the archive to the job staging directory for each job
submission. However if the archive path is not readable by all cluster users
then the archive will be localized separately for each user on each node
where tasks execute. This can cause unnecessary duplication in the
distributed cache.
When working with a large cluster it can be important to increase the
replication factor of the archive to increase its availability. This will
spread the load when the nodes in the cluster localize the archive for the
first time.
* MapReduce Archives and Classpath Configuration
Setting a proper classpath for the MapReduce archive depends upon the
composition of the archive and whether it has any additional dependencies.
For example, the archive can contain not only the MapReduce jars but also the
necessary YARN, HDFS, and Hadoop Common jars and all other dependencies. In
that case, <<<mapreduce.application.classpath>>> would be configured to
something like the following example, where the archive basename is
hadoop-mapreduce-${project.version}.tar.gz and the archive is organized
internally similar to the standard Hadoop distribution archive:
<<<$HADOOP_CONF_DIR,$PWD/hadoop-mapreduce-${project.version}.tar.gz/hadoop-mapreduce-${project.version}/share/hadoop/mapreduce/*,$PWD/hadoop-mapreduce-${project.version}.tar.gz/hadoop-mapreduce-${project.version}/share/hadoop/mapreduce/lib/*,$PWD/hadoop-mapreduce-${project.version}.tar.gz/hadoop-mapreduce-${project.version}/share/hadoop/common/*,$PWD/hadoop-mapreduce-${project.version}.tar.gz/hadoop-mapreduce-${project.version}/share/hadoop/common/lib/*,$PWD/hadoop-mapreduce-${project.version}.tar.gz/hadoop-mapreduce-${project.version}/share/hadoop/yarn/*,$PWD/hadoop-mapreduce-${project.version}.tar.gz/hadoop-mapreduce-${project.version}/share/hadoop/yarn/lib/*,$PWD/hadoop-mapreduce-${project.version}.tar.gz/hadoop-mapreduce-${project.version}/share/hadoop/hdfs/*,$PWD/hadoop-mapreduce-${project.version}.tar.gz/hadoop-mapreduce-${project.version}/share/hadoop/hdfs/lib/*>>>
Another possible approach is to have the archive consist of just the
MapReduce jars and have the remaining dependencies picked up from the Hadoop
distribution installed on the nodes. In that case, the above example would
change to something like the following:
<<<$HADOOP_CONF_DIR,$PWD/hadoop-mapreduce-${project.version}.tar.gz/hadoop-mapreduce-${project.version}/share/hadoop/mapreduce/*,$PWD/hadoop-mapreduce-${project.version}.tar.gz/hadoop-mapreduce-${project.version}/share/hadoop/mapreduce/lib/*,$HADOOP_COMMON_HOME/share/hadoop/common/*,$HADOOP_COMMON_HOME/share/hadoop/common/lib/*,$HADOOP_HDFS_HOME/share/hadoop/hdfs/*,$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*,$HADOOP_YARN_HOME/share/hadoop/yarn/*,$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*>>>
** NOTE:
If shuffle encryption is also enabled in the cluster, then we could meet the problem that MR job get failed with exception like below:
+---+
2014-10-10 02:17:16,600 WARN [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.Fetcher: Failed to connect to junpingdu-centos5-3.cs1cloud.internal:13562 with 1 map outputs
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1731)
at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Handshaker.java:241)
at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Handshaker.java:235)
at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1206)
at com.sun.net.ssl.internal.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:136)
at com.sun.net.ssl.internal.ssl.Handshaker.processLoop(Handshaker.java:593)
at com.sun.net.ssl.internal.ssl.Handshaker.process_record(Handshaker.java:529)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:925)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1170)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1197)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1181)
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:434)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.setNewClient(AbstractDelegateHttpsURLConnection.java:81)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.setNewClient(AbstractDelegateHttpsURLConnection.java:61)
at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:584)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1193)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:379)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:318)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:427)
....
+---+
This is because MR client (deployed from HDFS) cannot access ssl-client.xml in local FS under directory of $HADOOP_CONF_DIR. To fix the problem, we can add the directory with ssl-client.xml to the classpath of MR which is specified in "mapreduce.application.classpath" as mentioned above. To avoid MR application being affected by other local configurations, it is better to create a dedicated directory for putting ssl-client.xml, e.g. a sub-directory under $HADOOP_CONF_DIR, like: $HADOOP_CONF_DIR/security.

320
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/apt/EncryptedShuffle.apt.vm
View File

@ -1,320 +0,0 @@
~~ Licensed under the Apache License, Version 2.0 (the "License");
~~ you may not use this file except in compliance with the License.
~~ You may obtain a copy of the License at
~~
~~ http://www.apache.org/licenses/LICENSE-2.0
~~
~~ Unless required by applicable law or agreed to in writing, software
~~ distributed under the License is distributed on an "AS IS" BASIS,
~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~~ See the License for the specific language governing permissions and
~~ limitations under the License. See accompanying LICENSE file.
---
Hadoop Map Reduce Next Generation-${project.version} - Encrypted Shuffle
---
---
${maven.build.timestamp}
Hadoop MapReduce Next Generation - Encrypted Shuffle
* {Introduction}
The Encrypted Shuffle capability allows encryption of the MapReduce shuffle
using HTTPS and with optional client authentication (also known as
bi-directional HTTPS, or HTTPS with client certificates). It comprises:
* A Hadoop configuration setting for toggling the shuffle between HTTP and
HTTPS.
* A Hadoop configuration settings for specifying the keystore and truststore
properties (location, type, passwords) used by the shuffle service and the
reducers tasks fetching shuffle data.
* A way to re-load truststores across the cluster (when a node is added or
removed).
* {Configuration}
** <<core-site.xml>> Properties
To enable encrypted shuffle, set the following properties in core-site.xml of
all nodes in the cluster:
*--------------------------------------+---------------------+-----------------+
| <<Property>> | <<Default Value>> | <<Explanation>> |
*--------------------------------------+---------------------+-----------------+
| <<<hadoop.ssl.require.client.cert>>> | <<<false>>> | Whether client certificates are required |
*--------------------------------------+---------------------+-----------------+
| <<<hadoop.ssl.hostname.verifier>>> | <<<DEFAULT>>> | The hostname verifier to provide for HttpsURLConnections. Valid values are: <<DEFAULT>>, <<STRICT>>, <<STRICT_I6>>, <<DEFAULT_AND_LOCALHOST>> and <<ALLOW_ALL>> |
*--------------------------------------+---------------------+-----------------+
| <<<hadoop.ssl.keystores.factory.class>>> | <<<org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory>>> | The KeyStoresFactory implementation to use |
*--------------------------------------+---------------------+-----------------+
| <<<hadoop.ssl.server.conf>>> | <<<ssl-server.xml>>> | Resource file from which ssl server keystore information will be extracted. This file is looked up in the classpath, typically it should be in Hadoop conf/ directory |
*--------------------------------------+---------------------+-----------------+
| <<<hadoop.ssl.client.conf>>> | <<<ssl-client.xml>>> | Resource file from which ssl server keystore information will be extracted. This file is looked up in the classpath, typically it should be in Hadoop conf/ directory |
*--------------------------------------+---------------------+-----------------+
| <<<hadoop.ssl.enabled.protocols>>> | <<<TLSv1>>> | The supported SSL protocols (JDK6 can use <<TLSv1>>, JDK7+ can use <<TLSv1,TLSv1.1,TLSv1.2>>) |
*--------------------------------------+---------------------+-----------------+
<<IMPORTANT:>> Currently requiring client certificates should be set to false.
Refer the {{{ClientCertificates}Client Certificates}} section for details.
<<IMPORTANT:>> All these properties should be marked as final in the cluster
configuration files.
*** Example:
------
...
<property>
<name>hadoop.ssl.require.client.cert</name>
<value>false</value>
<final>true</final>
</property>
<property>
<name>hadoop.ssl.hostname.verifier</name>
<value>DEFAULT</value>
<final>true</final>
</property>
<property>
<name>hadoop.ssl.keystores.factory.class</name>
<value>org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory</value>
<final>true</final>
</property>
<property>
<name>hadoop.ssl.server.conf</name>
<value>ssl-server.xml</value>
<final>true</final>
</property>
<property>
<name>hadoop.ssl.client.conf</name>
<value>ssl-client.xml</value>
<final>true</final>
</property>
...
------
** <<<mapred-site.xml>>> Properties
To enable encrypted shuffle, set the following property in mapred-site.xml
of all nodes in the cluster:
*--------------------------------------+---------------------+-----------------+
| <<Property>> | <<Default Value>> | <<Explanation>> |
*--------------------------------------+---------------------+-----------------+
| <<<mapreduce.shuffle.ssl.enabled>>> | <<<false>>> | Whether encrypted shuffle is enabled |
*--------------------------------------+---------------------+-----------------+
<<IMPORTANT:>> This property should be marked as final in the cluster
configuration files.
*** Example:
------
...
<property>
<name>mapreduce.shuffle.ssl.enabled</name>
<value>true</value>
<final>true</final>
</property>
...
------
The Linux container executor should be set to prevent job tasks from
reading the server keystore information and gaining access to the shuffle
server certificates.
Refer to Hadoop Kerberos configuration for details on how to do this.
* {Keystore and Truststore Settings}
Currently <<<FileBasedKeyStoresFactory>>> is the only <<<KeyStoresFactory>>>
implementation. The <<<FileBasedKeyStoresFactory>>> implementation uses the
following properties, in the <<ssl-server.xml>> and <<ssl-client.xml>> files,
to configure the keystores and truststores.
** <<<ssl-server.xml>>> (Shuffle server) Configuration:
The mapred user should own the <<ssl-server.xml>> file and have exclusive
read access to it.
*---------------------------------------------+---------------------+-----------------+
| <<Property>> | <<Default Value>> | <<Explanation>> |
*---------------------------------------------+---------------------+-----------------+
| <<<ssl.server.keystore.type>>> | <<<jks>>> | Keystore file type |
*---------------------------------------------+---------------------+-----------------+
| <<<ssl.server.keystore.location>>> | NONE | Keystore file location. The mapred user should own this file and have exclusive read access to it. |
*---------------------------------------------+---------------------+-----------------+
| <<<ssl.server.keystore.password>>> | NONE | Keystore file password |
*---------------------------------------------+---------------------+-----------------+
| <<<ssl.server.truststore.type>>> | <<<jks>>> | Truststore file type |
*---------------------------------------------+---------------------+-----------------+
| <<<ssl.server.truststore.location>>> | NONE | Truststore file location. The mapred user should own this file and have exclusive read access to it. |
*---------------------------------------------+---------------------+-----------------+
| <<<ssl.server.truststore.password>>> | NONE | Truststore file password |
*---------------------------------------------+---------------------+-----------------+
| <<<ssl.server.truststore.reload.interval>>> | 10000 | Truststore reload interval, in milliseconds |
*--------------------------------------+----------------------------+-----------------+
*** Example:
------
<configuration>
<!-- Server Certificate Store -->
<property>
<name>ssl.server.keystore.type</name>
<value>jks</value>
</property>
<property>
<name>ssl.server.keystore.location</name>
<value>${user.home}/keystores/server-keystore.jks</value>
</property>
<property>
<name>ssl.server.keystore.password</name>
<value>serverfoo</value>
</property>
<!-- Server Trust Store -->
<property>
<name>ssl.server.truststore.type</name>
<value>jks</value>
</property>
<property>
<name>ssl.server.truststore.location</name>
<value>${user.home}/keystores/truststore.jks</value>
</property>
<property>
<name>ssl.server.truststore.password</name>
<value>clientserverbar</value>
</property>
<property>
<name>ssl.server.truststore.reload.interval</name>
<value>10000</value>
</property>
</configuration>
------
** <<<ssl-client.xml>>> (Reducer/Fetcher) Configuration:
The mapred user should own the <<ssl-client.xml>> file and it should have
default permissions.
*---------------------------------------------+---------------------+-----------------+
| <<Property>> | <<Default Value>> | <<Explanation>> |
*---------------------------------------------+---------------------+-----------------+
| <<<ssl.client.keystore.type>>> | <<<jks>>> | Keystore file type |
*---------------------------------------------+---------------------+-----------------+
| <<<ssl.client.keystore.location>>> | NONE | Keystore file location. The mapred user should own this file and it should have default permissions. |
*---------------------------------------------+---------------------+-----------------+
| <<<ssl.client.keystore.password>>> | NONE | Keystore file password |
*---------------------------------------------+---------------------+-----------------+
| <<<ssl.client.truststore.type>>> | <<<jks>>> | Truststore file type |
*---------------------------------------------+---------------------+-----------------+
| <<<ssl.client.truststore.location>>> | NONE | Truststore file location. The mapred user should own this file and it should have default permissions. |
*---------------------------------------------+---------------------+-----------------+
| <<<ssl.client.truststore.password>>> | NONE | Truststore file password |
*---------------------------------------------+---------------------+-----------------+
| <<<ssl.client.truststore.reload.interval>>> | 10000 | Truststore reload interval, in milliseconds |
*--------------------------------------+----------------------------+-----------------+
*** Example:
------
<configuration>
<!-- Client certificate Store -->
<property>
<name>ssl.client.keystore.type</name>
<value>jks</value>
</property>
<property>
<name>ssl.client.keystore.location</name>
<value>${user.home}/keystores/client-keystore.jks</value>
</property>
<property>
<name>ssl.client.keystore.password</name>
<value>clientfoo</value>
</property>
<!-- Client Trust Store -->
<property>
<name>ssl.client.truststore.type</name>
<value>jks</value>
</property>
<property>
<name>ssl.client.truststore.location</name>
<value>${user.home}/keystores/truststore.jks</value>
</property>
<property>
<name>ssl.client.truststore.password</name>
<value>clientserverbar</value>
</property>
<property>
<name>ssl.client.truststore.reload.interval</name>
<value>10000</value>
</property>
</configuration>
------
* Activating Encrypted Shuffle
When you have made the above configuration changes, activate Encrypted
Shuffle by re-starting all NodeManagers.
<<IMPORTANT:>> Using encrypted shuffle will incur in a significant
performance impact. Users should profile this and potentially reserve
1 or more cores for encrypted shuffle.
* {ClientCertificates} Client Certificates
Using Client Certificates does not fully ensure that the client is a
reducer task for the job. Currently, Client Certificates (their private key)
keystore files must be readable by all users submitting jobs to the cluster.
This means that a rogue job could read such those keystore files and use
the client certificates in them to establish a secure connection with a
Shuffle server. However, unless the rogue job has a proper JobToken, it won't
be able to retrieve shuffle data from the Shuffle server. A job, using its
own JobToken, can only retrieve shuffle data that belongs to itself.
* Reloading Truststores
By default the truststores will reload their configuration every 10 seconds.
If a new truststore file is copied over the old one, it will be re-read,
and its certificates will replace the old ones. This mechanism is useful for
adding or removing nodes from the cluster, or for adding or removing trusted
clients. In these cases, the client or NodeManager certificate is added to
(or removed from) all the truststore files in the system, and the new
configuration will be picked up without you having to restart the NodeManager
daemons.
* Debugging
<<NOTE:>> Enable debugging only for troubleshooting, and then only for jobs
running on small amounts of data. It is very verbose and slows down jobs by
several orders of magnitude. (You might need to increase mapred.task.timeout
to prevent jobs from failing because tasks run so slowly.)
To enable SSL debugging in the reducers, set <<<-Djavax.net.debug=all>>> in
the <<<mapreduce.reduce.child.java.opts>>> property; for example:
------
<property>
<name>mapred.reduce.child.java.opts</name>
<value>-Xmx-200m -Djavax.net.debug=all</value>
</property>
------
You can do this on a per-job basis, or by means of a cluster-wide setting in
the <<<mapred-site.xml>>> file.
To set this property in NodeManager, set it in the <<<yarn-env.sh>>> file:
------
YARN_NODEMANAGER_OPTS="-Djavax.net.debug=all $YARN_NODEMANAGER_OPTS"
------

1605
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/apt/MapReduceTutorial.apt.vm
View File

File diff suppressed because it is too large Load Diff

114
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/apt/MapReduce_Compatibility_Hadoop1_Hadoop2.apt.vm
View File

@ -1,114 +0,0 @@
~~ Licensed under the Apache License, Version 2.0 (the "License");
~~ you may not use this file except in compliance with the License.
~~ You may obtain a copy of the License at
~~
~~ http://www.apache.org/licenses/LICENSE-2.0
~~
~~ Unless required by applicable law or agreed to in writing, software
~~ distributed under the License is distributed on an "AS IS" BASIS,
~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~~ See the License for the specific language governing permissions and
~~ limitations under the License. See accompanying LICENSE file.
---
Hadoop Map Reduce Next Generation-${project.version} - Backward Compatibility
---
---
${maven.build.timestamp}
Apache Hadoop MapReduce - Migrating from Apache Hadoop 1.x to Apache Hadoop 2.x
* {Introduction}
This document provides information for users to migrate their Apache Hadoop
MapReduce applications from Apache Hadoop 1.x to Apache Hadoop 2.x.
In Apache Hadoop 2.x we have spun off resource management capabilities
into Apache Hadoop YARN, a general purpose, distributed application management
framework while Apache Hadoop MapReduce (aka MRv2) remains as a pure
distributed computation framework.
In general, the previous MapReduce runtime (aka MRv1) has been reused and
no major surgery has been conducted on it. Therefore, MRv2 is able to ensure
satisfactory compatibility with MRv1 applications. However, due to some
improvements and code refactorings, a few APIs have been rendered
backward-incompatible.
The remainder of this page will discuss the scope and the level of backward
compatibility that we support in Apache Hadoop MapReduce 2.x (MRv2).
* {Binary Compatibility}
First, we ensure binary compatibility to the applications that use old
<<mapred>> APIs. This means that applications which were built against MRv1
<<mapred>> APIs can run directly on YARN without recompilation, merely by
pointing them to an Apache Hadoop 2.x cluster via configuration.
* {Source Compatibility}
We cannot ensure complete binary compatibility with the applications that use
<<mapreduce>> APIs, as these APIs have evolved a lot since MRv1. However, we
ensure source compatibility for <<mapreduce>> APIs that break binary
compatibility. In other words, users should recompile their applications that
use <<mapreduce>> APIs against MRv2 jars. One notable binary incompatibility
break is Counter and CounterGroup.
* {Not Supported}
MRAdmin has been removed in MRv2 because because <<<mradmin>>> commands
no longer exist. They have been replaced by the commands in <<<rmadmin>>>. We
neither support binary compatibility nor source compatibility for the
applications that use this class directly.
* {Tradeoffs between MRv1 Users and Early MRv2 Adopters}
Unfortunately, maintaining binary compatibility for MRv1 applications may lead
to binary incompatibility issues for early MRv2 adopters, in particular Hadoop
0.23 users. For <<mapred>> APIs, we have chosen to be compatible with MRv1
applications, which have a larger user base. For <<mapreduce>> APIs, if they
don't significantly break Hadoop 0.23 applications, we still change them to be
compatible with MRv1 applications. Below is the list of MapReduce APIs which
are incompatible with Hadoop 0.23.
*-----------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------+
| <<Problematic Function>> | <<Incompatibility Issue>> |
*-----------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------+
| <<<org.apache.hadoop.util.ProgramDriver#drive>>> | Return type changes from <<<void>>> to <<<int>>> |
*-----------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------+
| <<<org.apache.hadoop.mapred.jobcontrol.Job#getMapredJobID>>> | Return type changes from <<<String>>> to <<<JobID>>> |
*-----------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------+
| <<<org.apache.hadoop.mapred.TaskReport#getTaskId>>> | Return type changes from <<<String>>> to <<<TaskID>>> |
*-----------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------+
| <<<org.apache.hadoop.mapred.ClusterStatus#UNINITIALIZED_MEMORY_VALUE>>> | Data type changes from <<<long>>> to <<<int>>> |
*-----------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------+
| <<<org.apache.hadoop.mapreduce.filecache.DistributedCache#getArchiveTimestamps>>> | Return type changes from <<<long[]>>> to <<<String[]>>> |
*-----------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------+
| <<<org.apache.hadoop.mapreduce.filecache.DistributedCache#getFileTimestamps>>> | Return type changes from <<<long[]>>> to <<<String[]>>> |
*-----------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------+
| <<<org.apache.hadoop.mapreduce.Job#failTask>>> | Return type changes from <<<void>>> to <<<boolean>>> |
*-----------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------+
| <<<org.apache.hadoop.mapreduce.Job#killTask>>> | Return type changes from <<<void>>> to <<<boolean>>> |
*-----------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------+
| <<<org.apache.hadoop.mapreduce.Job#getTaskCompletionEvents>>> | Return type changes from <<<o.a.h.mapred.TaskCompletionEvent[]>>> to <<<o.a.h.mapreduce.TaskCompletionEvent[]>>> |
*-----------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------+
* {Malicious}
For the users who are going to try <<<hadoop-examples-1.x.x.jar>>> on YARN,
please note that <<<hadoop -jar hadoop-examples-1.x.x.jar>>> will still use
<<<hadoop-mapreduce-examples-2.x.x.jar>>>, which is installed together with
other MRv2 jars. By default Hadoop framework jars appear before the users'
jars in the classpath, such that the classes from the 2.x.x jar will still be
picked. Users should remove <<<hadoop-mapreduce-examples-2.x.x.jar>>>
from the classpath of all the nodes in a cluster. Otherwise, users need to
set <<<HADOOP_USER_CLASSPATH_FIRST=true>>> and
<<<HADOOP_CLASSPATH=...:hadoop-examples-1.x.x.jar>>> to run their target
examples jar, and add the following configuration in <<<mapred-site.xml>>> to
make the processes in YARN containers pick this jar as well.
+---+
<property>
<name>mapreduce.job.user.classpath.first</name>
<value>true</value>
</property>
+---+

2709
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/apt/MapredAppMasterRest.apt.vm
View File

File diff suppressed because it is too large Load Diff

227
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/apt/MapredCommands.apt.vm
View File

@ -1,227 +0,0 @@
~~ Licensed to the Apache Software Foundation (ASF) under one or more
~~ contributor license agreements. See the NOTICE file distributed with
~~ this work for additional information regarding copyright ownership.
~~ The ASF licenses this file to You under the Apache License, Version 2.0
~~ (the "License"); you may not use this file except in compliance with
~~ the License. You may obtain a copy of the License at
~~
~~ http://www.apache.org/licenses/LICENSE-2.0
~~
~~ Unless required by applicable law or agreed to in writing, software
~~ distributed under the License is distributed on an "AS IS" BASIS,
~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~~ See the License for the specific language governing permissions and
~~ limitations under the License.
---
MapReduce Commands Guide
---
---
${maven.build.timestamp}
MapReduce Commands Guide
%{toc|section=1|fromDepth=2|toDepth=4}
* Overview
MapReduce commands are invoked by the <<<bin/mapred>>> script. Running the
script without any arguments prints the description for all commands.
Usage: <<<mapred [--config confdir] [--loglevel loglevel] COMMAND>>>
MapReduce has an option parsing framework that employs parsing generic
options as well as running classes.
*-------------------------+---------------------------------------------------+
|| COMMAND_OPTIONS || Description |
*-------------------------+---------------------------------------------------+
| --config confdir | Overwrites the default Configuration directory. Default
| | is $\{HADOOP_PREFIX\}/conf.
*-------------------------+---------------------------------------------------+
| --loglevel loglevel | Overwrites the log level. Valid log levels are FATAL,
| | ERROR, WARN, INFO, DEBUG, and TRACE. Default is INFO.
*-------------------------+---------------------------------------------------+
| COMMAND COMMAND_OPTIONS | Various commands with their options are described
| | in the following sections. The commands have been
| | grouped into {{User Commands}} and
| | {{Administration Commands}}.
*-------------------------+---------------------------------------------------+
* User Commands
Commands useful for users of a hadoop cluster.
** <<<pipes>>>
Runs a pipes job.
Usage: <<<mapred pipes [-conf <path>] [-jobconf <key=value>, <key=value>,
...] [-input <path>] [-output <path>] [-jar <jar file>] [-inputformat
<class>] [-map <class>] [-partitioner <class>] [-reduce <class>] [-writer
<class>] [-program <executable>] [-reduces <num>]>>>
*----------------------------------------+------------------------------------+
|| COMMAND_OPTION || Description
*----------------------------------------+------------------------------------+
| -conf <path> | Configuration for job
*----------------------------------------+------------------------------------+
| -jobconf <key=value>, <key=value>, ... | Add/override configuration for job
*----------------------------------------+------------------------------------+
| -input <path> | Input directory
*----------------------------------------+------------------------------------+
| -output <path> | Output directory
*----------------------------------------+------------------------------------+
| -jar <jar file> | Jar filename
*----------------------------------------+------------------------------------+
| -inputformat <class> | InputFormat class
*----------------------------------------+------------------------------------+
| -map <class> | Java Map class
*----------------------------------------+------------------------------------+
| -partitioner <class> | Java Partitioner
*----------------------------------------+------------------------------------+
| -reduce <class> | Java Reduce class
*----------------------------------------+------------------------------------+
| -writer <class> | Java RecordWriter
*----------------------------------------+------------------------------------+
| -program <executable> | Executable URI
*----------------------------------------+------------------------------------+
| -reduces <num> | Number of reduces
*----------------------------------------+------------------------------------+
** <<<job>>>
Command to interact with Map Reduce Jobs.
Usage: <<<mapred job
| [{{{../../hadoop-project-dist/hadoop-common/CommandsManual.html#Generic_Options}GENERIC_OPTIONS}}]
| [-submit <job-file>]
| [-status <job-id>]
| [-counter <job-id> <group-name> <counter-name>]
| [-kill <job-id>]
| [-events <job-id> <from-event-#> <#-of-events>]
| [-history [all] <jobOutputDir>] | [-list [all]]
| [-kill-task <task-id>] | [-fail-task <task-id>]
| [-set-priority <job-id> <priority>]>>>
*------------------------------+---------------------------------------------+
|| COMMAND_OPTION || Description
*------------------------------+---------------------------------------------+
| -submit <job-file> | Submits the job.
*------------------------------+---------------------------------------------+
| -status <job-id> | Prints the map and reduce completion
| percentage and all job counters.
*------------------------------+---------------------------------------------+
| -counter <job-id> <group-name> <counter-name> | Prints the counter value.
*------------------------------+---------------------------------------------+
| -kill <job-id> | Kills the job.
*------------------------------+---------------------------------------------+
| -events <job-id> <from-event-#> <#-of-events> | Prints the events' details
| received by jobtracker for the given range.
*------------------------------+---------------------------------------------+
| -history [all]<jobOutputDir> | Prints job details, failed and killed tip
| details. More details about the job such as
| successful tasks and task attempts made for
| each task can be viewed by specifying the
| [all] option.
*------------------------------+---------------------------------------------+
| -list [all] | Displays jobs which are yet to complete.
| <<<-list all>>> displays all jobs.
*------------------------------+---------------------------------------------+
| -kill-task <task-id> | Kills the task. Killed tasks are NOT counted
| against failed attempts.
*------------------------------+---------------------------------------------+
| -fail-task <task-id> | Fails the task. Failed tasks are counted
| against failed attempts.
*------------------------------+---------------------------------------------+
| -set-priority <job-id> <priority> | Changes the priority of the job. Allowed
| priority values are VERY_HIGH, HIGH, NORMAL,
| LOW, VERY_LOW
*------------------------------+---------------------------------------------+
** <<<queue>>>
command to interact and view Job Queue information
Usage: <<<mapred queue [-list] | [-info <job-queue-name> [-showJobs]]
| [-showacls]>>>
*-----------------+-----------------------------------------------------------+
|| COMMAND_OPTION || Description
*-----------------+-----------------------------------------------------------+
| -list | Gets list of Job Queues configured in the system.
| Along with scheduling information associated with the job
| queues.
*-----------------+-----------------------------------------------------------+
| -info <job-queue-name> [-showJobs] | Displays the job queue information and
| associated scheduling information of particular job queue.
| If <<<-showJobs>>> options is present a list of jobs
| submitted to the particular job queue is displayed.
*-----------------+-----------------------------------------------------------+
| -showacls | Displays the queue name and associated queue operations
| allowed for the current user. The list consists of only
| those queues to which the user has access.
*-----------------+-----------------------------------------------------------+
** <<<classpath>>>
Prints the class path needed to get the Hadoop jar and the required
libraries.
Usage: <<<mapred classpath>>>
** <<<distcp>>>
Copy file or directories recursively. More information can be found at
{{{./DistCp.html}Hadoop DistCp Guide}}.
** <<<archive>>>
Creates a hadoop archive. More information can be found at
{{{./HadoopArchives.html}Hadoop Archives Guide}}.
* Administration Commands
Commands useful for administrators of a hadoop cluster.
** <<<historyserver>>>
Start JobHistoryServer.
Usage: <<<mapred historyserver>>>
** <<<hsadmin>>>
Runs a MapReduce hsadmin client for execute JobHistoryServer administrative
commands.
Usage: <<<mapred hsadmin
[-refreshUserToGroupsMappings] |
[-refreshSuperUserGroupsConfiguration] |
[-refreshAdminAcls] |
[-refreshLoadedJobCache] |
[-refreshLogRetentionSettings] |
[-refreshJobRetentionSettings] |
[-getGroups [username]] | [-help [cmd]]>>>
*-----------------+-----------------------------------------------------------+
|| COMMAND_OPTION || Description
*-----------------+-----------------------------------------------------------+
| -refreshUserToGroupsMappings | Refresh user-to-groups mappings
*-----------------+-----------------------------------------------------------+
| -refreshSuperUserGroupsConfiguration| Refresh superuser proxy groups mappings
*-----------------+-----------------------------------------------------------+
| -refreshAdminAcls | Refresh acls for administration of Job history server
*-----------------+-----------------------------------------------------------+
| -refreshLoadedJobCache | Refresh loaded job cache of Job history server
*-----------------+-----------------------------------------------------------+
| -refreshJobRetentionSettings|Refresh job history period, job cleaner settings
*-----------------+-----------------------------------------------------------+
| -refreshLogRetentionSettings | Refresh log retention period and log retention
| | check interval
*-----------------+-----------------------------------------------------------+
| -getGroups [username] | Get the groups which given user belongs to
*-----------------+-----------------------------------------------------------+
| -help [cmd] | Displays help for the given command or all commands if none is
| | specified.
*-----------------+-----------------------------------------------------------+

98
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/apt/PluggableShuffleAndPluggableSort.apt.vm
View File

@ -1,98 +0,0 @@
~~ Licensed under the Apache License, Version 2.0 (the "License");
~~ you may not use this file except in compliance with the License.
~~ You may obtain a copy of the License at
~~
~~ http://www.apache.org/licenses/LICENSE-2.0
~~
~~ Unless required by applicable law or agreed to in writing, software
~~ distributed under the License is distributed on an "AS IS" BASIS,
~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~~ See the License for the specific language governing permissions and
~~ limitations under the License. See accompanying LICENSE file.
---
Hadoop Map Reduce Next Generation-${project.version} - Pluggable Shuffle and Pluggable Sort
---
---
${maven.build.timestamp}
Hadoop MapReduce Next Generation - Pluggable Shuffle and Pluggable Sort
* Introduction
The pluggable shuffle and pluggable sort capabilities allow replacing the
built in shuffle and sort logic with alternate implementations. Example use
cases for this are: using a different application protocol other than HTTP
such as RDMA for shuffling data from the Map nodes to the Reducer nodes; or
replacing the sort logic with custom algorithms that enable Hash aggregation
and Limit-N query.
<<IMPORTANT:>> The pluggable shuffle and pluggable sort capabilities are
experimental and unstable. This means the provided APIs may change and break
compatibility in future versions of Hadoop.
* Implementing a Custom Shuffle and a Custom Sort
A custom shuffle implementation requires a
<<<org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.AuxiliaryService>>>
implementation class running in the NodeManagers and a
<<<org.apache.hadoop.mapred.ShuffleConsumerPlugin>>> implementation class
running in the Reducer tasks.
The default implementations provided by Hadoop can be used as references:
* <<<org.apache.hadoop.mapred.ShuffleHandler>>>
* <<<org.apache.hadoop.mapreduce.task.reduce.Shuffle>>>
A custom sort implementation requires a <<<org.apache.hadoop.mapred.MapOutputCollector>>>
implementation class running in the Mapper tasks and (optionally, depending
on the sort implementation) a <<<org.apache.hadoop.mapred.ShuffleConsumerPlugin>>>
implementation class running in the Reducer tasks.
The default implementations provided by Hadoop can be used as references:
* <<<org.apache.hadoop.mapred.MapTask$MapOutputBuffer>>>
* <<<org.apache.hadoop.mapreduce.task.reduce.Shuffle>>>
* Configuration
Except for the auxiliary service running in the NodeManagers serving the
shuffle (by default the <<<ShuffleHandler>>>), all the pluggable components
run in the job tasks. This means, they can be configured on per job basis.
The auxiliary service servicing the Shuffle must be configured in the
NodeManagers configuration.
** Job Configuration Properties (on per job basis):
*--------------------------------------+---------------------+-----------------+
| <<Property>> | <<Default Value>> | <<Explanation>> |
*--------------------------------------+---------------------+-----------------+
| <<<mapreduce.job.reduce.shuffle.consumer.plugin.class>>> | <<<org.apache.hadoop.mapreduce.task.reduce.Shuffle>>> | The <<<ShuffleConsumerPlugin>>> implementation to use |
*--------------------------------------+---------------------+-----------------+
| <<<mapreduce.job.map.output.collector.class>>> | <<<org.apache.hadoop.mapred.MapTask$MapOutputBuffer>>> | The <<<MapOutputCollector>>> implementation(s) to use |
*--------------------------------------+---------------------+-----------------+
These properties can also be set in the <<<mapred-site.xml>>> to change the default values for all jobs.
The collector class configuration may specify a comma-separated list of collector implementations.
In this case, the map task will attempt to instantiate each in turn until one of the
implementations successfully initializes. This can be useful if a given collector
implementation is only compatible with certain types of keys or values, for example.
** NodeManager Configuration properties, <<<yarn-site.xml>>> in all nodes:
*--------------------------------------+---------------------+-----------------+
| <<Property>> | <<Default Value>> | <<Explanation>> |
*--------------------------------------+---------------------+-----------------+
| <<<yarn.nodemanager.aux-services>>> | <<<...,mapreduce_shuffle>>> | The auxiliary service name |
*--------------------------------------+---------------------+-----------------+
| <<<yarn.nodemanager.aux-services.mapreduce_shuffle.class>>> | <<<org.apache.hadoop.mapred.ShuffleHandler>>> | The auxiliary service class to use |
*--------------------------------------+---------------------+-----------------+
<<IMPORTANT:>> If setting an auxiliary service in addition the default
<<<mapreduce_shuffle>>> service, then a new service key should be added to the
<<<yarn.nodemanager.aux-services>>> property, for example <<<mapred.shufflex>>>.
Then the property defining the corresponding class must be
<<<yarn.nodemanager.aux-services.mapreduce_shufflex.class>>>.

119
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/DistributedCacheDeploy.md.vm Normal file
View File

@ -0,0 +1,119 @@
<!---
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
#set ( $H3 = '###' )
#set ( $H4 = '####' )
#set ( $H5 = '#####' )
Hadoop: Distributed Cache Deploy
================================
Introduction
------------
The MapReduce application framework has rudimentary support for deploying a new version of the MapReduce framework via the distributed cache. By setting the appropriate configuration properties, users can run a different version of MapReduce than the one initially deployed to the cluster. For example, cluster administrators can place multiple versions of MapReduce in HDFS and configure `mapred-site.xml` to specify which version jobs will use by default. This allows the administrators to perform a rolling upgrade of the MapReduce framework under certain conditions.
Preconditions and Limitations
-----------------------------
The support for deploying the MapReduce framework via the distributed cache currently does not address the job client code used to submit and query jobs. It also does not address the `ShuffleHandler` code that runs as an auxilliary service within each NodeManager. As a result the following limitations apply to MapReduce versions that can be successfully deployed via the distributed cache in a rolling upgrade fashion:
* The MapReduce version must be compatible with the job client code used to
submit and query jobs. If it is incompatible then the job client must be
upgraded separately on any node from which jobs using the new MapReduce
version will be submitted or queried.
* The MapReduce version must be compatible with the configuration files used
by the job client submitting the jobs. If it is incompatible with that
configuration (e.g.: a new property must be set or an existing property
value changed) then the configuration must be updated first.
* The MapReduce version must be compatible with the `ShuffleHandler`
version running on the nodes in the cluster. If it is incompatible then the
new `ShuffleHandler` code must be deployed to all the nodes in the
cluster, and the NodeManagers must be restarted to pick up the new
`ShuffleHandler` code.
Deploying a New MapReduce Version via the Distributed Cache
-----------------------------------------------------------
Deploying a new MapReduce version consists of three steps:
1. Upload the MapReduce archive to a location that can be accessed by the
job submission client. Ideally the archive should be on the cluster's default
filesystem at a publicly-readable path. See the archive location discussion
below for more details.
2. Configure `mapreduce.application.framework.path` to point to the
location where the archive is located. As when specifying distributed cache
files for a job, this is a URL that also supports creating an alias for the
archive if a URL fragment is specified. For example,
`hdfs:/mapred/framework/hadoop-mapreduce-${project.version}.tar.gz#mrframework`
will be localized as `mrframework` rather than
`hadoop-mapreduce-${project.version}.tar.gz`.
3. Configure `mapreduce.application.classpath` to set the proper
classpath to use with the MapReduce archive configured above. NOTE: An error
occurs if `mapreduce.application.framework.path` is configured but
`mapreduce.application.classpath` does not reference the base name of the
archive path or the alias if an alias was specified.
$H3 Location of the MapReduce Archive and How It Affects Job Performance
Note that the location of the MapReduce archive can be critical to job submission and job startup performance. If the archive is not located on the cluster's default filesystem then it will be copied to the job staging directory for each job and localized to each node where the job's tasks run. This will slow down job submission and task startup performance.
If the archive is located on the default filesystem then the job client will not upload the archive to the job staging directory for each job submission. However if the archive path is not readable by all cluster users then the archive will be localized separately for each user on each node where tasks execute. This can cause unnecessary duplication in the distributed cache.
When working with a large cluster it can be important to increase the replication factor of the archive to increase its availability. This will spread the load when the nodes in the cluster localize the archive for the first time.
MapReduce Archives and Classpath Configuration
----------------------------------------------
Setting a proper classpath for the MapReduce archive depends upon the composition of the archive and whether it has any additional dependencies. For example, the archive can contain not only the MapReduce jars but also the necessary YARN, HDFS, and Hadoop Common jars and all other dependencies. In that case, `mapreduce.application.classpath` would be configured to something like the following example, where the archive basename is hadoop-mapreduce-${project.version}.tar.gz and the archive is organized internally similar to the standard Hadoop distribution archive:
`$HADOOP_CONF_DIR,$PWD/hadoop-mapreduce-${project.version}.tar.gz/hadoop-mapreduce-${project.version}/share/hadoop/mapreduce/*,$PWD/hadoop-mapreduce-${project.version}.tar.gz/hadoop-mapreduce-${project.version}/share/hadoop/mapreduce/lib/*,$PWD/hadoop-mapreduce-${project.version}.tar.gz/hadoop-mapreduce-${project.version}/share/hadoop/common/*,$PWD/hadoop-mapreduce-${project.version}.tar.gz/hadoop-mapreduce-${project.version}/share/hadoop/common/lib/*,$PWD/hadoop-mapreduce-${project.version}.tar.gz/hadoop-mapreduce-${project.version}/share/hadoop/yarn/*,$PWD/hadoop-mapreduce-${project.version}.tar.gz/hadoop-mapreduce-${project.version}/share/hadoop/yarn/lib/*,$PWD/hadoop-mapreduce-${project.version}.tar.gz/hadoop-mapreduce-${project.version}/share/hadoop/hdfs/*,$PWD/hadoop-mapreduce-${project.version}.tar.gz/hadoop-mapreduce-${project.version}/share/hadoop/hdfs/lib/*`
Another possible approach is to have the archive consist of just the MapReduce jars and have the remaining dependencies picked up from the Hadoop distribution installed on the nodes. In that case, the above example would change to something like the following:
`$HADOOP_CONF_DIR,$PWD/hadoop-mapreduce-${project.version}.tar.gz/hadoop-mapreduce-${project.version}/share/hadoop/mapreduce/*,$PWD/hadoop-mapreduce-${project.version}.tar.gz/hadoop-mapreduce-${project.version}/share/hadoop/mapreduce/lib/*,$HADOOP_COMMON_HOME/share/hadoop/common/*,$HADOOP_COMMON_HOME/share/hadoop/common/lib/*,$HADOOP_HDFS_HOME/share/hadoop/hdfs/*,$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*,$HADOOP_YARN_HOME/share/hadoop/yarn/*,$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*`
$H3 NOTE:
If shuffle encryption is also enabled in the cluster, then we could meet the problem that MR job get failed with exception like below:
2014-10-10 02:17:16,600 WARN [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.Fetcher: Failed to connect to junpingdu-centos5-3.cs1cloud.internal:13562 with 1 map outputs
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1731)
at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Handshaker.java:241)
at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Handshaker.java:235)
at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1206)
at com.sun.net.ssl.internal.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:136)
at com.sun.net.ssl.internal.ssl.Handshaker.processLoop(Handshaker.java:593)
at com.sun.net.ssl.internal.ssl.Handshaker.process_record(Handshaker.java:529)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:925)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1170)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1197)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1181)
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:434)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.setNewClient(AbstractDelegateHttpsURLConnection.java:81)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.setNewClient(AbstractDelegateHttpsURLConnection.java:61)
at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:584)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1193)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:379)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:318)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:427)
....
This is because MR client (deployed from HDFS) cannot access ssl-client.xml in local FS under directory of $HADOOP\_CONF\_DIR. To fix the problem, we can add the directory with ssl-client.xml to the classpath of MR which is specified in "mapreduce.application.classpath" as mentioned above. To avoid MR application being affected by other local configurations, it is better to create a dedicated directory for putting ssl-client.xml, e.g. a sub-directory under $HADOOP\_CONF\_DIR, like: $HADOOP\_CONF\_DIR/security.

255
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/EncryptedShuffle.md Normal file
View File

@ -0,0 +1,255 @@
<!---
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
Hadoop: Encrypted Shuffle
=========================
Introduction
------------
The Encrypted Shuffle capability allows encryption of the MapReduce shuffle using HTTPS and with optional client authentication (also known as bi-directional HTTPS, or HTTPS with client certificates). It comprises:
* A Hadoop configuration setting for toggling the shuffle between HTTP and
HTTPS.
* A Hadoop configuration settings for specifying the keystore and truststore
properties (location, type, passwords) used by the shuffle service and the
reducers tasks fetching shuffle data.
* A way to re-load truststores across the cluster (when a node is added or
removed).
Configuration
-------------
### **core-site.xml** Properties
To enable encrypted shuffle, set the following properties in core-site.xml of all nodes in the cluster:
| **Property** | **Default Value** | **Explanation** |
|:---- |:---- |:---- |
| `hadoop.ssl.require.client.cert` | `false` | Whether client certificates are required |
| `hadoop.ssl.hostname.verifier` | `DEFAULT` | The hostname verifier to provide for HttpsURLConnections. Valid values are: **DEFAULT**, **STRICT**, **STRICT\_I6**, **DEFAULT\_AND\_LOCALHOST** and **ALLOW\_ALL** |
| `hadoop.ssl.keystores.factory.class` | `org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory` | The KeyStoresFactory implementation to use |
| `hadoop.ssl.server.conf` | `ssl-server.xml` | Resource file from which ssl server keystore information will be extracted. This file is looked up in the classpath, typically it should be in Hadoop conf/ directory |
| `hadoop.ssl.client.conf` | `ssl-client.xml` | Resource file from which ssl server keystore information will be extracted. This file is looked up in the classpath, typically it should be in Hadoop conf/ directory |
| `hadoop.ssl.enabled.protocols` | `TLSv1` | The supported SSL protocols (JDK6 can use **TLSv1**, JDK7+ can use **TLSv1,TLSv1.1,TLSv1.2**) |
**IMPORTANT:** Currently requiring client certificates should be set to false. Refer the [Client Certificates](#Client_Certificates) section for details.
**IMPORTANT:** All these properties should be marked as final in the cluster configuration files.
#### Example:
```xml
<property>
<name>hadoop.ssl.require.client.cert</name>
<value>false</value>
<final>true</final>
</property>
<property>
<name>hadoop.ssl.hostname.verifier</name>
<value>DEFAULT</value>
<final>true</final>
</property>
<property>
<name>hadoop.ssl.keystores.factory.class</name>
<value>org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory</value>
<final>true</final>
</property>
<property>
<name>hadoop.ssl.server.conf</name>
<value>ssl-server.xml</value>
<final>true</final>
</property>
<property>
<name>hadoop.ssl.client.conf</name>
<value>ssl-client.xml</value>
<final>true</final>
</property>
```
### `mapred-site.xml` Properties
To enable encrypted shuffle, set the following property in mapred-site.xml of all nodes in the cluster:
| **Property** | **Default Value** | **Explanation** |
|:---- |:---- |:---- |
| `mapreduce.shuffle.ssl.enabled` | `false` | Whether encrypted shuffle is enabled |
**IMPORTANT:** This property should be marked as final in the cluster configuration files.
#### Example:
```xml
<property>
<name>mapreduce.shuffle.ssl.enabled</name>
<value>true</value>
<final>true</final>
</property>
```
The Linux container executor should be set to prevent job tasks from reading the server keystore information and gaining access to the shuffle server certificates.
Refer to Hadoop Kerberos configuration for details on how to do this.
Keystore and Truststore Settings
--------------------------------
Currently `FileBasedKeyStoresFactory` is the only `KeyStoresFactory` implementation. The `FileBasedKeyStoresFactory` implementation uses the following properties, in the **ssl-server.xml** and **ssl-client.xml** files, to configure the keystores and truststores.
### `ssl-server.xml` (Shuffle server) Configuration:
The mapred user should own the **ssl-server.xml** file and have exclusive read access to it.
| **Property** | **Default Value** | **Explanation** |
|:---- |:---- |:---- |
| `ssl.server.keystore.type` | `jks` | Keystore file type |
| `ssl.server.keystore.location` | NONE | Keystore file location. The mapred user should own this file and have exclusive read access to it. |
| `ssl.server.keystore.password` | NONE | Keystore file password |
| `ssl.server.truststore.type` | `jks` | Truststore file type |
| `ssl.server.truststore.location` | NONE | Truststore file location. The mapred user should own this file and have exclusive read access to it. |
| `ssl.server.truststore.password` | NONE | Truststore file password |
| `ssl.server.truststore.reload.interval` | 10000 | Truststore reload interval, in milliseconds |
#### Example:
```xml
<configuration>
<!-- Server Certificate Store -->
<property>
<name>ssl.server.keystore.type</name>
<value>jks</value>
</property>
<property>
<name>ssl.server.keystore.location</name>
<value>${user.home}/keystores/server-keystore.jks</value>
</property>
<property>
<name>ssl.server.keystore.password</name>
<value>serverfoo</value>
</property>
<!-- Server Trust Store -->
<property>
<name>ssl.server.truststore.type</name>
<value>jks</value>
</property>
<property>
<name>ssl.server.truststore.location</name>
<value>${user.home}/keystores/truststore.jks</value>
</property>
<property>
<name>ssl.server.truststore.password</name>
<value>clientserverbar</value>
</property>
<property>
<name>ssl.server.truststore.reload.interval</name>
<value>10000</value>
</property>
</configuration>
```
### `ssl-client.xml` (Reducer/Fetcher) Configuration:
The mapred user should own the **ssl-client.xml** file and it should have default permissions.
| **Property** | **Default Value** | **Explanation** |
|:---- |:---- |:---- |
| `ssl.client.keystore.type` | `jks` | Keystore file type |
| `ssl.client.keystore.location` | NONE | Keystore file location. The mapred user should own this file and it should have default permissions. |
| `ssl.client.keystore.password` | NONE | Keystore file password |
| `ssl.client.truststore.type` | `jks` | Truststore file type |
| `ssl.client.truststore.location` | NONE | Truststore file location. The mapred user should own this file and it should have default permissions. |
| `ssl.client.truststore.password` | NONE | Truststore file password |
| `ssl.client.truststore.reload.interval` | 10000 | Truststore reload interval, in milliseconds |
#### Example:
```xml
<configuration>
<!-- Client certificate Store -->
<property>
<name>ssl.client.keystore.type</name>
<value>jks</value>
</property>
<property>
<name>ssl.client.keystore.location</name>
<value>${user.home}/keystores/client-keystore.jks</value>
</property>
<property>
<name>ssl.client.keystore.password</name>
<value>clientfoo</value>
</property>
<!-- Client Trust Store -->
<property>
<name>ssl.client.truststore.type</name>
<value>jks</value>
</property>
<property>
<name>ssl.client.truststore.location</name>
<value>${user.home}/keystores/truststore.jks</value>
</property>
<property>
<name>ssl.client.truststore.password</name>
<value>clientserverbar</value>
</property>
<property>
<name>ssl.client.truststore.reload.interval</name>
<value>10000</value>
</property>
</configuration>
```
Activating Encrypted Shuffle
----------------------------
When you have made the above configuration changes, activate Encrypted Shuffle by re-starting all NodeManagers.
**IMPORTANT:** Using encrypted shuffle will incur in a significant performance impact. Users should profile this and potentially reserve 1 or more cores for encrypted shuffle.
Client Certificates
-------------------
Using Client Certificates does not fully ensure that the client is a reducer task for the job. Currently, Client Certificates (their private key) keystore files must be readable by all users submitting jobs to the cluster. This means that a rogue job could read such those keystore files and use the client certificates in them to establish a secure connection with a Shuffle server. However, unless the rogue job has a proper JobToken, it won't be able to retrieve shuffle data from the Shuffle server. A job, using its own JobToken, can only retrieve shuffle data that belongs to itself.
Reloading Truststores
---------------------
By default the truststores will reload their configuration every 10 seconds. If a new truststore file is copied over the old one, it will be re-read, and its certificates will replace the old ones. This mechanism is useful for adding or removing nodes from the cluster, or for adding or removing trusted clients. In these cases, the client or NodeManager certificate is added to (or removed from) all the truststore files in the system, and the new configuration will be picked up without you having to restart the NodeManager daemons.
Debugging
---------
**NOTE:** Enable debugging only for troubleshooting, and then only for jobs running on small amounts of data. It is very verbose and slows down jobs by several orders of magnitude. (You might need to increase mapred.task.timeout to prevent jobs from failing because tasks run so slowly.)
To enable SSL debugging in the reducers, set `-Djavax.net.debug=all` in the `mapreduce.reduce.child.java.opts` property; for example:
<property>
<name>mapred.reduce.child.java.opts</name>
<value>-Xmx-200m -Djavax.net.debug=all</value>
</property>
You can do this on a per-job basis, or by means of a cluster-wide setting in the `mapred-site.xml` file.
To set this property in NodeManager, set it in the `yarn-env.sh` file:
YARN_NODEMANAGER_OPTS="-Djavax.net.debug=all"

1156
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/MapReduceTutorial.md Normal file
View File

File diff suppressed because it is too large Load Diff

69
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/MapReduce_Compatibility_Hadoop1_Hadoop2.md Normal file
View File

@ -0,0 +1,69 @@
<!---
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
Apache Hadoop MapReduce - Migrating from Apache Hadoop 1.x to Apache Hadoop 2.x
===============================================================================
Introduction
------------
This document provides information for users to migrate their Apache Hadoop MapReduce applications from Apache Hadoop 1.x to Apache Hadoop 2.x.
In Apache Hadoop 2.x we have spun off resource management capabilities into Apache Hadoop YARN, a general purpose, distributed application management framework while Apache Hadoop MapReduce (aka MRv2) remains as a pure distributed computation framework.
In general, the previous MapReduce runtime (aka MRv1) has been reused and no major surgery has been conducted on it. Therefore, MRv2 is able to ensure satisfactory compatibility with MRv1 applications. However, due to some improvements and code refactorings, a few APIs have been rendered backward-incompatible.
The remainder of this page will discuss the scope and the level of backward compatibility that we support in Apache Hadoop MapReduce 2.x (MRv2).
Binary Compatibility
--------------------
First, we ensure binary compatibility to the applications that use old **mapred** APIs. This means that applications which were built against MRv1 **mapred** APIs can run directly on YARN without recompilation, merely by pointing them to an Apache Hadoop 2.x cluster via configuration.
Source Compatibility
--------------------
We cannot ensure complete binary compatibility with the applications that use **mapreduce** APIs, as these APIs have evolved a lot since MRv1. However, we ensure source compatibility for **mapreduce** APIs that break binary compatibility. In other words, users should recompile their applications that use **mapreduce** APIs against MRv2 jars. One notable binary incompatibility break is Counter and CounterGroup.
Not Supported
-------------
MRAdmin has been removed in MRv2 because because `mradmin` commands no longer exist. They have been replaced by the commands in `rmadmin`. We neither support binary compatibility nor source compatibility for the applications that use this class directly.
Tradeoffs between MRv1 Users and Early MRv2 Adopters
----------------------------------------------------
Unfortunately, maintaining binary compatibility for MRv1 applications may lead to binary incompatibility issues for early MRv2 adopters, in particular Hadoop 0.23 users. For **mapred** APIs, we have chosen to be compatible with MRv1 applications, which have a larger user base. For **mapreduce** APIs, if they don't significantly break Hadoop 0.23 applications, we still change them to be compatible with MRv1 applications. Below is the list of MapReduce APIs which are incompatible with Hadoop 0.23.
| **Problematic Function** | **Incompatibility Issue** |
|:---- |:---- |
| `org.apache.hadoop.util.ProgramDriver#drive` | Return type changes from `void` to `int` |
| `org.apache.hadoop.mapred.jobcontrol.Job#getMapredJobID` | Return type changes from `String` to `JobID` |
| `org.apache.hadoop.mapred.TaskReport#getTaskId` | Return type changes from `String` to `TaskID` |
| `org.apache.hadoop.mapred.ClusterStatus#UNINITIALIZED_MEMORY_VALUE` | Data type changes from `long` to `int` |
| `org.apache.hadoop.mapreduce.filecache.DistributedCache#getArchiveTimestamps` | Return type changes from `long[]` to `String[]` |
| `org.apache.hadoop.mapreduce.filecache.DistributedCache#getFileTimestamps` | Return type changes from `long[]` to `String[]` |
| `org.apache.hadoop.mapreduce.Job#failTask` | Return type changes from `void` to `boolean` |
| `org.apache.hadoop.mapreduce.Job#killTask` | Return type changes from `void` to `boolean` |
| `org.apache.hadoop.mapreduce.Job#getTaskCompletionEvents` | Return type changes from `o.a.h.mapred.TaskCompletionEvent[]` to `o.a.h.mapreduce.TaskCompletionEvent[]` |
Malicious
---------
For the users who are going to try `hadoop-examples-1.x.x.jar` on YARN, please note that `hadoop -jar hadoop-examples-1.x.x.jar` will still use `hadoop-mapreduce-examples-2.x.x.jar`, which is installed together with other MRv2 jars. By default Hadoop framework jars appear before the users' jars in the classpath, such that the classes from the 2.x.x jar will still be picked. Users should remove `hadoop-mapreduce-examples-2.x.x.jar` from the classpath of all the nodes in a cluster. Otherwise, users need to set `HADOOP_USER_CLASSPATH_FIRST=true` and `HADOOP_CLASSPATH=...:hadoop-examples-1.x.x.jar` to run their target examples jar, and add the following configuration in `mapred-site.xml` to make the processes in YARN containers pick this jar as well.
<property>
<name>mapreduce.job.user.classpath.first</name>
<value>true</value>
</property>

2397
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/MapredAppMasterRest.md Normal file
View File

File diff suppressed because it is too large Load Diff

151
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/MapredCommands.md Normal file
View File

@ -0,0 +1,151 @@
<!---
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
MapReduce Commands Guide
========================
* [Overview](#Overview)
* [User Commands](#User_Commands)
* [archive](#archive)
* [classpath](#classpath)
* [distcp](#distcp)
* [job](#job)
* [pipes](#pipes)
* [queue](#queue)
* [version](#version)
* [Administration Commands](#Administration_Commands)
* [historyserver](#historyserver)
* [hsadmin](#hsadmin)
Overview
--------
All mapreduce commands are invoked by the `bin/mapred` script. Running the mapred script without any arguments prints the description for all commands.
Usage: `mapred [SHELL_OPTIONS] COMMAND [GENERIC_OPTIONS] [COMMAND_OPTIONS]`
Hadoop has an option parsing framework that employs parsing generic options as well as running classes.
| COMMAND\_OPTIONS | Description |
|:---- |:---- |
| SHELL\_OPTIONS | The common set of shell options. These are documented on the [Hadoop Commands Reference](../../hadoop-project-dist/hadoop-common/CommandsManual.html#Shell_Options) page. |
| GENERIC\_OPTIONS | The common set of options supported by multiple commands. See the [Hadoop Commands Reference](../../hadoop-project-dist/hadoop-common/CommandsManual.html#Generic_Options) for more information. |
| COMMAND COMMAND\_OPTIONS | Various commands with their options are described in the following sections. The commands have been grouped into [User Commands](#User_Commands) and [Administration Commands](#Administration_Commands). |
User Commands
-------------
Commands useful for users of a hadoop cluster.
### `archive`
Creates a hadoop archive. More information can be found at
[Hadoop Archives Guide](./HadoopArchives.html).
### `classpath`
Prints the class path needed to get the Hadoop jar and the required libraries.
Usage: `mapred classpath`
### `distcp`
Copy file or directories recursively. More information can be found at
[Hadoop DistCp Guide](./DistCp.html).
### `job`
Command to interact with Map Reduce Jobs.
Usage: `mapred job | [GENERIC_OPTIONS] | [-submit <job-file>] | [-status <job-id>] | [-counter <job-id> <group-name> <counter-name>] | [-kill <job-id>] | [-events <job-id> <from-event-#> <#-of-events>] | [-history [all] <jobOutputDir>] | [-list [all]] | [-kill-task <task-id>] | [-fail-task <task-id>] | [-set-priority <job-id> <priority>]`
| COMMAND\_OPTION | Description |
|:---- |:---- |
| -submit *job-file* | Submits the job. |
| -status *job-id* | Prints the map and reduce completion percentage and all job counters. |
| -counter *job-id* *group-name* *counter-name* | Prints the counter value. |
| -kill *job-id* | Kills the job. |
| -events *job-id* *from-event-\#* *\#-of-events* | Prints the events' details received by jobtracker for the given range. |
| -history [all]*jobOutputDir* | Prints job details, failed and killed tip details. More details about the job such as successful tasks and task attempts made for each task can be viewed by specifying the [all] option. |
| -list [all] | Displays jobs which are yet to complete. `-list all` displays all jobs. |
| -kill-task *task-id* | Kills the task. Killed tasks are NOT counted against failed attempts. |
| -fail-task *task-id* | Fails the task. Failed tasks are counted against failed attempts. |
| -set-priority *job-id* *priority* | Changes the priority of the job. Allowed priority values are VERY\_HIGH, HIGH, NORMAL, LOW, VERY\_LOW |
### `pipes`
Runs a pipes job.
Usage: `mapred pipes [-conf <path>] [-jobconf <key=value>, <key=value>, ...] [-input <path>] [-output <path>] [-jar <jar file>] [-inputformat <class>] [-map <class>] [-partitioner <class>] [-reduce <class>] [-writer <class>] [-program <executable>] [-reduces <num>]`
| COMMAND\_OPTION | Description |
|:---- |:---- |
| -conf *path* | Configuration for job |
| -jobconf *key=value*, *key=value*, ... | Add/override configuration for job |
| -input *path* | Input directory |
| -output *path* | Output directory |
| -jar *jar file* | Jar filename |
| -inputformat *class* | InputFormat class |
| -map *class* | Java Map class |
| -partitioner *class* | Java Partitioner |
| -reduce *class* | Java Reduce class |
| -writer *class* | Java RecordWriter |
| -program *executable* | Executable URI |
| -reduces *num* | Number of reduces |
### `queue`
command to interact and view Job Queue information
Usage: `mapred queue [-list] | [-info <job-queue-name> [-showJobs]] | [-showacls]`
| COMMAND\_OPTION | Description |
|:---- |:---- |
| -list | Gets list of Job Queues configured in the system. Along with scheduling information associated with the job queues. |
| -info *job-queue-name* [-showJobs] | Displays the job queue information and associated scheduling information of particular job queue. If `-showJobs` options is present a list of jobs submitted to the particular job queue is displayed. |
| -showacls | Displays the queue name and associated queue operations allowed for the current user. The list consists of only those queues to which the user has access. |
### `version`
Prints the version.
Usage: `mapred version`
Administration Commands
-----------------------
Commands useful for administrators of a hadoop cluster.
### `historyserver`
Start JobHistoryServer.
Usage: `mapred historyserver`
### `hsadmin`
Runs a MapReduce hsadmin client for execute JobHistoryServer administrative commands.
Usage: `mapred hsadmin [-refreshUserToGroupsMappings] | [-refreshSuperUserGroupsConfiguration] | [-refreshAdminAcls] | [-refreshLoadedJobCache] | [-refreshLogRetentionSettings] | [-refreshJobRetentionSettings] | [-getGroups [username]] | [-help [cmd]]`
| COMMAND\_OPTION | Description |
|:---- |:---- |
| -refreshUserToGroupsMappings | Refresh user-to-groups mappings |
| -refreshSuperUserGroupsConfiguration | Refresh superuser proxy groups mappings |
| -refreshAdminAcls | Refresh acls for administration of Job history server |
| -refreshLoadedJobCache | Refresh loaded job cache of Job history server |
| -refreshJobRetentionSettings | Refresh job history period, job cleaner settings |
| -refreshLogRetentionSettings | Refresh log retention period and log retention check interval |
| -getGroups [username] | Get the groups which given user belongs to |
| -help [cmd] | Displays help for the given command or all commands if none is specified. |

73
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/PluggableShuffleAndPluggableSort.md Normal file
View File

@ -0,0 +1,73 @@
<!---
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
Hadoop: Pluggable Shuffle and Pluggable Sort
============================================
Introduction
------------
The pluggable shuffle and pluggable sort capabilities allow replacing the built in shuffle and sort logic with alternate implementations. Example use cases for this are: using a different application protocol other than HTTP such as RDMA for shuffling data from the Map nodes to the Reducer nodes; or replacing the sort logic with custom algorithms that enable Hash aggregation and Limit-N query.
**IMPORTANT:** The pluggable shuffle and pluggable sort capabilities are experimental and unstable. This means the provided APIs may change and break compatibility in future versions of Hadoop.
Implementing a Custom Shuffle and a Custom Sort
-----------------------------------------------
A custom shuffle implementation requires a
`org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.AuxiliaryService`
implementation class running in the NodeManagers and a
`org.apache.hadoop.mapred.ShuffleConsumerPlugin`
implementation class running in the Reducer tasks.
The default implementations provided by Hadoop can be used as references:
* `org.apache.hadoop.mapred.ShuffleHandler`
* `org.apache.hadoop.mapreduce.task.reduce.Shuffle`
A custom sort implementation requires a `org.apache.hadoop.mapred.MapOutputCollector` implementation class running in the Mapper tasks and (optionally, depending on the sort implementation) a `org.apache.hadoop.mapred.ShuffleConsumerPlugin` implementation class running in the Reducer tasks.
The default implementations provided by Hadoop can be used as references:
* `org.apache.hadoop.mapred.MapTask$MapOutputBuffer`
* `org.apache.hadoop.mapreduce.task.reduce.Shuffle`
Configuration
-------------
Except for the auxiliary service running in the NodeManagers serving the shuffle (by default the `ShuffleHandler`), all the pluggable components run in the job tasks. This means, they can be configured on per job basis. The auxiliary service servicing the Shuffle must be configured in the NodeManagers configuration.
### Job Configuration Properties (on per job basis):
| **Property** | **Default Value** | **Explanation** |
|:---- |:---- |:---- |
| `mapreduce.job.reduce.shuffle.consumer.plugin.class` | `org.apache.hadoop.mapreduce.task.reduce.Shuffle` | The `ShuffleConsumerPlugin` implementation to use |
| `mapreduce.job.map.output.collector.class` | `org.apache.hadoop.mapred.MapTask$MapOutputBuffer` | The `MapOutputCollector` implementation(s) to use |
These properties can also be set in the `mapred-site.xml` to change the default values for all jobs.
The collector class configuration may specify a comma-separated list of collector implementations. In this case, the map task will attempt to instantiate each in turn until one of the implementations successfully initializes. This can be useful if a given collector implementation is only compatible with certain types of keys or values, for example.
### NodeManager Configuration properties, `yarn-site.xml` in all nodes:
| **Property** | **Default Value** | **Explanation** |
|:---- |:---- |:---- |
| `yarn.nodemanager.aux-services` | `...,mapreduce_shuffle` | The auxiliary service name |
| `yarn.nodemanager.aux-services.mapreduce_shuffle.class` | `org.apache.hadoop.mapred.ShuffleHandler` | The auxiliary service class to use |
**IMPORTANT:** If setting an auxiliary service in addition the default
`mapreduce_shuffle` service, then a new service key should be added to the
`yarn.nodemanager.aux-services` property, for example `mapred.shufflex`.
Then the property defining the corresponding class must be
`yarn.nodemanager.aux-services.mapreduce_shufflex.class`.

2672
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/site/apt/HistoryServerRest.apt.vm
View File

File diff suppressed because it is too large Load Diff

2361
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/site/markdown/HistoryServerRest.md Normal file
View File

File diff suppressed because it is too large Load Diff

73
hadoop-project/src/site/apt/index.apt.vm
View File

@ -1,73 +0,0 @@
~~ Licensed under the Apache License, Version 2.0 (the "License");
~~ you may not use this file except in compliance with the License.
~~ You may obtain a copy of the License at
~~
~~ http://www.apache.org/licenses/LICENSE-2.0
~~
~~ Unless required by applicable law or agreed to in writing, software
~~ distributed under the License is distributed on an "AS IS" BASIS,
~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~~ See the License for the specific language governing permissions and
~~ limitations under the License. See accompanying LICENSE file.
---
Apache Hadoop ${project.version}
---
---
${maven.build.timestamp}
Apache Hadoop ${project.version}
Apache Hadoop ${project.version} is a minor release in the 2.x.y release
line, building upon the previous stable release 2.4.1.
Here is a short overview of the major features and improvements.
* {Common}
* Authentication improvements when using an HTTP proxy server. This is
useful when accessing WebHDFS via a proxy server.
* A new Hadoop metrics sink that allows writing directly to Graphite.
* {{{./hadoop-project-dist/hadoop-common/filesystem/index.html}Specification work}}
related to the Hadoop Compatible Filesystem (HCFS) effort.
* {HDFS}
* Support for POSIX-style filesystem extended attributes. See the
{{{./hadoop-project-dist/hadoop-hdfs/ExtendedAttributes.html}user documentation}}
for more details.
* Using the OfflineImageViewer, clients can now browse an fsimage via
the WebHDFS API.
* The NFS gateway received a number of supportability improvements and
bug fixes. The Hadoop portmapper is no longer required to run the gateway,
and the gateway is now able to reject connections from unprivileged ports.
* The SecondaryNameNode, JournalNode, and DataNode web UIs have been
modernized with HTML5 and Javascript.
* {YARN}
* YARN's REST APIs now support write/modify operations. Users can submit and
kill applications through REST APIs.
* The timeline store in YARN, used for storing generic and
application-specific information for applications, supports authentication
through Kerberos.
* The Fair Scheduler supports dynamic hierarchical user queues, user queues
are created dynamically at runtime under any specified parent-queue.
Getting Started
The Hadoop documentation includes the information you need to get started using
Hadoop. Begin with the
{{{./hadoop-project-dist/hadoop-common/SingleCluster.html}Single Node Setup}} which
shows you how to set up a single-node Hadoop installation. Then move on to the
{{{./hadoop-project-dist/hadoop-common/ClusterSetup.html}Cluster Setup}} to learn how
to set up a multi-node Hadoop installation.

69
hadoop-project/src/site/markdown/index.md.vm Normal file
View File

@ -0,0 +1,69 @@
<!---
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
Apache Hadoop ${project.version}
================================
Apache Hadoop ${project.version} is a minor release in the 2.x.y release
line, building upon the previous stable release 2.4.1.
Here is a short overview of the major features and improvements.
* Common
* Authentication improvements when using an HTTP proxy server. This is
useful when accessing WebHDFS via a proxy server.
* A new Hadoop metrics sink that allows writing directly to Graphite.
* [Specification work](./hadoop-project-dist/hadoop-common/filesystem/index.html)
related to the Hadoop Compatible Filesystem (HCFS) effort.
* HDFS
* Support for POSIX-style filesystem extended attributes. See the
[user documentation](./hadoop-project-dist/hadoop-hdfs/ExtendedAttributes.html)
for more details.
* Using the OfflineImageViewer, clients can now browse an fsimage via
the WebHDFS API.
* The NFS gateway received a number of supportability improvements and
bug fixes. The Hadoop portmapper is no longer required to run the gateway,
and the gateway is now able to reject connections from unprivileged ports.
* The SecondaryNameNode, JournalNode, and DataNode web UIs have been
modernized with HTML5 and Javascript.
* YARN
* YARN's REST APIs now support write/modify operations. Users can submit and
kill applications through REST APIs.
* The timeline store in YARN, used for storing generic and
application-specific information for applications, supports authentication
through Kerberos.
* The Fair Scheduler supports dynamic hierarchical user queues, user queues
are created dynamically at runtime under any specified parent-queue.
Getting Started
===============
The Hadoop documentation includes the information you need to get started using
Hadoop. Begin with the
[Single Node Setup](./hadoop-project-dist/hadoop-common/SingleCluster.html) which
shows you how to set up a single-node Hadoop installation. Then move on to the
[Cluster Setup](./hadoop-project-dist/hadoop-common/ClusterSetup.html) to learn how
to set up a multi-node Hadoop installation.

686
hadoop-tools/hadoop-openstack/src/site/apt/index.apt.vm
View File

@ -1,686 +0,0 @@
~~ Licensed under the Apache License, Version 2.0 (the "License");
~~ you may not use this file except in compliance with the License.
~~ You may obtain a copy of the License at
~~
~~ http://www.apache.org/licenses/LICENSE-2.0
~~
~~ Unless required by applicable law or agreed to in writing, software
~~ distributed under the License is distributed on an "AS IS" BASIS,
~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~~ See the License for the specific language governing permissions and
~~ limitations under the License. See accompanying LICENSE file.
---
Hadoop OpenStack Support: Swift Object Store
---
---
${maven.build.timestamp}
%{toc|section=1|fromDepth=0}
Hadoop OpenStack Support: Swift Object Store
* {Introduction}
{{{http://www.openstack.org/}OpenStack}} is an open source cloud infrastructure
which can be accessed
from multiple public IaaS providers, and deployed privately. It offers
infrastructure services such as VM hosting (Nova), authentication (Keystone)
and storage of binary objects (Swift).
This module enables Apache Hadoop applications -including MapReduce jobs,
read and write data to and from instances of the
{{{http://www.openstack.org/software/openstack-storage/}OpenStack Swift object store}}.
* Features
* Read and write of data stored in a Swift object store
* Support of a pseudo-hierachical file system (directories, subdirectories and
files)
* Standard filesystem operations: <<<create>>>, <<<delete>>>, <<<mkdir>>>,
<<<ls>>>, <<<mv>>>, <<<stat>>>.
* Can act as a source of data in a MapReduce job, or a sink.
* Support for multiple OpenStack services, and multiple containers from a
single service.
* Supports in-cluster and remote access to Swift data.
* Supports OpenStack Keystone authentication with password or token.
* Released under the Apache Software License
* Tested against the Hadoop 3.x and 1.x branches, against multiple public
OpenStack clusters: Rackspace US, Rackspace UK, HP Cloud.
* Tested against private OpenStack clusters, including scalability tests of
large file uploads.
* Using the Hadoop Swift Filesystem Client
** Concepts: services and containers
OpenStack swift is an <Object Store>; also known as a <blobstore>. It stores
arbitrary binary objects by name in a <container>.
The Hadoop Swift filesystem library adds another concept, the <service>, which
defines which Swift blobstore hosts a container -and how to connect to it.
** Containers and Objects
* Containers are created by users with accounts on the Swift filestore, and hold
<objects>.
* Objects can be zero bytes long, or they can contain data.
* Objects in the container can be up to 5GB; there is a special support for
larger files than this, which merges multiple objects in to one.
* Each object is referenced by it's <name>; there is no notion of directories.
* You can use any characters in an object name that can be 'URL-encoded'; the
maximum length of a name is 1034 characters -after URL encoding.
* Names can have <<</>>> characters in them, which are used to create the illusion of
a directory structure. For example <<<dir/dir2/name>>>. Even though this looks
like a directory, <it is still just a name>. There is no requirement to have
any entries in the container called <<<dir>>> or <<<dir/dir2>>>
* That said. if the container has zero-byte objects that look like directory
names above other objects, they can pretend to be directories. Continuing the
example, a 0-byte object called <<<dir>>> would tell clients that it is a
directory while <<<dir/dir2>>> or <<<dir/dir2/name>>> were present. This creates an
illusion of containers holding a filesystem.
Client applications talk to Swift over HTTP or HTTPS, reading, writing and
deleting objects using standard HTTP operations (GET, PUT and DELETE,
respectively). There is also a COPY operation, that creates a new object in the
container, with a new name, containing the old data. There is no rename
operation itself, objects need to be copied -then the original entry deleted.
** Eventual Consistency
The Swift Filesystem is *eventually consistent*: an operation on an object may
not be immediately visible to that client, or other clients. This is a
consequence of the goal of the filesystem: to span a set of machines, across
multiple datacenters, in such a way that the data can still be available when
many of them fail. (In contrast, the Hadoop HDFS filesystem is *immediately
consistent*, but it does not span datacenters.)
Eventual consistency can cause surprises for client applications that expect
immediate consistency: after an object is deleted or overwritten, the object
may still be visible -or the old data still retrievable. The Swift Filesystem
client for Apache Hadoop attempts to handle this, in conjunction with the
MapReduce engine, but there may be still be occasions when eventual consistency
causes surprises.
** Non-atomic "directory" operations.
Hadoop expects some
operations to be atomic, especially <<<rename()>>>, which is something
the MapReduce layer relies on to commit the output of a job, renaming data
from a temp directory to the final path. Because a rename
is implemented as a copy of every blob under the directory's path, followed
by a delete of the originals, the intermediate state of the operation
will be visible to other clients. If two Reducer tasks to rename their temp
directory to the final path, both operations may succeed, with the result that
output directory contains mixed data. This can happen if MapReduce jobs
are being run with <speculation> enabled and Swift used as the direct output
of the MR job (it can also happen against Amazon S3).
Other consequences of the non-atomic operations are:
1. If a program is looking for the presence of the directory before acting
on the data -it may start prematurely. This can be avoided by using
other mechanisms to co-ordinate the programs, such as the presence of a file
that is written <after> any bulk directory operations.
2. A <<<rename()>>> or <<<delete()>>> operation may include files added under
the source directory tree during the operation, may unintentionally delete
it, or delete the 0-byte swift entries that mimic directories and act
as parents for the files. Try to avoid doing this.
The best ways to avoid all these problems is not using Swift as
the filesystem between MapReduce jobs or other Hadoop workflows. It
can act as a source of data, and a final destination, but it doesn't meet
all of Hadoop's expectations of what a filesystem is -it's a <blobstore>.
* Working with Swift Object Stores in Hadoop
Once installed, the Swift FileSystem client can be used by any Hadoop application
to read from or write to data stored in a Swift container.
Data stored in Swift can be used as the direct input to a MapReduce job
-simply use the <<<swift:>>> URL (see below) to declare the source of the data.
This Swift Filesystem client is designed to work with multiple
Swift object stores, both public and private. This allows the client to work
with different clusters, reading and writing data to and from either of them.
It can also work with the same object stores using multiple login details.
These features are achieved by one basic concept: using a service name in
the URI referring to a swift filesystem, and looking up all the connection and
login details for that specific service. Different service names can be defined
in the Hadoop XML configuration file, so defining different clusters, or
providing different login details for the same object store(s).
** Swift Filesystem URIs
Hadoop uses URIs to refer to files within a filesystem. Some common examples
are:
+--
local://etc/hosts
hdfs://cluster1/users/example/data/set1
hdfs://cluster2.example.org:8020/users/example/data/set1
+--
The Swift Filesystem Client adds a new URL type <<<swift>>>. In a Swift Filesystem
URL, the hostname part of a URL identifies the container and the service to
work with; the path the name of the object. Here are some examples
+--
swift://container.rackspace/my-object.csv
swift://data.hpcloud/data/set1
swift://dmitry.privatecloud/out/results
+--
In the last two examples, the paths look like directories: it is not, they are
simply the objects named <<<data/set1>>> and <<<out/results>>> respectively.
** Installing
The <<<hadoop-openstack>>> JAR must be on the classpath of the Hadoop program trying to
talk to the Swift service. If installed in the classpath of the Hadoop
MapReduce service, then all programs started by the MR engine will pick up the
JAR automatically. This is the easiest way to give all Hadoop jobs access to
Swift.
Alternatively, the JAR can be included as one of the JAR files that an
application uses. This lets the Hadoop jobs work with a Swift object store even
if the Hadoop cluster is not pre-configured for this.
The library also depends upon the Apache HttpComponents library, which
must also be on the classpath.
** Configuring
To talk to a swift service, the user must must provide:
[[1]] The URL defining the container and the service.
[[1]] In the cluster/job configuration, the login details of that service.
Multiple service definitions can co-exist in the same configuration file: just
use different names for them.
*** Example: Rackspace US, in-cluster access using API key
This service definition is for use in a Hadoop cluster deployed within Rackspace's
US infrastructure.
+--
<property>
<name>fs.swift.service.rackspace.auth.url</name>
<value>https://auth.api.rackspacecloud.com/v2.0/tokens</value>
<description>Rackspace US (multiregion)</description>
</property>
<property>
<name>fs.swift.service.rackspace.username</name>
<value>user4</value>
</property>
<property>
<name>fs.swift.service.rackspace.region</name>
<value>DFW</value>
</property>
<property>
<name>fs.swift.service.rackspace.apikey</name>
<value>fe806aa86dfffe2f6ed8</value>
</property>
+--
Here the API key visible in the account settings API keys page is used to log
in. No property for public/private access -the default is to use the private
endpoint for Swift operations.
This configuration also selects one of the regions, DFW, for its data.
A reference to this service would use the <<<rackspace>>> service name:
---
swift://hadoop-container.rackspace/
---
*** Example: Rackspace UK: remote access with password authentication
This connects to Rackspace's UK ("LON") datacenter.
+--
<property>
<name>fs.swift.service.rackspaceuk.auth.url</name>
<value>https://lon.identity.api.rackspacecloud.com/v2.0/tokens</value>
<description>Rackspace UK</description>
</property>
<property>
<name>fs.swift.service.rackspaceuk.username</name>
<value>user4</value>
</property>
<property>
<name>fs.swift.service.rackspaceuk.password</name>
<value>insert-password-here/value>
</property>
<property>
<name>fs.swift.service.rackspace.public</name>
<value>true</value>
</property>
+--
This is a public access point connection, using a password over an API key.
A reference to this service would use the <<<rackspaceuk>>> service name:
+--
swift://hadoop-container.rackspaceuk/
+--
Because the public endpoint is used, if this service definition is used within
the London datacenter, all accesses will be billed at the public
upload/download rates, <irrespective of where the Hadoop cluster is>.
*** Example: HP cloud service definition
Here is an example that connects to the HP Cloud object store.
+--
<property>
<name>fs.swift.service.hpcloud.auth.url</name>
<value>https://region-a.geo-1.identity.hpcloudsvc.com:35357/v2.0/tokens
</value>
<description>HP Cloud</description>
</property>
<property>
<name>fs.swift.service.hpcloud.tenant</name>
<value>FE806AA86</value>
</property>
<property>
<name>fs.swift.service.hpcloud.username</name>
<value>FE806AA86DFFFE2F6ED8</value>
</property>
<property>
<name>fs.swift.service.hpcloud.password</name>
<value>secret-password-goes-here</value>
</property>
<property>
<name>fs.swift.service.hpcloud.public</name>
<value>true</value>
</property>
+--
A reference to this service would use the <<<hpcloud>>> service name:
+--
swift://hadoop-container.hpcloud/
+--
** General Swift Filesystem configuration options
Some configuration options apply to the Swift client, independent of
the specific Swift filesystem chosen.
*** Blocksize fs.swift.blocksize
Swift does not break up files into blocks, except in the special case of files
over 5GB in length. Accordingly, there isn't a notion of a "block size"
to define where the data is kept.
Hadoop's MapReduce layer depends on files declaring their block size,
so that it knows how to partition work. Too small a blocksize means that
many mappers work on small pieces of data; too large a block size means
that only a few mappers get started.
The block size value reported by Swift, therefore, controls the basic workload
partioning of the MapReduce engine -and can be an important parameter to
tune for performance of the cluster.
The property has a unit of kilobytes; the default value is <<<32*1024>>>: 32 MB
+--
<property>
<name>fs.swift.blocksize</name>
<value>32768</value>
</property>
+--
This blocksize has no influence on how files are stored in Swift; it only controls
what the reported size of blocks are - a value used in Hadoop MapReduce to
divide work.
Note that the MapReduce engine's split logic can be tuned independently by setting
the <<<mapred.min.split.size>>> and <<<mapred.max.split.size>>> properties,
which can be done in specific job configurations.
+--
<property>
<name>mapred.min.split.size</name>
<value>524288</value>
</property>
<property>
<name>mapred.max.split.size</name>
<value>1048576</value>
</property>
+--
In an Apache Pig script, these properties would be set as:
---
mapred.min.split.size 524288
mapred.max.split.size 1048576
---
*** Partition size fs.swift.partsize
The Swift filesystem client breaks very large files into partitioned files,
uploading each as it progresses, and writing any remaning data and an XML
manifest when a partitioned file is closed.
The partition size defaults to 4608 MB; 4.5GB, the maximum filesize that
Swift can support.
It is possible to set a smaller partition size, in the <<<fs.swift.partsize>>>
option. This takes a value in KB.
+--
<property>
<name>fs.swift.partsize</name>
<value>1024</value>
<description>upload every MB</description>
</property>
+--
When should this value be changed from its default?
While there is no need to ever change it for basic operation of
the Swift filesystem client, it can be tuned
* If a Swift filesystem is location aware, then breaking a file up into
smaller partitions scatters the data round the cluster. For best performance,
the property <<<fs.swift.blocksize>>> should be set to a smaller value than the
partition size of files.
* When writing to an unpartitioned file, the entire write is done in the
<<<close()>>> operation. When a file is partitioned, the outstanding data to
be written whenever the outstanding amount of data is greater than the
partition size. This means that data will be written more incrementally
*** Request size fs.swift.requestsize
The Swift filesystem client reads files in HTTP GET operations, asking for
a block of data at a time.
The default value is 64KB. A larger value may be more efficient over faster
networks, as it reduces the overhead of setting up the HTTP operation.
However, if the file is read with many random accesses, requests for
data will be made from different parts of the file -discarding some of the
previously requested data. The benefits of larger request sizes may be wasted.
The property <<<fs.swift.requestsize>>> sets the request size in KB.
+--
<property>
<name>fs.swift.requestsize</name>
<value>128</value>
</property>
+--
*** Connection timeout fs.swift.connect.timeout
This sets the timeout in milliseconds to connect to a Swift service.
+--
<property>
<name>fs.swift.connect.timeout</name>
<value>15000</value>
</property>
+--
A shorter timeout means that connection failures are raised faster -but
may trigger more false alarms. A longer timeout is more resilient to network
problems -and may be needed when talking to remote filesystems.
*** Connection timeout fs.swift.socket.timeout
This sets the timeout in milliseconds to wait for data from a connected socket.
+--
<property>
<name>fs.swift.socket.timeout</name>
<value>60000</value>
</property>
+--
A shorter timeout means that connection failures are raised faster -but
may trigger more false alarms. A longer timeout is more resilient to network
problems -and may be needed when talking to remote filesystems.
*** Connection Retry Count fs.swift.connect.retry.count
This sets the number of times to try to connect to a service whenever
an HTTP request is made.
+--
<property>
<name>fs.swift.connect.retry.count</name>
<value>3</value>
</property>
+--
The more retries, the more resilient it is to transient outages -and the
less rapid it is at detecting and reporting server connectivity problems.
*** Connection Throttle Delay fs.swift.connect.throttle.delay
This property adds a delay between bulk file copy and delete operations,
to prevent requests being throttled or blocked by the remote service
+--
<property>
<name>fs.swift.connect.throttle.delay</name>
<value>0</value>
</property>
+--
It is measured in milliseconds; "0" means do not add any delay.
Throttling is enabled on the public endpoints of some Swift services.
If <<<rename()>>> or <<<delete()>>> operations fail with
<<<SwiftThrottledRequestException>>>
exceptions, try setting this property.
*** HTTP Proxy
If the client can only access the Swift filesystem via a web proxy
server, the client configuration must specify the proxy via
the <<<fs.swift.connect.proxy.host>>> and <<<fs.swift.connect.proxy.port>>>
properties.
+--
<property>
<name>fs.swift.proxy.host</name>
<value>web-proxy</value>
</property>
<property>
<name>fs.swift.proxy.port</name>
<value>8088</value>
</property>
+--
If the host is declared, the proxy port must be set to a valid integer value.
** Troubleshooting
*** ClassNotFoundException
The <<<hadoop-openstack>>> JAR -or any dependencies- may not be on your classpath.
If it is a remote MapReduce job that is failing, make sure that the JAR is
installed on the servers in the cluster -or that the job submission process
uploads the JAR file to the distributed cache.
*** Failure to Authenticate
A <<<SwiftAuthenticationFailedException>>> is thrown when the client
cannot authenticate with the OpenStack keystone server. This could be
because the URL in the service definition is wrong, or because
the supplied credentials are invalid.
[[1]] Check the authentication URL through <<<curl>>> or your browser
[[1]] Use a Swift client such as CyberDuck to validate your credentials
[[1]] If you have included a tenant ID, try leaving it out. Similarly,
try adding it if you had not included it.
[[1]] Try switching from API key authentication to password-based authentication,
by setting the password.
[[1]] Change your credentials. As with Amazon AWS clients, some credentials
don't seem to like going over the network.
*** Timeout connecting to the Swift Service
This happens if the client application is running outside an OpenStack cluster,
where it does not have access to the private hostname/IP address for filesystem
operations. Set the <<<public>>> flag to true -but remember to set it to false
for use in-cluster.
** Warnings
[[1]] Do not share your login details with anyone, which means do not log the
details, or check the XML configuration files into any revision control system
to which you do not have exclusive access.
[[1]] Similarly, do not use your real account details in any documentation *or any
bug reports submitted online*
[[1]] Prefer the apikey authentication over passwords as it is easier
to revoke a key -and some service providers allow you to set
an automatic expiry date on a key when issued.
[[1]] Do not use the public service endpoint from within a public OpenStack
cluster, as it will run up large bills.
[[1]] Remember: it's not a real filesystem or hierarchical directory structure.
Some operations (directory rename and delete) take time and are not atomic or
isolated from other operations taking place.
[[1]] Append is not supported.
[[1]] Unix-style permissions are not supported. All accounts with write access to
a repository have unlimited access; the same goes for those with read access.
[[1]] In the public clouds, do not make the containers public unless you are happy
with anyone reading your data, and are prepared to pay the costs of their
downloads.
** Limits
* Maximum length of an object path: 1024 characters
* Maximum size of a binary object: no absolute limit. Files > 5GB are
partitioned into separate files in the native filesystem, and merged during
retrieval. <Warning:> the partitioned/large file support is the
most complex part of the Hadoop/Swift FS integration, and, along with
authentication, the most troublesome to support.
** Testing the hadoop-openstack module
The <<<hadoop-openstack>>> can be remotely tested against any public
or private cloud infrastructure which supports the OpenStack Keystone
authentication mechanism. It can also be tested against private
OpenStack clusters. OpenStack Development teams are strongly encouraged to test
the Hadoop swift filesystem client against any version of Swift that they
are developing or deploying, to stress their cluster and to identify
bugs early.
The module comes with a large suite of JUnit tests -tests that are
only executed if the source tree includes credentials to test against a
specific cluster.
After checking out the Hadoop source tree, create the file:
+--
hadoop-tools/hadoop-openstack/src/test/resources/auth-keys.xml
+--
Into this file, insert the credentials needed to bond to the test filesystem,
as decribed above.
Next set the property <<<test.fs.swift.name>>> to the URL of a
swift container to test against. The tests expect exclusive access
to this container -do not keep any other data on it, or expect it
to be preserved.
+--
<property>
<name>test.fs.swift.name</name>
<value>swift://test.myswift/</value>
</property>
+--
In the base hadoop directory, run:
+--
mvn clean install -DskipTests
+--
This builds a set of Hadoop JARs consistent with the <<<hadoop-openstack>>>
module that is about to be tested.
In the <<<hadoop-tools/hadoop-openstack>>> directory run
+--
mvn test -Dtest=TestSwiftRestClient
+--
This runs some simple tests which include authenticating
against the remote swift service. If these tests fail, so will all
the rest. If it does fail: check your authentication.
Once this test succeeds, you can run the full test suite
+--
mvn test
+--
Be advised that these tests can take an hour or more, especially against a
remote Swift service -or one that throttles bulk operations.
Once the <<<auth-keys.xml>>> file is in place, the <<<mvn test>>> runs from
the Hadoop source base directory will automatically run these OpenStack tests
While this ensures that no regressions have occurred, it can also add significant
time to test runs, and may run up bills, depending on who is providing\
the Swift storage service. We recommend having a separate source tree
set up purely for the Swift tests, and running it manually or by the CI tooling
at a lower frequency than normal test runs.
Finally: Apache Hadoop is an open source project. Contributions of code
-including more tests- are very welcome.

544
hadoop-tools/hadoop-openstack/src/site/markdown/index.md Normal file
View File

@ -0,0 +1,544 @@
<!---
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
* [Hadoop OpenStack Support: Swift Object Store](#Hadoop_OpenStack_Support:_Swift_Object_Store)
* [Introduction](#Introduction)
* [Features](#Features)
* [Using the Hadoop Swift Filesystem Client](#Using_the_Hadoop_Swift_Filesystem_Client)
* [Concepts: services and containers](#Concepts:_services_and_containers)
* [Containers and Objects](#Containers_and_Objects)
* [Eventual Consistency](#Eventual_Consistency)
* [Non-atomic "directory" operations.](#Non-atomic_directory_operations.)
* [Working with Swift Object Stores in Hadoop](#Working_with_Swift_Object_Stores_in_Hadoop)
* [Swift Filesystem URIs](#Swift_Filesystem_URIs)
* [Installing](#Installing)
* [Configuring](#Configuring)
* [Example: Rackspace US, in-cluster access using API key](#Example:_Rackspace_US_in-cluster_access_using_API_key)
* [Example: Rackspace UK: remote access with password authentication](#Example:_Rackspace_UK:_remote_access_with_password_authentication)
* [Example: HP cloud service definition](#Example:_HP_cloud_service_definition)
* [General Swift Filesystem configuration options](#General_Swift_Filesystem_configuration_options)
* [Blocksize fs.swift.blocksize](#Blocksize_fs.swift.blocksize)
* [Partition size fs.swift.partsize](#Partition_size_fs.swift.partsize)
* [Request size fs.swift.requestsize](#Request_size_fs.swift.requestsize)
* [Connection timeout fs.swift.connect.timeout](#Connection_timeout_fs.swift.connect.timeout)
* [Connection timeout fs.swift.socket.timeout](#Connection_timeout_fs.swift.socket.timeout)
* [Connection Retry Count fs.swift.connect.retry.count](#Connection_Retry_Count_fs.swift.connect.retry.count)
* [Connection Throttle Delay fs.swift.connect.throttle.delay](#Connection_Throttle_Delay_fs.swift.connect.throttle.delay)
* [HTTP Proxy](#HTTP_Proxy)
* [Troubleshooting](#Troubleshooting)
* [ClassNotFoundException](#ClassNotFoundException)
* [Failure to Authenticate](#Failure_to_Authenticate)
* [Timeout connecting to the Swift Service](#Timeout_connecting_to_the_Swift_Service)
* [Warnings](#Warnings)
* [Limits](#Limits)
* [Testing the hadoop-openstack module](#Testing_the_hadoop-openstack_module)
Hadoop OpenStack Support: Swift Object Store
============================================
Introduction
------------
[OpenStack](http://www.openstack.org/) is an open source cloud infrastructure which can be accessed from multiple public IaaS providers, and deployed privately. It offers infrastructure services such as VM hosting (Nova), authentication (Keystone) and storage of binary objects (Swift).
This module enables Apache Hadoop applications -including MapReduce jobs, read and write data to and from instances of the [OpenStack Swift object store](http://www.openstack.org/software/openstack-storage/).
Features
--------
* Read and write of data stored in a Swift object store
* Support of a pseudo-hierachical file system (directories, subdirectories and
files)
* Standard filesystem operations: `create`, `delete`, `mkdir`,
`ls`, `mv`, `stat`.
* Can act as a source of data in a MapReduce job, or a sink.
* Support for multiple OpenStack services, and multiple containers from a
single service.
* Supports in-cluster and remote access to Swift data.
* Supports OpenStack Keystone authentication with password or token.
* Released under the Apache Software License
* Tested against the Hadoop 3.x and 1.x branches, against multiple public
OpenStack clusters: Rackspace US, Rackspace UK, HP Cloud.
* Tested against private OpenStack clusters, including scalability tests of
large file uploads.
Using the Hadoop Swift Filesystem Client
----------------------------------------
### Concepts: services and containers
OpenStack swift is an *Object Store*; also known as a *blobstore*. It stores arbitrary binary objects by name in a *container*.
The Hadoop Swift filesystem library adds another concept, the *service*, which defines which Swift blobstore hosts a container -and how to connect to it.
### Containers and Objects
* Containers are created by users with accounts on the Swift filestore, and hold
*objects*.
* Objects can be zero bytes long, or they can contain data.
* Objects in the container can be up to 5GB; there is a special support for
larger files than this, which merges multiple objects in to one.
* Each object is referenced by it's *name*; there is no notion of directories.
* You can use any characters in an object name that can be 'URL-encoded'; the
maximum length of a name is 1034 characters -after URL encoding.
* Names can have `/` characters in them, which are used to create the illusion of
a directory structure. For example `dir/dir2/name`. Even though this looks
like a directory, *it is still just a name*. There is no requirement to have
any entries in the container called `dir` or `dir/dir2`
* That said. if the container has zero-byte objects that look like directory
names above other objects, they can pretend to be directories. Continuing the
example, a 0-byte object called `dir` would tell clients that it is a
directory while `dir/dir2` or `dir/dir2/name` were present. This creates an
illusion of containers holding a filesystem.
Client applications talk to Swift over HTTP or HTTPS, reading, writing and deleting objects using standard HTTP operations (GET, PUT and DELETE, respectively). There is also a COPY operation, that creates a new object in the container, with a new name, containing the old data. There is no rename operation itself, objects need to be copied -then the original entry deleted.
### Eventual Consistency
The Swift Filesystem is \*eventually consistent\*: an operation on an object may not be immediately visible to that client, or other clients. This is a consequence of the goal of the filesystem: to span a set of machines, across multiple datacenters, in such a way that the data can still be available when many of them fail. (In contrast, the Hadoop HDFS filesystem is \*immediately consistent\*, but it does not span datacenters.)
Eventual consistency can cause surprises for client applications that expect immediate consistency: after an object is deleted or overwritten, the object may still be visible -or the old data still retrievable. The Swift Filesystem client for Apache Hadoop attempts to handle this, in conjunction with the MapReduce engine, but there may be still be occasions when eventual consistency causes surprises.
### Non-atomic "directory" operations.
Hadoop expects some operations to be atomic, especially `rename()`, which is something the MapReduce layer relies on to commit the output of a job, renaming data from a temp directory to the final path. Because a rename is implemented as a copy of every blob under the directory's path, followed by a delete of the originals, the intermediate state of the operation will be visible to other clients. If two Reducer tasks to rename their temp directory to the final path, both operations may succeed, with the result that output directory contains mixed data. This can happen if MapReduce jobs are being run with *speculation* enabled and Swift used as the direct output of the MR job (it can also happen against Amazon S3).
Other consequences of the non-atomic operations are:
1. If a program is looking for the presence of the directory before acting
on the data -it may start prematurely. This can be avoided by using
other mechanisms to co-ordinate the programs, such as the presence of a file
that is written *after* any bulk directory operations.
2. A `rename()` or `delete()` operation may include files added under
the source directory tree during the operation, may unintentionally delete
it, or delete the 0-byte swift entries that mimic directories and act
as parents for the files. Try to avoid doing this.
The best ways to avoid all these problems is not using Swift as the filesystem between MapReduce jobs or other Hadoop workflows. It can act as a source of data, and a final destination, but it doesn't meet all of Hadoop's expectations of what a filesystem is -it's a *blobstore*.
Working with Swift Object Stores in Hadoop
------------------------------------------
Once installed, the Swift FileSystem client can be used by any Hadoop application to read from or write to data stored in a Swift container.
Data stored in Swift can be used as the direct input to a MapReduce job -simply use the `swift:` URL (see below) to declare the source of the data.
This Swift Filesystem client is designed to work with multiple Swift object stores, both public and private. This allows the client to work with different clusters, reading and writing data to and from either of them.
It can also work with the same object stores using multiple login details.
These features are achieved by one basic concept: using a service name in the URI referring to a swift filesystem, and looking up all the connection and login details for that specific service. Different service names can be defined in the Hadoop XML configuration file, so defining different clusters, or providing different login details for the same object store(s).
### Swift Filesystem URIs
Hadoop uses URIs to refer to files within a filesystem. Some common examples are:
local://etc/hosts
hdfs://cluster1/users/example/data/set1
hdfs://cluster2.example.org:8020/users/example/data/set1
The Swift Filesystem Client adds a new URL type `swift`. In a Swift Filesystem URL, the hostname part of a URL identifies the container and the service to work with; the path the name of the object. Here are some examples
swift://container.rackspace/my-object.csv
swift://data.hpcloud/data/set1
swift://dmitry.privatecloud/out/results
In the last two examples, the paths look like directories: it is not, they are simply the objects named `data/set1` and `out/results` respectively.
### Installing
The `hadoop-openstack` JAR must be on the classpath of the Hadoop program trying to talk to the Swift service. If installed in the classpath of the Hadoop MapReduce service, then all programs started by the MR engine will pick up the JAR automatically. This is the easiest way to give all Hadoop jobs access to Swift.
Alternatively, the JAR can be included as one of the JAR files that an application uses. This lets the Hadoop jobs work with a Swift object store even if the Hadoop cluster is not pre-configured for this.
The library also depends upon the Apache HttpComponents library, which must also be on the classpath.
### Configuring
To talk to a swift service, the user must must provide:
1. The URL defining the container and the service.
2. In the cluster/job configuration, the login details of that service.
Multiple service definitions can co-exist in the same configuration file: just use different names for them.
#### Example: Rackspace US, in-cluster access using API key
This service definition is for use in a Hadoop cluster deployed within Rackspace's US infrastructure.
<property>
<name>fs.swift.service.rackspace.auth.url</name>
<value>https://auth.api.rackspacecloud.com/v2.0/tokens</value>
<description>Rackspace US (multiregion)</description>
</property>
<property>
<name>fs.swift.service.rackspace.username</name>
<value>user4</value>
</property>
<property>
<name>fs.swift.service.rackspace.region</name>
<value>DFW</value>
</property>
<property>
<name>fs.swift.service.rackspace.apikey</name>
<value>fe806aa86dfffe2f6ed8</value>
</property>
Here the API key visible in the account settings API keys page is used to log in. No property for public/private access -the default is to use the private endpoint for Swift operations.
This configuration also selects one of the regions, DFW, for its data.
A reference to this service would use the `rackspace` service name:
swift://hadoop-container.rackspace/
#### Example: Rackspace UK: remote access with password authentication
This connects to Rackspace's UK ("LON") datacenter.
<property>
<name>fs.swift.service.rackspaceuk.auth.url</name>
<value>https://lon.identity.api.rackspacecloud.com/v2.0/tokens</value>
<description>Rackspace UK</description>
</property>
<property>
<name>fs.swift.service.rackspaceuk.username</name>
<value>user4</value>
</property>
<property>
<name>fs.swift.service.rackspaceuk.password</name>
<value>insert-password-here/value>
</property>
<property>
<name>fs.swift.service.rackspace.public</name>
<value>true</value>
</property>
This is a public access point connection, using a password over an API key.
A reference to this service would use the `rackspaceuk` service name:
swift://hadoop-container.rackspaceuk/
Because the public endpoint is used, if this service definition is used within the London datacenter, all accesses will be billed at the public upload/download rates, *irrespective of where the Hadoop cluster is*.
#### Example: HP cloud service definition
Here is an example that connects to the HP Cloud object store.
<property>
<name>fs.swift.service.hpcloud.auth.url</name>
<value>https://region-a.geo-1.identity.hpcloudsvc.com:35357/v2.0/tokens
</value>
<description>HP Cloud</description>
</property>
<property>
<name>fs.swift.service.hpcloud.tenant</name>
<value>FE806AA86</value>
</property>
<property>
<name>fs.swift.service.hpcloud.username</name>
<value>FE806AA86DFFFE2F6ED8</value>
</property>
<property>
<name>fs.swift.service.hpcloud.password</name>
<value>secret-password-goes-here</value>
</property>
<property>
<name>fs.swift.service.hpcloud.public</name>
<value>true</value>
</property>
A reference to this service would use the `hpcloud` service name:
swift://hadoop-container.hpcloud/
### General Swift Filesystem configuration options
Some configuration options apply to the Swift client, independent of the specific Swift filesystem chosen.
#### Blocksize fs.swift.blocksize
Swift does not break up files into blocks, except in the special case of files over 5GB in length. Accordingly, there isn't a notion of a "block size" to define where the data is kept.
Hadoop's MapReduce layer depends on files declaring their block size, so that it knows how to partition work. Too small a blocksize means that many mappers work on small pieces of data; too large a block size means that only a few mappers get started.
The block size value reported by Swift, therefore, controls the basic workload partioning of the MapReduce engine -and can be an important parameter to tune for performance of the cluster.
The property has a unit of kilobytes; the default value is `32*1024`: 32 MB
<property>
<name>fs.swift.blocksize</name>
<value>32768</value>
</property>
This blocksize has no influence on how files are stored in Swift; it only controls what the reported size of blocks are - a value used in Hadoop MapReduce to divide work.
Note that the MapReduce engine's split logic can be tuned independently by setting the `mapred.min.split.size` and `mapred.max.split.size` properties, which can be done in specific job configurations.
<property>
<name>mapred.min.split.size</name>
<value>524288</value>
</property>
<property>
<name>mapred.max.split.size</name>
<value>1048576</value>
</property>
In an Apache Pig script, these properties would be set as:
mapred.min.split.size 524288
mapred.max.split.size 1048576
#### Partition size fs.swift.partsize
The Swift filesystem client breaks very large files into partitioned files, uploading each as it progresses, and writing any remaning data and an XML manifest when a partitioned file is closed.
The partition size defaults to 4608 MB; 4.5GB, the maximum filesize that Swift can support.
It is possible to set a smaller partition size, in the `fs.swift.partsize` option. This takes a value in KB.
<property>
<name>fs.swift.partsize</name>
<value>1024</value>
<description>upload every MB</description>
</property>
When should this value be changed from its default?
While there is no need to ever change it for basic operation of the Swift filesystem client, it can be tuned
* If a Swift filesystem is location aware, then breaking a file up into
smaller partitions scatters the data round the cluster. For best performance,
the property `fs.swift.blocksize` should be set to a smaller value than the
partition size of files.
* When writing to an unpartitioned file, the entire write is done in the
`close()` operation. When a file is partitioned, the outstanding data to
be written whenever the outstanding amount of data is greater than the
partition size. This means that data will be written more incrementally
#### Request size fs.swift.requestsize
The Swift filesystem client reads files in HTTP GET operations, asking for a block of data at a time.
The default value is 64KB. A larger value may be more efficient over faster networks, as it reduces the overhead of setting up the HTTP operation.
However, if the file is read with many random accesses, requests for data will be made from different parts of the file -discarding some of the previously requested data. The benefits of larger request sizes may be wasted.
The property `fs.swift.requestsize` sets the request size in KB.
<property>
<name>fs.swift.requestsize</name>
<value>128</value>
</property>
#### Connection timeout fs.swift.connect.timeout
This sets the timeout in milliseconds to connect to a Swift service.
<property>
<name>fs.swift.connect.timeout</name>
<value>15000</value>
</property>
A shorter timeout means that connection failures are raised faster -but may trigger more false alarms. A longer timeout is more resilient to network problems -and may be needed when talking to remote filesystems.
#### Connection timeout fs.swift.socket.timeout
This sets the timeout in milliseconds to wait for data from a connected socket.
<property>
<name>fs.swift.socket.timeout</name>
<value>60000</value>
</property>
A shorter timeout means that connection failures are raised faster -but may trigger more false alarms. A longer timeout is more resilient to network problems -and may be needed when talking to remote filesystems.
#### Connection Retry Count fs.swift.connect.retry.count
This sets the number of times to try to connect to a service whenever an HTTP request is made.
<property>
<name>fs.swift.connect.retry.count</name>
<value>3</value>
</property>
The more retries, the more resilient it is to transient outages -and the less rapid it is at detecting and reporting server connectivity problems.
#### Connection Throttle Delay fs.swift.connect.throttle.delay
This property adds a delay between bulk file copy and delete operations, to prevent requests being throttled or blocked by the remote service
<property>
<name>fs.swift.connect.throttle.delay</name>
<value>0</value>
</property>
It is measured in milliseconds; "0" means do not add any delay.
Throttling is enabled on the public endpoints of some Swift services. If `rename()` or `delete()` operations fail with `SwiftThrottledRequestException` exceptions, try setting this property.
#### HTTP Proxy
If the client can only access the Swift filesystem via a web proxy server, the client configuration must specify the proxy via the `fs.swift.connect.proxy.host` and `fs.swift.connect.proxy.port` properties.
<property>
<name>fs.swift.proxy.host</name>
<value>web-proxy</value>
</property>
<property>
<name>fs.swift.proxy.port</name>
<value>8088</value>
</property>
If the host is declared, the proxy port must be set to a valid integer value.
### Troubleshooting
#### ClassNotFoundException
The `hadoop-openstack` JAR -or any dependencies- may not be on your classpath.
If it is a remote MapReduce job that is failing, make sure that the JAR is installed on the servers in the cluster -or that the job submission process uploads the JAR file to the distributed cache.
#### Failure to Authenticate
A `SwiftAuthenticationFailedException` is thrown when the client cannot authenticate with the OpenStack keystone server. This could be because the URL in the service definition is wrong, or because the supplied credentials are invalid.
1. Check the authentication URL through `curl` or your browser
2. Use a Swift client such as CyberDuck to validate your credentials
3. If you have included a tenant ID, try leaving it out. Similarly,
try adding it if you had not included it.
4. Try switching from API key authentication to password-based authentication,
by setting the password.
5. Change your credentials. As with Amazon AWS clients, some credentials
don't seem to like going over the network.
#### Timeout connecting to the Swift Service
This happens if the client application is running outside an OpenStack cluster, where it does not have access to the private hostname/IP address for filesystem operations. Set the `public` flag to true -but remember to set it to false for use in-cluster.
### Warnings
1. Do not share your login details with anyone, which means do not log the
details, or check the XML configuration files into any revision control system
to which you do not have exclusive access.
2. Similarly, do not use your real account details in any
documentation \*or any bug reports submitted online\*
3. Prefer the apikey authentication over passwords as it is easier
to revoke a key -and some service providers allow you to set
an automatic expiry date on a key when issued.
4. Do not use the public service endpoint from within a public OpenStack
cluster, as it will run up large bills.
5. Remember: it's not a real filesystem or hierarchical directory structure.
Some operations (directory rename and delete) take time and are not atomic or
isolated from other operations taking place.
6. Append is not supported.
7. Unix-style permissions are not supported. All accounts with write access to
a repository have unlimited access; the same goes for those with read access.
8. In the public clouds, do not make the containers public unless you are happy
with anyone reading your data, and are prepared to pay the costs of their
downloads.
### Limits
* Maximum length of an object path: 1024 characters
* Maximum size of a binary object: no absolute limit. Files \> 5GB are
partitioned into separate files in the native filesystem, and merged during
retrieval. *Warning:* the partitioned/large file support is the
most complex part of the Hadoop/Swift FS integration, and, along with
authentication, the most troublesome to support.
### Testing the hadoop-openstack module
The `hadoop-openstack` can be remotely tested against any public or private cloud infrastructure which supports the OpenStack Keystone authentication mechanism. It can also be tested against private OpenStack clusters. OpenStack Development teams are strongly encouraged to test the Hadoop swift filesystem client against any version of Swift that they are developing or deploying, to stress their cluster and to identify bugs early.
The module comes with a large suite of JUnit tests -tests that are only executed if the source tree includes credentials to test against a specific cluster.
After checking out the Hadoop source tree, create the file:
hadoop-tools/hadoop-openstack/src/test/resources/auth-keys.xml
Into this file, insert the credentials needed to bond to the test filesystem, as decribed above.
Next set the property `test.fs.swift.name` to the URL of a swift container to test against. The tests expect exclusive access to this container -do not keep any other data on it, or expect it to be preserved.
<property>
<name>test.fs.swift.name</name>
<value>swift://test.myswift/</value>
</property>
In the base hadoop directory, run:
mvn clean install -DskipTests
This builds a set of Hadoop JARs consistent with the `hadoop-openstack` module that is about to be tested.
In the `hadoop-tools/hadoop-openstack` directory run
mvn test -Dtest=TestSwiftRestClient
This runs some simple tests which include authenticating against the remote swift service. If these tests fail, so will all the rest. If it does fail: check your authentication.
Once this test succeeds, you can run the full test suite
mvn test
Be advised that these tests can take an hour or more, especially against a remote Swift service -or one that throttles bulk operations.
Once the `auth-keys.xml` file is in place, the `mvn test` runs from the Hadoop source base directory will automatically run these OpenStack tests While this ensures that no regressions have occurred, it can also add significant time to test runs, and may run up bills, depending on who is providingthe Swift storage service. We recommend having a separate source tree set up purely for the Swift tests, and running it manually or by the CI tooling at a lower frequency than normal test runs.
Finally: Apache Hadoop is an open source project. Contributions of code -including more tests- are very welcome.

29
hadoop-tools/hadoop-openstack/src/site/resources/css/site.css Normal file
View File

@ -0,0 +1,29 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
#banner {
height: 93px;
background: none;
}
#bannerLeft img {
margin-left: 30px;
margin-top: 10px;
}
#bannerRight img {
margin: 17px;
}

439
hadoop-tools/hadoop-sls/src/site/apt/SchedulerLoadSimulator.apt.vm
View File

@ -1,439 +0,0 @@
~~ Licensed under the Apache License, Version 2.0 (the "License");
~~ you may not use this file except in compliance with the License.
~~ You may obtain a copy of the License at
~~
~~ http://www.apache.org/licenses/LICENSE-2.0
~~
~~ Unless required by applicable law or agreed to in writing, software
~~ distributed under the License is distributed on an "AS IS" BASIS,
~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~~ See the License for the specific language governing permissions and
~~ limitations under the License.
---
Yarn Scheduler Load Simulator (SLS)
---
---
${maven.build.timestamp}
Yarn Scheduler Load Simulator (SLS)
%{toc|section=1|fromDepth=0}
* Overview
** Overview
The Yarn scheduler is a fertile area of interest with different
implementations, e.g., Fifo, Capacity and Fair schedulers. Meanwhile, several
optimizations are also made to improve scheduler performance for different
scenarios and workload. Each scheduler algorithm has its own set of features,
and drives scheduling decisions by many factors, such as fairness, capacity
guarantee, resource availability, etc. It is very important to evaluate a
scheduler algorithm very well before we deploy in a production cluster.
Unfortunately, currently it is non-trivial to evaluate a scheduler algorithm.
Evaluating in a real cluster is always time and cost consuming, and it is
also very hard to find a large-enough cluster. Hence, a simulator which can
predict how well a scheduler algorithm for some specific workload would be
quite useful.
The Yarn Scheduler Load Simulator (SLS) is such a tool, which can simulate
large-scale Yarn clusters and application loads in a single machine.This
simulator would be invaluable in furthering Yarn by providing a tool for
researchers and developers to prototype new scheduler features and predict
their behavior and performance with reasonable amount of confidence,
thereby aiding rapid innovation.
The simulator will exercise the real Yarn <<<ResourceManager>>> removing the
network factor by simulating <<<NodeManagers>>> and <<<ApplicationMasters>>>
via handling and dispatching <<<NM>>>/<<<AMs>>> heartbeat events from within
the same JVM. To keep tracking of scheduler behavior and performance, a
scheduler wrapper will wrap the real scheduler.
The size of the cluster and the application load can be loaded from
configuration files, which are generated from job history files directly by
adopting {{{https://hadoop.apache.org/docs/stable/rumen.html}Apache Rumen}}.
The simulator will produce real time metrics while executing, including:
* Resource usages for whole cluster and each queue, which can be utilized to
configure cluster and queue's capacity.
* The detailed application execution trace (recorded in relation to simulated
time), which can be analyzed to understand/validate the scheduler behavior
(individual jobs turn around time, throughput, fairness, capacity guarantee,
etc.).
* Several key metrics of scheduler algorithm, such as time cost of each
scheduler operation (allocate, handle, etc.), which can be utilized by Hadoop
developers to find the code spots and scalability limits.
** Goals
* Exercise the scheduler at scale without a real cluster using real job
traces.
* Being able to simulate real workloads.
** Architecture
The following figure illustrates the implementation architecture of the
simulator.
[images/sls_arch.png] The architecture of the simulator
The simulator takes input of workload traces, and fetches the cluster and
applications information. For each NM and AM, the simulator builds a simulator
to simulate their running. All NM/AM simulators run in a thread pool. The
simulator reuses Yarn Resource Manager, and builds a wrapper out of the
scheduler. The Scheduler Wrapper can track the scheduler behaviors and
generates several logs, which are the outputs of the simulator and can be
further analyzed.
** Usecases
* Engineering
* Verify correctness of scheduler algorithm under load
* Cheap/practical way for finding code hotspots/critical-path.
* Validate the impact of changes and new features.
* Determine what drives the scheduler scalability limits.
[]
* QA
* Validate scheduler behavior for "large" clusters and several workload
profiles.
* Solutions/Sales.
* Sizing model for predefined/typical workloads.
* Cluster sizing tool using real customer data (job traces).
* Determine minimum SLAs under a particular workload.
* Usage
This section will show how to use the simulator. Here let <<<$HADOOP_ROOT>>>
represent the Hadoop install directory. If you build Hadoop yourself,
<<<$HADOOP_ROOT>>> is <<<hadoop-dist/target/hadoop-$VERSION>>>. The simulator
is located at <<<$HADOOP_ROOT/share/hadoop/tools/sls>>>. The fold <<<sls>>>
containers four directories: <<<bin>>>, <<<html>>>, <<<sample-conf>>>, and
<<<sample-data>>>
* <<<bin>>>: contains running scripts for the simulator.
* <<<html>>>: contains several html/css/js files we needed for real-time
tracking.
* <<<sample-conf>>>: specifies the simulator configurations.
* <<<sample-data>>>: provides an example rumen trace, which can be used to
generate inputs of the simulator.
[]
The following sections will describe how to use the simulator step by step.
Before start, make sure that command <<<hadoop>>> is included in your
<<<$PATH>>> environment parameter.
** Step 1: Configure Hadoop and the simulator
Before we start, make sure Hadoop and the simulator are configured well.
All configuration files for Hadoop and the simulator should be placed in
directory <<<$HADOOP_ROOT/etc/hadoop>>>, where the <<<ResourceManager>>>
and Yarn scheduler load their configurations. Directory
<<<$HADOOP_ROOT/share/hadoop/tools/sls/sample-conf/>>> provides several
example configurations, that can be used to start a demo.
For configuration of Hadoop and Yarn scheduler, users can refer to Yarn’s
website ({{{http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/}
http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/}}).
For the simulator, it loads configuration information from file
<<<$HADOOP_ROOT/etc/hadoop/sls-runner.xml>>>.
Here we illustrate each configuration parameter in <<<sls-runner.xml>>>.
Note that <<<$HADOOP_ROOT/share/hadoop/tools/sls/sample-conf/sls-runner.xml>>>
contains all the default values for these configuration parameters.
* <<<yarn.sls.runner.pool.size>>>
The simulator uses a thread pool to simulate the <<<NM>>> and <<<AM>>> running
, and this parameter specifies the number of threads in the pool.
* <<<yarn.sls.nm.memory.mb>>>
The total memory for each <<<NMSimulator>>>.
* <<<yarn.sls.nm.vcores>>>
The total vCores for each <<<NMSimulator>>>.
* <<<yarn.sls.nm.heartbeat.interval.ms>>>
The heartbeat interval for each <<<NMSimulator>>>.
* <<<yarn.sls.am.heartbeat.interval.ms>>>
The heartbeat interval for each <<<AMSimulator>>>.
* <<<yarn.sls.am.type.mapreduce>>>
The <<<AMSimulator>>> implementation for MapReduce-like applications.
Users can specify implementations for other type of applications.
* <<<yarn.sls.container.memory.mb>>>
The memory required for each container simulator.
* <<<yarn.sls.container.vcores>>>
The vCores required for each container simulator.
* <<<yarn.sls.runner.metrics.switch>>>
The simulator introduces {{{http://metrics.codahale.com/}Metrics}} to measure
the behaviors of critical components and operations. This field specifies
whether we open (<<<ON>>>) or close (<<<OFF>>>) the Metrics running.
* <<<yarn.sls.metrics.web.address.port>>>
The port used by simulator to provide real-time tracking. The default value is
10001.
* <<<org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler>>>
The implementation of scheduler metrics of Fifo Scheduler.
* <<<org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler>>>
The implementation of scheduler metrics of Fair Scheduler.
* <<<org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler>>>
The implementation of scheduler metrics of Capacity Scheduler.
** Step 2: Run the simulator
The simulator supports two types of input files: the rumen traces and its own
input traces. The script to start the simulator is <<<slsrun.sh>>>.
+----+
$ cd $HADOOP_ROOT/share/hadoop/tools/sls
$ bin/slsrun.sh
--input-rumen|--input-sls=<TRACE_FILE1,TRACE_FILE2,...>
--output-dir=<SLS_SIMULATION_OUTPUT_DIRECTORY> [--nodes=<SLS_NODES_FILE>]
[--track-jobs=<JOBID1,JOBID2,...>] [--print-simulation]
+----+
* <<<--input-rumen>>>: The input rumen trace files. Users can input multiple
files, separated by comma. One example trace is provided in
<<<$HADOOP_ROOT/share/hadoop/tools/sls/sample-data/2jobs2min-rumen-jh.json>>>.
* <<<--input-sls>>>: Simulator its own file format. The simulator also
provides a tool to convert rumen traces to sls traces (<<<rumen2sls.sh>>>).
Refer to appendix for an example of sls input json file.
* <<<--output-dir>>>: The output directory for generated running logs and
metrics.
* <<<--nodes>>>: The cluster topology. By default, the simulator will use the
topology fetched from the input json files. Users can specifies a new topology
by setting this parameter. Refer to the appendix for the topology file format.
* <<<--track-jobs>>>: The particular jobs that will be tracked during
simulator running, spearated by comma.
* <<<--print-simulation>>>: Whether to print out simulation information
before simulator running, including number of nodes, applications, tasks,
and information for each application.
In comparison to rumen format, here the sls format is much simpler and users
can easily generate various workload. The simulator also provides a tool to
convert rumen traces to sls traces.
+----+
$ bin/rumen2sls.sh
--rumen-file=<RUMEN_FILE>
--output-dir=<SLS_OUTPUT_DIRECTORY>
[--output-prefix=<SLS_FILE_PREFIX>]
+----+
* <<<--rumen-file>>>: The rumen format file. One example trace is provided
in directory <<<sample-data>>>.
* <<<--output-dir>>>: The output directory of generated simulation traces.
Two files will be generated in this output directory, including one trace
file including all job and task information, and another file showing the
topology information.
* <<<--output-prefix>>>: The prefix of the generated files. The default value
is ”sls”, and the two generated files are <<<sls-jobs.json>>> and
<<<sls-nodes.json>>>.
* Metrics
The Yarn Scheduler Load Simulator has integrated
{{{http://metrics.codahale.com/}Metrics}} to measure the behaviors of critical
components and operations, including running applications and containers,
cluster available resources, scheduler operation timecost, et al. If the
switch <<<yarn.sls.runner.metrics.switch>>> is set <<<ON>>>, <<<Metrics>>>
will run and output it logs in <<<--output-dir>>> directory specified by users.
Users can track these information during simulator running, and can also
analyze these logs after running to evaluate the scheduler performance.
** Real-time Tracking
The simulator provides an interface for tracking its running in real-time.
Users can go to <<<http://host:port/simulate>>> to track whole running,
and <<<http://host:port/track>>> to track a particular job or queue. Here
the <<<host>>> is the place when we run the simulator, and <<<port>>> is
the value configured by <<<yarn.sls.metrics.web.address.port>>> (default value
is 10001).
Here we'll illustrate each chart shown in the webpage.
The first figure describes the number of running applications and containers.
[images/sls_running_apps_containers.png] Number of running applications/containers
The second figure describes the allocated and available resources (memory)
in the cluster.
[images/sls_cluster_memory.png] Cluster Resource (Memory)
The third figure describes the allocated resource for each queue. Here we have
three queues: sls_queue_1, sls_queue_2, and sls_queue_3.The first two queues
are configured with 25% share, while the last one has 50% share.
[images/sls_queue_allocated_memory.png] Queue Allocated Resource (Memory)
The fourth figure describes the timecost for each scheduler operation.
[images/sls_scheduler_operation_timecost.png] Scheduler Opertion Timecost
Finally, we measure the memory used by the simulator.
[images/sls_JVM.png] JVM Memory
The simulator also provides an interface for tracking some particular
jobs and queues. Go to <<<http://<Host>:<Port>/track>>> to get these
information.
Here the first figure illustrates the resource usage information for queue
<<<SLS_Queue_1>>>.
[images/sls_track_queue.png] Tracking Queue <<<sls_queue_3>>>
The second figure illustrates the resource usage information for job
<<<job_1369942127770_0653>>>.
[images/sls_track_job.png] Tracking Job <<<job_1369942127770_0653>>>
** Offline Analysis
After the simulator finishes, all logs are saved in the output directory
specified by <<<--output-dir>>> in
<<<$HADOOP_ROOT/share/hadoop/tools/sls/bin/slsrun.sh>>>.
* File <<<realtimetrack.json>>>: records all real-time tracking logs every 1
second.
* File <<<jobruntime.csv>>>: records all jobs’ start and end time in the
simulator.
* Folder <<<metrics>>>: logs generated by the Metrics.
[]
Users can also reproduce those real-time tracking charts in offline mode.
Just upload the <<<realtimetrack.json>>> to
<<<$HADOOP_ROOT/share/hadoop/tools/sls/html/showSimulationTrace.html>>>.
For browser security problem, need to put files <<<realtimetrack.json>>> and
<<<showSimulationTrace.html>>> in the same directory.
* Appendix
** Resources
{{{https://issues.apache.org/jira/browse/YARN-1021}YARN-1021}} is the main
JIRA that introduces Yarn Scheduler Load Simulator to Hadoop Yarn project.
** SLS JSON input file format
Here we provide an example format of the sls json file, which contains 2 jobs.
The first job has 3 map tasks and the second one has 2 map tasks.
+----+
{
"am.type" : "mapreduce",
"job.start.ms" : 0,
"job.end.ms" : 95375,
"job.queue.name" : "sls_queue_1",
"job.id" : "job_1",
"job.user" : "default",
"job.tasks" : [ {
"container.host" : "/default-rack/node1",
"container.start.ms" : 6664,
"container.end.ms" : 23707,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node3",
"container.start.ms" : 6665,
"container.end.ms" : 21593,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node2",
"container.start.ms" : 68770,
"container.end.ms" : 86613,
"container.priority" : 20,
"container.type" : "map"
} ]
}
{
"am.type" : "mapreduce",
"job.start.ms" : 105204,
"job.end.ms" : 197256,
"job.queue.name" : "sls_queue_2",
"job.id" : "job_2",
"job.user" : "default",
"job.tasks" : [ {
"container.host" : "/default-rack/node1",
"container.start.ms" : 111822,
"container.end.ms" : 133985,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node2",
"container.start.ms" : 111788,
"container.end.ms" : 131377,
"container.priority" : 20,
"container.type" : "map"
} ]
}
+----+
** Simulator input topology file format
Here is an example input topology file which has 3 nodes organized in 1 rack.
+----+
{
"rack" : "default-rack",
"nodes" : [ {
"node" : "node1"
}, {
"node" : "node2"
}, {
"node" : "node3"
}]
}
+----+

357
hadoop-tools/hadoop-sls/src/site/markdown/SchedulerLoadSimulator.md Normal file
View File

@ -0,0 +1,357 @@
<!---
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
Yarn Scheduler Load Simulator (SLS)
===================================
* [Yarn Scheduler Load Simulator (SLS)](#Yarn_Scheduler_Load_Simulator_SLS)
* [Overview](#Overview)
* [Overview](#Overview)
* [Goals](#Goals)
* [Architecture](#Architecture)
* [Usecases](#Usecases)
* [Usage](#Usage)
* [Step 1: Configure Hadoop and the simulator](#Step_1:_Configure_Hadoop_and_the_simulator)
* [Step 2: Run the simulator](#Step_2:_Run_the_simulator)
* [Metrics](#Metrics)
* [Real-time Tracking](#Real-time_Tracking)
* [Offline Analysis](#Offline_Analysis)
* [Appendix](#Appendix)
* [Resources](#Resources)
* [SLS JSON input file format](#SLS_JSON_input_file_format)
* [Simulator input topology file format](#Simulator_input_topology_file_format)
Overview
--------
### Overview
The Yarn scheduler is a fertile area of interest with different implementations, e.g., Fifo, Capacity and Fair schedulers. Meanwhile, several optimizations are also made to improve scheduler performance for different scenarios and workload. Each scheduler algorithm has its own set of features, and drives scheduling decisions by many factors, such as fairness, capacity guarantee, resource availability, etc. It is very important to evaluate a scheduler algorithm very well before we deploy in a production cluster. Unfortunately, currently it is non-trivial to evaluate a scheduler algorithm. Evaluating in a real cluster is always time and cost consuming, and it is also very hard to find a large-enough cluster. Hence, a simulator which can predict how well a scheduler algorithm for some specific workload would be quite useful.
The Yarn Scheduler Load Simulator (SLS) is such a tool, which can simulate large-scale Yarn clusters and application loads in a single machine.This simulator would be invaluable in furthering Yarn by providing a tool for researchers and developers to prototype new scheduler features and predict their behavior and performance with reasonable amount of confidence, thereby aiding rapid innovation.
o
The simulator will exercise the real Yarn `ResourceManager` removing the network factor by simulating `NodeManagers` and `ApplicationMasters` via handling and dispatching `NM`/`AMs` heartbeat events from within the same JVM. To keep tracking of scheduler behavior and performance, a scheduler wrapper will wrap the real scheduler.
The size of the cluster and the application load can be loaded from configuration files, which are generated from job history files directly by adopting [Apache Rumen](https://hadoop.apache.org/docs/stable/rumen.html).
The simulator will produce real time metrics while executing, including:
* Resource usages for whole cluster and each queue, which can be utilized to
configure cluster and queue's capacity.
* The detailed application execution trace (recorded in relation to simulated
time), which can be analyzed to understand/validate the scheduler behavior
(individual jobs turn around time, throughput, fairness, capacity guarantee,
etc.).
* Several key metrics of scheduler algorithm, such as time cost of each
scheduler operation (allocate, handle, etc.), which can be utilized by Hadoop
developers to find the code spots and scalability limits.
### Goals
* Exercise the scheduler at scale without a real cluster using real job
traces.
* Being able to simulate real workloads.
### Architecture
The following figure illustrates the implementation architecture of the simulator.
![The architecture of the simulator](images/sls_arch.png)
The simulator takes input of workload traces, and fetches the cluster and applications information. For each NM and AM, the simulator builds a simulator to simulate their running. All NM/AM simulators run in a thread pool. The simulator reuses Yarn Resource Manager, and builds a wrapper out of the scheduler. The Scheduler Wrapper can track the scheduler behaviors and generates several logs, which are the outputs of the simulator and can be further analyzed.
### Usecases
* Engineering
* Verify correctness of scheduler algorithm under load
* Cheap/practical way for finding code hotspots/critical-path.
* Validate the impact of changes and new features.
* Determine what drives the scheduler scalability limits.
* QA
* Validate scheduler behavior for "large" clusters and several workload profiles.
* Solutions/Sales.
* Sizing model for predefined/typical workloads.
* Cluster sizing tool using real customer data (job traces).
* Determine minimum SLAs under a particular workload.
Usage
-----
This section will show how to use the simulator. Here let `$HADOOP_ROOT` represent the Hadoop install directory. If you build Hadoop yourself, `$HADOOP_ROOT` is `hadoop-dist/target/hadoop-$VERSION`. The simulator is located at `$HADOOP_ROOT/share/hadoop/tools/sls`. The fold `sls` containers four directories: `bin`, `html`, `sample-conf`, and `sample-data`
* `bin`: contains running scripts for the simulator.
* `html`: contains several html/css/js files we needed for real-time tracking.
* `sample-conf`: specifies the simulator configurations.
* `sample-data`: provides an example rumen trace, which can be used to
generate inputs of the simulator.
The following sections will describe how to use the simulator step by step. Before start, make sure that command `hadoop` is included in your `$PATH` environment parameter.
### Step 1: Configure Hadoop and the simulator
Before we start, make sure Hadoop and the simulator are configured well. All configuration files for Hadoop and the simulator should be placed in directory `$HADOOP_ROOT/etc/hadoop`, where the `ResourceManager` and Yarn scheduler load their configurations. Directory `$HADOOP_ROOT/share/hadoop/tools/sls/sample-conf/` provides several example configurations, that can be used to start a demo.
For configuration of Hadoop and Yarn scheduler, users can refer to Yarn’s website (<http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/>).
For the simulator, it loads configuration information from file `$HADOOP_ROOT/etc/hadoop/sls-runner.xml`.
Here we illustrate each configuration parameter in `sls-runner.xml`. Note that `$HADOOP_ROOT/share/hadoop/tools/sls/sample-conf/sls-runner.xml` contains all the default values for these configuration parameters.
* `yarn.sls.runner.pool.size`
The simulator uses a thread pool to simulate the `NM` and `AM` running,
and this parameter specifies the number of threads in the pool.
* `yarn.sls.nm.memory.mb`
The total memory for each `NMSimulator`.
* `yarn.sls.nm.vcores`
The total vCores for each `NMSimulator`.
* `yarn.sls.nm.heartbeat.interval.ms`
The heartbeat interval for each `NMSimulator`.
* `yarn.sls.am.heartbeat.interval.ms`
The heartbeat interval for each `AMSimulator`.
* `yarn.sls.am.type.mapreduce`
The `AMSimulator` implementation for MapReduce-like applications.
Users can specify implementations for other type of applications.
* `yarn.sls.container.memory.mb`
The memory required for each container simulator.
* `yarn.sls.container.vcores`
The vCores required for each container simulator.
* `yarn.sls.runner.metrics.switch`
The simulator introduces [Metrics](http://metrics.codahale.com/) to measure
the behaviors of critical components and operations. This field specifies
whether we open (`ON`) or close (`OFF`) the Metrics running.
* `yarn.sls.metrics.web.address.port`
The port used by simulator to provide real-time tracking. The default value is
10001.
* `org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler`
The implementation of scheduler metrics of Fifo Scheduler.
* `org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler`
The implementation of scheduler metrics of Fair Scheduler.
* `org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler`
The implementation of scheduler metrics of Capacity Scheduler.
### Step 2: Run the simulator
The simulator supports two types of input files: the rumen traces and its own input traces. The script to start the simulator is `slsrun.sh`.
$ cd $HADOOP_ROOT/share/hadoop/tools/sls
$ bin/slsrun.sh
--input-rumen |--input-sls=<TRACE_FILE1,TRACE_FILE2,...>
--output-dir=<SLS_SIMULATION_OUTPUT_DIRECTORY> [--nodes=<SLS_NODES_FILE>]
[--track-jobs=<JOBID1,JOBID2,...>] [--print-simulation]
* `--input-rumen`: The input rumen trace files. Users can input multiple
files, separated by comma. One example trace is provided in
`$HADOOP_ROOT/share/hadoop/tools/sls/sample-data/2jobs2min-rumen-jh.json`.
* `--input-sls`: Simulator its own file format. The simulator also
provides a tool to convert rumen traces to sls traces (`rumen2sls.sh`).
Refer to appendix for an example of sls input json file.
* `--output-dir`: The output directory for generated running logs and
metrics.
* `--nodes`: The cluster topology. By default, the simulator will use the
topology fetched from the input json files. Users can specifies a new topology
by setting this parameter. Refer to the appendix for the topology file format.
* `--track-jobs`: The particular jobs that will be tracked during
simulator running, spearated by comma.
* `--print-simulation`: Whether to print out simulation information
before simulator running, including number of nodes, applications, tasks,
and information for each application.
In comparison to rumen format, here the sls format is much simpler and users
can easily generate various workload. The simulator also provides a tool to
convert rumen traces to sls traces.
$ bin/rumen2sls.sh
--rumen-file=<RUMEN_FILE>
--output-dir=<SLS_OUTPUT_DIRECTORY>
[--output-prefix=<SLS_FILE_PREFIX>]
* `--rumen-file`: The rumen format file. One example trace is provided
in directory `sample-data`.
* `--output-dir`: The output directory of generated simulation traces.
Two files will be generated in this output directory, including one trace
file including all job and task information, and another file showing the
topology information.
* `--output-prefix`: The prefix of the generated files. The default value
is "sls", and the two generated files are `sls-jobs.json` and
`sls-nodes.json`.
Metrics
-------
The Yarn Scheduler Load Simulator has integrated [Metrics](http://metrics.codahale.com/) to measure the behaviors of critical components and operations, including running applications and containers, cluster available resources, scheduler operation timecost, et al. If the switch `yarn.sls.runner.metrics.switch` is set `ON`, `Metrics` will run and output it logs in `--output-dir` directory specified by users. Users can track these information during simulator running, and can also analyze these logs after running to evaluate the scheduler performance.
### Real-time Tracking
The simulator provides an interface for tracking its running in real-time. Users can go to `http://host:port/simulate` to track whole running, and `http://host:port/track` to track a particular job or queue. Here the `host` is the place when we run the simulator, and `port` is the value configured by `yarn.sls.metrics.web.address.port` (default value is 10001).
Here we'll illustrate each chart shown in the webpage.
The first figure describes the number of running applications and containers.
![Number of running applications/containers](images/sls_running_apps_containers.png)
The second figure describes the allocated and available resources (memory) in the cluster.
![Cluster Resource (Memory)](images/sls_cluster_memory.png)
The third figure describes the allocated resource for each queue. Here we have three queues: sls\_queue\_1, sls\_queue\_2, and sls\_queue\_3.The first two queues are configured with 25% share, while the last one has 50% share.
![Queue Allocated Resource (Memory)](images/sls_queue_allocated_memory.png)
The fourth figure describes the timecost for each scheduler operation.
![Scheduler Opertion Timecost](images/sls_scheduler_operation_timecost.png)
Finally, we measure the memory used by the simulator.
![JVM Memory](images/sls_JVM.png)
The simulator also provides an interface for tracking some particular jobs and queues. Go to `http://<Host>:<Port>/track` to get these information.
Here the first figure illustrates the resource usage information for queue `SLS_Queue_1`.
![Tracking Queue `sls_queue_3`](images/sls_track_queue.png)
The second figure illustrates the resource usage information for job `job_1369942127770_0653`.
![Tracking Job `job_1369942127770_0653`](images/sls_track_job.png)
### Offline Analysis
After the simulator finishes, all logs are saved in the output directory specified by `--output-dir` in `$HADOOP_ROOT/share/hadoop/tools/sls/bin/slsrun.sh`.
* File `realtimetrack.json`: records all real-time tracking logs every 1
second.
* File `jobruntime.csv`: records all jobs’ start and end time in the
simulator.
* Folder `metrics`: logs generated by the Metrics.
Users can also reproduce those real-time tracking charts in offline mode. Just upload the `realtimetrack.json` to `$HADOOP_ROOT/share/hadoop/tools/sls/html/showSimulationTrace.html`. For browser security problem, need to put files `realtimetrack.json` and `showSimulationTrace.html` in the same directory.
Appendix
--------
### Resources
[YARN-1021](https://issues.apache.org/jira/browse/YARN-1021) is the main JIRA that introduces Yarn Scheduler Load Simulator to Hadoop Yarn project.
### SLS JSON input file format
Here we provide an example format of the sls json file, which contains 2 jobs. The first job has 3 map tasks and the second one has 2 map tasks.
{
"am.type" : "mapreduce",
"job.start.ms" : 0,
"job.end.ms" : 95375,
"job.queue.name" : "sls_queue_1",
"job.id" : "job_1",
"job.user" : "default",
"job.tasks" : [ {
"container.host" : "/default-rack/node1",
"container.start.ms" : 6664,
"container.end.ms" : 23707,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node3",
"container.start.ms" : 6665,
"container.end.ms" : 21593,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node2",
"container.start.ms" : 68770,
"container.end.ms" : 86613,
"container.priority" : 20,
"container.type" : "map"
} ]
}
{
"am.type" : "mapreduce",
"job.start.ms" : 105204,
"job.end.ms" : 197256,
"job.queue.name" : "sls_queue_2",
"job.id" : "job_2",
"job.user" : "default",
"job.tasks" : [ {
"container.host" : "/default-rack/node1",
"container.start.ms" : 111822,
"container.end.ms" : 133985,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node2",
"container.start.ms" : 111788,
"container.end.ms" : 131377,
"container.priority" : 20,
"container.type" : "map"
} ]
}
### Simulator input topology file format
Here is an example input topology file which has 3 nodes organized in 1 rack.
{
"rack" : "default-rack",
"nodes" : [ {
"node" : "node1"
}, {
"node" : "node2"
}, {
"node" : "node3"
}]
}

792
hadoop-tools/hadoop-streaming/src/site/apt/HadoopStreaming.apt.vm
View File

@ -1,792 +0,0 @@
~~ Licensed under the Apache License, Version 2.0 (the "License");
~~ you may not use this file except in compliance with the License.
~~ You may obtain a copy of the License at
~~
~~ http://www.apache.org/licenses/LICENSE-2.0
~~
~~ Unless required by applicable law or agreed to in writing, software
~~ distributed under the License is distributed on an "AS IS" BASIS,
~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~~ See the License for the specific language governing permissions and
~~ limitations under the License. See accompanying LICENSE file.
---
Hadoop Streaming
---
---
${maven.build.timestamp}
Hadoop Streaming
%{toc|section=1|fromDepth=0|toDepth=4}
* Hadoop Streaming
Hadoop streaming is a utility that comes with the Hadoop distribution. The
utility allows you to create and run Map/Reduce jobs with any executable or
script as the mapper and/or the reducer. For example:
+---+
hadoop jar hadoop-streaming-${project.version}.jar \
-input myInputDirs \
-output myOutputDir \
-mapper /bin/cat \
-reducer /usr/bin/wc
+---+
* How Streaming Works
In the above example, both the mapper and the reducer are executables that
read the input from stdin (line by line) and emit the output to stdout. The
utility will create a Map/Reduce job, submit the job to an appropriate
cluster, and monitor the progress of the job until it completes.
When an executable is specified for mappers, each mapper task will launch the
executable as a separate process when the mapper is initialized. As the
mapper task runs, it converts its inputs into lines and feed the lines to the
stdin of the process. In the meantime, the mapper collects the line oriented
outputs from the stdout of the process and converts each line into a
key/value pair, which is collected as the output of the mapper. By default,
the <prefix of a line up to the first tab character> is the <<<key>>> and the
rest of the line (excluding the tab character) will be the <<<value>>>. If
there is no tab character in the line, then entire line is considered as key
and the value is null. However, this can be customized by setting
<<<-inputformat>>> command option, as discussed later.
When an executable is specified for reducers, each reducer task will launch
the executable as a separate process then the reducer is initialized. As the
reducer task runs, it converts its input key/values pairs into lines and
feeds the lines to the stdin of the process. In the meantime, the reducer
collects the line oriented outputs from the stdout of the process, converts
each line into a key/value pair, which is collected as the output of the
reducer. By default, the prefix of a line up to the first tab character is
the key and the rest of the line (excluding the tab character) is the value.
However, this can be customized by setting <<<-outputformat>>> command
option, as discussed later.
This is the basis for the communication protocol between the Map/Reduce
framework and the streaming mapper/reducer.
User can specify <<<stream.non.zero.exit.is.failure>>> as <<<true>>> or
<<<false>>> to make a streaming task that exits with a non-zero status to be
<<<Failure>>> or <<<Success>>> respectively. By default, streaming tasks
exiting with non-zero status are considered to be failed tasks.
* Streaming Command Options
Streaming supports streaming command options as well as
{{{Generic_Command_Options}generic command options}}. The general command
line syntax is shown below.
<<Note:>> Be sure to place the generic options before the streaming options,
otherwise the command will fail. For an example, see
{{{Making_Archives_Available_to_Tasks}Making Archives Available to Tasks}}.
+---+
hadoop command [genericOptions] [streamingOptions]
+---+
The Hadoop streaming command options are listed here:
*-------------*--------------------*------------------------------------------*
|| Parameter || Optional/Required || Description |
*-------------+--------------------+------------------------------------------+
| -input directoryname or filename | Required | Input location for mapper
*-------------+--------------------+------------------------------------------+
| -output directoryname | Required | Output location for reducer
*-------------+--------------------+------------------------------------------+
| -mapper executable or JavaClassName | Required | Mapper executable
*-------------+--------------------+------------------------------------------+
| -reducer executable or JavaClassName | Required | Reducer executable
*-------------+--------------------+------------------------------------------+
| -file filename | Optional | Make the mapper, reducer, or combiner executable
| | | available locally on the compute nodes
*-------------+--------------------+------------------------------------------+
| -inputformat JavaClassName | Optional | Class you supply should return
| | | key/value pairs of Text class. If not
| | | specified, TextInputFormat is used as
| | | the default
*-------------+--------------------+------------------------------------------+
| -outputformat JavaClassName | Optional | Class you supply should take
| | | key/value pairs of Text class. If
| | | not specified, TextOutputformat is
| | | used as the default
*-------------+--------------------+------------------------------------------+
| -partitioner JavaClassName | Optional | Class that determines which reduce a
| | | key is sent to
*-------------+--------------------+------------------------------------------+
| -combiner streamingCommand | Optional | Combiner executable for map output
| or JavaClassName | |
*-------------+--------------------+------------------------------------------+
| -cmdenv name=value | Optional | Pass environment variable to streaming
| | | commands
*-------------+--------------------+------------------------------------------+
| -inputreader | Optional | For backwards-compatibility: specifies a record
| | | reader class (instead of an input format class)
*-------------+--------------------+------------------------------------------+
| -verbose | Optional | Verbose output
*-------------+--------------------+------------------------------------------+
| -lazyOutput | Optional | Create output lazily. For example, if the output
| | | format is based on FileOutputFormat, the output file
| | | is created only on the first call to Context.write
*-------------+--------------------+------------------------------------------+
| -numReduceTasks | Optional | Specify the number of reducers
*-------------+--------------------+------------------------------------------+
| -mapdebug | Optional | Script to call when map task fails
*-------------+--------------------+------------------------------------------+
| -reducedebug | Optional | Script to call when reduce task fails
*-------------+--------------------+------------------------------------------+
** Specifying a Java Class as the Mapper/Reducer
You can supply a Java class as the mapper and/or the reducer.
+---+
hadoop jar hadoop-streaming-${project.version}.jar \
-input myInputDirs \
-output myOutputDir \
-inputformat org.apache.hadoop.mapred.KeyValueTextInputFormat \
-mapper org.apache.hadoop.mapred.lib.IdentityMapper \
-reducer /usr/bin/wc
+---+
You can specify <<<stream.non.zero.exit.is.failure>>> as <<<true>>> or
<<<false>>> to make a streaming task that exits with a non-zero status to be
<<<Failure>>> or <<<Success>>> respectively. By default, streaming tasks
exiting with non-zero status are considered to be failed tasks.
** Packaging Files With Job Submissions
You can specify any executable as the mapper and/or the reducer. The
executables do not need to pre-exist on the machines in the cluster; however,
if they don't, you will need to use "-file" option to tell the framework to
pack your executable files as a part of job submission. For example:
+---+
hadoop jar hadoop-streaming-${project.version}.jar \
-input myInputDirs \
-output myOutputDir \
-mapper myPythonScript.py \
-reducer /usr/bin/wc \
-file myPythonScript.py
+---+
The above example specifies a user defined Python executable as the mapper.
The option "-file myPythonScript.py" causes the python executable shipped
to the cluster machines as a part of job submission.
In addition to executable files, you can also package other auxiliary files
(such as dictionaries, configuration files, etc) that may be used by the
mapper and/or the reducer. For example:
+---+
hadoop jar hadoop-streaming-${project.version}.jar \
-input myInputDirs \
-output myOutputDir \
-mapper myPythonScript.py \
-reducer /usr/bin/wc \
-file myPythonScript.py \
-file myDictionary.txt
+---+
** Specifying Other Plugins for Jobs
Just as with a normal Map/Reduce job, you can specify other plugins for a
streaming job:
+---+
-inputformat JavaClassName
-outputformat JavaClassName
-partitioner JavaClassName
-combiner streamingCommand or JavaClassName
+---+
The class you supply for the input format should return key/value pairs of
Text class. If you do not specify an input format class, the TextInputFormat
is used as the default. Since the TextInputFormat returns keys of
LongWritable class, which are actually not part of the input data, the keys
will be discarded; only the values will be piped to the streaming mapper.
The class you supply for the output format is expected to take key/value
pairs of Text class. If you do not specify an output format class, the
TextOutputFormat is used as the default.
** Setting Environment Variables
To set an environment variable in a streaming command use:
+---+
-cmdenv EXAMPLE_DIR=/home/example/dictionaries/
+---+
* Generic Command Options
Streaming supports {{{Streaming_Command_Options}streaming command options}}
as well as generic command options. The general command line syntax is shown
below.
<<Note:>> Be sure to place the generic options before the streaming options,
otherwise the command will fail. For an example, see
{{{Making_Archives_Available_to_Tasks}Making Archives Available to Tasks}}.
+---+
hadoop command [genericOptions] [streamingOptions]
+---+
The Hadoop generic command options you can use with streaming are listed
here:
*-------------*--------------------*------------------------------------------*
|| Parameter || Optional/Required || Description |
*-------------+--------------------+------------------------------------------+
| -conf configuration_file | Optional | Specify an application configuration
| | | file
*-------------+--------------------+------------------------------------------+
| -D property=value | Optional | Use value for given property
*-------------+--------------------+------------------------------------------+
| -fs host:port or local | Optional | Specify a namenode
*-------------+--------------------+------------------------------------------+
| -files | Optional | Specify comma-separated files to be copied to the
| | | Map/Reduce cluster
*-------------+--------------------+------------------------------------------+
| -libjars | Optional | Specify comma-separated jar files to include in the
| | | classpath
*-------------+--------------------+------------------------------------------+
| -archives | Optional | Specify comma-separated archives to be unarchived on
| | | the compute machines
*-------------+--------------------+------------------------------------------+
** Specifying Configuration Variables with the -D Option
You can specify additional configuration variables by using
"-D \<property\>=\<value\>".
*** Specifying Directories
To change the local temp directory use:
+---+
-D dfs.data.dir=/tmp
+---+
To specify additional local temp directories use:
+---+
-D mapred.local.dir=/tmp/local
-D mapred.system.dir=/tmp/system
-D mapred.temp.dir=/tmp/temp
+---+
<<Note:>> For more details on job configuration parameters see:
{{{./mapred-default.xml}mapred-default.xml}}
*** Specifying Map-Only Jobs
Often, you may want to process input data using a map function only. To do
this, simply set <<<mapreduce.job.reduces>>> to zero. The Map/Reduce
framework will not create any reducer tasks. Rather, the outputs of the
mapper tasks will be the final output of the job.
+---+
-D mapreduce.job.reduces=0
+---+
To be backward compatible, Hadoop Streaming also supports the "-reducer NONE"
option, which is equivalent to "-D mapreduce.job.reduces=0".
*** Specifying the Number of Reducers
To specify the number of reducers, for example two, use:
+---+
hadoop jar hadoop-streaming-${project.version}.jar \
-D mapreduce.job.reduces=2 \
-input myInputDirs \
-output myOutputDir \
-mapper /bin/cat \
-reducer /usr/bin/wc
+---+
*** Customizing How Lines are Split into Key/Value Pairs
As noted earlier, when the Map/Reduce framework reads a line from the stdout
of the mapper, it splits the line into a key/value pair. By default, the
prefix of the line up to the first tab character is the key and the rest of
the line (excluding the tab character) is the value.
However, you can customize this default. You can specify a field separator
other than the tab character (the default), and you can specify the nth
(n >= 1) character rather than the first character in a line (the default) as
the separator between the key and value. For example:
+---+
hadoop jar hadoop-streaming-${project.version}.jar \
-D stream.map.output.field.separator=. \
-D stream.num.map.output.key.fields=4 \
-input myInputDirs \
-output myOutputDir \
-mapper /bin/cat \
-reducer /bin/cat
+---+
In the above example, "-D stream.map.output.field.separator=." specifies "."
as the field separator for the map outputs, and the prefix up to the fourth
"." in a line will be the key and the rest of the line (excluding the fourth
".") will be the value. If a line has less than four "."s, then the whole
line will be the key and the value will be an empty Text object (like the one
created by new Text("")).
Similarly, you can use "-D stream.reduce.output.field.separator=SEP" and
"-D stream.num.reduce.output.fields=NUM" to specify the nth field separator
in a line of the reduce outputs as the separator between the key and the
value.
Similarly, you can specify "stream.map.input.field.separator" and
"stream.reduce.input.field.separator" as the input separator for Map/Reduce
inputs. By default the separator is the tab character.
** Working with Large Files and Archives
The -files and -archives options allow you to make files and archives
available to the tasks. The argument is a URI to the file or archive that you
have already uploaded to HDFS. These files and archives are cached across
jobs. You can retrieve the host and fs_port values from the fs.default.name
config variable.
<<Note:>> The -files and -archives options are generic options. Be sure to
place the generic options before the command options, otherwise the command
will fail.
*** Making Files Available to Tasks
The -files option creates a symlink in the current working directory of the
tasks that points to the local copy of the file.
In this example, Hadoop automatically creates a symlink named testfile.txt in
the current working directory of the tasks. This symlink points to the local
copy of testfile.txt.
+---+
-files hdfs://host:fs_port/user/testfile.txt
+---+
User can specify a different symlink name for -files using #.
+---+
-files hdfs://host:fs_port/user/testfile.txt#testfile
+---+
Multiple entries can be specified like this:
+---+
-files hdfs://host:fs_port/user/testfile1.txt,hdfs://host:fs_port/user/testfile2.txt
+---+
*** Making Archives Available to Tasks
The -archives option allows you to copy jars locally to the current working
directory of tasks and automatically unjar the files.
In this example, Hadoop automatically creates a symlink named testfile.jar in
the current working directory of tasks. This symlink points to the directory
that stores the unjarred contents of the uploaded jar file.
+---+
-archives hdfs://host:fs_port/user/testfile.jar
+---+
User can specify a different symlink name for -archives using #.
+---+
-archives hdfs://host:fs_port/user/testfile.tgz#tgzdir
+---+
In this example, the input.txt file has two lines specifying the names of the
two files: cachedir.jar/cache.txt and cachedir.jar/cache2.txt. "cachedir.jar"
is a symlink to the archived directory, which has the files "cache.txt" and
"cache2.txt".
+---+
hadoop jar hadoop-streaming-${project.version}.jar \
-archives 'hdfs://hadoop-nn1.example.com/user/me/samples/cachefile/cachedir.jar' \
-D mapreduce.job.maps=1 \
-D mapreduce.job.reduces=1 \
-D mapreduce.job.name="Experiment" \
-input "/user/me/samples/cachefile/input.txt" \
-output "/user/me/samples/cachefile/out" \
-mapper "xargs cat" \
-reducer "cat"
$ ls test_jar/
cache.txt cache2.txt
$ jar cvf cachedir.jar -C test_jar/ .
added manifest
adding: cache.txt(in = 30) (out= 29)(deflated 3%)
adding: cache2.txt(in = 37) (out= 35)(deflated 5%)
$ hdfs dfs -put cachedir.jar samples/cachefile
$ hdfs dfs -cat /user/me/samples/cachefile/input.txt
cachedir.jar/cache.txt
cachedir.jar/cache2.txt
$ cat test_jar/cache.txt
This is just the cache string
$ cat test_jar/cache2.txt
This is just the second cache string
$ hdfs dfs -ls /user/me/samples/cachefile/out
Found 2 items
-rw-r--r-- 1 me supergroup 0 2013-11-14 17:00 /user/me/samples/cachefile/out/_SUCCESS
-rw-r--r-- 1 me supergroup 69 2013-11-14 17:00 /user/me/samples/cachefile/out/part-00000
$ hdfs dfs -cat /user/me/samples/cachefile/out/part-00000
This is just the cache string
This is just the second cache string
+---+
* More Usage Examples
** Hadoop Partitioner Class
Hadoop has a library class,
{{{../../api/org/apache/hadoop/mapred/lib/KeyFieldBasedPartitioner.html}
KeyFieldBasedPartitioner}}, that is useful for many applications. This class
allows the Map/Reduce framework to partition the map outputs based on certain
key fields, not the whole keys. For example:
+---+
hadoop jar hadoop-streaming-${project.version}.jar \
-D stream.map.output.field.separator=. \
-D stream.num.map.output.key.fields=4 \
-D map.output.key.field.separator=. \
-D mapreduce.partition.keypartitioner.options=-k1,2 \
-D mapreduce.job.reduces=12 \
-input myInputDirs \
-output myOutputDir \
-mapper /bin/cat \
-reducer /bin/cat \
-partitioner org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner
+---+
Here, <-D stream.map.output.field.separator=.> and
<-D stream.num.map.output.key.fields=4> are as explained in previous example.
The two variables are used by streaming to identify the key/value pair of
mapper.
The map output keys of the above Map/Reduce job normally have four fields
separated by ".". However, the Map/Reduce framework will partition the map
outputs by the first two fields of the keys using the
<-D mapred.text.key.partitioner.options=-k1,2> option. Here,
<-D map.output.key.field.separator=.> specifies the separator for the
partition. This guarantees that all the key/value pairs with the same first
two fields in the keys will be partitioned into the same reducer.
<This is effectively equivalent to specifying the first two fields as the
primary key and the next two fields as the secondary. The primary key is used
for partitioning, and the combination of the primary and secondary keys is
used for sorting.> A simple illustration is shown here:
Output of map (the keys)
+---+
11.12.1.2
11.14.2.3
11.11.4.1
11.12.1.1
11.14.2.2
+---+
Partition into 3 reducers (the first 2 fields are used as keys for partition)
+---+
11.11.4.1
-----------
11.12.1.2
11.12.1.1
-----------
11.14.2.3
11.14.2.2
+---+
Sorting within each partition for the reducer(all 4 fields used for sorting)
+---+
11.11.4.1
-----------
11.12.1.1
11.12.1.2
-----------
11.14.2.2
11.14.2.3
+---+
** Hadoop Comparator Class
Hadoop has a library class,
{{{../../api/org/apache/hadoop/mapreduce/lib/partition/KeyFieldBasedComparator.html}
KeyFieldBasedComparator}}, that is useful for many applications. This class
provides a subset of features provided by the Unix/GNU Sort. For example:
+---+
hadoop jar hadoop-streaming-${project.version}.jar \
-D mapreduce.job.output.key.comparator.class=org.apache.hadoop.mapreduce.lib.partition.KeyFieldBasedComparator \
-D stream.map.output.field.separator=. \
-D stream.num.map.output.key.fields=4 \
-D mapreduce.map.output.key.field.separator=. \
-D mapreduce.partition.keycomparator.options=-k2,2nr \
-D mapreduce.job.reduces=1 \
-input myInputDirs \
-output myOutputDir \
-mapper /bin/cat \
-reducer /bin/cat
+---+
The map output keys of the above Map/Reduce job normally have four fields
separated by ".". However, the Map/Reduce framework will sort the outputs by
the second field of the keys using the
<-D mapreduce.partition.keycomparator.options=-k2,2nr> option. Here, <-n>
specifies that the sorting is numerical sorting and <-r> specifies that the
result should be reversed. A simple illustration is shown below:
Output of map (the keys)
+---+
11.12.1.2
11.14.2.3
11.11.4.1
11.12.1.1
11.14.2.2
+---+
Sorting output for the reducer (where second field used for sorting)
+---+
11.14.2.3
11.14.2.2
11.12.1.2
11.12.1.1
11.11.4.1
+---+
** Hadoop Aggregate Package
Hadoop has a library package called
{{{../../org/apache/hadoop/mapred/lib/aggregate/package-summary.html}
Aggregate}}. Aggregate provides a special reducer class and a special
combiner class, and a list of simple aggregators that perform aggregations
such as "sum", "max", "min" and so on over a sequence of values. Aggregate
allows you to define a mapper plugin class that is expected to generate
"aggregatable items" for each input key/value pair of the mappers. The
combiner/reducer will aggregate those aggregatable items by invoking the
appropriate aggregators.
To use Aggregate, simply specify "-reducer aggregate":
+---+
hadoop jar hadoop-streaming-${project.version}.jar \
-input myInputDirs \
-output myOutputDir \
-mapper myAggregatorForKeyCount.py \
-reducer aggregate \
-file myAggregatorForKeyCount.py \
+---+
The python program myAggregatorForKeyCount.py looks like:
+---+
#!/usr/bin/python
import sys;
def generateLongCountToken(id):
return "LongValueSum:" + id + "\t" + "1"
def main(argv):
line = sys.stdin.readline();
try:
while line:
line = line&#91;:-1];
fields = line.split("\t");
print generateLongCountToken(fields&#91;0]);
line = sys.stdin.readline();
except "end of file":
return None
if __name__ == "__main__":
main(sys.argv)
+---+
** Hadoop Field Selection Class
Hadoop has a library class,
{{{../../api/org/apache/hadoop/mapred/lib/FieldSelectionMapReduce.html}
FieldSelectionMapReduce}}, that effectively allows you to process text data
like the unix "cut" utility. The map function defined in the class treats
each input key/value pair as a list of fields. You can specify the field
separator (the default is the tab character). You can select an arbitrary
list of fields as the map output key, and an arbitrary list of fields as the
map output value. Similarly, the reduce function defined in the class treats
each input key/value pair as a list of fields. You can select an arbitrary
list of fields as the reduce output key, and an arbitrary list of fields as
the reduce output value. For example:
+---+
hadoop jar hadoop-streaming-${project.version}.jar \
-D mapreduce.map.output.key.field.separator=. \
-D mapreduce.partition.keypartitioner.options=-k1,2 \
-D mapreduce.fieldsel.data.field.separator=. \
-D mapreduce.fieldsel.map.output.key.value.fields.spec=6,5,1-3:0- \
-D mapreduce.fieldsel.reduce.output.key.value.fields.spec=0-2:5- \
-D mapreduce.map.output.key.class=org.apache.hadoop.io.Text \
-D mapreduce.job.reduces=12 \
-input myInputDirs \
-output myOutputDir \
-mapper org.apache.hadoop.mapred.lib.FieldSelectionMapReduce \
-reducer org.apache.hadoop.mapred.lib.FieldSelectionMapReduce \
-partitioner org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner
+---+
The option "-D
mapreduce.fieldsel.map.output.key.value.fields.spec=6,5,1-3:0-" specifies
key/value selection for the map outputs. Key selection spec and value
selection spec are separated by ":". In this case, the map output key will
consist of fields 6, 5, 1, 2, and 3. The map output value will consist of all
fields (0- means field 0 and all the subsequent fields).
The option "-D mapreduce.fieldsel.reduce.output.key.value.fields.spec=0-2:5-"
specifies key/value selection for the reduce outputs. In this case, the
reduce output key will consist of fields 0, 1, 2 (corresponding to the
original fields 6, 5, 1). The reduce output value will consist of all fields
starting from field 5 (corresponding to all the original fields).
* Frequently Asked Questions
** How do I use Hadoop Streaming to run an arbitrary set of (semi) independent
tasks?
Often you do not need the full power of Map Reduce, but only need to run
multiple instances of the same program - either on different parts of the
data, or on the same data, but with different parameters. You can use Hadoop
Streaming to do this.
** How do I process files, one per map?
As an example, consider the problem of zipping (compressing) a set of files
across the hadoop cluster. You can achieve this by using Hadoop Streaming
and custom mapper script:
* Generate a file containing the full HDFS path of the input files. Each map
task would get one file name as input.
* Create a mapper script which, given a filename, will get the file to local
disk, gzip the file and put it back in the desired output directory.
** How many reducers should I use?
See MapReduce Tutorial for details: {{{./MapReduceTutorial.html#Reducer}
Reducer}}
** If I set up an alias in my shell script, will that work after -mapper?
For example, say I do: alias c1='cut -f1'. Will -mapper "c1" work?
Using an alias will not work, but variable substitution is allowed as shown
in this example:
+---+
$ hdfs dfs -cat /user/me/samples/student_marks
alice 50
bruce 70
charlie 80
dan 75
$ c2='cut -f2'; hadoop jar hadoop-streaming-${project.version}.jar \
-D mapreduce.job.name='Experiment' \
-input /user/me/samples/student_marks \
-output /user/me/samples/student_out \
-mapper "$c2" -reducer 'cat'
$ hdfs dfs -cat /user/me/samples/student_out/part-00000
50
70
75
80
+---+
** Can I use UNIX pipes?
For example, will -mapper "cut -f1 | sed s/foo/bar/g" work?
Currently this does not work and gives an "java.io.IOException: Broken pipe"
error. This is probably a bug that needs to be investigated.
** What do I do if I get the "No space left on device" error?
For example, when I run a streaming job by distributing large executables
(for example, 3.6G) through the -file option, I get a "No space left on
device" error.
The jar packaging happens in a directory pointed to by the configuration
variable stream.tmpdir. The default value of stream.tmpdir is /tmp. Set the
value to a directory with more space:
+---+
-D stream.tmpdir=/export/bigspace/...
+---+
** How do I specify multiple input directories?
You can specify multiple input directories with multiple '-input' options:
+---+
hadoop jar hadoop-streaming-${project.version}.jar \
-input '/user/foo/dir1' -input '/user/foo/dir2' \
(rest of the command)
+---+
** How do I generate output files with gzip format?
Instead of plain text files, you can generate gzip files as your generated
output. Pass '-D mapreduce.output.fileoutputformat.compress=true -D
mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.GzipCodec'
as option to your streaming job.
** How do I provide my own input/output format with streaming?
You can specify your own custom class by packing them and putting the custom
jar to \$\{HADOOP_CLASSPATH\}.
** How do I parse XML documents using streaming?
You can use the record reader StreamXmlRecordReader to process XML documents.
+---+
hadoop jar hadoop-streaming-${project.version}.jar \
-inputreader "StreamXmlRecord,begin=BEGIN_STRING,end=END_STRING" \
(rest of the command)
+---+
Anything found between BEGIN_STRING and END_STRING would be treated as one
record for map tasks.
** How do I update counters in streaming applications?
A streaming process can use the stderr to emit counter information.
<<<reporter:counter:\<group\>,\<counter\>,\<amount\>>>> should be sent to
stderr to update the counter.
** How do I update status in streaming applications?
A streaming process can use the stderr to emit status information. To set a
status, <<<reporter:status:\<message\>>>> should be sent to stderr.
** How do I get the Job variables in a streaming job's mapper/reducer?
See {{{./MapReduceTutorial.html#Configured_Parameters}
Configured Parameters}}. During the execution of a streaming job, the names
of the "mapred" parameters are transformed. The dots ( . ) become underscores
( _ ). For example, mapreduce.job.id becomes mapreduce_job_id and
mapreduce.job.jar becomes mapreduce_job_jar. In your code, use the parameter
names with the underscores.

559
hadoop-tools/hadoop-streaming/src/site/markdown/HadoopStreaming.md.vm Normal file
View File

@ -0,0 +1,559 @@
%<!---
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
#set ( $H3 = '###' )
#set ( $H4 = '####' )
#set ( $H5 = '#####' )
Hadoop Streaming
================
* [Hadoop Streaming](#Hadoop_Streaming)
* [Hadoop Streaming](#Hadoop_Streaming)
* [How Streaming Works](#How_Streaming_Works)
* [Streaming Command Options](#Streaming_Command_Options)
* [Specifying a Java Class as the Mapper/Reducer](#Specifying_a_Java_Class_as_the_MapperReducer)
* [Packaging Files With Job Submissions](#Packaging_Files_With_Job_Submissions)
* [Specifying Other Plugins for Jobs](#Specifying_Other_Plugins_for_Jobs)
* [Setting Environment Variables](#Setting_Environment_Variables)
* [Generic Command Options](#Generic_Command_Options)
* [Specifying Configuration Variables with the -D Option](#Specifying_Configuration_Variables_with_the_-D_Option)
* [Specifying Directories](#Specifying_Directories)
* [Specifying Map-Only Jobs](#Specifying_Map-Only_Jobs)
* [Specifying the Number of Reducers](#Specifying_the_Number_of_Reducers)
* [Customizing How Lines are Split into Key/Value Pairs](#Customizing_How_Lines_are_Split_into_KeyValue_Pairs)
* [Working with Large Files and Archives](#Working_with_Large_Files_and_Archives)
* [Making Files Available to Tasks](#Making_Files_Available_to_Tasks)
* [Making Archives Available to Tasks](#Making_Archives_Available_to_Tasks)
* [More Usage Examples](#More_Usage_Examples)
* [Hadoop Partitioner Class](#Hadoop_Partitioner_Class)
* [Hadoop Comparator Class](#Hadoop_Comparator_Class)
* [Hadoop Aggregate Package](#Hadoop_Aggregate_Package)
* [Hadoop Field Selection Class](#Hadoop_Field_Selection_Class)
* [Frequently Asked Questions](#Frequently_Asked_Questions)
* [How do I use Hadoop Streaming to run an arbitrary set of (semi) independent tasks?](#How_do_I_use_Hadoop_Streaming_to_run_an_arbitrary_set_of_semi_independent_tasks)
* [How do I process files, one per map?](#How_do_I_process_files_one_per_map)
* [How many reducers should I use?](#How_many_reducers_should_I_use)
* [If I set up an alias in my shell script, will that work after -mapper?](#If_I_set_up_an_alias_in_my_shell_script_will_that_work_after_-mapper)
* [Can I use UNIX pipes?](#Can_I_use_UNIX_pipes)
* [What do I do if I get the "No space left on device" error?](#What_do_I_do_if_I_get_the_No_space_left_on_device_error)
* [How do I specify multiple input directories?](#How_do_I_specify_multiple_input_directories)
* [How do I generate output files with gzip format?](#How_do_I_generate_output_files_with_gzip_format)
* [How do I provide my own input/output format with streaming?](#How_do_I_provide_my_own_inputoutput_format_with_streaming)
* [How do I parse XML documents using streaming?](#How_do_I_parse_XML_documents_using_streaming)
* [How do I update counters in streaming applications?](#How_do_I_update_counters_in_streaming_applications)
* [How do I update status in streaming applications?](#How_do_I_update_status_in_streaming_applications)
* [How do I get the Job variables in a streaming job's mapper/reducer?](#How_do_I_get_the_Job_variables_in_a_streaming_jobs_mapperreducer)
Hadoop Streaming
----------------
Hadoop streaming is a utility that comes with the Hadoop distribution. The utility allows you to create and run Map/Reduce jobs with any executable or script as the mapper and/or the reducer. For example:
hadoop jar hadoop-streaming-${project.version}.jar \
-input myInputDirs \
-output myOutputDir \
-mapper /bin/cat \
-reducer /usr/bin/wc
How Streaming Works
-------------------
In the above example, both the mapper and the reducer are executables that read the input from stdin (line by line) and emit the output to stdout. The utility will create a Map/Reduce job, submit the job to an appropriate cluster, and monitor the progress of the job until it completes.
When an executable is specified for mappers, each mapper task will launch the executable as a separate process when the mapper is initialized. As the mapper task runs, it converts its inputs into lines and feed the lines to the stdin of the process. In the meantime, the mapper collects the line oriented outputs from the stdout of the process and converts each line into a key/value pair, which is collected as the output of the mapper. By default, the *prefix of a line up to the first tab character* is the `key` and the rest of the line (excluding the tab character) will be the `value`. If there is no tab character in the line, then entire line is considered as key and the value is null. However, this can be customized by setting `-inputformat` command option, as discussed later.
When an executable is specified for reducers, each reducer task will launch the executable as a separate process then the reducer is initialized. As the reducer task runs, it converts its input key/values pairs into lines and feeds the lines to the stdin of the process. In the meantime, the reducer collects the line oriented outputs from the stdout of the process, converts each line into a key/value pair, which is collected as the output of the reducer. By default, the prefix of a line up to the first tab character is the key and the rest of the line (excluding the tab character) is the value. However, this can be customized by setting `-outputformat` command option, as discussed later.
This is the basis for the communication protocol between the Map/Reduce framework and the streaming mapper/reducer.
User can specify `stream.non.zero.exit.is.failure` as `true` or `false` to make a streaming task that exits with a non-zero status to be `Failure` or `Success` respectively. By default, streaming tasks exiting with non-zero status are considered to be failed tasks.
Streaming Command Options
-------------------------
Streaming supports streaming command options as well as [generic command options](#Generic_Command_Options). The general command line syntax is shown below.
**Note:** Be sure to place the generic options before the streaming options, otherwise the command will fail. For an example, see [Making Archives Available to Tasks](#Making_Archives_Available_to_Tasks).
hadoop command [genericOptions] [streamingOptions]
The Hadoop streaming command options are listed here:
| Parameter | Optional/Required | Description |
|:---- |:---- |:---- |
| -input directoryname or filename | Required | Input location for mapper |
| -output directoryname | Required | Output location for reducer |
| -mapper executable or JavaClassName | Required | Mapper executable |
| -reducer executable or JavaClassName | Required | Reducer executable |
| -file filename | Optional | Make the mapper, reducer, or combiner executable available locally on the compute nodes |
| -inputformat JavaClassName | Optional | Class you supply should return key/value pairs of Text class. If not specified, TextInputFormat is used as the default |
| -outputformat JavaClassName | Optional | Class you supply should take key/value pairs of Text class. If not specified, TextOutputformat is used as the default |
| -partitioner JavaClassName | Optional | Class that determines which reduce a key is sent to |
| -combiner streamingCommand or JavaClassName | Optional | Combiner executable for map output |
| -cmdenv name=value | Optional | Pass environment variable to streaming commands |
| -inputreader | Optional | For backwards-compatibility: specifies a record reader class (instead of an input format class) |
| -verbose | Optional | Verbose output |
| -lazyOutput | Optional | Create output lazily. For example, if the output format is based on FileOutputFormat, the output file is created only on the first call to Context.write |
| -numReduceTasks | Optional | Specify the number of reducers |
| -mapdebug | Optional | Script to call when map task fails |
| -reducedebug | Optional | Script to call when reduce task fails |
$H3 Specifying a Java Class as the Mapper/Reducer
You can supply a Java class as the mapper and/or the reducer.
hadoop jar hadoop-streaming-${project.version}.jar \
-input myInputDirs \
-output myOutputDir \
-inputformat org.apache.hadoop.mapred.KeyValueTextInputFormat \
-mapper org.apache.hadoop.mapred.lib.IdentityMapper \
-reducer /usr/bin/wc
You can specify `stream.non.zero.exit.is.failure` as `true` or `false` to make a streaming task that exits with a non-zero status to be `Failure` or `Success` respectively. By default, streaming tasks exiting with non-zero status are considered to be failed tasks.
$H3 Packaging Files With Job Submissions
You can specify any executable as the mapper and/or the reducer. The executables do not need to pre-exist on the machines in the cluster; however, if they don't, you will need to use "-file" option to tell the framework to pack your executable files as a part of job submission. For example:
hadoop jar hadoop-streaming-${project.version}.jar \
-input myInputDirs \
-output myOutputDir \
-mapper myPythonScript.py \
-reducer /usr/bin/wc \
-file myPythonScript.py
The above example specifies a user defined Python executable as the mapper. The option "-file myPythonScript.py" causes the python executable shipped to the cluster machines as a part of job submission.
In addition to executable files, you can also package other auxiliary files (such as dictionaries, configuration files, etc) that may be used by the mapper and/or the reducer. For example:
hadoop jar hadoop-streaming-${project.version}.jar \
-input myInputDirs \
-output myOutputDir \
-mapper myPythonScript.py \
-reducer /usr/bin/wc \
-file myPythonScript.py \
-file myDictionary.txt
$H3 Specifying Other Plugins for Jobs
Just as with a normal Map/Reduce job, you can specify other plugins for a streaming job:
-inputformat JavaClassName
-outputformat JavaClassName
-partitioner JavaClassName
-combiner streamingCommand or JavaClassName
The class you supply for the input format should return key/value pairs of Text class. If you do not specify an input format class, the TextInputFormat is used as the default. Since the TextInputFormat returns keys of LongWritable class, which are actually not part of the input data, the keys will be discarded; only the values will be piped to the streaming mapper.
The class you supply for the output format is expected to take key/value pairs of Text class. If you do not specify an output format class, the TextOutputFormat is used as the default.
$H3 Setting Environment Variables
To set an environment variable in a streaming command use:
-cmdenv EXAMPLE_DIR=/home/example/dictionaries/
Generic Command Options
-----------------------
Streaming supports [streaming command options](#Streaming_Command_Options) as well as generic command options. The general command line syntax is shown below.
**Note:** Be sure to place the generic options before the streaming options, otherwise the command will fail. For an example, see [Making Archives Available to Tasks](#Making_Archives_Available_to_Tasks).
hadoop command [genericOptions] [streamingOptions]
The Hadoop generic command options you can use with streaming are listed here:
| Parameter | Optional/Required | Description |
|:---- |:---- |:---- |
| -conf configuration\_file | Optional | Specify an application configuration file |
| -D property=value | Optional | Use value for given property |
| -fs host:port or local | Optional | Specify a namenode |
| -files | Optional | Specify comma-separated files to be copied to the Map/Reduce cluster |
| -libjars | Optional | Specify comma-separated jar files to include in the classpath |
| -archives | Optional | Specify comma-separated archives to be unarchived on the compute machines |
$H3 Specifying Configuration Variables with the -D Option
You can specify additional configuration variables by using "-D \<property\>=\<value\>".
$H4 Specifying Directories
To change the local temp directory use:
-D dfs.data.dir=/tmp
To specify additional local temp directories use:
-D mapred.local.dir=/tmp/local
-D mapred.system.dir=/tmp/system
-D mapred.temp.dir=/tmp/temp
**Note:** For more details on job configuration parameters see: [mapred-default.xml](./mapred-default.xml)
$H4 Specifying Map-Only Jobs
Often, you may want to process input data using a map function only. To do this, simply set `mapreduce.job.reduces` to zero. The Map/Reduce framework will not create any reducer tasks. Rather, the outputs of the mapper tasks will be the final output of the job.
-D mapreduce.job.reduces=0
To be backward compatible, Hadoop Streaming also supports the "-reducer NONE" option, which is equivalent to "-D mapreduce.job.reduces=0".
$H4 Specifying the Number of Reducers
To specify the number of reducers, for example two, use:
hadoop jar hadoop-streaming-${project.version}.jar \
-D mapreduce.job.reduces=2 \
-input myInputDirs \
-output myOutputDir \
-mapper /bin/cat \
-reducer /usr/bin/wc
$H4 Customizing How Lines are Split into Key/Value Pairs
As noted earlier, when the Map/Reduce framework reads a line from the stdout of the mapper, it splits the line into a key/value pair. By default, the prefix of the line up to the first tab character is the key and the rest of the line (excluding the tab character) is the value.
However, you can customize this default. You can specify a field separator other than the tab character (the default), and you can specify the nth (n \>= 1) character rather than the first character in a line (the default) as the separator between the key and value. For example:
hadoop jar hadoop-streaming-${project.version}.jar \
-D stream.map.output.field.separator=. \
-D stream.num.map.output.key.fields=4 \
-input myInputDirs \
-output myOutputDir \
-mapper /bin/cat \
-reducer /bin/cat
In the above example, "-D stream.map.output.field.separator=." specifies "." as the field separator for the map outputs, and the prefix up to the fourth "." in a line will be the key and the rest of the line (excluding the fourth ".") will be the value. If a line has less than four "."s, then the whole line will be the key and the value will be an empty Text object (like the one created by new Text("")).
Similarly, you can use "-D stream.reduce.output.field.separator=SEP" and "-D stream.num.reduce.output.fields=NUM" to specify the nth field separator in a line of the reduce outputs as the separator between the key and the value.
Similarly, you can specify "stream.map.input.field.separator" and "stream.reduce.input.field.separator" as the input separator for Map/Reduce inputs. By default the separator is the tab character.
$H3 Working with Large Files and Archives
The -files and -archives options allow you to make files and archives available to the tasks. The argument is a URI to the file or archive that you have already uploaded to HDFS. These files and archives are cached across jobs. You can retrieve the host and fs\_port values from the fs.default.name config variable.
**Note:** The -files and -archives options are generic options. Be sure to place the generic options before the command options, otherwise the command will fail.
$H4 Making Files Available to Tasks
The -files option creates a symlink in the current working directory of the tasks that points to the local copy of the file.
In this example, Hadoop automatically creates a symlink named testfile.txt in the current working directory of the tasks. This symlink points to the local copy of testfile.txt.
-files hdfs://host:fs_port/user/testfile.txt
User can specify a different symlink name for -files using \#.
-files hdfs://host:fs_port/user/testfile.txt#testfile
Multiple entries can be specified like this:
-files hdfs://host:fs_port/user/testfile1.txt,hdfs://host:fs_port/user/testfile2.txt
$H4 Making Archives Available to Tasks
The -archives option allows you to copy jars locally to the current working directory of tasks and automatically unjar the files.
In this example, Hadoop automatically creates a symlink named testfile.jar in the current working directory of tasks. This symlink points to the directory that stores the unjarred contents of the uploaded jar file.
-archives hdfs://host:fs_port/user/testfile.jar
User can specify a different symlink name for -archives using \#.
-archives hdfs://host:fs_port/user/testfile.tgz#tgzdir
In this example, the input.txt file has two lines specifying the names of the two files: cachedir.jar/cache.txt and cachedir.jar/cache2.txt. "cachedir.jar" is a symlink to the archived directory, which has the files "cache.txt" and "cache2.txt".
hadoop jar hadoop-streaming-${project.version}.jar \
-archives 'hdfs://hadoop-nn1.example.com/user/me/samples/cachefile/cachedir.jar' \
-D mapreduce.job.maps=1 \
-D mapreduce.job.reduces=1 \
-D mapreduce.job.name="Experiment" \
-input "/user/me/samples/cachefile/input.txt" \
-output "/user/me/samples/cachefile/out" \
-mapper "xargs cat" \
-reducer "cat"
$ ls test_jar/
cache.txt cache2.txt
$ jar cvf cachedir.jar -C test_jar/ .
added manifest
adding: cache.txt(in = 30) (out= 29)(deflated 3%)
adding: cache2.txt(in = 37) (out= 35)(deflated 5%)
$ hdfs dfs -put cachedir.jar samples/cachefile
$ hdfs dfs -cat /user/me/samples/cachefile/input.txt
cachedir.jar/cache.txt
cachedir.jar/cache2.txt
$ cat test_jar/cache.txt
This is just the cache string
$ cat test_jar/cache2.txt
This is just the second cache string
$ hdfs dfs -ls /user/me/samples/cachefile/out
Found 2 items
-rw-r--r-* 1 me supergroup 0 2013-11-14 17:00 /user/me/samples/cachefile/out/_SUCCESS
-rw-r--r-* 1 me supergroup 69 2013-11-14 17:00 /user/me/samples/cachefile/out/part-00000
$ hdfs dfs -cat /user/me/samples/cachefile/out/part-00000
This is just the cache string
This is just the second cache string
More Usage Examples
-------------------
$H3 Hadoop Partitioner Class
Hadoop has a library class, [KeyFieldBasedPartitioner](../../api/org/apache/hadoop/mapred/lib/KeyFieldBasedPartitioner.html), that is useful for many applications. This class allows the Map/Reduce framework to partition the map outputs based on certain key fields, not the whole keys. For example:
hadoop jar hadoop-streaming-${project.version}.jar \
-D stream.map.output.field.separator=. \
-D stream.num.map.output.key.fields=4 \
-D map.output.key.field.separator=. \
-D mapreduce.partition.keypartitioner.options=-k1,2 \
-D mapreduce.job.reduces=12 \
-input myInputDirs \
-output myOutputDir \
-mapper /bin/cat \
-reducer /bin/cat \
-partitioner org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner
Here, *-D stream.map.output.field.separator=.* and *-D stream.num.map.output.key.fields=4* are as explained in previous example. The two variables are used by streaming to identify the key/value pair of mapper.
The map output keys of the above Map/Reduce job normally have four fields separated by ".". However, the Map/Reduce framework will partition the map outputs by the first two fields of the keys using the *-D mapred.text.key.partitioner.options=-k1,2* option. Here, *-D map.output.key.field.separator=.* specifies the separator for the partition. This guarantees that all the key/value pairs with the same first two fields in the keys will be partitioned into the same reducer.
*This is effectively equivalent to specifying the first two fields as the primary key and the next two fields as the secondary. The primary key is used for partitioning, and the combination of the primary and secondary keys is used for sorting.* A simple illustration is shown here:
Output of map (the keys)
11.12.1.2
11.14.2.3
11.11.4.1
11.12.1.1
11.14.2.2
Partition into 3 reducers (the first 2 fields are used as keys for partition)
11.11.4.1
-----------
11.12.1.2
11.12.1.1
-----------
11.14.2.3
11.14.2.2
Sorting within each partition for the reducer(all 4 fields used for sorting)
11.11.4.1
-----------
11.12.1.1
11.12.1.2
-----------
11.14.2.2
11.14.2.3
$H3 Hadoop Comparator Class
Hadoop has a library class, [KeyFieldBasedComparator](../../api/org/apache/hadoop/mapreduce/lib/partition/KeyFieldBasedComparator.html), that is useful for many applications. This class provides a subset of features provided by the Unix/GNU Sort. For example:
hadoop jar hadoop-streaming-${project.version}.jar \
-D mapreduce.job.output.key.comparator.class=org.apache.hadoop.mapreduce.lib.partition.KeyFieldBasedComparator \
-D stream.map.output.field.separator=. \
-D stream.num.map.output.key.fields=4 \
-D mapreduce.map.output.key.field.separator=. \
-D mapreduce.partition.keycomparator.options=-k2,2nr \
-D mapreduce.job.reduces=1 \
-input myInputDirs \
-output myOutputDir \
-mapper /bin/cat \
-reducer /bin/cat
The map output keys of the above Map/Reduce job normally have four fields separated by ".". However, the Map/Reduce framework will sort the outputs by the second field of the keys using the *-D mapreduce.partition.keycomparator.options=-k2,2nr* option. Here, *-n* specifies that the sorting is numerical sorting and *-r* specifies that the result should be reversed. A simple illustration is shown below:
Output of map (the keys)
11.12.1.2
11.14.2.3
11.11.4.1
11.12.1.1
11.14.2.2
Sorting output for the reducer (where second field used for sorting)
11.14.2.3
11.14.2.2
11.12.1.2
11.12.1.1
11.11.4.1
$H3 Hadoop Aggregate Package
Hadoop has a library package called [Aggregate](../../org/apache/hadoop/mapred/lib/aggregate/package-summary.html). Aggregate provides a special reducer class and a special combiner class, and a list of simple aggregators that perform aggregations such as "sum", "max", "min" and so on over a sequence of values. Aggregate allows you to define a mapper plugin class that is expected to generate "aggregatable items" for each input key/value pair of the mappers. The combiner/reducer will aggregate those aggregatable items by invoking the appropriate aggregators.
To use Aggregate, simply specify "-reducer aggregate":
hadoop jar hadoop-streaming-${project.version}.jar \
-input myInputDirs \
-output myOutputDir \
-mapper myAggregatorForKeyCount.py \
-reducer aggregate \
-file myAggregatorForKeyCount.py \
The python program myAggregatorForKeyCount.py looks like:
#!/usr/bin/python
import sys;
def generateLongCountToken(id):
return "LongValueSum:" + id + "\t" + "1"
def main(argv):
line = sys.stdin.readline();
try:
while line:
line = line&#91;:-1];
fields = line.split("\t");
print generateLongCountToken(fields&#91;0]);
line = sys.stdin.readline();
except "end of file":
return None
if __name__ == "__main__":
main(sys.argv)
$H3 Hadoop Field Selection Class
Hadoop has a library class, [FieldSelectionMapReduce](../../api/org/apache/hadoop/mapred/lib/FieldSelectionMapReduce.html), that effectively allows you to process text data like the unix "cut" utility. The map function defined in the class treats each input key/value pair as a list of fields. You can specify the field separator (the default is the tab character). You can select an arbitrary list of fields as the map output key, and an arbitrary list of fields as the map output value. Similarly, the reduce function defined in the class treats each input key/value pair as a list of fields. You can select an arbitrary list of fields as the reduce output key, and an arbitrary list of fields as the reduce output value. For example:
hadoop jar hadoop-streaming-${project.version}.jar \
-D mapreduce.map.output.key.field.separator=. \
-D mapreduce.partition.keypartitioner.options=-k1,2 \
-D mapreduce.fieldsel.data.field.separator=. \
-D mapreduce.fieldsel.map.output.key.value.fields.spec=6,5,1-3:0- \
-D mapreduce.fieldsel.reduce.output.key.value.fields.spec=0-2:5- \
-D mapreduce.map.output.key.class=org.apache.hadoop.io.Text \
-D mapreduce.job.reduces=12 \
-input myInputDirs \
-output myOutputDir \
-mapper org.apache.hadoop.mapred.lib.FieldSelectionMapReduce \
-reducer org.apache.hadoop.mapred.lib.FieldSelectionMapReduce \
-partitioner org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner
The option "-D mapreduce.fieldsel.map.output.key.value.fields.spec=6,5,1-3:0-" specifies key/value selection for the map outputs. Key selection spec and value selection spec are separated by ":". In this case, the map output key will consist of fields 6, 5, 1, 2, and 3. The map output value will consist of all fields (0- means field 0 and all the subsequent fields).
The option "-D mapreduce.fieldsel.reduce.output.key.value.fields.spec=0-2:5-" specifies key/value selection for the reduce outputs. In this case, the reduce output key will consist of fields 0, 1, 2 (corresponding to the original fields 6, 5, 1). The reduce output value will consist of all fields starting from field 5 (corresponding to all the original fields).
Frequently Asked Questions
--------------------------
$H3 How do I use Hadoop Streaming to run an arbitrary set of (semi) independent tasks?
Often you do not need the full power of Map Reduce, but only need to run multiple instances of the same program - either on different parts of the data, or on the same data, but with different parameters. You can use Hadoop Streaming to do this.
$H3 How do I process files, one per map?
As an example, consider the problem of zipping (compressing) a set of files across the hadoop cluster. You can achieve this by using Hadoop Streaming and custom mapper script:
* Generate a file containing the full HDFS path of the input files. Each map
task would get one file name as input.
* Create a mapper script which, given a filename, will get the file to local
disk, gzip the file and put it back in the desired output directory.
$H3 How many reducers should I use?
See MapReduce Tutorial for details: [Reducer](./MapReduceTutorial.html#Reducer)
$H3 If I set up an alias in my shell script, will that work after -mapper?
For example, say I do: alias c1='cut -f1'. Will -mapper "c1" work?
Using an alias will not work, but variable substitution is allowed as shown in this example:
$ hdfs dfs -cat /user/me/samples/student_marks
alice 50
bruce 70
charlie 80
dan 75
$ c2='cut -f2'; hadoop jar hadoop-streaming-${project.version}.jar \
-D mapreduce.job.name='Experiment' \
-input /user/me/samples/student_marks \
-output /user/me/samples/student_out \
-mapper "$c2" -reducer 'cat'
$ hdfs dfs -cat /user/me/samples/student_out/part-00000
50
70
75
80
$H3 Can I use UNIX pipes?
For example, will -mapper "cut -f1 | sed s/foo/bar/g" work?
Currently this does not work and gives an "java.io.IOException: Broken pipe" error. This is probably a bug that needs to be investigated.
$H3 What do I do if I get the "No space left on device" error?
For example, when I run a streaming job by distributing large executables (for example, 3.6G) through the -file option, I get a "No space left on device" error.
The jar packaging happens in a directory pointed to by the configuration variable stream.tmpdir. The default value of stream.tmpdir is /tmp. Set the value to a directory with more space:
-D stream.tmpdir=/export/bigspace/...
$H3 How do I specify multiple input directories?
You can specify multiple input directories with multiple '-input' options:
hadoop jar hadoop-streaming-${project.version}.jar \
-input '/user/foo/dir1' -input '/user/foo/dir2' \
(rest of the command)
$H3 How do I generate output files with gzip format?
Instead of plain text files, you can generate gzip files as your generated output. Pass '-D mapreduce.output.fileoutputformat.compress=true -D mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.GzipCodec' as option to your streaming job.
$H3 How do I provide my own input/output format with streaming?
You can specify your own custom class by packing them and putting the custom jar to `$HADOOP_CLASSPATH`.
$H3 How do I parse XML documents using streaming?
You can use the record reader StreamXmlRecordReader to process XML documents.
hadoop jar hadoop-streaming-${project.version}.jar \
-inputreader "StreamXmlRecord,begin=BEGIN_STRING,end=END_STRING" \
(rest of the command)
Anything found between BEGIN\_STRING and END\_STRING would be treated as one record for map tasks.
$H3 How do I update counters in streaming applications?
A streaming process can use the stderr to emit counter information. `reporter:counter:<group>,<counter>,<amount>` should be sent to stderr to update the counter.
$H3 How do I update status in streaming applications?
A streaming process can use the stderr to emit status information. To set a status, `reporter:status:<message>` should be sent to stderr.
$H3 How do I get the Job variables in a streaming job's mapper/reducer?
See [Configured Parameters](./MapReduceTutorial.html#Configured_Parameters). During the execution of a streaming job, the names of the "mapred" parameters are transformed. The dots ( . ) become underscores ( \_ ). For example, mapreduce.job.id becomes mapreduce\_job\_id and mapreduce.job.jar becomes mapreduce\_job\_jar. In your code, use the parameter names with the underscores.